rounakbende/rh-swe-bench

Name: rh-swe-bench
Creator: rounakbende

rh-swe-bench: Harbor dataset on Hugging Face with 357 tasks. A Red Hat-specific benchmark for evaluating coding agents, packaged in Harbor format. Contains 357 quality-verified tasks across 25 Red Hat ecosystem repositories. Pre-built GHCR images for fast execution.

Tasks

YAML parsing is completely broken for playbook verification. Mapping nodes come out empty — all key-value pairs are silently dropped…
YAML round-trip parsing is broken for flow-style sequences. When loading and dumping YAML that uses flow-style collections like inline…
Netstat parser is misclassifying lines that contain both a colon and numeric values. Lines like "destination unreachable: 107" are being…
Grubby combiner isn't expanding grubenv variables correctly in boot entries. When boot entries contain things like $kernelopts or $tuned…
The egg/core update fetch is broken for non-legacy upload configs. The URL construction ignores the platform path stripping that was there…
ResolvConf parser is broken for resolv.conf files that have both search and domain keywords. The old logic correctly tracked mutual…
EarliestSSLCertExpireDate combiner is broken after a recent change to how it iterates certificate info. It now expects a dict-like…
XFS quota state parsing is broken for certain field mappings. The parser fails to correctly extract device names and key-value pairs from…
Ansible Galaxy dependency resolution is silently broken after some recent changes to the collection dataclasses. Hashing for requirements…
task-0023
Ansible galaxy collection requirement parsing is silently swallowing a bunch of validation logic. Invalid collection specs that should…
Templating access notifications are silently dropped after a recent change. When iterating over notify contexts in reverse, the masking…
Error chaining for captured Ansible errors seems broken. When an AnsibleCapturedError is involved in an exception chain, the cause event…
Jinja2 filter and test plugins from Ansible collections are silently failing to load. When you reference a filter or test plugin via its…
Molecule's symlink handling for inventory vars is broken — if a symlink already exists from a previous run, the provisioner crashes…
Molecule's log level formatting is broken after a recent change to the ANSI output handling. The bracketed format like INFO is now leaking…
Log level formatting in Molecule's output is broken after a recent change. The colored and plain versions of log level labels are now…
Molecule's destroy command crashes when --destroy=never is passed or when no provisioner is configured. It used to skip gracefully in…
Molecule's inventory verification behavior changed in a confusing way — when no platform instances are defined, it used to exit with a…
Molecule's bordered output stream has a subtle bug where flushing partial lines can produce garbled or duplicated output. If a line is…
Molecule's create command silently skips provisioning when instances are already marked as created in state, even after a destroy. The…
Molecule's scenario execution logic is silently broken after a recent refactor. Shared state detection is now based on a heuristic that…
Molecule's shared state handling between scenarios is broken when using a default scenario. The subcommand sequence check got dropped, so…
Click argument names are being uppercased when they shouldn't be. Positional arguments defined via CliOption are getting their names…
Molecule's platform instances are missing default values for groups and children when they're not explicitly defined in the config. If…
Molecule's scenario pruning is broken when the provisioner isn't configured. The safe files list now unconditionally accesses provisioner…
Molecule stops warning users when molecule.yml has been modified since the scenario was created. That stale config warning was genuinely…
Molecule's dict2args conversion is broken for single-character flags. Short options like -v or -q are getting turned into --v and --q…
Molecule's scenario execution is broken when running multiple scenarios with shared infrastructure. The create/destroy lifecycle for the…
Molecule's StrEnum compatibility layer is too strict about value types. Non-string enum values that should be coerced automatically are…
Molecule's playbook resolution is broken after a recent refactor. The bundled playbook directory lookup no longer respects the…
Molecule's scenario directory resolution is wrong when a molecule config file is present. It falls back to the current working directory…
Delegated driver isn't preserving the instance config file during cleanup. When using the delegated driver, the instance config gets wiped…
Molecule's Ansible provisioner is dropping important default options that were previously always applied. Things like skip-tags…
Molecule swallows the actual exit code and error message when a command fails. If something goes wrong during a scenario run, you just get…
Molecule's VCS root detection is too strict when checking for .git directories. If you're in a repo where the .git dir exists but doesn't…
Molecule's scenario discovery breaks when running from inside an Ansible collection. The glob pattern resolution for finding scenarios is…
Molecule's exclude patterns stop working when you pass glob-style wildcards like appliance to --exclude . It also breaks scenario…
Molecule silently swallows error messages when a MoleculeError is raised. Instead of seeing what actually went wrong, you just get a blank…
The --report and --command-borders flags are now on by default but the docs still show them being passed explicitly in every example…
Copier's chroot and eval logic is completely broken on Unix. The error checks are wrong — successful operations are being treated as…
Symlink resolution in the copier is completely broken — it never actually follows symlinks because the loop condition is wrong, so…
Symlink resolution in the copier is completely broken — it never actually follows symlinks because the loop condition is wrong, so…
Symlink resolution in the copier is broken for absolute paths inside a container root. When a symlink target is an absolute path, it…
Buildah's copier is applying chroot path adjustments to the wrong branch of logic. When running in a chrooted environment, the directory…
Copier's put logic has the directory/file ownership and permission handling completely wrong. ChownDirs is being applied to files and…
Symlink resolution in the copier package is completely broken. Paths that should resolve correctly through symlinks are failing, and the…
The copier subprocess in buildah/podman is completely broken for file copy operations. Errors that should cause the process to exit are…
Hook configuration parsing is silently accepting invalid configs that should be rejected. When both 'stage' and 'stages' are set in a hook…
Blob caching in containers/image seems broken after a recent refactor. When pushing images, blobs that should be cached locally aren't…
UID/GID mapping parsing is completely broken when using the @ parent mapping syntax. The @<id notation and plain numeric IDs are being…
Quadlet label parsing silently mangles values that contain unquoted whitespace. If you write something like…
Unpacking tar archives is broken after the recent changes to the archive handling code. Directory timestamps are no longer being restored…
The CreateFromTemplate test in the graph driver is not actually verifying isolation between the template layer and derived layers…
Cluster membership logic in etcd is badly broken after a recent change. Adding a member now skips actually saving it to the backend and…
Several bugs crept into the storage and WAL layers that are causing serious breakage. The backend initialization is skipping bolt options…
Corruption checking in etcd is badly broken in a few related ways. The hash mismatch detection has its log messages mixed up — compact…
Several bugs crept into etcd's server code that are going to cause real pain. Member version collection is skipping the local member and…
Benchmarking stats in etcd's report package are completely wrong. The stddev calculation runs before any latency data has been collected…
Several bugs in etcd's cluster management code are causing subtle but serious misbehavior. Version collection skips remote members and…
etcd's v2store event queue has a broken wrap-around calculation when the queue is at capacity. When the queue fills up and needs to…
Host normalization for URLs is producing wrong results in some cases. IPv4 and IPv6 addresses aren't being canonicalized consistently, and…
Benchmark reporting stats are completely wrong — average, stddev, RPS, and total time are all being calculated in the wrong order, so…
Cluster membership operations in etcd are badly broken — removing a member proceeds when the permission check fails and skips when it…
Feature gate alpha handling is broken — gates that should only apply to Alpha features are being applied to everything else instead. This…
Interval tree comparisons are returning wrong results for both string and int64 keys. Lookups that should find a match come back empty…
Promoting a learner member in etcd can panic or silently corrupt cluster state. If the member ID being promoted doesn't exist in the…
Several bugs crept into the etcd server cluster coordination code. Version negotiation during cluster upgrades is behaving incorrectly —…
Several bugs crept into etcd's backend and cluster membership code that are going to cause real pain. The backend initialization has nil…
Host normalization for URLs is returning wrong results in some cases. IPv4 addresses with ports, hostnames without ports, and similar…
Interval tree comparisons for both string and int64 types are returning incorrect results. Lookups that should find overlapping intervals…
Feature gate alpha handling is broken — gates that should only apply to Alpha features are instead being applied to everything else. This…
Removing a member from an etcd cluster isn't working right — the permission check and quorum safety check are both broken, so valid…
Several stats calculations in the benchmark reporter are completely broken. The stddev is being computed before any latency data is…
etcd's v2store watcher registration is broken — when a key already has existing watchers, new watchers for that key are incorrectly…
Several bugs crept into etcd that are causing serious misbehavior. The event queue wraps its front pointer incorrectly when at capacity…
etcd's v2store watcher registration is broken — when a key already has watchers, instead of appending the new watcher to the existing…
Several bugs crept into etcd's server code that are causing pretty serious misbehavior. Member self-registration during discovery retries…
etcd cluster membership operations are badly broken — RemoveMember is allowing removals through when permission checks fail and blocking…
etcd cluster restore is broken for single-member clusters. The special case that allowed adding a new voting member to a 1-member cluster…
Interval tree deletions are producing corrupted results in etcd's adt package. After removing nodes, subsequent range queries return wrong…
task-0208
URL validation for comma-separated flag values is too permissive. When passing multiple URLs to etcd flags, invalid URLs that should be…
PageWriter is producing corrupted output when writing data that crosses page boundaries. Writes that would overflow the buffer watermark…
etcd's auth checks are too strict for read-only cluster and maintenance operations. Listing members or getting alarm status requires full…
Lease permission checks are being skipped inside transactions when auth is enabled. If a put request within a txn (including nested txns)…
Cluster member raft attributes aren't actually being persisted when updated. When a member's peer URLs change, the update gets applied to…
Protobuf deserialization for API resource metadata is completely broken. Decoding APIResource messages from the wire format silently…
JSON stream encoding for typed lists is silently falling through to the default serializer instead of being handled correctly. Objects…
GroupVersion kind resolution is returning wrong kinds in some cases. When converting between API versions with a multi-group versioner…
Quantity's String() method is producing garbled or empty output in some cases. When a Quantity hasn't been serialized before, the byte…
ResourcesFor in the REST mapper is returning duplicate results when looking up resources. If a resource is registered under both its…
kubectl taint is completely broken right now. Node names get ignored, the --all flag behaves wrong, dry-run applies changes when it…
kubectl logs is broken when using a label selector together with a pod name argument. The two should be mutually exclusive but the…
kubectl exec argument parsing is broken when a pod name is specified without a type prefix. Running something like kubectl exec mypod --…
kubectl apply is badly broken right now. ApplySet label handling is wrong — labels are being added when there's no ApplySet instead of…
kubectl get --sort-by is sorting results in reverse order. When sorting by numeric fields like resource quantities, integers, or string…
kubectl debug validation logic is broken for --copy-to vs non-copy-to cases. When you use --copy-to, the flags that are supposed to be…
kubectl create deployment is broken when specifying a port — the port never gets applied to the container spec. Also, kubectl create…
kubectl apply validation is dropping several checks that used to be there. Using --prune without --server-side no longer warns you…
Draining nodes seems broken in a few ways after recent changes. The eviction support check no longer looks at the core v1 API group, so…
kubectl debug's restricted profile isn't applying the seccomp RuntimeDefault setting to debug containers. When you use the restricted…
Cordon/drain node state is getting mutated in place during patch operations instead of working on a copy, which means if the API call…
kubectl create secret is badly broken right now. Combining --from-literal with --from-env-file no longer errors out like it should, and…
kubectl delete is badly broken in several ways right now. Grace period of 0 is being ignored entirely, cluster-scope warnings fire on…
kubectl get is pretty badly broken right now. Custom columns parsing rejects valid specs instead of invalid ones, yaml output no longer…
kubectl create clusterrolebinding silently drops service account subjects. When you pass --serviceaccount=namespace:name , the binding…
kubectl apply is silently skipping deletes during pruning and mishandling null fields in strategic merge patches. Resources that should be…
kubectl delete is pretty badly broken right now. Grace period handling is wrong so resources aren't getting deleted cleanly, the…
kubectl get is broken when sorting with --sort-by . The chunk size override that disabled chunking during sorted requests got removed, so…
kubectl create token validation is rejecting valid inputs and accepting invalid ones. Passing --bound-object-uid without…
kubectl debug pod copy behavior is broken after a recent refactor. When using --copy-to , labels and annotations aren't being preserved…
kubectl expose is silently dropping session affinity validation. When you pass an invalid session affinity value like…
kubectl cp silently drops symlinks when copying files to a pod. If your source directory contains any symlinks, they just get skipped…
kubectl describe is no longer showing SeccompProfile info for containers. If you have a pod with a seccomp profile configured at the…
kubectl apply is silently corrupting last-applied-configuration annotations in some cases. When using a custom field manager or when…
kubectl rollout status errors out for DaemonSets using OnDelete update strategy. If your DaemonSet isn't using RollingUpdate, the command…
kubectl apply is completely broken — errors are being swallowed instead of returned, so failures in validation, namespace resolution, REST…
kubectl apply validation logic is completely broken — flags that should be compatible are getting rejected, and flags that are actually…
kubectl taint is broken after a recent refactor — the operation string (tainted/untainted/modified) is being calculated incorrectly, and…
kubectl set env and kubectl set resources are both behaving incorrectly in ways that seem related to some botched length checks. Setting…
kubectl top pod --all-namespaces isn't actually querying across all namespaces when metrics aren't available yet. The all-namespaces flag…
kubectl cp is broken in a few ways after a recent change. Copying from a local path to a pod fails with a confusing "one of src or dest…
kubectl debug pod copy is broken after a recent refactor. When using kubectl debug with --copy-to, the copied pod ends up with unexpected…
kubectl debug pod copy mode is broken in a bunch of subtle ways. When using --copy-to , the copied pod ends up with wrong labels (a…
kubectl debug is passing the wrong pod to the applier when generating ephemeral containers. When you use kubectl debug with a custom…
kubectl apply is silently swallowing errors during initialization. Validation setup, REST mapper creation, namespace resolution, and a few…
kubectl apply validation is dropping several checks that should still be there. Using --prune without --all or a selector gives a…
kubectl apply is silently dropping some validation checks after a recent change. Using --prune without --applyset no longer catches the…
kubectl expose is silently dropping the selector requirement when creating services. Previously if you didn't pass a selector it would…
kubectl rollout status errors out for DaemonSets that use a non-rolling update strategy. If your DaemonSet is configured with OnDelete…
kubectl expose is silently dropping selector validation and the service creation logic got scrambled. When you run kubectl expose without…
kubectl debug pod copy is broken after a recent refactor. When using kubectl debug with --copy-to, the copied pod gets a hardcoded…
kubectl cp is completely broken for any pod file path containing a colon. Passing something like pod:/some/path fails to parse correctly —…
kubectl taint is broken in a few ways after a recent refactor. The operation string (tainted/untainted/modified) isn't being computed…
kubectl exec argument parsing is broken when you don't use the -- separator. Running exec with just a pod name and command directly…
Validation logic for kubectl create job and kubectl create service is completely broken. Creating a job with just --image or just --from…
kubectl create is badly broken after a recent change. The --edit flag does nothing useful anymore, extra args that should be rejected are…
kubectl create configmap silently ignores file sources and env files when they're provided. Passing --from-file or --from-env-file to…
kubectl apply validation is dropping several checks that should still be there. Using --prune without --all or a selector gives a…
kubectl cp silently drops the namespace when copying to/from a pod without an explicit namespace in the path spec. Instead of falling back…
kubectl logs is broken when using a label selector together with a pod name argument. Passing both -l and a pod name used to give a clear…
kubectl debug is passing the wrong pod spec to the applier when generating ephemeral containers. The original pod object gets used instead…
kubectl taint has a couple of broken behaviors around validation and error reporting. Using --all together with a selector no longer…
kubectl applyset has a few bugs that cause pretty confusing behavior. Custom resource parent type validation silently swallows errors when…
kubectl wait condition checking has some subtle but painful bugs. The observedGeneration comparison logic is broken — it's not correctly…
kubectl set image is completely broken right now. Error handling logic is wrong so errors are being swallowed instead of returned, and…
kubectl api-resources filtering is completely broken. The namespaced flag does the opposite of what you'd expect, verb and category…
kubectl create clusterrole is broken when using aggregation rules — the AggregationRule map never gets initialized, so passing…
kubectl set image validation is dropping some important checks. Using --all together with --selector no longer raises an error like it…

207 more tasks.

rounakbende/rh-swe-bench on the Hugging Face Hub