harborframework/terminal-bench-2.1
Terminal-Bench 2.1 (Harbor git-repos dataset): Harbor dataset on Hugging Face with 89 tasks. See what each task asks, how it is graded and what it runs in, then run an agent on it.
Tasks
- Evaluates the ability to implement an adaptive rejection sampler in R with proper statistical algorithms, modular design, input…
- Evaluates the ability to recover a Bayesian Network DAG structure from data, perform causal interventions, and sample from the modified…
- Evaluates the agent's ability to bypass an HTML sanitization filter by crafting malicious HTML that triggers JavaScript execution after…
- Evaluates the ability to compile and install a Python package with Cython extensions from source while fixing NumPy 2.x compatibility…
- Evaluates the ability to build pMARS from Debian source packages with X11 support disabled, requiring Makefile modification and headless…
- Evaluates the ability to locate, download, patch, and compile legacy POV-Ray 2.2 raytracer from 1990s source archives on a modern system.
- Evaluates the ability to install and configure BVLC Caffe 1.0.0, train a CNN on CIFAR-10 for exactly 500 iterations in CPU-only mode, and…
- Evaluates the ability to implement async task concurrency control with proper cleanup on cancellation, including the edge case of queued…
- Evaluates the agent's ability to analyze a chess position from an image, use a chess engine to find the best move(s), and handle multiple…
- Evaluates the agent's ability to implement complex mathematical functions (Fibonacci of integer square root) using only combinational and…
- Evaluates the ability to reverse-engineer and reimplement a COBOL program's business logic in Python with exact output reproduction.
- Evaluates an agent's ability to extract code from an image using OCR or vision models, implement the pseudocode logic with cryptographic…
- Evaluates the ability to build the CompCert verified C compiler from source with proper configuration for the host architecture and…
- Evaluates the ability to configure a Git server with automatic deployment to an nginx web server using post-receive hooks.
- Find an optimal 1-hour meeting slot for three people with complex availability constraints by parsing ICS calendars and applying…
- Evaluates the ability to count tokens in a filtered HuggingFace dataset using a specific tokenizer.
- Evaluates the ability to crack a password-protected 7z archive using John the Ripper and extract secret contents.
- Evaluates the ability to debug and fix a C++ program that crashes in release mode due to a static initialization order issue with custom…
- Tests the ability to decrypt an XOR-encrypted SQLite WAL file and recover complete database contents including write-ahead log changes.
- Tests the ability to find a probability distribution satisfying precise dual KL divergence constraints through numerical optimization.
- Evaluates the ability to design PCR primers for Golden Gate assembly by applying molecular biology knowledge and bioinformatics tools to…
- Evaluates the ability to design PCR primers for site-directed mutagenesis by analyzing plasmid sequences and applying molecular biology…
- Evaluates ability to parse ELF binary format and extract memory values from executable sections using Node.js.
- Evaluates the agent's ability to download a YouTube video, extract text commands through OCR or transcription, and produce a formatted…
- Evaluates the ability to implement differential cryptanalysis on a FEAL-like cipher to recover a round key through chosen plaintext attacks.
- Evaluates the ability to perform linear cryptanalysis on a FEAL-like cipher to recover encryption keys from known plaintext-ciphertext…
- Evaluates the agent's ability to create a robust XSS filter that removes JavaScript from HTML files while preserving legitimate HTML…
- Evaluates OCR, document classification, financial data extraction from mixed JPG/PDF documents, and CSV generation with totals.
- Evaluates the ability to identify and fix a CRLF injection vulnerability (CWE-93) in HTTP header handling code by adding input validation…
- Evaluates the ability to recover lost Git commits from a detached HEAD state and merge them back into the master branch.
- Evaluates ability to debug and fix a runtime crash in the OCaml garbage collector's C implementation, requiring low-level debugging skills…
- Tests the agent's ability to extract and decode text from a 3D printer G-code file by parsing movement commands, rendering them visually…
- Evaluates the ability to recover secrets from unreachable git objects and completely remove them from repository history while preserving…
- Evaluates the ability to set up a Git server with SSH authentication, implement post-receive hooks for automated multi-branch deployment…
- Evaluates the ability to implement a minimal, dependency-free C program that performs GPT-2 inference from TensorFlow checkpoints in under…
- Implement a Python class that provides a headless terminal interface supporting interactive bash shells, modifier keys, startup file…
- Evaluates the ability to download a Hugging Face transformer model, create a Flask API for sentiment analysis, and run the service in the…
- Evaluates the ability to configure and run Windows 3.11 in QEMU with VNC display, web interface, and programmatic keyboard control for…
- Evaluates the ability to build and deploy a gRPC-based key-value store server with Protocol Buffers, including service definition, code…
- Evaluates the ability to efficiently transform a 1-million-row CSV file using keystroke-efficient Vim macros with strict command…
- Evaluates the agent's ability to optimize eigenvalue computation by implementing a faster alternative to numpy's default algorithm while…
- Implement a shape-aware batching scheduler for static-graph LLM inference that optimally packs requests into batches while meeting strict…
- Evaluates the ability to analyze date-stamped log files, calculate counts across multiple date ranges, and generate structured CSV output.
- Evaluates the ability to configure a functional mailing list server by integrating postfix and mailman3 with proper join/leave/announce…
- Evaluates ability to cross-compile the DOOM game engine for MIPS architecture using LLVM toolchain and verify execution in a JavaScript…
- Implement a complete MIPS interpreter in JavaScript that can execute a DOOM ELF binary, handle system calls, and render the first game…
- Evaluates the ability to implement and run a hierarchical Bayesian model using R and Stan, including package installation, model…
- Evaluates git bundle merging, conflict resolution, and ARC-AGI style pattern recognition by requiring agents to fetch two git bundles…
- Extracts hidden layer weights from a black-box ReLU neural network by querying outputs and identifying critical points where neurons…
- Evaluates the ability to migrate legacy Python 2.7 scientific computing code to modern Python 3 with proper dependencies and data…
- Evaluates the ability to research and identify the best-performing embedding model on the Scandinavian MTEB leaderboard using data science…
- Evaluates the agent's ability to perform semantic text retrieval using MTEB embeddings, computing cosine similarities and correctly…
- Evaluates an agent's ability to merge multi-format data sources (JSON, CSV, Parquet) with inconsistent schemas, applying field mappings…
- Evaluates the ability to install and configure Nginx with advanced request logging, rate limiting, and custom error pages.
- Evaluates an agent's ability to generate self-signed TLS certificates using OpenSSL, manage cryptographic keys with proper permissions…
- Evaluates the ability to fix LaTeX overfull hbox warnings by replacing words with valid synonyms while satisfying compilation and…
- Evaluates an agent's ability to perform digital forensics by recovering a deleted password from fragmented data within a disk image using…
- Evaluates the ability to reverse-engineer and implement a path tracing renderer in C by analyzing a reference image and recreating it…
- Evaluates the ability to reverse-engineer a compiled path tracing renderer and recreate functionally identical C source code under size…
- Create a single polyglot source file that computes Fibonacci numbers when executed as both Python 3 and C code.
- Evaluates the ability to write a polyglot program that compiles and runs correctly as both Rust and C++ code, computing Fibonacci numbers.
- Evaluates the ability to implement a high-performance C extension for Python that performs portfolio risk and return calculations at least…
- Evaluates the ability to design a fusion protein gBlock by querying bioinformatics APIs, selecting proteins based on spectral properties…
- Evaluates the ability to complete an incomplete Coq proof of addition commutativity using inductive reasoning and formal verification…
- Evaluates the ability to create a Python package, build it, set up a local PyPI server, and make the package installable from the server.
- Evaluates the ability to convert PyTorch model weights to JSON, implement neural network inference in C, and create a command-line tool…
- Evaluates the ability to reverse-engineer a PyTorch Transformer model architecture from a state dictionary, load pre-trained weights, and…
- Evaluates the ability to start an Alpine Linux VM in QEMU and configure SSH server access with proper networking and authentication.
- Evaluates the agent's ability to configure and start a QEMU virtual machine with telnet-accessible serial console, requiring knowledge of…
- Evaluates the ability to optimize a slow SQL query with correlated subqueries by rewriting it using CTEs and window functions while…
- Evaluates the ability to fit Lorentzian curves to Raman spectroscopy data, extract peak parameters, and perform scientific data analysis…
- Evaluates the ability to implement a complete chess move generator using only regular expression transformations on FEN notation.
- Tests the ability to construct a complex regular expression that matches dates in log lines containing valid IPv4 addresses while handling…
- Evaluates the ability to create Python scripts for bidirectional data resharding with file size and directory constraints, using proper…
- Evaluates the ability to convert an RStan Gaussian Process script to functionally equivalent PyStan 3.10.0 code, including complex…
- Evaluates the ability to implement a histopathology image segmentation pipeline using MobileSAM to convert rectangular cell masks to…
- Evaluates the ability to identify and sanitize sensitive API keys and tokens from a Git repository by replacing them with placeholders…
- Evaluates the ability to implement a metacircular evaluator in Scheme that can interpret itself and a comprehensive suite of Scheme…
- Evaluates the ability to write complex SPARQL queries with multiple constraints, aggregations, and date filtering against an RDF knowledge…
- Evaluates the ability to recover data from a corrupted SQLite database using binary file analysis and data recovery techniques.
- Evaluates the ability to compile SQLite from source with gcov instrumentation and make it available in the system PATH.
- Evaluates the ability to implement pipeline parallel training for LLaMA using PyTorch distributed primitives with all-forward-all-backward…
- Evaluates the ability to implement tensor parallelism for PyTorch linear layers with correct weight sharding, distributed forward/backward…
- Train a FastText text classification model on Yelp review data that achieves 0.62 accuracy while staying under 150MB in size.
- Evaluates the ability to optimize MuJoCo physics simulation parameters to achieve a 40% speedup while maintaining physical accuracy within…
- Evaluates the ability to build a computer vision script that analyzes hurdle jump videos and extracts takeoff/landing frame numbers using…
- Evaluates the agent's ability to analyze a binary executable, identify and exploit a buffer overflow vulnerability to bypass…
- Evaluates the ability to write a competitive CoreWars Redcode warrior that achieves specific win rates against five diverse opponent…
- Evaluates the agent's ability to reverse-engineer a custom compression format and write a compatible compressor program.