Run Autonomous Science Bench

Every campaign is a standard Harbor task, in the same format as Terminal-Bench-Science. A trial starts two containers, the agent and the lab, so you need Docker (4 CPUs and 8 GB of RAM per trial) or a cloud sandbox on Modal or Daytona, and uv.

git clone https://github.com/Yibo-Wen/as-bench.git
cd as-bench
uv tool install "harbor[modal,daytona]"

The 0.1 dataset has 6 campaigns. Each fixes its rounds, batch sizes, and budget, and is scored on its own metrics. Run one by its path, tasks/<path>. The campaigns in archive/ run the same way but are outside the dataset.

• Nuclease active learningbiology/deepmind/nuclease-active-learningreplay · 4 × 96 enzyme variants
• Biosensor active learningbiology/jewett-lab/biosensor-active-learningreplay · 3 × 10 protein variants
• Propylene active learningchemistry/sargent-lab/propylene-active-learningtwin · 3 × 20 dilute-alloy designs
• Suzuki condition screenchemistry/pfizer/suzuki-condition-screenreplay · 4 × 48 reaction-condition wells
• OER composition screenmaterials/gregoire-lab/oer-composition-screenreplay · 4 × 48 oxide compositions
• Coercivity peak searchmaterials/kusne-lab/coercivity-peak-searchreplay · 4 × 16 thin-film compositions

A campaign fixes its backend. Swapping it changes only what answers the agent’s experiments, never the instruction, the asb commands, or the ledger format. See the roadmap.

• v0replayReplayPublic experimental data, held privately by the lab service and replayed. Five campaigns use it.
• v1twinDigital twinA hidden process model standing in for the lab. Propylene active learning uses it.
• v2liveLive labA physical lab, reached through asynchronous jobs. The contract is defined; no campaign uses it yet.

Check the setup first, then point an agent at a campaign. Inside a trial the agent reaches the lab through the asb command: asb campaign shows the rules, asb status the remaining budget, asb run orders a batch, and asb deliver submits the final model.

# Check the setup: the oracle solution should score 1, a do-nothing agent 0
harbor run -p tasks/biology/jewett-lab/biosensor-active-learning --agent oracle
harbor run -p tasks/biology/jewett-lab/biosensor-active-learning --agent nop

# Run an agent on one campaign
harbor run -p tasks/biology/jewett-lab/biosensor-active-learning \
  -a claude-code -m anthropic/claude-opus-5-5

# Run every campaign in the dataset, in cloud sandboxes
harbor run -d as-bench/as-bench \
  -a claude-code -m anthropic/claude-opus-5-5 --env modal

To compare agents and models, use the eval runner. Its presets pair each agent (Claude Code, Codex, Grok Build, or mini-SWE-agent) with a model and reasoning effort, following the Terminal-Bench-Science leaderboard. Results go to results/.

uv sync --group eval
cp .env.example .env    # then add the API keys for the models you run

uv run python evals/run.py --list                                   # presets, suites, tasks
uv run python evals/run.py --suite sanity                           # oracle = 1, nop = 0
uv run python evals/run.py --suite small --agent-timeout-multiplier 0.25
uv run python evals/run.py --preset claude-opus-5.5 --preset gpt-5.6-terra -k 3 --env modal

A trial can run up to 8 hours; --agent-timeout-multiplier 0.25 cuts that to 2. Start with --suite small and one attempt, and check the cost column before scaling up. Stop a job with uv run python evals/stop.py, which also removes its containers.

Each trial leaves its reward, the agent’s trajectory, and the verifier’s metrics under results/jobs/<job>/. The summary groups trials by task and by agent, model, and effort, with pass rate, metrics, cost, and time. The metrics are task-specific, such as discovery hits and NDCG, plus campaign progress (stages completed, designs measured, deliverables submitted), which shows how far an agent got even when it fails.

uv run python evals/summarize.py     # rebuild results/trials.csv and results/summary.md
cat results/summary.md

harbor analyze -m <model> results/jobs/<job>   # qualitative review of trajectories

Results stay local by default. To share a job, upload it to Harbor Hub; it is private unless you add --public. Name the full agent, model, and reasoning-effort configuration with any result.

uv run harbor upload results/jobs/<job>

If you enjoy the benchmark or have feedback on how to improve it, join the community (link coming soon) and let us know.

If you find this work useful, please cite it.

@article{wen2026autonomous,
  title={Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery},
  author={Yibo Wen and Han Liu},
  journal={arXiv preprint arXiv:2610.18449},
  year={2026},
  url={https://arxiv.org/abs/2610.18449}
}