Call for Contributions

Contribute closed-loop experimental tasks from your lab to Autonomous Science Bench.

Autonomous Science Bench 0.1Rolling submissions
  1. 01Propose
    Open an issue describing the experiment, its data, and what success looks like. Propose a task
  2. 02Implement
    Build the campaign as a task in the repository: lab service, design catalogs, oracle, and verifier.
  3. 03Verify
    Pass the automated checks and a review with your lab, then the task joins the next release.

About

Autonomous Science Bench grows one task at a time, and each task comes from a lab. A task is a budgeted campaign built on that lab’s published experiments: the agent orders batches of measurements, and it passes only if it both discovers good designs and learns a model that ranks designs it never measured. Every new task widens what the benchmark can tell about agents that will one day run real labs.

The 6 tasks in version 0.1 come from these groups.

  • DeepMindNuclease active learningBiology
  • Jewett labBiosensor active learningBiology
  • Sargent labPropylene active learningChemistry
  • PfizerSuzuki condition screenChemistry
  • Gregoire groupOER composition screenMaterials
  • Kusne groupCoercivity peak searchMaterials

Tasks are grouped by domain, then by the lab they come from. All three domains are open to new tasks, in the fields below and beyond them, and semiconductors is open for its first.

  • Biology
    Protein engineeringDirected evolutionCell-free expressionBiosensors
  • Chemistry
    ElectrocatalysisCO₂ reductionReaction screening
  • Materials
    ElectrocatalysisMagnetic materialsCombinatorial thin films

Contribute

What we're looking for

Good tasks come from optimization and screening campaigns a lab has already run: protein or enzyme variants, reaction conditions, catalyst or alloy compositions, process recipes. The data should cover enough of the design space that designs the agent never measured can still be scored.

  • Closed-loop

    Each round's results change what is worth measuring next, so the task rewards learning from evidence, not a single prediction.

  • Experimentally verified

    Measurements come from real experiments, published with a DOI and a license that allows sharing derived tables.

  • Fully budgeted

    Rounds, batch sizes, and costs are fixed up front, and both gates are calibrated against baselines on the real data.

What each node supplies

A node is a lab contributing tasks. With each task it hands over five things.

  • 01Task specificationObjective, rounds, budget, and deliverable
  • 02DataPublished measurements, with DOI and license
  • 03Experiment interfaceThe designs the lab can measure each round
  • 04Reference methodAn oracle that passes both gates
  • 05Independent verifierHidden truth, scored with no network access

Why contribute?

Your experiments become a standard test for AI systems that aim to run labs like yours. Every model and agent evaluated on Autonomous Science Bench is scored on your task, so you see exactly where today’s agents succeed and fail on your own workflow before trusting one with your instruments. Building a task is also a close look at your campaign: what each round was worth, and what a fair baseline would have found.

How to contribute

  1. 01Propose. Open a GitHub issue with the objective, the published data behind it, the budget a real campaign would have, and what counts as a hit. We reply on the issue, usually with questions about scope and gates.
  2. 02Register your lab as a source. A source card names your group, its publications, and the datasets the task replays, each with a DOI and license.
  3. 03Build the campaign. Start from the biosensor task as a template. Write the campaign's stages, catalogs, and deliverable, the private replay table, and a script that rebuilds every data file from its public source.
  4. 04Write the verifier and oracle. The verifier scores the lab's ledger and the delivered model against hidden truth; the oracle drives the campaign through the same interface an agent uses.
  5. 05Validate. The oracle must score 1 and a do-nothing agent 0, and the task must pass every repository check for isolation, pinning, and format.
  6. 06Open a pull request. We review the task with you, calibrate both gates against baselines, and run the sanity and model evaluations. Accepted tasks join the next release.

Proposals are reviewed as they arrive, with no deadline. The contributing guide has every step in full.

Credit and responsibilities

Contributors of an accepted task are co-authors on the paper for the release it joins, and their lab is named as the task’s source in the repository, on the leaderboard, and in the paper.

Contributors make sure their data may be shared under its license, with attribution in the task, and that no answer reaches the agent. After release, they help answer issues about their task and review fixes to it, so its results stay comparable across versions.

Resources