RSI4Science
English

This page is available in English. Choose a language to open its dataset guide.

Benchmarks and demos

Experiment planning · proposed pilot

Improve how a system tests its next research idea

Build an offline experiment-planning scenario in which a system can revise how it compares scientific methods, then use that revised procedure to make its next self-modification.

Your first contribution

You can begin with a source, dataset, method or diagnostic. Researchers supply and run any proposed RSI system in their own workflow.

Scenario proposal and review criteria

For a complete research scenario

A future study would also need a recoverable workspace, scientific feedback and independent improvement tests. A source listing can be reviewed separately from that study.

Study protocol · Available data

Proposed research question

Does revising the system’s own experiment-comparison procedure improve its ability to produce further scientific gains, under a fixed total budget?

Worked example: comparing planner revisions

Proposed sequence, not an executed experiment or a prescribed solution.

  1. Recover a scientific starting point. A reasonable planner chooses candidate conditions to meet a fixed objective and constraints. Repeated baseline runs establish whether apparently useful revisions fail on an independent experimental series.
  2. Let the improvement procedure change. The initial improver, I₀, may revise how it allocates comparison queries, diagnoses constraint failures or handles uncertainty, producing I₁. The dataset supplies evidence and outcome checks, not a gold patch.
  3. Install and actually use the revision. After interface and execution checks, I₁ takes over later improvement decisions, including proposing a further revision, I₂. Record which installed procedure made each decision. Installation alone does not mean improvement.
  4. Test what the new improver can do. Give I₀ and I₁ the same fresh planning workspace and budget on an unseen experimental series. Compare the scientific gains they produce; separately compare full runs with and without descendant improvers taking over.

This measures offline experiment planning. It cannot establish a new material discovery. Realistic limitations must be supported by data; injected software bugs or invented noise do not supply that evidence.

Where recursion would happen

The starting system contains an experiment planner and an editable improvement procedure. The latter diagnoses results, proposes changes and designs comparisons. Its installed revision must drive subsequent system modifications.

A planner updating predictions from new measurements is active learning. A fixed external researcher repeatedly optimizing the planner is a necessary control. Neither alone establishes a recursive return path.

Scientific feedback and evaluation

Start with a finite pool of candidate conditions and verified outcomes, revealed through a query service. Preserve measurement conditions and missingness. Surrogate outputs are simulations; query counts are not measured laboratory costs.

Fix the scientific objective and constraints. Test old and new improvers on independent experimental series, each with the same reset planner, permitted history and resource envelope. Report scientific performance, improvement ability and the effect of the return path separately.

The comparison we need

Compare against a fixed improver that can still update the planner, memory without procedure changes, budget-matched fixed search and strong scientific planning methods. Include the cost of producing the new improver and all rejected attempts.

Read the shared evaluation design.

Available now

  • A worked scenario design and contribution scope

Needed for a future study

  • A selected source, verified rights and independent experimental series
  • A replayed starting planner and observed limitation evidence
  • The feedback service, recursive runner, controls and run records

A bounded contribution · Scientific source and scenario design

Propose one experiment-planning scenario

This scope is for a research scenario proposal. To contribute a dataset or source alone, use the dataset contribution form; a complete RSI workspace is optional.

What to deliver

  1. Identify a source and version, reuse terms, one representative experiment and its measured or simulated origin.
  2. Name a recoverable planning method. Supply any observed limitation and its evidence; mark untested limitations as hypotheses.
  3. Describe available queries, costs or query-count rules, and independent experimental series for a future improvement test. Record unknowns.

How it will be reviewed

A domain reviewer checks scientific meaning, source rights and whether the feedback and independent series can support the proposed test. After selection, contributors and maintainers assign owners for the planner, evaluator and recursive runner.

Sources and contributors

Accepted sources keep their original citations. Reviewed curation, checking tools and documentation are acknowledged by contributor and version.

Source and reuse notes.

Propose a scientific task

Agree a manageable scope and a reviewer before starting. The form saves a draft on your device; contact the project lead to discuss it.

Project-led design. No independent scientific approval or completed RSI experiment is claimed. Accepted contributions and review decisions will be recorded as that work occurs.