How benchmarks support RSI
Recursive self-improvement (RSI) uses a system’s current capabilities to improve itself, then brings the updated capabilities into further rounds of improvement.
We provide tasks, data and checks. You bring the model and improvement method.
From feedback to a testable revision
- 1
Do the task
Your model makes predictions on a versioned scientific task.
- 2
Get feedback
The benchmark checker records outcomes and output validity.
- 3
Find a possible cause
Your workflow tests possible causes using diagnostic cases.
- 4
Revise the system
Your workflow updates the model, code, tools or prompts.
- 5
Check the revision
Use additional development cases to check gains and regressions.
Feed the new development evidence into the next round. Preserve versions, feedback, decisions and costs.
Scientific search spaces are vast. Learning which direction to try next may reduce wasted trials. Benchmarks help test progress through checkable outcomes, useful feedback and fresh evaluation cases. The data supplies evidence; researchers build and run the improvement method.
Your experiment defines the revision process. Depending on the task rules, you may revise model weights, agent code, tools, memory, prompts or training strategies. Record the allowed edits, versions and costs; keep final assessment separate from revision feedback.
A molecular example
MolP2SBench starts from known molecule–property records, masks a structural fragment and asks the model to recover it from valid candidates. The original record retains the reference for scoring. In this pair, the candidates stay fixed while the property evidence and target choice change.
This isolates one part of molecular design: using properties to reason about structure. Researchers can use development feedback to improve a solver and test it on new molecular families. A longer-term question is whether that learning transfers to generating structures under property constraints. From recovery to generation.
Teaching walkthrough: real source records, your choices and a proposed revision plan. The starter returns scored outcomes; diagnostic experiments are part of your workflow.
What makes it recursive?
The proposed protocol below studies a specific form of RSI: the system revises its improvement process and uses that revision in later rounds. Here, P does the scientific task; I diagnoses, proposes and tests changes to P and to itself. Researchers supply and run both parts.
I: improvement process
A revised improver makes the next change.
Compare with a fixed improver
In the recursive path, the improver changes and is reused. In the fixed-improver control, the original process stays in charge. Model weights may remain fixed while programs, tools or procedures change.
Technical definition and evaluation protocolWhat evidence would show recursive improvement?
Three separate questions keep the claims precise:
- Performance: how well does the system solve the scientific task?
- Transfer: does a revision help on independent problems?
- Recursion: does a revised improvement procedure produce better later improvements?
Run records must show the revised procedure being used. Compare old and new procedures from common starting systems and budgets, then compare complete runs against a fixed improver. Final scientific checks remain independent. Report failures, costs and uncertainty alongside gains.
Evaluation protocol and controls. The current teaching starter demonstrates scored feedback; controlled RSI results are still to be established.
Contribute to the collection
Share original data, curate an existing resource or propose a scientific task. A paper and a completed RSI study are optional. Useful contributions describe a checkable scientific capability, the available feedback and the data others can use.
Collection in development. You can try the molecular teaching demo. The reviewed benchmark collection and controlled RSI evaluations are future work.
How this builds on existing benchmarks
SciCode checks scientific code; MLAgentBench supports interactive experiments. We build on these ideas by connecting task performance to feedback for revisions and independent comparisons of improvement methods. Dataset requirements.
Research behind this design
- STOP studies a language-model-based program that modifies its own improvement scaffold. Its authors explicitly distinguish this from full RSI of the underlying language model.
- HyperAgents studies editable task and meta agents, including subsequent improvement capability and transfer. This motivates the proposed checks.
- RSIBench-Data evaluates data-centric research using a fixed post-training stack. It illustrates how a benchmark can test capabilities needed for RSI without guaranteeing a successful recursive system.