Start with tasks, feedback and a fresh test.
Use a versioned task package to generate predictions and score outputs. Compare errors across controlled evidence changes; treat possible causes as hypotheses to test. Your own workflow then makes and records a revision.
Evaluate the submitted version on independent scientific families under matched budgets. Report performance, regressions, failed outputs and costs. Keep final evaluation separate from the feedback used to revise or select a system.
The platform provides the benchmark interfaces and review criteria. The local molecular starter demonstrates input and scoring; researchers implement their model and experiment runner. Browse teaching examples and proposed studies.
For studies of recursive self-improvement. This proposed protocol tests changes to the improvement method. Its requirements apply to claims evaluated under this protocol. Dataset contributions follow the resource review criteria.
The recursive mechanism tested by this protocol
Each scenario starts with a scientific system and a process that diagnoses, proposes, tests or selects changes to that system. The system may revise this improvement process. A subsequent revision must actually be produced using the new version.
This optional protocol studies system-level recursive self-improvement through editable programs, tools and research procedures. Model weights may remain fixed while these parts of a user’s system change.
A concrete example: an experiment-planning agent changes how it validates proposed changes to itself. The updated validation process then helps it develop its next version. The protocol then tests whether this leads to better improvements on independent scientific problems.
Installation rules and version records
Record whole-system versions separately from improver artifacts and parent references. A renamed file, comment edit or P-only update does not establish a new improvement mechanism. Evidence must link an actual mechanism revision to the later decisions it makes.
The improver may change its internal testing and selection policy; record that policy’s version for each attempt. Your runner mechanically installs its selected candidate only when fixed format, executability, permitted scientific-constraint and resource checks pass. Installation checks use only public or permitted query information, never protected audit outcomes. Those external rules and the final judge remain outside the editable system.
Repeatedly updating a predictor from new observations can be active learning. Repeatedly editing a solver with an unchanged external researcher can be optimization. Both are useful comparisons; a recursive-mechanism claim additionally requires a revised improvement process to drive later changes.
What would a multi-generation execution record show?
An illustrative accepted path is I₀ creates S₁ → installed I₁ creates S₂ → installed I₂ creates S₃, where S = (P, I, H) contains the scientific program, improver and permitted history. The recursive-loop diagram shows the P and I parts of these system states. Your runner must verify the installed version was invoked and link its output to the later revision.
The illustrative chain assumes distinct mechanism revisions; artifact differences and actual return calls must support it.
This proposed multi-generation test seeks two post-installation return calls. A shorter trace supports a narrower claim. A changed file is not evidence of changed decisions: paired probes can test observed differences, while matched checkpoint branches can test a particular revision’s effect. Runs with no accepted self-change remain reportable.
Three separate outcomes.
Scientific performance
Does the resulting system achieve valid scientific outcomes on unseen cases, within the stated budget?
Ability to improve
From the same starting system and evidence, does the new improvement process produce better changes than the old one?
Effect of recursion
Does letting revised improvement processes take over help, compared with keeping the original process in charge?
Use task-appropriate scientific measures: constraint satisfaction, prediction quality or valid discoveries per query, for example. Report these separately from the evidence that a recursive process ran. Versions can regress; all outcomes belong in the record.
Compare new and old processes from the same starting point.
Reserve independent scientific workspaces for improvement tests. Each contains an initial scientific program, permitted development/query cases and separate protected audit cases, with a declared barrier against using audit scores for selection. Independence comes from the outer scientific-group split. Subcases inside a workspace may remain correlated. All groups in these tests must be held out from development and selection of the tested improver. Give old and new processes the same workspace and resource envelope.
For this diagnostic, hold each improvement process fixed while measuring the scientific changes it produces. In the main recursive run, allow the process to keep modifying itself. This separates acquired improvement capability from the overall effect of allowing the improver to be revised and reused.
Comparison conditions
| Condition | What changes |
|---|---|
| Recursive system | The scientific program and its improvement process may change. Installed versions drive later revisions. |
| Fixed improvement process | The scientific program P may change. Edits to I are disabled; I₀ stays in charge. This is the primary fixed-improver arm. |
| History / memory control | A fixed process receives comparable accumulated evidence, isolating the value of retained experience. |
| Budget-matched search | A fixed search strategy explores its declared edit space under comparable resource limits. State whether it can edit P, I, or both. |
| Strong scientific reference | An established method shows whether there is meaningful headroom on the scientific task. |
Controls for external tuning and takeover claims
Additional comparison for claims beyond external tuning: let a fixed external optimizer edit I under a matched total development budget, then evaluate its selected improver in the same capability tests and declared complete-run setting. Without this arm, do not claim recursion beats external tuning of the improver.
If a secondary control may propose and test I revisions but cannot install them, label it separately and count those attempts. It estimates the effect of takeover permission, rather than the full permission to modify I; do not pool the two control definitions.
View the two comparison designs
Better at improving, or better because of recursion?
Test the old and new improvers
Improve P*
Improve P*
Audit their scientific gains independently
Each tested improver stays fixed. Within each fresh workspace, separate permitted development feedback from protected scientific audit cases; count internal queries in its budget.
Compare complete runs
Recursive run
Fixed-process run
Audit selected systems independently
Does allowing I to be revised and reused help? Include their development cost, plus memory and fixed-search controls. This estimates the effect of the complete recursive procedure.
A concrete improvement-capability diagnostic
Choose the new improver using only allowed development or selection evidence, before these independent tests. For each fresh workspace, freeze old and new improvers separately. Each receives the same starting scientific program P*, history and resource budget B, including all internal tests and scientific queries.
Subsequent gain at budget B = independent scientific score of the development-selected output − score of the common starting program.
Orient the scientific measure so that larger gains mean better outcomes. Predeclare the selection interface and immutable validity rules; log the policy actually executed by each improver. The independent score cannot choose the best candidate. If no update is accepted, evaluate the unchanged starting program when valid. Crashes, invalid outputs, missing results and budget violations remain explicit outcomes in the full run denominator, with a predefined handling rule.
Compare paired gains over independent workspaces and report costs and uncertainty. This adapts a common-start improvement-capability diagnostic used in HyperAgents.
Test the effect of a particular revision
Keep base models, feedback permissions and scientific judges comparable. Record which version actually made each modification. To attribute a later decision or gain to a particular revision, branch from the same scientific program and history, run old and new improvers under matched randomness and resources, and independently assess their outputs. Record both branches, including identical decisions or regression. This local diagnostic is required for that specific causal claim; it is not required after every edit or a substitute for the complete-run comparison.
What must be checked before a claim counts?
These are proposed acceptance requirements. The research validators have not been implemented or executed. Website build checks and the molecular and oscillator teaching scorers check formats and scoring. Research execution evidence needs separate validation.
Replay the scientific start
- Claim
- The scientific problem is meaningful and its starting workflow can be reproduced.
- Evidence
- Source records, units, independent groups, pinned workflow and actual baseline evidence.
- Check
- An independent reviewer resets and reruns the workflow, checking declared tolerances, constraints and failure handling.
- If unsupported
- The scenario is not ready for research evaluation; this is not evidence that RSI failed.
Verify recursive use
- Claim
- A changed improvement mechanism actually drives a later modification of P, I, or both; a P-only edit creates no new I generation.
- Evidence
- Separate system versions and improver artifact references, hashes and parents; actual diffs and runner records of installation and later calls.
- Check
- Use documented diff review and execution evidence to verify the changed mechanism actually executes. Target-only edits or renamed identical improvers create no new improver generation. Optional frozen paired probes with matched randomness can reveal behavior differences; finite matching outputs do not prove universal semantic equivalence.
- If unsupported
- A label, unused revision or inert edit does not demonstrate a changed improvement mechanism or positive benefit.
Verify actual execution
- Claim
- Reported attempts and results correspond to real runs.
- Evidence
- Runner records linking inputs, artifacts, visible outputs, failures, evaluator results and costs.
- Check
- Reconcile records with execution receipts and artifacts, and independently replay the declared checks. Self-reported hashes alone do not prove execution.
- If unsupported
- Do not treat the records as verified experimental evidence or verified training traces.
Test the ability to improve
- Claim
- The new improver produces better subsequent scientific changes than the old one.
- Evidence
- Common-start workspaces with permitted query/development subcases and protected audit subcases; scientific grouping rationale and a prespecified effect and uncertainty plan.
- Check
- Outer workspace groups must be held out from development and selection of the tested improver. Inner audit subcases are isolated from selection; do not assume they are statistically independent. Freeze each improver, select using permitted feedback, then audit independently. Count internal scientific queries; record independent-group counts, repetitions and analysis units.
- If unsupported
- A better solver does not establish a better improver. A reportable null does not rule out meaningful effects; underpowered early evidence is descriptive.
Test the benefit of recursion
- Claim
- Letting revised improvers take over helps under comparable resource limits.
- Evidence
- Matched comparison runs and total resource ledgers; claim scope and shared-checkpoint branch references when attributing an effect to a particular revision.
- Check
- The primary frozen-I arm disables edits to I while allowing P edits; compare it with recursion, memory, fixed search and a scientific reference. Label propose-but-do-not-install controls separately. A particular-revision causal claim needs paired old/new-improver checkpoint branches, not every generation. Beating external tuning additionally needs a fixed external optimizer allowed to edit the improver under matched total development budgets.
- If unsupported
- Do not make the unsupported causal or external-tuning claim. Overall matched-arm effects do not require every-generation branches; null results need adequate precision before excluding meaningful benefit.
Test transfer independently
- Claim
- The observed benefit extends beyond the cases used to develop or select the system.
- Evidence
- Frozen scientific groups, overlap audits, runtime access records and a human/team final-partition exposure ledger across runs.
- Check
- Keep related records together. Record who saw final material, when, and subsequent changes. Changes informed by final results are exploratory and need new protected assessment for confirmation; new seeds alone are not new scientific tasks.
- If unsupported
- Limit the claim to observed cases. Runtime isolation alone cannot establish independent confirmation after human or team exposure.
Protect the scientific judge
- Claim
- Higher scores reflect valid science under the agreed rules.
- Evidence
- Immutable external evaluator and validity/access/budget checks, plus the separately versioned editable improver testing and selection policy.
- Check
- Record selection_rule_version. The runner mechanically installs only valid, in-budget candidates selected by the currently executed improver policy. That policy cannot change the external judge or access rules; unauthorized interference must be rejected or invalidate the run.
- If unsupported
- Score gains may reflect changed grading or unauthorized access rather than scientific progress.
Counterexamples the check suite must catch
Build deliberate fixtures for these cases before trusting the measurement system. A checksum or self-reported success label alone cannot establish that a run happened.
- Improve the scientific program while the original improver remains in charge.Scientific progress may pass; the recursive-use claim must not pass.
- Save a revised improver but keep loading or calling the old version.Installation and later-use checks must detect that the revision never took over.
- Alter a reported version or score, or omit a failed attempt, without matching execution evidence.Reconciliation must flag the mismatch or missing evidence; a self-reported hash must not make the record valid.
- Attempt to change the final evaluator or read protected assessment answers or scores.The boundary must reject the operation or mark the affected run invalid; no resulting gain counts.
- Use extra model calls, training or scientific queries outside the declared resource ledger.Accounting must detect the discrepancy; the run cannot support a matched-budget benefit claim.
- Place duplicate or related experimental records, or derived run records, across development and final partitions.The group audit must reject the split as independent; changing row IDs or random seeds does not repair it.
- Execute a renamed identical improver, or a revision with only inert changes, while advancing the system version.Do not infer a changed improvement mechanism or causal benefit from the label or execution alone. Actual diffs and execution evidence are required; finite probe equality cannot prove universal equivalence.
- Let an editable improver selection policy accept every candidate that passes immutable validity, access and budget checks.Record real recursive use if present and expose any observed regressions through the fixed audit. A poor policy is not automatically invalid when rules are followed, and cannot fabricate a positive result.
Keep the evidence independent.
- Split before running. Separate development, version selection and final assessment by scientific source or problem family. Keep related system descendants and experiment series together.
- Distinguish experiment feedback from audit scores. A planner may receive the measurements allowed by its task. It may not use the final independent assessment to select or rewrite its submitted system.
- Track human access too. Keep a final-partition exposure ledger across runs and researchers. Changes made after viewing final results are exploratory; a new protected assessment is needed for a confirmatory claim.
- Count the complete cost. Include model calls, training, scientific queries, validation and rejected attempts. Equal iteration counts do not establish equal resource use.
- Preserve failures and uncertainty. Report invalid attempts, regressions and missing runs. Analyze independent workspaces or complete runs; sibling versions are correlated observations.
Predeclare the analysis unit, independent-group count, repetitions, smallest scientifically meaningful effect and uncertainty analysis. Do not invent a universal sample-size threshold. Small pilot results may be descriptive; failure to detect a gain is not evidence of equivalence.
Public source data may have been seen before. Isolation during a run does not establish freedom from prior exposure. Measured, computed and simulated observations must remain distinguishable.
Before a release: what a resource needs.
Source listings record the original link and version, scientific use, citation, access terms and which metadata was checked. Full datasets can remain in their original repositories.
Checkable task packages also need defined inputs and outputs, a runnable checker, documented feedback, independent grouping, a usage example and verified reuse terms. A package must pass format checks and human scientific review before being described as reviewed.
Researchers use these resources with their own models and improvement methods. A complete RSI study is optional; claims evaluated under this protocol require the controlled experiments described above.
Report gains, regressions and null results. A well-controlled null result is reportable. Its informativeness depends on independent coverage and uncertainty; an underpowered pilot cannot establish that recursion has no useful effect.
The molecular starter illustrates task interfaces and checks; the synthetic oscillator package illustrates the checker format only.
Research foundations.
Self-modifying agents and improvement-capability tests have prior work. Our contribution is scientific tasks, diagnostic evidence and evaluation resources that researchers can use to study improvement and transfer.
- STOP: a program that improves programs, including itself.
- HyperAgents: editable improvement mechanisms and tests of subsequent improvement capability.
- RSIBench-Data: controlled evaluation of data-centric research for model improvement.
- RSI-Exam: executable research tasks with independent final evaluation.