# Build recovery tasks from known scientific records

A construction recipe illustrated by [MolP2SBench](https://arxiv.org/abs/2610.12522). This guide describes how to prepare resources for learning and improvement experiments. Researchers supply and run their own models and improvement methods.

## The design idea

Start with a complete scientific record whose reference is known. Hide a meaningful part from the solver, retain the evidence needed to infer it, and preserve the reference for checking. MolP2SBench turns structure–property records into local molecular recovery tasks: a masked scaffold, supplied physical properties and a short set of valid completions.

This construction gives an auditable target while concentrating the task on a controlled scientific inference. The target is known to the evaluator before masking. Whether the visible evidence distinguishes that target is a separate scientific check.

## Prepare a task family

1. **Keep a traceable reference.** Preserve the complete structure, aligned observations, source version, units, conditions and reuse terms. Distinguish measured, computed and simulated quantities. The website's dipole example uses computed QM9S values.
2. **Choose a meaningful mask.** Hide a property-relevant connected fragment while retaining the surrounding scaffold and attachment constraints. Document which information the solver receives and keep reference fields out of its input. Declare permitted tools and retrieval: the source benchmark's closed comparison supplies task text only, without tools, web search or literature retrieval.
3. **Assemble valid alternatives.** Include the source completion and chemically valid, structurally distinct candidates compatible with the scaffold. MolP2SBench uses two to four candidates; its manuscript (Section 3.1 and Appendix B.2) describes property-specific separation checks. Candidate validity and evidence distinguishability require scientific review.
4. **Define scoring and development feedback.** Store the target separately. Report correct and incorrect choices, with diagnostics for missing, malformed or out-of-range outputs, under a fixed scoring policy. Retain the family ID, evidence condition and prediction so outcomes can be compared. Specify when development labels and feedback become available.
5. **Add controlled views and independent tests.** Build scientifically justified interventions, keep all related views and descendants together, and separate development from final assessment before release. Donor pools used for shuffled controls must respect that boundary. Record versions and budgets in the user experiment.

Known reference identity does not establish unique inverse identification over all possible molecules. If the evidence leaves multiple candidates indistinguishable, add evidence, change the candidate set, accept a scientifically justified set of answers, or label the case as indeterminate. Record the treatment and exclusion criteria before evaluation.

## Reference answers and explanations

A checkable task must provide a verifiable reference answer, ground-truth label or scientific evaluation rule, together with its source and validation limits. A complete canonical reasoning trace is **optional**. A contributor can provide an outcome-checkable task without a complete step-by-step solution, but must state what the checker establishes and what remains unchecked.

Reference labels establish candidate recovery; they do not establish the truth of an explanation or reveal why a model chose an answer. When a task also assesses explanations, provide separate reference facts and a declared checking method. Label unverified rationales as hypotheses, and never treat generated text as a validated scientific explanation.

## What the evidence controls test

MolP2SBench releases three evidence conditions—correct, shuffled and counterfactual—over the task families. They describe evidence interventions, not the number of questions or candidates. A structure-only comparison is an additional control. The checked source has 136 families with 2–4 candidates per task; the website uses 2 observations and 2 candidates from 1 family.

| View | Construction and scoring | Question |
| --- | --- | --- |
| Correctly paired | Supply the source molecule's properties; score against its target. | Can the model recover the reference with matched evidence? |
| Structure only | Keep the scaffold and candidates; omit properties; retain the source target. | How much performance comes from structural context alone? |
| Shuffled | Supply properties from another family; retain the source target. | What benefit comes from correct property–structure pairing across cases? |
| Counterfactual | Supply aligned properties of another valid candidate in the same family; update the target to that candidate. | Does recovery track the corresponding change in evidence? |

Keep scaffold, candidate order and evaluation cohort matched across paired views. A shuffled donor may match none of the listed candidates; it is an aggregate control, not an ordinary additional matching question. Report correctly paired recovery together with the effect of shuffling and movement toward the specific counterfactual candidate. A shuffle-related drop plus a donor-directed counterfactual response supports a behavioral evidence-use claim scoped to the tested model and property. Compare source and counterfactual correctness together; simply depressing shuffled performance or changing to an unrelated candidate does not establish better recovery.

## How this supports learning and RSI research

Development labels can support supervised learning or a reward signal for candidate recovery. Controlled-view outcomes can help researchers choose diagnostic tests, locate behavioral weaknesses and compare proposed revisions. An outcome score validates the selected candidate; checks on scientific explanations require separate references and criteria.

A researcher can connect the package to a workflow that predicts, scores, tests a failure hypothesis, updates a model or agent, and evaluates on independent families. Improvements measured on exposed examples must remain separate from transfer results. Record failed attempts, regressions and total cost alongside successful changes.

An RSI study also checks whether an updated capability helps the system make later improvements. One stronger test under the draft protocol updates the procedure that proposes or selects improvements and uses that updated procedure again. Compare old and new procedures from common starting systems and budgets, and compare full runs against a fixed-procedure control. This resource design supports those experiments; it does not establish their outcome.

## Adapt the recipe to your research

The reusable pattern is **known record → meaningful missing information → justified alternatives → verifiable recovery → controlled feedback**. Candidate extensions include recovering a missing equation term from a documented dynamical system, or identifying a local material-structure completion from measured properties. These are proposed applications; each needs its own identifiability, validity and leakage checks.

Begin a contribution with one representative record, what would be hidden, which evidence would remain, how candidates and targets would be constructed, and what the checker would verify. Include the original paper or source, reuse terms and a grouping plan. Partial resources can enter the resource-review route before a complete task package exists. Automatic format checks precede human scientific, usability and rights review.

Audit answer lookup separately from scientific ambiguity. Public property values can act as database lookup keys; returning a stored source structure then tests a different capability from inferring a completion from the allowed evidence. For a tool-using extension, declare permitted scientific computation and retrieval, isolate protected references, and record access. Keep its results distinct from the source task-only comparison. Public records can also have appeared in model training; grouping alone cannot establish absence of prior exposure.

## Source and current scope

**Benchmark paper and task design:** Yuhang Xie, Zhiyi Shi, Zeyi Zhang, Zhisheng Qi, Yushun Dong, Yu Wang, and Xiongye Xiao (2026). [*MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?*](https://arxiv.org/abs/2610.12522). arXiv:2610.12522.

**Underlying molecular property source:** Zihan Zou et al. (2023). [*A deep learning model for predicting selected organic molecular spectra*](https://doi.org/10.1038/s43588-023-00550-y). *Nature Computational Science* 3, 957–964. Identify the [QM9S dataset release](https://figshare.com/articles/dataset/QM9S_dataset/24235333) used as well. These attributions serve different roles; citing the benchmark does not replace attribution of the underlying data.

- [MolP2SBench source and task format](https://github.com/DUAL-Xiao/MolP2SBench/tree/1ede82bf68f634097b296b1937d66a9b04bbf269), pinned for this illustration. The accompanying manuscript describes masking and candidate construction in Sections 3.1–3.2 and Appendix B.
- The website walkthrough contains two observations and two candidates from one teaching family. It provides choice feedback and a proposed revision/test plan; the downloadable starter scores a user's model predictions. Independent evaluation families and controlled RSI results remain future work.
- Follow the original data and benchmark reuse terms. This platform’s teaching adaptation is shared with project-lead authorization. Original source terms remain separate; this grants no blanket license to the full upstream benchmark.