Prepare a dataset for review
Prepare a checkable scientific data package. A paper, model or RSI experiment is optional. To list an existing external dataset, start with a source suggestion; the structured-file checks below apply to benchmark packages.
Local checks are available now. For human review, contact us with a report and public source links to arrange review. Your full dataset can stay with its current host; see the external-resource and package review routes.
Current checker format: JSON records with scalar fields (number, integer, string or boolean) and reference targets. Images, arrays, nested records, simulators and unlabeled data need a format discussion first; preserve their native format and suggest the source. Protected final-test data stays with its custodian.
Start with two files
manifest.json- Describe the source, version, fields, units, related families, coverage, splits and release/access plan. If no reserved evaluation set exists, use
protected_evaluation: {"availability":"none"}. Otherwise describe its actual custody and access. records.json- Provide the declared development/selection records. Each has an ID, family, split, input and reference target.
For human review, add a usage card with a readable case, loading instructions and expected output, plus links to the original data and available checker.
What makes a resource useful?
These criteria apply to checkable data and benchmark packages. Dataset, task and benchmark definitions. Source suggestions can begin with fewer pieces.
- Traceable science
- Original source and version; measured, computed or synthetic origin; units, missing cases and access terms.
- Checkable outcomes
- A reference target or scientific check, its scoring rule and its limits. Explain what a correct result establishes.
- Useful feedback
- A readable record and loading example. For benchmark tasks, add a prediction, score and feedback example, with the development outcomes users may inspect. Complete reasoning traces are optional.
- Fresh comparisons
- Group related scientific cases before splitting. Describe available development cases and any independent evaluation set; mark missing evaluation data explicitly.
- Reproducible use
- A versioned access route, setup instructions and expected output. A reviewer should be able to follow the guide from a clean download.
Automatic checks inspect format, records and declared contracts. Human reviewers check scientific meaning, useful feedback, independence, reproducibility and source permissions. A resource can be listed with a limited, clearly documented scope.
How dataset quality will be reported
We plan to report the criteria above as a version-specific quality profile. Each finding will identify the evidence, reviewer, review date and coverage. Contributor declarations, completed checks, independent verification and unresolved questions will be shown separately.
The current checker reports technical issues and readiness for human review. Numerical quality ratings are a future design question: format compliance alone cannot establish scientific validity or predict an RSI gain. A missing independent test set limits evaluation use; a useful development dataset can still be listed with that scope.
Additional requirements for an RSI study package
Choose track: "rsi_task" only when also supplying a proposed executable study. It adds baseline replay, permitted edits, feedback access, budgets, revision/install/reuse contracts, logs, checker fixtures and independent comparisons with a fixed improver. The built-in fixture checks support exact match and numeric tolerance. Domain-specific scoring needs a reviewed extension; no submitted code is executed. Ordinary data contributors can use the resource route.
RSI study manifest example · Matching study records · Research protocol
Check a package before review
Choose your manifest and records, or try an example. Files stay in your browser; the result is a downloadable check report.
What to fix
What the human reviewers will check
Read or copy the complete report
Run the same check offline
The standalone ZIP includes both examples, the format guide and exact commands. It uses Node.js 22.12 or later with no package installation, account or network connection.
Local limits: 1 MiB for the manifest and 64 MiB for records. Larger packages need an agreed review route; contact the maintainers with their format and size. Keep restricted data and final-test answers private.
After the check
- Changes needed
- Use the report’s rule IDs and locations to fix the files, then rerun.
- Sample checked
- A sample helps prepare a proposal. Review of a full package requires a check over all declared development/selection records. Reviewers verify the declared scope against the source.
- Ready for human review
- Save the report. To request review, contact us with the report, version, usage guide and public source links. Agree a permitted transfer route before sharing restricted files.
- Approved release
- Reviewers rerun the checker and review the science, reproducibility and reuse permissions for the exact version. A maintainer then approves its listing and permitted downloads.
Running the checker sends nothing. Passing it records a local result; human approval is a separate decision. Changed files need a fresh report.