A research loop that shows its work.
You steer. The model reads, proposes, experiments and writes.
Code checks every citation, number and claim against the record before it reaches the page.
Quick start · The web UI · What it does · How it works · The rules · Recorded runs
Ask a language model to "do research" and you get a confident paper. The trouble starts when you check it: a citation that does not exist, a number nobody measured, a headline idea that was never actually run.
ScholarLoop keeps the model doing the thinking and puts a harness around it that keeps the books. When an output breaks a rule, the harness rejects it, hands the errors back, and the model tries again. You stay at the decisions that need judgment.
|
🧭 Ideation you steer It will not start from a bare topic. You say what you want to find out and your hunch; it maps directions from the literature, and at least one must challenge you. |
🧪 Ideas that actually get tested The idea you pick has to run as a treatment against a matched control. The loop cannot wander off and tune hyperparameters instead. |
|
🔒 Scores the model cannot touch Training only saves weights. A separate, frozen scorer produces every reported number, and the held-out split stays sealed during exploration. |
⚖️ Review before prose Four critics question every claim against the measurements. They can demand a control, which is frozen as a protocol before it runs. Nothing is written until you approve. |
|
🔗 Every sentence traceable Each sentence of the paper cites an approved statement, each statement its runs, each run its code version. Change the evidence and your approval is void. |
🖥️ A web UI, under a hard budget Run every stage from a local web UI or the command line. A dollar cap is checked before each model request, and the UI tells you which tools are free to run. |
git clone https://github.com/r2m-ai/scholar-loop.git && cd scholar-loop
pip install -e ".[dev]"
python3 -m scholarloop.ui # Windows: python -m scholarloop.uiYour browser opens on http://127.0.0.1:8765. Click Explore a real run to walk through four
recorded runs, from the author's steer to the reviewed paper and the
evidence behind each sentence. No API key, no GPU.
| I want to… | Run | Costs |
|---|---|---|
| browse finished research | python3 -m scholarloop.ui, then Explore a real run |
free |
| watch the loop on a scripted model | python examples/quickstart.py · ideation_demo.py · writing_demo.py |
free |
| run real torch training, scripted model | pip install -e ".[dev,engines]", then python examples/campaign_demo.py |
free |
| do live research | pip install -e ".[dev,engines,llm]", add your Anthropic key under Connections in the UI |
API spend, capped |
Every script is listed in examples/README.md. pytest -q runs the
offline tests.
The UI runs on your machine only. It starts the same stage commands as the command line, keeps their logs, and takes each of your decisions as an explicit action. A recorded run opens read-only, so you can look at everything without spending anything.
- Start the UI and click Explore a real run. Four studies appear under Your research.
- Research Ideas: read each idea's hypothesis, compare tournament ratings, open a source on arXiv.
- Experiments: see one bar per arm and the record of every run.
- Scientific Review: read the verdict, then unfold any issue to see how it was closed.
- Writing: open the manuscript, then the sentence-by-sentence trace to see the approved statement behind each sentence, and the evidence map to reach its runs and code versions.
- Click New research, pick a domain, and set a spending cap.
- Add your Anthropic key under Connections (it stays in memory on your machine).
- Literature & Directions: write what you want to find out, run the review, choose directions.
- Research Ideas: generate ideas, refine them with feedback or add your own, and select one.
- Experiments: start the campaign and watch runs arrive; each stage runs as a background job with a live log, and you can stop it at any time.
- Scientific Review: let the four critics examine the argument, answer what they raise, and approve.
- Writing: step through outline, key points, manuscript and panel review.
| Page | What you find there |
|---|---|
| Literature & Directions | your steer, the literature review, and the direction map with each direction's open question |
| Research Ideas | the candidate ideas, their tournament, and the one you picked |
| Experiments | every run with its stage, role and score, and one bar per fully evaluated arm |
| Scientific Review | the verdict, each critic's summary, and the issues grouped by state and severity |
| Writing | the manuscript and what each writing stage produced |
| Artifacts | every file of the run, grouped by the stage that wrote it |
| Evaluation & Tools | diagnostics and reliability checks, split into free ones and ones that call the model |
- Your steer comes first. What you want to find out, the hunches you want tested, what to stay away from. It shapes the search queries.
- A literature review you can push back on. Papers come from arXiv and OpenAlex. Every cited identifier must be a paper that was retrieved.
- A direction map. Three to five lines of inquiry, each with an open question, a cheap first experiment and the most likely way it turns out dull. It may not only agree with you. You choose directions, or write your own.
- A tournament the model cannot rig. Judges compare every pair of ideas twice with the order swapped; a win in one order only is a draw. Elo is computed in code.
- Your pick. Choose one, reject them all, or bring your own idea.
Each round the Reasoner proposes runs, three critics vote on whether each deserves compute, and the survivors climb a funnel of one seed, then three, then five. Most ideas die after the first cheap run.
- Your idea cannot be tuned around. Every proposal is labelled a
treatmentor its matchedcontrol. The loop may not change direction until a treatment has run. If no hyperparameter expresses the idea, the Reasoner rewrites the training script in an isolated copy. - The model never reports its own score. A frozen scorer in a separate process does.
- Predictions are checked. Each proposal predicts its effect before it runs, and each agent sees its track record.
- The test set stays sealed. Opening it ends exploration for that campaign.
- Scientific review. The argument is built as statements: claims bound to the measurements that carry their numbers, setup, related work, limitations. Four critics go through it claim by claim, and issues stay open across rounds until a critic records why they are settled.
- Requested controls become frozen protocols. Arms, seeds, the prediction and the outcome that would falsify it are hashed before any arm trains. Negative results are kept.
- Your approval is bound to the evidence. Change a run, the argument or the code afterwards and it is void.
- Writing in stages. A panel argues over the outline, then the key points, and you sign off on each. Sections are then drafted so that every sentence restates an approved statement.
- What you get. A reader's edition, a traced edition with the statement behind each sentence, an evidence map, and three simulated reviewers' reports.
Command lines and scripted modes for each stage are in REFERENCE.md.
These live in code. A prompt can ask for good behaviour; a check can refuse the output.
| Rule | What gets rejected |
|---|---|
| Citations must exist | a paper id that is not in the retrieved corpus, from the review to the final prose |
| Numbers must be measured | a number no measurement supports; in the paper, a number that is not in the statement the sentence cites |
| Evidence must exist | an unknown run or statement id; a result claim with no evidence behind it |
| Scores come from the frozen scorer | a metric from anywhere else; an edit outside the training script; a scorer file that changed mid-run |
| The idea gets tested | a second control before the first treatment |
| Protocols are frozen first | a protocol run whose arms, code or seeds differ from what was hashed |
| The test set stays sealed | scoring it during exploration; exploring after it was opened |
| Approval follows the evidence | writing against a review whose evidence, argument or code has changed |
| Rankings must be robust | a tournament win that holds in only one order |
| Budgets hold | requests past a call cap, training runs past a reservation, spend past a dollar cap |
Every attempt by every agent is logged with its output, its validation errors and its outcome, so the rejected attempts stay on the record too.
The checks are tested, not asserted. On three complete runs with claude-opus-5-5:
| fabricated citations, numbers and run ids rejected and 0 of 41 clean artifacts rejected |
faults injected one at a time into accepted artifacts run it yourself from Evaluation & Tools, free |
claims that go beyond the evidence ScholarLoop paper vs a single-call writer |
Each went from a one-sentence steer to a reviewed paper. All three open in the web UI, and their
files are in examples/sample_runs/.
| Run | What was asked | What the evidence said | Review rounds / approvals | Cost |
|---|---|---|---|---|
| Distillation | does an adaptive switch from teacher to student prefixes help | the registered prediction was falsified | 10 / 3 | $34.40 |
| AG News | which boosting regularizers transfer to a small text classifier | partly held; the named comparator never ran | 2 / 1 | $13.39 |
| Digits | does a skip connection change a shallow MLP's final error | neither supported nor falsified | 7 / 2 | $25.66 |
The full measurements, with every table, are in REFERENCE.md.
A domain is a YAML profile plus two files: a frozen prepare.py that owns the data, the model
definition and the scorer, and a train.py the agent may rewrite. Adding a domain does not
touch the orchestrator.
| Profile | Task | Notes |
|---|---|---|
digits-mlp |
8x8 digit classification with a small MLP | no download · 1-2 s per run on CPU |
agnews-bow |
AG News topic classification, bag of embeddings | about 1 s per run · one-time download (".[nlp]") |
seq-distill |
distilling a small sequence model from a frozen teacher | no download · trains the teacher once |
image-classification |
deterministic stub | powers the free demos and the tests |
| Control | What it does |
|---|---|
SCHOLARLOOP_SPEND_CAP + SCHOLARLOOP_SPEND_FILE |
a hard dollar cap shared by every command that points at the same file, checked before each request |
--max-llm-calls, --max-calls |
a cap on model requests that survives restarts and counts retries |
| training reservations | a cap on training attempts; failed attempts still count |
In the web UI the cap is a field on each study. With a cap set, a model that has no listed price is refused.
scholarloop/
├── core/ agent harness, model clients, profiles, ledger, registry, budgets
├── ideation/ retrieval, literature review, steer and direction map, ideator, tournament
├── experiment/ orchestrator, runner, reasoner, debate, governor, experiment agenda
├── review/ evidence dossier, critics, issue history, author release
├── writing/ staged writing session and writing guides
├── ui/ the local web UI: server, job runner, run store, static front end
├── reliability.py fault injection, attempt-log analysis, single-call baseline, claim audit
├── fidelity_study.py does a campaign test the idea it was handed, with and without enforcement
├── human_eval.py builds the blinded rater packet for the claim audit and scores the returns
└── evaluation.py read-only run diagnostics
engines/ prepare.py (frozen) + train.py (editable) per domain
profiles/ domain profiles
examples/ runnable scripts, each listed in examples/README.md
└── sample_runs/ four recorded runs with their records; browse them in the web UI
tests/ mirrors the package and runs offline
REFERENCE.md: command lines for every stage, scripted modes, release and final-test rules, comparison harnesses, environment variables.examples/README.md: what each example script does and what it needs.examples/sample_runs/: the recorded runs.
MIT.



