Some time in April, Google put out a paper called PaperOrchestra (arXiv:2604.05018). The pitch is straightforward. You give the system five things: an idea, an experimental log, a venue template, conference guidelines, and optionally some figures. It produces a submission-ready LaTeX manuscript, with a real bibliography and synthesised figures. The architecture is a five-agent DAG. Outline, then literature and plotting in parallel, then section writing, then a refinement loop with simulated peer review. The authors publish their prompts verbatim in Appendix F.1, which is generous. It means you can reproduce the paper from the preprint alone.

I did that mostly to stress-test the Claude Agent SDK's multi-agent primitives. Seven distinct subagents with different tool surfaces, structured outputs across stage boundaries, a refinement loop that bypasses the master orchestrator entirely. If the SDK can hold that shape, it can hold most of what I want to do at work.

The agents themselves are not the interesting part. Those come from the paper. The interesting parts are the bits the paper doesn't talk about: how you make a multi-agent system safe to re-run when it crashes in wave three, what catches the LLM when it makes a number up, how you verify that citations actually exist, and why a single reviewer kept rubber-stamping its own writer.

The shape

The orchestrator is a flat async state machine. Each wave writes a checkpoint when it finishes. A crashed run resumes from the last completed wave instead of starting over. That sounds boring until you watch the literature stage spend twenty minutes hitting the Semantic Scholar API, only to die in the plotting stage and need to rerun the lit search from scratch. Then it sounds essential.

The master prompt for each wave is intentionally tiny. Wave one is three lines telling Claude to call the outline subagent and reply DONE. All real work lives in the subagent. Master context stays small, subagents get long focused prompts, and there's nowhere for the master to second-guess the work.

A short list of things I added that the paper doesn't describe.

The retrieval bit

The literature stage is the part I'd point a working RAG engineer at. The agent is given a structured search plan from the outline. It issues web searches, one per query, and collects candidates with title, snippet, and URL. It deduplicates them by normalised title (lowercase, strip non-word characters, collapse whitespace). Then it sends every survivor to Semantic Scholar and only keeps the ones S2 can verify: title matches within a fuzzy threshold, abstract exists, year is before the cutoff.

There are no embeddings anywhere in this. The task is "does this paper exist, with this title, before this cutoff?" That's a string-similarity problem against a canonical source. Adding an embedding index would be carry without benefit.

Two cheap checks close the loop after the writing agent does its draft. One asks whether the draft cites most of what was verified, so the agent doesn't quietly drop ninety percent of the work the lit stage did. The other asks whether every cite in the draft has a matching verified record, so the agent doesn't invent a key that happens to compile because a BibTeX entry exists for it.

That second check matters more than it sounds. Hallucinated cites are the failure mode that survives a casual read. The text looks fluent, the BibTeX entry exists, the paper compiles, and the citation points nowhere. The integrity gate catches that by demanding a traceable identifier all the way back to Semantic Scholar.

The reviewer pool, and why it exists

The paper hands peer review to a single scorer and treats whatever comes back as ground truth. When I plugged a single Opus reviewer into the same loop, it kept agreeing with the writer. The writer would propose an edit. The reviewer would say nice things. The writer would propose another edit. The reviewer would say more nice things. Scores crept up monotonically. The paper got worse. There's a name for that failure mode and I don't think it's flattering.

So I made the reviewer two reviewers with different priors. One leans on empirical rigour, the kind of voice that flags missing baselines and weak ablations. The other leans on theoretical clarity, the kind that wants problem statements and assumptions nailed down. They score the same iteration independently. If their overall scores land within a couple of points of each other, that's consensus and the loop moves on. If they diverge by more than three points, a third call (call it the area chair) reads both prior reviews and breaks the tie. That's the same shape an actual program committee runs.

It's a small piece of theatre. But the disagreement-as-signal pattern made the loop honest. The writer now had to address consensus weaknesses, not whatever the single reviewer happened to fixate on this turn.

One more thing about the loop. The reviser can produce a worse paper. So at the end, the system promotes the best-scoring snapshot, not the last iteration. A real run on the smoke-test workspace did exactly this: iteration one tried to rewrite too much and was rejected by a diff-size guard, iteration two passed and scored slightly above baseline, iteration three regressed and was reverted. Final paper is iteration two.

What is still hard

Five things I'd tell you if you asked.

The diagram pipeline is wired but only really tested end-to-end against one workspace. There's a live integration test that runs out of band when the diagram service is reachable.

Wave two is supposed to run literature and plotting in parallel. Mine runs them sequentially because the master Claude sometimes drops one of the two subagents when both are offered in the same turn. Correctness beats parallelism, but I'd like that back eventually.

The reviewer occasionally emits JSON in a slightly wrong shape, nesting a field where it should sit at the top level. I have a repair helper for truncated JSON. I don't yet have one for structurally wrong but complete JSON.

The LaTeX compile-fix loop is half-wired. The agent currently reads compiler errors and edits in-LLM, which works, but a small auto-fixer would be tidier.

The intent-check only runs against the single top weakness per iteration. Bounding it that way caps Opus cost at one call per iteration, but it leaves multi-weakness verification on the table.

None of these are blockers. They're the kind of things you find when the loop runs against real inputs for hours, and the kind of things that don't show up in the paper because the paper isn't trying to run on real inputs for hours.