A published meta-analysis is a frozen artifact. The day after it appears, new trials start coming out, and within a few years the conclusions drift. The job I had was building a system that takes one of these papers and produces an updated version using the evidence that arrived after its cutoff. A clinical statistician would do this by hand over weeks: redo the search, screen thousands of abstracts, pull full texts, extract outcomes, harmonise against the original review's data table, re-run the pooled statistics. The mechanical work is enormous. The clinical judgment underneath is small but irreducible. Our goal was never to replace the statistician, it was to take the mechanical work to a place where one could sign off on it.

I built this pipeline once and got it wrong. The way I got it wrong is the interesting part.

The shape of the problem

Seven phases, in both versions: parse, search, screen, retrieve, extract, combine, pool. The interesting thing isn't any single phase. It's that every phase produces structured artifacts the next phase reads, and every paper finds a new way for those artifacts to be wrong.

Some things stay with the human by design: GRADE certainty, clinical interpretation, funnel-plot reading. Crossing into clinical judgment turns the tool into a medical device, with regulatory consequences. That line stayed bright in both versions.

The first attempt

The first version was a tidy state machine. A long orchestrator walked through the states one at a time. Each state loaded its checkpoint, ran its phase, validated the output, saved a section, moved on. No branching, no anomaly detection, no recovery beyond "halt and log degradation". A full run took thirty to sixty minutes, so checkpointing was non-negotiable.

One genuinely good idea lived in this version. Two independent agents would read the source paper, a third would reconcile them, and the reconciliation was done as a stream of patches against a canonical artifact. The patches were keyed by structural identity, not list index. A deterministic merger applied them, rejecting any patch that referred to something it couldn't find, or that would create a duplicate, or that landed outside scope. That boundary, the line between "an agent said so" and "the system believes it", is the most reusable thing in the entire first attempt.

The rest of it was a hazard waiting to bite.

Retrieval, the original sin

The first version's retrieval was a single function. The entire surface, for a system that needed full-text access to dozens of papers per run, was: if the user dropped a PDF in the right folder, use it; otherwise wait. It was a deliberate scope cut. Retrieval was treated as out-of-band so we could focus on the LLM-heavy phases.

The trade-off compounded everywhere. User PDFs don't bypass screening. If a paper never made it into the search results, having its PDF doesn't help, because the study must enter the pipeline through search first. So retrieval succeeded for the studies we happened to have local copies of, and silently failed for the rest, and the downstream phases had no way to tell which was which. The whole "every phase boundary is a contract" problem lives here.

How a tidy pipeline corrupts data

Three classes of failure dominated. They weren't bugs. They were structural properties of the architecture.

The first was ordering. Trial resolution, the step where you figure out which papers are different reports of the same trial, ran after screening. Screening therefore stamped every candidate as a new trial by default. Trial resolution later correctly clustered the papers, but by then the wrong role had already propagated into retrieval and extraction. A follow-up report of an already-included trial would get through screening as a new trial, get extracted, get pooled, and the original entry would still be sitting in the pool because the "replacement" lookup used a name resolved one way and the original baseline data used another. Same patients counted twice. I rewrote the combine phase three times trying to fix it. The problem wasn't combine. The problem was the order.

The second was schema rigidity. Per-phase schemas were tight, which was correct, because they're the contract between phases. But real papers contained things the schemas didn't anticipate. Statistical analyses with no valid category. Pooling methods not in the enum. Confidence scores defined as words and returned as numbers. Validation rejected. The repair agent was told to "read the schema" but never received the enum values. The repair used values the schema rejected. The whole repair was rolled back atomically, including the parts that were correct.

The third was the fallback trap, and this one taught me the most. When a secondary model failed to return decisions for a chunk of candidates, those defaulted to "uncertain" and got dumped on the adjudicator. The adjudicator's prompt had no signal that these were technical failures rather than real disagreements, so it made guesses on candidates that should have been re-screened. Degraded screening masquerading as completion. The data looked correct. There was no error log. It was wrong.

I had a one-line rule taped to the side of the project: a fallback is giving up and hiding it. The first version violated it everywhere it touched.

The rewrite

The second version's orchestrator is roughly a third the size of the first. The shrinkage isn't because the system got simpler. It's because the orchestrator is now genuinely just a coordinator. It runs through the phases, and after each one, it does something the first version never did: it looks at the world state, checks for things that shouldn't be true, and decides whether to keep going. If anomalies turn up, it surfaces structured options to a human and records the choice in an audit log.

Three things about the new shape mattered.

First, the phases overlap. In the old version, screening produced the full set of included studies, then retrieval ran, then extraction ran. In the new one, as soon as a screening batch produces include decisions, the matching studies queue for extraction while the next batch starts. The throughput change is real, but the change in the failure profile matters more. If screening fails partway through, the studies that already finished extraction are saved.

Second, the checkpoints are fingerprinted. Each phase hashes the canonical JSON of its inputs. On resume, if the stored fingerprint doesn't match the current inputs, the phase refuses to use stale progress unless you explicitly force a restart. The first version trusted its checkpoints blindly across input changes, and that was a quiet source of every "why is this run using old data" mystery.

Third, scope is a contract, resolved once at the start of the run, and frozen. Whether you're replicating the original study set or expanding into new evidence is a property the whole pipeline reads from one place. The old version let each phase make its own scope decisions, and they drifted apart.

Retrieval, take two

The new retrieval is a fallback chain. Try the canonical source first, then the open-access mirror, then a few smaller archives. If none of them yield enough text, hand back the abstract with a flag saying we couldn't get the full paper.

Each source has its own small helper behind a common interface, so reordering is a one-line edit. Every successful hit returns the text, the character count, and the name of the source that produced it. The attribution survives all the way to the final report.

The architectural point is in the fall-through. When source three fails, we try source four. We don't pretend source three answered the question. The terminal state, if every source fails, is a structured "needs human PDF", not a hallucinated abstract.

A note on vector retrieval. There isn't an embedding model anywhere in this chain. Vector search is excellent for finding documents that look like other documents. The agent in this system already has an identifier (a PubMed ID or a DOI). The question isn't "what's similar to this paper", it's "where can I get the full text of this exact paper". For that, you want a deterministic priority chain over structured sources, not a similarity score.

The schema underneath

The pipeline's data contract is a layered schema. One layer for the paper, one for each included study, one for each pooled analysis. The pivot from study-organised data to outcome-organised data happens between layers two and three. This solves a specific problem: extraction asks "for this paper, what did it report"; pooling asks "for this outcome, which studies contribute and what are their numbers". The first version didn't separate these cleanly. The second version makes the split a first-class schema decision.

Two things about this schema turned out to matter beyond clinical work.

Every methodological field that a paper might or might not explicitly report carries a small triple: the value, whether the paper stated it, and what we assumed if it didn't. The model has to declare whether it's reading or inferring. That distinction is structured uncertainty, written into the data, which means a later phase can read it and behave differently for stated facts versus assumptions.

Each field has a lifecycle tag that says when it gets written and by what. Some fields are written once at parse time and never change. Some get filled in per phase. Some are computed at the end and explicitly forbidden from being written by a model. The schema is the declaration of write semantics for the system's memory, which is the kind of thing any agent infrastructure has to decide eventually.

Evaluation as part of the pipeline

Every completed run produces a directory of checkpoints, audit logs, trace events, and CSVs. A separate evaluator reads all of it and applies a set of integrity checks. Did the original studies survive through combine when we were replicating a previously-published meta-analysis? Do the counts in the pooled output match the combined CSV? Are the effect measures in the report consistent with what the source paper declared? Are there any outcomes in the report that weren't in the methodology's declared list? Did any failed quality gates get silently accepted by a human override?

When we're using a previously-published meta-analysis as a validation gate, the evaluator compares pooled estimates against the original within a small tolerance. If the system can't reproduce the original numbers, it can't be trusted with new data. The replication check is codified, not aspirational.

A separate audit log records every decision the orchestrator makes, with timestamps, costs, and escalations. When a study unexpectedly fails to pool, you can read the exact sequence of decisions that led to the failure.

What changed, in one picture

The first version was sequential and all-or-nothing. The second interleaves the phases where it can, persists checkpoints with input-fingerprint validation, and ends with the evaluator gate.

What's still hard

Risk-of-bias assessment is the weakest automated link. The standard tool requires clinical reasoning across several signalling questions per domain, with conditional logic between them, and answers depend on inferences a model can't reliably make from a paper alone. The system drafts. A human signs off. That trade-off is explicit.

Cross-study population reconciliation is harder than it looks. Different trials report the same outcome over different denominators (intention-to-treat, safety-evaluable, per-protocol). Pooling these together inflates or deflates the effect. The pipeline doesn't yet separate them. This is also, interestingly, a place where vectors might actually help. Denominator-definition matching across heterogeneous prose is a label-semantic problem, and a domain-tuned embedding could earn its keep. Credit where due: similarity is the right primitive for that sub-problem, just not for retrieving the artifact.

Outcome label matching is partly solved and partly not. The current method handles token-level synonymy, where common medical abbreviations expand into their full forms. It doesn't handle conceptual synonymy where the words don't overlap, like "change from baseline" versus "mean difference". A threshold papers over some of this. A proper fix needs a domain-tuned embedding for the labels themselves.

GRADE certainty is out of scope by design. It requires synthesising risk of bias, inconsistency, indirectness, imprecision, and publication bias into a single rating, which is a clinical judgment the regulatory framework explicitly assigns to humans.

The lesson, plainly

The first version taught me that ordering is a hidden contract. When you put one phase before another in a pipeline, you're committing to the assumption that the earlier phase can't be wrong about the things the later phase reads. It usually is. The second version's design doesn't trust the order. It looks back at the world state after each step, and where it can't separate work cleanly, it lets the phases overlap so they can correct each other.

The other thing it taught me is that every fallback the first version had was a lie I was telling myself. A retrieval system that silently degrades to a worse source produces data that looks correct, has no error log, and is wrong. The fix isn't to be more careful. The fix is to make that kind of failure impossible to express in the architecture. The retrieval chain falls through, not down. The terminal state for failure is a structured "we couldn't get this", not a synthesised guess.

That's the rule that ended up holding the whole second version together: degradation has to be loud and structured. Everything else flows from that.