Draw the architecture of a knowledge system and you will draw a box labeled ingestion, somewhere on the left, with an arrow into the interesting part. The interesting part is the graph, the reasoning, the agent. The box is where the work goes to look finished. It is also where the system actually dies, quietly, months before anyone notices, because a fact that was parsed wrong or extracted wrong or resolved to the wrong entity does not announce itself. It sits in the graph looking exactly like a correct fact, and every clever thing downstream inherits the error and amplifies it. You cannot reason your way out of garbage you ingested. The model can only work with the structure it was handed.
This is the essay about the box. Not the conceptual version, which is essay two on bi-temporal modeling and essay three on the contradiction engine. This is the engineering version: the specific stages a document passes through on its way to becoming a set of resolved entities and sourced claims, and the specific way each stage fails when you skip the unglamorous work it demands. The pipeline is a sequence, and a sequence is only as strong as its weakest stage, because the output of each stage is the input to the next and an error introduced early is invisible by the time it reaches the part you were watching.
The sources make it harder than it looks. A production knowledge system does not ingest one clean format. It ingests books, transcripts, academic papers, and social posts, and each of those is a different kind of mess with a different failure mode, and a pipeline that handles one well will mangle the other three unless it knows the difference. So the right way to understand the pipeline is to walk it stage by stage and name, at each one, exactly how systems die there and what the discipline is that keeps them alive.
Four sources, four different deaths
The first mistake is treating ingestion as a single parser. The four common source types are four separate problems that happen to end in the same graph, each with its own failure mode.
A book or long PDF dies on layout. Multi-column pages, footnotes, running headers, and page furniture scramble reading order the moment you dump text naively, so sentences break mid-column and interleave with the next one, captions weld onto body paragraphs, and a copyright footer repeats into every chunk where it poisons extraction. The discipline is layout-aware parsing that detects columns and strips the furniture before a single token reaches the next stage. An academic paper is worse, because its meaning is bound to its sections: a claim in Methods means something different from the same sentence in Results, and a chunk that straddles the boundary into the reference list will have the extractor reading citations as claims. Tables carry the quantitative facts, and a generic parser either drops them or flattens them into a row of unlabeled cells that destroys the relationships they encoded.
A transcript dies on attribution and noise. Automatic speech recognition gives you imperfect speaker boundaries, so a claim gets pinned to the wrong speaker and the whole who-said-what structure inverts, and disfluencies and misrecognized terms manufacture false entities out of an "uh, you" or flip "no evidence of" into "evidence of." The discipline is diarizing and segmenting by speaker turn before chunking, and cleaning lightly enough that you normalize the noise without correcting the meaning into its opposite. A social post dies on context collapse. A single post is elliptical, its "this is wrong" or "he lied" pointing at a parent it no longer carries, so reading posts in isolation extracts claims with missing subjects, and a reply chain reassembled in the wrong order from paginated, partly-deleted data fabricates a conversation that never happened. The discipline is ingesting threads as trees with parent pointers and chunking at the level of a conversation segment, never a lone post. Four sources, four deaths, and the only defense is a pipeline that knows which one it is holding.
| Source type | How it dies | The discipline |
|---|---|---|
| Book or long PDF | Layout. Multi-column pages, footnotes, running headers and page furniture scramble reading order, captions weld onto body paragraphs, and a copyright footer repeats into every chunk. | Layout-aware parsing that detects columns and strips the furniture before a token reaches the next stage. |
| Academic paper | Section binding. A claim in Methods means something different from the same sentence in Results, a chunk straddling into the reference list has the extractor reading citations as claims, and a generic parser flattens tables into unlabeled cells. | Section-aware chunking that respects the boundary, and table handling that keeps the relationships the cells encoded. |
| Transcript | Attribution and noise. Imperfect speaker boundaries pin a claim to the wrong speaker, and misrecognized terms manufacture false entities or flip "no evidence of" into "evidence of." | Diarize and segment by speaker turn before chunking, and clean lightly enough to normalize noise without correcting meaning into its opposite. |
| Social post | Context collapse. A single post is elliptical, its "this is wrong" pointing at a parent it no longer carries, and a reply chain reassembled out of order fabricates a conversation that never happened. | Ingest threads as trees with parent pointers, and chunk at the level of a conversation segment. |
Chunking decides the atomic unit of truth
After parsing, the text gets cut into chunks, and this quiet step decides more than almost anything else in the pipeline, because the chunk is the atomic unit of evidence the rest of the system reasons over. Get the chunk wrong and everything downstream is capped, the way a foundation poured crooked caps how straight a building can ever be.
The naive approach is fixed-size chunks, a fixed number of tokens per slice, and it is popular because it is simple and it plays nicely with embeddings. It also produces the single most common ingestion bug, which is the chunk boundary that splits a claim. A statement that runs "in the 2020 trial the drug reduced mortality, therefore the guideline changed" gets cut down the middle, and now each half looks incomplete on its own. The extractor reading the first half omits the conclusion; the extractor reading the second half hallucinates the premise it cannot see. A qualifier severed from its claim is worse: cut "before 2019" or "if the temperature exceeds forty degrees" away from the statement it conditions, and an unconditional, wrong fact lands in the graph wearing the confidence of a measured one.
The fixes are known and they are not exotic. Overlapping windows, where each chunk repeats a stretch of the previous one, mean most claims appear whole in at least one chunk even when a boundary cuts them in another. Semantic chunking, where you segment on real boundaries, sentences, headings, speaker turns, discourse units, rather than on a token count, keeps the cut off the syntactic and discourse seams that carry meaning. The trap on the other side is over-segmentation: cut down to single sentences and you strip the context that disambiguates a "he" or a "this method," and the extractor starts guessing antecedents that live in a chunk it cannot see. The pattern that holds is to segment on natural boundaries and then group up to a token budget without ever splitting a paragraph or a turn mid-sentence, and to tune the size against your own corpus by reading the wrong answers and the chunks that produced them, not by inheriting a number from folklore.
| Chunking strategy | What it does | What it costs |
|---|---|---|
| Fixed-size | A fixed number of tokens per slice. Simple, and it plays nicely with embeddings. | Boundaries cut claims in half. One extractor omits the conclusion, the other hallucinates the premise, and a severed qualifier lands an unconditional wrong fact in the graph. |
| Overlapping windows | Each chunk repeats a stretch of the previous one. | Most claims appear whole in at least one chunk even when a boundary cuts them in another. |
| Semantic or structural | Segment on real boundaries: sentences, headings, speaker turns, discourse units. | Keeps the cut off the seams that carry meaning. |
| Over-segmented | Cut down to single sentences. | Strips the context that disambiguates a "he" or a "this method," and the extractor starts guessing antecedents from a chunk it cannot see. |
| The pattern that holds | Segment on natural boundaries, then group up to a token budget without splitting a paragraph or a turn mid-sentence. | The size gets tuned against your own corpus, by reading the wrong answers and the chunks that produced them. |
Extraction is the cap on everything
Now the chunk goes to a model that pulls out the entities and the relationships, and this is the stage that sets the ceiling for the whole system. A relationship the extractor never produces cannot be invented downstream without hallucinating it, and a relationship it produces wrong cannot be reasoned back to correct without first detecting that it is wrong. Extraction quality sets the cap, and every stage after it operates under that cap.
The dial you are setting is precision against recall, and it is a real fork, because you do not get both for free. Tune for precision, instruct the model to extract only what the text states outright, and you get a sparse, reliable graph that misses the implicit and the paraphrased. Tune for recall, let the model capture every plausible relation, and you get a dense graph laced with hallucinations: relationships that are true in the world but unsupported by this text, because the model knows from training that insulin treats diabetes and writes the edge even when the document never said it. Most teams run a high-recall prompt without realizing it and then blame the model for the noise they asked for.
The discipline that makes recall safe is evidence-anchored extraction. You require, for every triple, the surface form of the subject, predicate, and object plus the exact span of text it came from, and at ingest time you discard any triple whose evidence is missing or thin. A strict JSON schema does the structural half: controlled types for predicates, type constraints on dates and numbers, a parser that rejects anything that does not validate and re-queries it. The shape that works in practice is two stages: a higher-recall generative pass that casts wide, then a verification pass, a second model or a rule set, that checks each candidate triple is actually quoted in the chunk and drops the ones that are not. The recall pass finds the implicit relations; the verify pass kills the hallucinated ones; and the graph you keep is the intersection of what was said and what can be shown.
One coreference problem deserves its own mention because it fails quietly across the seam between chunks. Within a chunk, the model resolves "the company" and "he" well enough. Across chunks, the same real entity appears as "Microsoft," then "the company," then "MSFT," then "we," and without a resolution step that spans chunks you get four nodes where there should be one and the relationships fragment across them. The pattern is local-then-global: extract entities per chunk with local identifiers, carry the document, section, and speaker as features on each mention, and let a separate resolution stage merge the local mentions into global nodes using those features as signal. Which is the next stage, and the next place systems die.
The over-linking trap
Before the resolution stage, one specific extraction failure is worth isolating because it is subtle and common and it corrupts the graph in a way that looks plausible. Put two unrelated true facts in one chunk and a high-recall extractor will often invent a relationship between them, because they are near each other and the model is rewarded for finding connections. "Barack Obama visited Paris. Macron is the president of France." Two correct claims, and the extractor, reading them together, writes a third that nobody said: Obama met Macron. The edge is consistent with world knowledge, which is exactly why it slips past a careless reviewer, and it is unsupported by the text, which is exactly why it is wrong.
Over-linking is why same-chunk co-mention cannot be treated as evidence of a relationship, and why the evidence-anchored rule from the previous stage earns its cost. The defense is the same span requirement applied with teeth: a relationship that cannot point at the clause that states it does not enter the graph, no matter how reasonable it sounds. The cost of the false edge is high precisely because it is reasonable, since the reasonable wrong edge is the one that survives review and quietly skews every traversal that crosses it.
Entity resolution: blocking before you compare
The resolution stage takes all the local entity mentions and decides which ones are the same real thing, and it is where toy systems that worked on a thousand documents hit the wall on a million. The wall is combinatorial. Comparing every mention against every other mention is quadratic, and quadratic is a synonym for impossible at scale, so the first move is never comparison. It is blocking: a cheap pass that partitions mentions into candidate groups so you only ever compare within a group, never across the whole set.
Blocking is borrowed straight from classical record linkage and it still works. Cheap keys, a normalized name, a phonetic encoding for person names, a hashed prefix plus a coarse location, sort mentions into blocks, and you compare only within a block. The modern addition is embedding-based blocking: embed each entity label with its context and use an approximate-nearest-neighbor index to fetch the top handful of candidates for a new mention, then combine that with cheap lexical keys to drop the absurd comparisons. The danger is in the tuning. Block too strictly and true matches land in different blocks and become permanent duplicates the system can never reconcile. Block too loosely and you rebuild a block so large it is the quadratic problem again. Once you have candidate pairs, a scoring model, string similarity, embedding cosine, attribute overlap, source-type weight, decides same or different. Blocking is what makes that scoring affordable, and getting it wrong is how resolution either misses real matches or never finishes.
| Blocking key | What it buckets on |
|---|---|
| Normalized name | The cheapest lexical key, and the one that catches most true pairs. |
| Phonetic encoding | Person names that sound alike and are spelled apart. |
| Hashed prefix plus coarse location | A second cheap axis, so a single weak key does not decide the bucket alone. |
| Embedding neighbours | Embed the entity label with its context and pull the top handful from an approximate-nearest-neighbor index, combined with the lexical keys to drop the absurd comparisons. |
| Tuning failure | What happens |
|---|---|
| Block too strictly | True matches land in different blocks and become permanent duplicates the system can never reconcile. |
| Block too loosely | The block grows until it is the quadratic problem again, and resolution never finishes. |
The transitivity cascade
Pairwise scores are not the end, because matches have to be consistent with each other, and that consistency is where resolution turns dangerous. If A matches B and B matches C, you usually want A, B, and C in one cluster, one entity node. But real data is noisy, and the case that breaks systems is the one where A matches B strongly, B matches C strongly, and A against C is borderline or outright wrong, two different John Smiths who each resemble a third record from different angles. Naive union-find on the match edges merges all three anyway, and now two distinct people share a node, every fact about either contaminates the other, and the merge is almost impossible to unwind because the system kept no record of why it joined them.
The fix is to treat resolution as graph clustering rather than chained merging. Mentions are nodes, pairwise scores are weighted edges, and you cluster with thresholds that refuse to let a weak link force a merge, correlation clustering rather than blind connected-components. Every cluster carries a confidence and a provenance trail of which mentions joined and why, so that when a later document contradicts the merge you can split it instead of living with it. And because re-resolving the entire graph on every new document is too expensive, resolution runs incrementally: stable internal ids on the entities you have, candidate generation for each new mention against the existing nodes, a decision to attach to an existing entity or mint a new one, and a change log of every resolution decision so you can re-run a slice of the work when you improve the model. The system that survives is the one that can admit a merge was wrong and take it back.
Conflict, change, or a merge gone wrong
When two claims about the same entity disagree, the pipeline has to decide what the disagreement means, and there are three causes that look identical at the surface and demand opposite responses. Calling all three a contradiction is how the contradiction engine fills with noise.
The first and most common cause is a resolution error wearing a contradiction's clothes: you merged two different entities, and the conflict you are seeing is the seam between them, two birth dates that disagree only because one node is secretly two people. Before flagging any contradiction, the check is whether the conflicting claims hang off the same resolved entity and whether that entity's resolution confidence is high; a low-confidence merge is a suspect, not a contradiction. The second cause is temporal change. A company had one chief executive, then another, and their tenures do not overlap, so the two names sit together without conflict. This is the bi-temporal model from essay two doing its job: claims carry valid-time intervals, and two values for a functional property are a contradiction only when their intervals overlap without explanation. The third cause is the genuine article: the same entity, the same property, an overlapping time window, and incompatible values, one source saying the drug reduces mortality and another saying it does not, for the same population at the same time.
Telling them apart takes structural work that no prompt supplies. You distinguish functional properties, the ones with at most one true value at a time, a birth date, a current employer, from multi-valued ones, awards, subsidiaries, interests, because only functional properties produce contradictions from a second value. You attach source, publication time, and extraction evidence to every claim so a conflict resolution policy has something to weigh: prefer the more recent source for time-sensitive facts, prefer the higher-authority source, or, the move the contradiction engine essay argues for, refuse to force a single truth and instead let incompatible claims coexist marked as disputed, separating what the sources say from what the system asserts. The mechanics can be as low as numeric-range and negation rules or as high as a model classifying two statements as same, refinement, temporal change, or contradiction. What matters is that the pipeline asks the resolution-error and temporal-change questions before it ever writes the word contradiction.
| The structural work | What it buys the pipeline |
|---|---|
| Separate functional properties from multi-valued ones | A birth date or a current employer holds at most one true value at a time, so a second value is a real signal. Awards, subsidiaries and interests take many values, and a second one means nothing. |
| Attach source, publication time, and extraction evidence to every claim | A conflict-resolution policy has something to weigh: prefer the more recent source for time-sensitive facts, prefer the higher-authority source, or let incompatible claims coexist marked as disputed. |
| Check resolution confidence before anything else | A low-confidence merge is a suspect, not a contradiction. Two birth dates that disagree may be one node that is secretly two people. |
| Check the valid-time intervals next | Two values for a functional property are a contradiction only when their intervals overlap without explanation. Two chief executives whose tenures do not overlap sit together without conflict. |
The engineering that keeps it alive
Everything above is algorithm. The reason well-designed algorithms still die in production is the operational layer underneath them, and it comes down to a few disciplines that are pure engineering and entirely unglamorous. The first is idempotency, the property that reprocessing a document does not duplicate it. It rests on stable identifiers: a document id derived from a canonical key or a hash of the raw bytes, a chunk id that is a deterministic function of the document id, the chunk index, and the document version, an extraction id built from those. Reprocess a document version and you overwrite every chunk and triple tied to it through those ids, expressed as upserts, and you leave tombstones for deleted documents so their stale embeddings and triples get purged instead of lingering as ghosts. Without stable ids, every reprocessing run, and there will be many, doubles your data.
The second discipline is surviving partial failure, because a multi-stage pipeline with model calls in it fails unevenly, a timeout here, a rate limit there, a model update that breaks a prompt. You persist the output of each stage, parse, clean, chunk, extract, resolve, write, so a failure retries only the stage that broke rather than the whole document. You distinguish transient failures, which you retry with backoff, from permanent ones, a schema bug, a pathological PDF, which you route to a dead-letter queue for a human instead of retrying forever. The third is ordering and supersession: documents carry a version or an as-of timestamp, and when a newer version arrives its claims supersede the old ones by an explicit per-predicate rule, replace the current employer, append to the list of past employers, so that even a late-arriving older document lands with the right temporal semantics rather than corrupting the present.
The fourth discipline is flow. The stages have wildly different throughput, optical character recognition and model extraction are slow and expensive, graph writes are fast, so you put queues between stages and let backpressure slow the upstream when the downstream saturates, and you batch model calls where you safely can without mixing unrelated chunks into one prompt and reintroducing the over-linking problem. A cheap pre-filter that drops spam and duplicates before the expensive stages is worth more than any optimization inside them, because the cheapest document to process is the one you decided not to.
How you find the death before it spreads
The last discipline is the one that lets you find all the others when they fail, and it is observability, because a pipeline you cannot see into dies in the dark. Every fact in the graph has to trace back to the source document and version, the chunk and its text, and the extraction call that produced it, so that when an edge is wrong you can walk back to exactly where it was born and reprocess the slice that produced it. On top of that traceability you watch per-stage metrics, counts and latencies and error rates for parse, chunk, extract, resolve, and write, and you watch the data-quality distributions: chunks per document, triples per chunk, triples per entity. Those distributions are the early warning. When triples-per-chunk drops toward zero, something upstream broke, a parser change, a prompt regression, and the graph stopped learning before anyone filed a bug.
Reprocessing ties it together, because reprocessing is a constant, the price of tuning a prompt or upgrading a model or changing the ontology. The discipline is to scope it. Version every part of the pipeline, the parsing config, the chunking strategy, the extraction prompt and model, the resolution model, and stamp that version onto every triple, so you can reprocess exactly the slice produced by the buggy extractor and audit which facts came from it. Change detection reprocesses only the documents that changed; a shadow run compares old and new extraction on a sample before you roll a new prompt across the whole corpus. The system that can reprocess a narrow, named slice is the one that can improve without rebuilding, and the one that cannot is the one where every fix is a full re-ingest nobody wants to authorize.
The clear-eyed accounting is what separates this from a list of best practices. What is proven across production systems is the spine: layout-aware and section-aware parsing, speaker-turn handling for transcripts, thread-aware ingestion for social, semantic chunking with overlap tuned against your own corpus, evidence-anchored extraction with schema validation, blocking before pairwise comparison, resolution as confidence-bounded clustering with incremental updates, explicit versioning and supersession, idempotent ingestion on stable ids with tombstones, queues with backpressure, and per-stage observability. What stays a design choice, genuinely contingent on your domain, is the exact chunk size and overlap, how far you push recall before the verify pass, whether entity matching runs on classical features or a model, which predicates you treat as functional, how often you re-cluster versus update incrementally, and how much history you keep. The spine is not optional and the choices are not free, and the system that lives is the one whose builders did the unglamorous engineering at every stage instead of drawing a box and trusting the arrow.
| The spine, proven across production systems | The choices, contingent on your domain |
|---|---|
| Layout-aware and section-aware parsing | The exact chunk size and overlap |
| Speaker-turn handling for transcripts, thread-aware ingestion for social | How far you push recall before the verify pass |
| Semantic chunking with overlap tuned against your own corpus | Whether entity matching runs on classical features or a model |
| Evidence-anchored extraction with schema validation | Which predicates you treat as functional |
| Blocking before pairwise comparison | How often you re-cluster versus update incrementally |
| Resolution as confidence-bounded clustering with incremental updates | How much history you keep |
| Explicit versioning and supersession | |
| Idempotent ingestion on stable ids, with tombstones | |
| Queues with backpressure | |
| Per-stage observability |
The next essay leaves the pipeline that builds the knowledge for the discipline that designs the agent which will use it. Agent Redwood, the twelve-dimension blueprint that scopes an agent before it is built, so the memory, evaluation, and constraints that read a graph like this one are decided on purpose rather than improvised. The graph is the substrate. Redwood is how you decide what reads from it.
END OF ESSAY 06 · CONTINUE TO ESSAY 07 →