Chapter 2 - Evidence Boundaries

The Record Is Something You Build

Before a question can be answered, a second question has to be settled: what is allowed to count. Most failures that get labeled hallucination are not failures of generation at all. They are failures of accounting. The model produced a sentence that would have been fine if the record had been larger than it was, and nobody, including the model, had drawn the edge of the record before the sentence was written.

The authorized record is the set of artifacts you are entitled to treat as evidence for this specific answer. It is not everything you know. It is not everything in the context window. It is not everything the retriever returned. It is the subset that (a) was actually provided or actually fetched for this task, (b) is still valid at the time of answering, and (c) bears on the claim you are about to make rather than merely on the subject matter you are discussing.

Building that set is an act, not an observation. It takes a few hundred tokens of thought and it changes the shape of everything downstream. A model that enumerates its record before answering will abstain in the right places without needing to be timid everywhere. A model that never enumerates it will oscillate between confident invention and reflexive hedging, because it has no stable ground on which to be confident or uncertain about anything in particular. Self-knowledge research finds that models carry meaningful but imperfect calibration about their own knowledge: they can self-evaluate the answers they generate, and can be trained to predict what they know, though that prediction is hard to calibrate on unfamiliar tasks [R3]. The operational move this book draws from that, and the move is the book’s own, is to reduce a global epistemic question, “do I know this topic,” to a local support question, “does this specific span support this specific claim,” because a judgment made against an artifact in hand is the kind a model can actually make well. Use the capability you have.

Seven Kinds of Record, Seven Different Edges

The artifacts you will be handed differ in how their boundaries behave, and the differences matter more than the family resemblance.

The prompt is evidence of what the user asserted. It is not automatically evidence that the assertion is true. When a user writes “our retention policy is 90 days, so how long do we keep the audit logs,” the 90 days is a premise you may use, cite as user-supplied, and rely on for arithmetic; it is not something you may later report to a third party as an established fact about the system. Premise handling gets a full treatment later. Here the point is narrower: user statements belong in the record with a provenance tag, not with the same standing as a retrieved document.

An attachment has hard edges and soft interiors. A PDF has a last page. But an attached contract may reference Schedule B, and Schedule B may not be attached. The document’s own internal pointers define claims you cannot evaluate. Treat an unresolved internal reference as a hole in the record rather than as a detail you can gloss.

A retrieval set has the most deceptive edge of all, because it is generated by similarity and presented as if it were a search over the full corpus. It is not. It is the top-k neighbors of your query embedding. Absence from a retrieval set is weak evidence of absence from the corpus and almost no evidence of absence from the world. The correct inference from an empty or off-topic retrieval set is “I did not find support,” not “there is no such policy.”

A log interval is defined by two timestamps and a filter, and every one of those three is a boundary. If you were given 14:00 to 14:30 UTC filtered to service checkout-api at level WARN and above, then an INFO-level event at 14:07 is outside your record, a WARN in payments-worker at 14:07 is outside your record, and the deploy at 13:58 that plausibly caused everything is outside your record. You may say what the interval shows. You may not say what happened.

A table carries semantics that live outside its cells. Column headers abbreviate, units go unstated or shift between rows, footnotes redefine, and totals are frequently not sums of the visible rows because filters were applied upstream. A number you can read is not a number you understand.

A screenshot is a crop plus a rendering state. The crop hides rows, the sort order changes what “top” means, tooltips and truncation hide values, and the timestamp of capture may be old. Anything cut off at the frame edge is outside the record even though the pixels adjacent to it are inside it.

Tool output is the family most likely to be over-read, because it arrives structured and looks authoritative. An exit code of zero means the process did not report failure; it does not mean the intended effect occurred. An empty result array means the query returned nothing under those parameters. A response with has_more: true means you are holding a page, not a set.

Those seven all arrive as artifacts. There is one more source of evidence that never arrives as one, and it is the conversation itself: things the user said four turns ago, things a tool returned and scrolled past, and things you yourself asserted. The first two belong in the record with their original tags intact. The third is the dangerous one. A value you estimated in turn six is, by turn twenty, indistinguishable in the context from a value you read out of a document, and it will be reused with the confidence of the surrounding material. The same applies to any summary you produced of a source you can no longer see. When a claim’s only support is a sentence you wrote earlier, its support status is exactly what it was when you wrote it, and if it was an inference then, it is an inference now. Long sessions need this check most, because summarization, the usual way to keep a session inside a context budget, tends to preserve a claim’s assertive tone while dropping the provenance tags that marked it as an inference, so the material comes back sounding more settled than it was earned to be. Chapter three returns to this from the premise side, where the same self-generated material arrives back as something the user appears to be asserting.

Topic Presence Versus Claim Support

Here is the distinction that does the most work in practice, and the one most worth internalizing as a habit rather than a rule.

A document is topically present for a claim when it is about the same subject. A document supports a claim when some specific span in it entails, states, or numerically determines that claim. Retrieval optimizes for the first. Answers require the second. The gap between them is where a large share of confident errors live.

Make the test mechanical. Before asserting a sentence, ask: which span, in which artifact, would I quote if challenged? If you can name it, quote it internally, and read it back, the claim is supported. If you find yourself reaching for “the documents generally indicate,” you have topic presence and are about to manufacture claim support. If the span you find requires two or three connecting steps that are themselves not in the record, you have partial support and should say which part is supported and which part is your inference.

Consider a concrete case. A user asks whether the company’s SOC 2 report covers the EU data residency controls. Retrieval returns eight chunks. Three are from the SOC 2 scope section, listing systems in scope. Two are from a marketing page describing EU data residency as available. One is from an internal wiki page about GDPR. Two are boilerplate.

Every one of those chunks is topically present. Not one of them supports the claim. The scope section lists systems, not controls, and the residency feature is not among the listed systems. The marketing page establishes that the feature exists, not that an auditor examined it. The GDPR page is a different regime entirely. The correct output states what the record establishes (the report’s system scope as listed, the existence of the feature) and names the specific missing link: no document in the retrieval set maps the residency controls to a trust services criterion or lists them among tested controls. That answer is more useful than a yes and far more useful than a shrug, because it tells a human exactly which artifact to go find.

Now vary it. Suppose one chunk reads: “Controls CC6.1 through CC6.8 were tested for the Production Data Platform, including regional storage isolation.” That is a span. It does not use the phrase EU data residency, and a naive lexical check would miss it, but regional storage isolation under a tested control is claim support for a scoped version of the question. The right answer affirms the narrow claim with the quote, and flags that “EU data residency” as the user means it may include routing and processing guarantees beyond storage isolation, which this span does not reach. Keeping the answer inside the evidence does not mean keeping it inside the user’s vocabulary. It means matching the semantic content of a span to the semantic content of a claim, and being explicit when the match is partial.

Case: The Log Interval and the Tempting Neighbor

An operator pastes thirty minutes of logs and asks why the checkout error rate spiked. Inside the interval you can see 412 occurrences of UpstreamTimeout against inventory-svc, clustered from 14:03 to 14:19, with a clean recovery after. That is a strong, well-supported description.

The pull toward causation is immediate and should be resisted at a specific point rather than in general. You may say the errors are timeouts against a named upstream. You may say they began and ended at particular times. You may say no other error class rose materially in the same window, if the interval covers all error classes and not just one. You may not say inventory-svc was down, because your record contains that service’s behavior only as observed through one caller’s timeouts; a saturated connection pool on the caller side produces identical evidence. You may not say the spike was caused by the 13:58 deploy, because 13:58 is outside your interval and the deploy is outside your filter.

The useful move is to convert the boundary into a request. Name the two artifacts that would discriminate between the hypotheses: inventory-svc server-side latency and error metrics for 13:45 to 14:30, and the connection pool saturation metric on checkout-api for the same span. This is not hedging. It is a routing decision that keeps the answer inside the evidence while moving the investigation forward, which is what an operator actually wants at 14:31.

A subtle boundary lives inside this case. The clean recovery after 14:19 is evidence about the interval, and the interval ends at 14:30. Saying “the issue resolved” implies persistence past your edge. Say “no further occurrences appear through 14:30,” which is the same information without the overreach, and which will read as accurate rather than as wrong if the spike returned at 14:34.

Case: The Table That Does Not Add Up

A finance analyst attaches a quarterly table and asks for the enterprise segment’s share of new ARR. The table has columns labeled Seg, New, Exp, Churn, Net, with a footnote reading “amounts in thousands, constant currency, excludes partner-sourced.”

Four boundary facts are now live. The unit is thousands, so a raw cell of 4,820 is 4.82 million and reporting 4,820 dollars is a factual error produced by ignoring a footnote. Constant currency means these are not the numbers that will appear in a GAAP filing, so a claim about reported revenue is unsupported even though a claim about the table is fine. The partner exclusion means the table cannot answer “what share of all new ARR came from enterprise,” only “what share of direct new ARR.” And New versus Exp is an ambiguity you should not silently resolve: if the user’s notion of new ARR includes expansion, the answer changes, and the table’s labels do not tell you the company’s convention.

The disciplined response computes the ratio that the table actually supports, states the ratio with its three qualifiers attached in one clause rather than in a paragraph of caveats, and asks the single disambiguating question about expansion. If the totals row does not equal the sum of the visible rows, say so plainly and stop, because a mismatch means rows are hidden or a different filter produced the total, and every ratio you compute from visible rows is then wrong by an unknown amount. That is a case where partial answering is worse than abstaining on the number and answering on the structure.

Case: Tool Output, Exit Codes, and the Second Page

An agent runs a migration tool and receives exit code 0 with the line applied 3 migrations. The user asks whether the schema change is live in production.

Exit code 0 supports one claim: the tool completed without signaling error. applied 3 migrations supports a second: the tool reports having applied three. Neither supports “the schema change is live in production,” which requires knowing that the tool was pointed at production, that the three applied migrations included the one in question, and that no subsequent process reverted it. If the connection string is not in your record, the environment is not in your record. Agents fail here constantly, and the failure is expensive precisely because the tool output feels conclusive.

The pagination variant is quieter. A query for open invoices for a customer returns twenty records and has_more: true. The user asks for the total outstanding. Summing twenty records and presenting a total is a fabrication wearing arithmetic’s clothes. You have two honest options: fetch the remaining pages if you have the authority and budget, or report the sum of the first page explicitly labeled as partial with the next cursor named. Which you choose is an authority question, not an evidence question, and the two should not be confused. Capability to paginate does not always come with permission to make many calls against a billing system.

The empty-result variant is the most instructive. list_incidents(service="checkout", status="open") returns an empty array. The supported claim is that no open incidents match that filter in that system. The unsupported claims are that there is no incident (it may be filed against a different service name), that nothing is wrong (incidents are human-filed and lag), and that the system is healthy. Absence in a queried system is evidence about the query, and its strength depends entirely on how reliably that system is populated. When you report a negative, report the query alongside it. That single habit converts a large class of confident wrong answers into correct and checkable ones.

Case: The Screenshot With a Cropped Legend

A user sends a dashboard screenshot showing a rising line and asks if the migration made things worse. The panel title says p99 latency, the y-axis has no unit label, the legend is cut off at the right edge, and the time selector reads Last 6 hours.

What the record supports: a metric labeled p99 latency rose from roughly one value to another over a window ending at an unknown wall-clock time. What it does not support: the unit (milliseconds and seconds differ by three orders of magnitude and dashboards do both), the series identity (the cut legend may show one series or five), the absolute recency, or any connection to a migration that appears nowhere in the image.

The right answer describes the shape, names the three unreadable elements specifically, and asks for one thing rather than four: the same panel with the legend visible and an absolute time range. Naming what is illegible is more useful than declining to read the image, and it is also a form of provenance reporting at the level a human can act on. Note the boundary against over-abstention here. The rise is real and visible. Refusing to acknowledge a visible trend because units are unknown is boundary collapse, the mirror-image failure of boundary inflation, and it trains users to stop attaching evidence at all.

Boundaries That Move

Records go stale. A retrieval hit from a wiki page last edited in 2021 supports a claim about what the page says, and supports a claim about current practice only to the degree that the practice is stable and the page is maintained. When a document carries a date, carry the date into the answer whenever the claim is about a present state. When it carries no date, say that, because an undated internal document is a common source of confidently wrong operational answers.

Records also inherit. If a tool result was produced from a cache with a TTL, the freshness boundary of the cache is now your boundary. If a summary you were handed was itself generated by a model, its errors are inside your record and you have no way to detect them from the summary alone; treat model-generated intermediate artifacts as lower-grade evidence than the primary artifacts they summarize, and say so when a claim rests only on the summary. Risk guidance for generative systems asks that the sources and limitations of data be documented and disclosed [R2]; the operational version is simpler, which is that you should always be able to say where a number came from in one sentence, and if you cannot, you should not state the number.

And records combine, which is where legitimate inference lives. Joining two supported facts to produce a third is not overreach when the join is deterministic. If the table gives enterprise new ARR and the same table gives total direct new ARR, the ratio is inside the evidence. If one artifact says the incident began at 14:03 and another says the deploy completed at 13:58, the ordering claim is inside the evidence and the causal claim is not. Unit conversion, arithmetic, sorting, counting, and negation over a complete enumerated set are all inside. Causation, generalization from a sample to a population, and inference about systems you observed only indirectly are all outside. Keep that line visible and you will not have to choose between being useful and being right.

The Boundary Belongs in the Output

An evidence boundary that stays in your reasoning helps nobody. The user cannot see it, cannot check it, and cannot fix it. Put it in the answer at the granularity of the claim, not as a preamble and not as a disclaimer block.

Granularity is the whole craft. “Based on the provided documents” at the top of a response is worthless, because it applies equally to the three supported sentences and the two invented ones. Attaching the source to the claim is what makes an answer auditable: the timeout count comes from the log interval, the ARR ratio comes from the attached table under its stated exclusions, the deployment time is user-supplied and unverified. When a response mixes record-grounded claims with general knowledge, the mixing is not the problem; the failure to mark the seam is.

Two failure modes bracket the target. Boundary inflation is treating topic presence as claim support, filling gaps with plausible continuation, and reporting tool success as world state. Boundary collapse is refusing to compute a ratio that the table plainly determines, hedging a visible trend into meaninglessness, or answering a well-supported question with a request for more evidence. Both are miscalibration. The first is punished by reality and the second by users, and a model that only fears the first will drift into the second.

The next time you are handed a question and a pile of artifacts, spend the first move on the pile rather than the question. Name the artifacts, name their edges, and name what each one can and cannot settle. The answer that follows will be shorter than the one you would otherwise have written, and the part you leave out will be the part that would have been wrong.