Two months ago I built a report that told me my system worked. Every pair processed. Nothing unresolved. No failures. One metric came back at a perfect 1.0 across every case in the set.
The number was real. The code that produced it was correct. And it could not have returned anything else, no matter how badly the system underneath it had failed.
I want to walk through how I found that, because the interesting part is not the bug. The interesting part is that the failure and the success looked identical from the outside, and I had built the thing that made them look identical.
What the system does
It reads the risk factors section of a company's annual report, splits it into individual risks, and compares this year's version against last year's to find what changed. If a company quietly rewrote its supply chain risk or dropped a factor entirely, the system is supposed to surface that.
The splitting step matters more than anything else. Everything downstream, the comparison, the evidence, the scoring, assumes the section arrived as a list of separate risks. If that assumption breaks, everything after it is comparing two blobs of text and reporting the result with full confidence.
The number that cannot move
The metric is called evidence resolution rate. Every claim the system makes carries references back to the source text, and the metric checks what fraction of those references actually resolve to something real. If the system cites evidence that does not exist, the rate drops. That was the idea.
Across the whole evaluation set it scored 1.0. Every reference resolved. Zero unresolved, zero foreign.
Here is why that happened. The detector creates each evidence reference from the chunks it has just loaded out of the vector store, copying the ids straight off them. The resolution check then takes those same ids and looks them up in the same store. The rate is the fraction that come back.
So the metric is asking whether ids I just wrote down still exist in the place I copied them from. It measures round trip consistency. It cannot measure whether the evidence is any good, and it certainly cannot notice that the section was never split, because a single unit spanning the entire section produces references that resolve just as perfectly as a correctly split one. In fact it produces more of them.
A metric that cannot fail is not a passing grade. It is a blank space shaped like one.
What the honest numbers said
The same evaluation report carries other metrics, ones scored against labels I wrote by hand after reading the filings myself. They sit in the same file, a few lines apart.
Change precision: 11 of 24, so 0.458. Change recall: 11 of 21, so 0.524. Unchanged false positive rate: 4 of 7. Pairs where the system got every single change right: 1 of 9.
One in nine. Next to a 1.0.
If I had shipped only the metrics that were cheap to compute, this project would have a perfect score and a completely broken core.
How wide the damage was
Once I stopped trusting the report I went and measured the splitting directly, side by side, on the raw build output rather than on anything the pipeline summarized for me.
The evaluation set has nine company pairs, eighteen sides in total once you count both years. Six of those sides never got split at all: the entire risk factors section, tens of thousands of characters, sitting in one undifferentiated unit. Six more got split, but one unit swallowed between 65 and 81 percent of the section, so the split is technically real and practically meaningless. Six sides look genuinely clean.
Six of eighteen. That is the honest version.
The root cause is embarrassingly small. My heading detector looked for lines ending in "Risk" or "Risks". Most filers do not write headings that way. They write "Risks Related to Our Business", which starts with the word and ends with something else, and five of the nine companies in the set do exactly that. Another writes "General Risk Factors", which ends in Factors. One more failed only because it wrote "Operational and Compliance/Legal Risks" and my character class did not include a slash.
Nineteen real headings, sitting right there in the text, none of them matched.
The part that still bothers me
There is a count in the report for undetermined comparisons, cases where the system could not decide what happened. You would expect it to spike where the splitting collapsed.
It runs the other way. Every single one of the collapsed sides reports zero undetermined. The well split ones report the most: a section that came apart into fourteen units reported twenty four undetermined comparisons, and one that came apart into nine reported two.
That is the whole problem in one line. The system expresses doubt in proportion to how much structure it managed to find. Where it found none, it had nothing to be unsure about, so it reported certainty.
Nobody caught it, including the parts built to catch it
There was exactly one signal anywhere in the system. When a section collapses, the detector labels the resulting blob with a synthetic heading of its own invention, and a validation check that verifies cited headings actually appear in the source text fails, because a heading I made up appears in no filing. Every validation failure in the entire corpus is that one check.
Three things about that signal, all of which I had built myself:
It is not a segmentation signal. It fires whenever that introductory blob changed year over year, and stays silent when the text is byte identical. So silence means the text did not change, not that the split worked.
Nothing downstream reads it. The evaluator does not look at it. The review packets I generated for my own manual labeling do not carry validation status at all, so sitting there labeling by hand, I saw no warning. It never reaches any committed file.
There is a governance layer whose configured policy is to hold the result whenever any check fails. It would have caught this. The pipeline that builds the evaluation set never calls it. Not a single governance evaluation row exists in any of the workspace databases, across all of them.
So the control exists, the policy is correct, the code works, and it runs nowhere near the place where it was needed. That is the second time this project has produced that exact shape. I will come back to the first one.
Meanwhile each collapsed pair reported the same three things: lifecycle detected, zero undetermined, no failure code.
The mirror image
Earlier this year the same system did the opposite thing, and I want to put it next to this one because together they say more than either does alone.
The audit layer truncates the excerpt it stores for each retrieved chunk at 700 characters, purely to keep log size sane. The grounding validator, whose job is to check whether numbers in an answer are supported by the source, was scoring against those truncated excerpts. So a figure sitting at character 900 of a chunk read as unsupported. The compliance layer was accusing its own correct answers of hallucination.
I fixed it on July 19. The validator now reads the full chunk text and only falls back to the capped excerpt when it cannot match. On the frozen 32 case baseline, the unsupported claim rate went from 0.0639 to 0.0293 and the grounded answer rate went from 0.75 to 0.84375.
Those before numbers are the only ones in this whole post you can reproduce yourself without my machine, because they are sitting in git history. That is not a small detail. It is most of the point.
And while we are here: the baseline file those numbers live in carries no provenance at all. No timestamp, no model id, no commit hash, no corpus hash. Just the numbers. I have been quoting a file that cannot tell you when it was written or what it was written against. That is on the fix list, and I am putting it in the post because leaving it out while writing about verifiability would be absurd.
What I actually changed
I did not touch the parser. The failure is detectable at build time from fields the pipeline already writes to disk, with no new parsing, no regex work, no models.
Two independent rules. The first: the section produced exactly one unit, and that unit is the synthetic introductory one. The second: some unit's character count equals the whole section's character count. They test different things and they agree on every side in both of my evaluation sets. The script exits with a distinct code if they ever stop agreeing, because the day they diverge is the day one of them is wrong and I want to be told rather than reassured.
Then a third column, which is a judgment and not a fact: flag any side where a single unit holds more than half the section. Half is my choice. There is nothing on disk that says half. The script refuses to run without being handed a threshold explicitly, so nobody, including future me, can mistake my preference for a measurement.
That third column is where I got humbled a second time. My first version computed the dominant unit share as "first unit divided by section", which quietly assumes the first unit is the introductory blob. On one company that assumption is false, and on two sides there is no introductory unit at all. So the column meant one thing on most rows and something else on a few, with nothing on the screen to tell them apart. Fixing it moved the flag count on the first evaluation set from eight to twelve. Two companies where a non introductory unit had eaten 65 and 80 percent of the section were reading as one percent.
The rule columns did not move. The judgment column was wrong by a third, in the direction that made things look better.
What I still cannot tell you
I ran the detector against a second, later evaluation set. Four of its sixteen usable sides collapsed the same way. I am not going to tell you that means the failure is getting better or worse over time, because I have not verified whether the two sets even cover the same companies, and until I do, comparing them is a story rather than a result.
Some of what is above you can check and some you cannot. The code paths, the rules, the before and after baseline numbers, the absence of any governance call in the build pipeline: all in the public repo. The per unit character counts, the workspace databases, the audit log: those live in gitignored working directories on my laptop. I have tried to be clear about which is which rather than letting the verifiable parts lend credibility to the rest.
This is also a work in progress, not a shipped product. The evaluation stage this all belongs to is unfinished, and nothing here should be read as a system in production.
The thing I keep relearning
I wrote a while back about companies measuring AI token usage and getting exactly what they measured, which was more tokens. It is easy to write that about someone else's leaderboard.
This is the same mistake in my own repository, and it is worse, because I did not just pick a lazy metric. I built a metric whose value was determined before any work happened, and then let it sit in a report next to metrics that were doing real work, formatted identically, indistinguishable at a glance.
The test I use now is short. If this number went bad, what would have had to happen? If I cannot answer that, the number is decoration. A metric that has never once told me something I did not want to hear has not been validated. It has just never been asked a real question.
If you want to check any of this
Repo: github.com/JODGEW/Financial-Document-Intelligence-Agent
- Evidence references are minted from loaded chunks in
comparison_detector.py(around lines 648 to 663); the resolution check re-looks them up inscripts/build_real_filing_benchmark.py(around 524 to 535); the rate is computed inscripts/eval_real_filing_benchmark.py(around 1665 to 1671). - The heading pattern that started all of this is
_HEADING_REincomparison_detector.py, and the synthetic heading it falls back to is a few lines below it. - The validation check that fails is the citation support check in
comparison_validators.py. Grep for its call sites; the governance module that would hold on it iscomparison_governance.py, and the build pipeline contains no reference to it. - The truncation cap is
MAX_EXCERPT_CHARSinaudit.py. The fix is commit9fbaa62. For the before numbers:git show a5111b2:eval/baseline.json. - The detector I describe at the end is
scripts/detect_unsegmented_sections.py. It is read only, standard library only, and takes a workspace path and a threshold.