For the last two weeks I've been stuck on the same bug in ResumeMatch. The resume rewrite pass was producing edits that looked plausible at first glance and turned out to be nonsense on a second read. "Datadog logs and analytics" — Datadog is analytics. "TypeScript contract validation and attention to detail" — those things don't go together. The model was technically following my prompt and still producing garbage.
I spent ten rounds rewriting the prompt to fix it. Trailing-clause rules, source rules, self-check questions, edit budgets, examples of bad outputs. Each round fixed one failure mode and surfaced another. Eventually I gave up trying to prompt my way out and swapped the model. Bad-edit rate dropped from over half to about one in four overnight.
This is the story of how I got there, and what I'd do differently next time.
How the Pipeline Works
ResumeMatch runs a 4-pass pipeline on every analysis:
- Pass 1 — Extract keywords from the job description.
- Pass 2 — Score the resume against those keywords. Flag what's missing.
- Pass 2.5 — Rank the top 5 most important missing keywords.
- Pass 3 — Rewrite resume bullets to incorporate the missing keywords. This is the one that broke.
Originally everything ran on Claude Haiku 4.5. Cheap, fast, good enough for parsing and scoring. But Pass 3 is different — it has to make semantic judgments about whether an inserted phrase actually fits in a sentence. That turned out to be where Haiku ran out of headroom.
The Failure Modes I Saw
Each round of prompt iteration killed one failure and revealed another. Some examples from production:
Failure mode 1 — JD hallucination. Early versions of the prompt didn't strongly enforce that the "original" line had to come from the resume. The model would copy a sentence from the job description, edit it slightly, and return that as if it were a resume edit. Fixed with a strict source rule and a 40-word cap on original lines.
Failure mode 2 — Trailing clauses. Once the source rule held, the model started appending clauses to the end of bullet points to sneak keywords in. Things like:
Original: "Built a Jest test framework with TypeScript contract validation"
Modified: "Built a Jest test framework with TypeScript contract validation,
with CI/CD integration for automated testing"That trailing comma-clause is a classic AI-writing tell. I added a forbidden-pattern rule explicitly naming it and the model stopped doing it.
Failure mode 3 — Inline gibberish. This is the one I couldn't beat with prompting. Once trailing clauses were forbidden, the model found a new loophole — inserting keywords inline in places where the result didn't parse:
Original: "tracing API errors through Datadog logs"
Modified: "tracing API errors through Datadog logs and analytics"That edit follows every structural rule. It's an inline insertion. It's grammatically valid. It's also nonsense, because Datadog is the analytics platform — there's no separate "analytics" being added. The model couldn't tell.
I tried adding a self-check rule: "Read the modified line as if a hiring manager wrote it. Does the inserted phrase actually belong?" It helped a little. Not enough.
The Decision to Swap Models
The pattern across these failures was the same: Haiku could follow structural rules but couldn't reliably judge semantic fit. Prompt engineering can fix structural problems. It can't add capability that isn't there.
I'd already been using Sonnet 4.6 for Pass 2.5 (the keyword ranking step) because that pass needed nuance Haiku couldn't deliver either. So the path forward was obvious — route Pass 3 to Sonnet too, keep Haiku for Passes 1, 2, and the experience parser where it works fine.
| Pass | What it does | Model |
|---|---|---|
| 1 | Extract keywords from the JD | Haiku 4.5 |
| 2 | Score resume, flag what's missing | Haiku 4.5 |
| 2.5 | Rank top 5 missing keywords | Sonnet 4.6 |
| 3 | Rewrite bullets | Sonnet 4.6 (was Haiku) |
| — | Experience parser | Haiku 4.5 |
The implementation was a one-line config change. I added a PASS_REWRITE_MODEL_ID environment variable, defaulted it to the existing model, and let the env var override per-pass:
PASS_REWRITE_MODEL_ID = os.environ.get('PASS_REWRITE_MODEL_ID', MODEL_ID)
# In the Pass 3 call:
call_bedrock(pass3_system, pass3_user,
cached_prefix=SHARED_CTX,
model_id=PASS_REWRITE_MODEL_ID)
I also added a separate rewrite_usage token bucket so the cost dashboard could show Sonnet spend separately from Haiku spend. Without that, the cost log would attribute Sonnet tokens to Haiku pricing and lie to me.
What Changed
I ran the same three job descriptions through v9 (all-Haiku) and v10 (Sonnet for Pass 3). The most telling result was on the second JD — Sonnet returned zero edits, correctly judging that none of the missing keywords fit anywhere in the resume. Haiku had been forcing 2-3 bad edits per run on this same JD across ten prompt iterations. It never returned zero.
That moment told me everything. The fix wasn't "make Haiku try harder." It was "let Sonnet refuse when nothing fits."
On the runs where edits did make sense, the bad-edit rate dropped from about half (v9 Haiku) to about one in four (v10 Sonnet). Some semantic-judgment misses still happened, but the gibberish category was gone.
The Cost Tradeoff
Sonnet 4.6 is roughly 3x the per-token cost of Haiku 4.5. Average analysis cost went from about $0.021 to about $0.042. Annoying, but for a job-seeker tool with low volume, the absolute number is tiny — we're talking pennies per analysis.
| Metric | v9 — all Haiku | v10 — Sonnet on Pass 3 |
|---|---|---|
| Pass 3 model | Haiku 4.5 | Sonnet 4.6 |
| Per-token cost (Pass 3) | 1× | ~3× |
| Avg cost / analysis | ~$0.021 | ~$0.042 |
| Bad-edit rate | ~1 in 2 | ~1 in 4 |
The right way to think about this isn't "Sonnet costs more." It's "the rewrite pass is the semantically hardest part of the pipeline, and routing only that pass to a stronger model captures most of the quality gain at a fraction of the cost of routing everything to Sonnet." Haiku stays on the parsing/scoring passes where it works fine.
Lessons Learned
There's a ceiling to prompt engineering. Some failures are model-capability failures, not prompt failures. Structural rules respond to prompting: don't append trailing clauses, preserve the source wording, keep edits under a budget. Semantic judgment is different. If the model can't reliably tell whether a phrase belongs in a resume bullet, more prompt rules usually just move the failure around.
Compare models before you overfit the prompt. Now I run a model comparison early, usually after two or three prompt iterations fail to remove the same failure mode. The key signal is not whether the stronger model is simply “better,” but whether it behaves differently. When Sonnet returned an empty array because no edit fit, while the weaker model kept forcing bad rewrites, that told me I had hit a model-capability ceiling. If both models fail the same way, then I know the prompt is still the thing to fix.
Multi-model pipelines beat single-model pipelines. Different passes have different difficulty levels. Routing each pass to the cheapest model that can handle it is almost always better than picking one model for everything. In my case, the infrastructure overhead was just one environment variable per pass.
Track cost per pass, not just total cost. When I added the separate rewrite_usage bucket, I could finally see that Pass 3 was the most expensive call in the pipeline. Without that visibility, I would've made routing decisions blind.
The work is live at resumematchapp.com if you want to see the pipeline in action.
Until next time!