Most of the time, my resume tool refuses to rewrite your resume. That is the feature. It took an offline evaluation and two failed attempts at the same fix to get there.
While testing Pass 3, the rewrite step, I found it suggesting skills the resume never mentioned. I've been building a deterministic validator for this app from the beginning. In case you did not know what that is, check out my previous blog post, How I Fixed False Negative Keyword Matching in My LLM-Powered Resume Analyzer . Long story short, it stops the model from reporting a keyword as missing when the resume actually contains it.
This time, a similar issue happened again. I should have caught this earlier.
What the Evaluation Showed
To debug this specific finding, I conducted an offline evaluation:
10 resumes × 3 model configurations × 2 runs = 60 evaluations
The model proposed a total of 31 rewrites; 22 of these lacked any basis in the original resumes.
A second sweep in August, same fixtures and same shape, returned 29 proposed, 25 of them unevidenced and 3 surviving. The remaining one was rejected for a different reason: it tried to smuggle in words that were not part of any requested keyword.
What the Filter Is Allowed to Say Yes To
The rewrite step is an LLM. Left alone, it will add a keyword to your resume because the job posting asked for it, whether or not you have ever done the work. It reads as helpful, and it is the kind of claim a hiring manager finds in the first five minutes of a phone screen.
So the model's output doesn't go to you. It goes through a deterministic filter, and that filter has exactly two ways to say yes. Both functions are below, with their docstrings removed and the order swapped to match the explanation that follows:
def _keyword_supported_elsewhere(keyword, resume_text):
return word_boundary_match((keyword or '').strip(), resume_text or '')
def _concept_anchor_present(keyword, original_line):
kw = (keyword or '').strip().lower()
if not kw or not original_line:
return False
anchors = CONCEPT_ANCHORS.get(kw)
if anchors is None:
for key, val in CONCEPT_ANCHORS.items():
if kw.startswith(key) or kw.rstrip('s') == key.rstrip('s'):
anchors = val
break
if not anchors:
return False
return any(word_boundary_match(a, original_line) for a in anchors)An edit ships only if one of those returns true. Anything else is rejected, and an unrecognized keyword fails closed.
The first is literal promotion: the word is already somewhere in your resume, so moving it onto the line an applicant tracking system reads adds no new claim.
The second is a vocabulary reframe, and it is the one I got wrong twice. Some concepts can be genuinely evidenced without using the word: if your resume says you built dashboards and alerts, you did observability. Others cannot. You either used Docker or you didn't, and no synonym proves it. So concepts need an allowlist with anchor words:
CONCEPT_ANCHORS = {
'observability': {
'logging', 'logs', 'monitoring', 'metrics', 'dashboard', 'dashboards',
'alert', 'alerts', 'alerting', 'tracing', 'telemetry',
'datadog', 'grafana', 'prometheus', 'cloudwatch', 'splunk', 'kibana',
},
}
Note what _concept_anchor_present searches. It takes original_line, not the resume. The anchor has to be present in the specific line being edited.
That scoping is what cost me two rounds. When I searched the whole resume instead, "pipeline" matched a sentence about an event-driven data pipeline built on S3 and Textract, and on that evidence the tool appended "CI/CD" to a tools list belonging to someone with no CI work at all. I tightened it to the edited line and it happened again on a weaker anchor: "release" turned "improve release quality" into "improve release quality using CI/CD."
Both of those edits were fluent, within the edit budget, and passed every automated check I had. Both were caught by reading the output by eye.
What Production Actually Showed
After the guard shipped, 32 production analyses ran. Five were cache replays where Pass 3 never executed, which leaves 27 I can attribute. Of those 27, only 2 produced a rewrite.
Those 32 analyses came from four accounts, three of which are mine. It is not 32 independent observations, it is closer to four, mostly me running my own resume against real postings. I am giving you the decomposition, not a rate.
My immediate instinct was to blame the guard for being overly strict. But the CloudWatch logs showed it was only part of the story: 15 times the model proposed nothing to rewrite in the first place (56%), and 10 times the guard rejected the proposal entirely (37%). What the logs cannot tell me is why the model proposed nothing in those 15. It could be the rewrite prompt's own instruction to skip when in doubt, or it could be that those resumes had nothing worth changing. The logs record the outcome, not the cause. Those 10 analyses account for 19 individual rejected edits. Most of the silence is not the filter rejecting things. It is the model proposing nothing for the filter to look at.
Whether the concept-anchor half of the design ever rescued a single rewrite in production remains unknown. Across those 19 rejections, it did nothing by definition; for the 2 edits that passed, the filter dropped the evidence without logging which execution path approved them. It would be inaccurate to say it failed to work. The precise truth is stronger: I cannot determine whether the half of the design I rewrote twice has ever executed in production.
For the full context, see the complete technical write-up on GitHub. You can also explore resumematchapp.com/sample directly without creating an account. The single highlighted edit on that sample report is one of only two edits approved by the guard in production since launch.
I am currently looking for full-stack and AI engineering roles, primarily in NYC, hybrid or on-site.