How I built a job application assistant that cites your CV and checks every claim

The scariest thing an AI writing tool can do on a job application is make something up about you. A wrong number or an employer you never worked for, sent under your name, is worse than no help at all. So when I built job-application-assistant, I started from one rule: every claim in the draft must cite a line of your CV, and a deterministic checker, not another model, decides first whether that citation actually backs the claim.

It takes a CV and a job post, pulls out the requirements, finds evidence in the CV, drafts answers to the application questions plus a short cover note, and then checks every sentence. It is a portfolio demo. The default provider is a deterministic mock, so it runs with no API keys, and with JAA_PROVIDER=mock nothing leaves your machine. The sample CVs, people and employers in the repo (Alex Doe, Acme Logistics, Initech and friends) are fictional. All code here is from commit d88b7a1 (full SHA d88b7a1968ee73bf8d343d5a8c0ca05e90d652f5).

Run it first

git clone https://github.com/applycoding/job-application-assistant && cd job-application-assistant
uv sync
uv run jaa seed
uv run jaa serve

Open http://127.0.0.1:8000 and open the sample job. The job page lists the requirements the app extracted (must and nice to have), the application questions, and a checkbox for the optional LLM judge:

Image: job-application-assistant repo, docs/screenshots/job.png at commit d88b7a1.

Generate a draft and the run page shows a badge. For this sample job, the golden run in tests/golden/job01_run.json reads 1 unsupported, 0 not verified (of 10 claims). The flagged sentence is At Initech, I was part of a 15-person platform team. I planted that one on purpose, and catching it is the whole point of the demo.

Step 1: stable ids for every line

Citations are only useful if they point at something stable. The ingest step turns PDF, DOCX or Markdown into lines with a section path such as Experience > Acme Corp, and each chunk gets an id built from a hash of its section and text, in src/jaa/ingest/chunker.py:

src/jaa/ingest/chunker.py:

def chunk_id(
    section: str,
    text: str,
    line_no: int,
    seen: dict[str, str],
    *,
    doc_scope: str,
) -> str:
    """Stable id from document content scope (file sha prefix) + line position + body."""
    content = f"{doc_scope}{_CHUNK_SEP}{line_no}{_CHUNK_SEP}{section}{_CHUNK_SEP}{text}"
    digest = hashlib.sha256(content.encode()).hexdigest()[:16]
    prior = seen.get(digest)
    if prior is not None and prior != content:
        raise ChunkIdCollision(f"Chunk id collision for digest {digest}")
    seen[digest] = content
    return digest

The id is the first 16 hex characters of a SHA-256 over a prefix of the file's own hash, the line number, the section and the text. An earlier version mixed in the random document id, so uploading the same CV twice gave new ids. Now the random id is left out, and the same file bytes always give the same chunk ids. If two different bodies ever collide on that prefix, ChunkIdCollision stops the run rather than letting a citation silently point at the wrong line.

Step 2: boring, transparent retrieval

I did not use embeddings. Retrieval is lexical BM25, which keeps tests deterministic and lets me explain every match. Before ranking, a chunk must contain at least one of the requirement's terms. That is eligible() in src/jaa/retrieval/eligibility.py:

src/jaa/retrieval/eligibility.py:

def eligible(chunk: Chunk, req: Requirement, lexicon: SkillLexicon) -> list[str]:
    matched: list[str] = []
    for term in req.terms:
        if _term_in_chunk(term, chunk, lexicon):
            matched.append(term)
    return matched

Skill terms go through a lexicon, and short names like Go, R, .NET and Node.js are matched case sensitively with boundary rules, so the everyday word "go" does not count as the Go language. Eligible chunks are then ranked in src/jaa/retrieval/bm25.py:

src/jaa/retrieval/bm25.py:

        self._bm25 = BM25Plus(corpus, k1=1.5, b=0.75, delta=1.0)

src/jaa/limits.py caps the pick at 5 chunks per requirement (K_PER_REQ = 5) and 20 in the whole evidence pool (POOL_MAX = 20). When a requirement has no eligible chunk at all, it becomes a Gap, and the UI tells you instead of letting the model improvise.

Step 3: the model only sees redacted evidence

build_outbound_draft() sends only the requirements and the selected evidence, with emails, phone numbers, URLs and street addresses removed by regex. That is best effort redaction, not anonymisation, and a "What will be sent" view shows the exact outbound JSON first. The model must answer in a strict schema where every sentence lists the chunk ids it relies on, and validate_draft() rejects ids that were not in the evidence it was given.

Step 4: the deterministic truth checker

This is the heart of the project, src/jaa/truth/deterministic.py. Each sentence is either framing (the opening and closing lines) or a claim. Framing gets a pass, but only if it has no citations and matches a server side template exactly:

src/jaa/truth/deterministic.py:

    if s.kind == "framing" and not s.citations:
        expected = _framing_template_text(s.template_id, ctx) if s.template_id else None
        if expected is not None and s.text == expected:
            return SentenceCheck(
                sentence_id=s.id,
                status="checks_passed",
                failures=[],
                checks_run=[],
                judge=None,
            )

The allowed templates live in src/jaa/generation/templates.py, for example I am applying for the {title} role at {company}. with the real job title and company filled in. This was a real hole I closed: before, anything labelled framing skipped every check, so a model could have smuggled a claim past the checker just by calling it framing. Now a "framing" line that does not match a template falls through to the normal checks. A claim with no citation fails at once:

src/jaa/truth/deterministic.py:

    if not s.citations:
        failures.append(
            CheckFailure(check="no_citation", token=None, detail="No citations", outcome="fail")
        )
        return SentenceCheck(
            sentence_id=s.id,
            status="unsupported",
            failures=failures,
            checks_run=["no_citation"],
            judge=None,
        )

Next, if a sentence says which requirement it addresses, at least one cited chunk must actually be eligible for that requirement, using the same eligible() function as retrieval. Otherwise it fails with Cited evidence does not support linked requirements. Then the checker pulls facts out of the claim with regex extractors: numbers, years, organisations, ownership verbs like led, managed, owned and built, skill versions and durations. It looks for one cited chunk that supports all of them. If the facts exist but are spread across different citations, that is a mixed_citations failure, because stitching two true lines into one sentence can still produce a false claim. Otherwise each fact is checked against the cited text. Here is the number check:

src/jaa/truth/deterministic.py:

            for n in numbers:
                if not _number_in_any(n, cited):
                    failures.append(
                        CheckFailure(
                            check="number",
                            token=str(int(n.value)) if n.value == int(n.value) else str(n.value),
                            detail=f"Number {n.raw} not in evidence",
                            outcome="fail",
                        )
                    )

That is exactly what catches the planted sentence. The mock cites a real chunk, so the citation check passes, but the golden output in tests/golden/job01_run.json records two failures: Number 15 not in evidence and Employer Initech not in evidence. Nothing in the cited line says 15 or Initech.

The rules carry a version, CHECK_RULES_VERSION = "2026-10-24.2", which is mixed into the corpus version so old runs are never confused with new rules.

Step 5: four verdicts, and why the wording matters

Failures map to a status in src/jaa/truth/verdict.py:

src/jaa/truth/verdict.py:

def status_from_checks(
    failures: list[CheckFailure],
    *,
    judge_enabled: bool,
    judge: JudgeVerdict | None,
) -> ClaimStatus:
    if any(f.outcome == "fail" for f in failures):
        return "unsupported"
    if any(f.outcome == "abstain" for f in failures):
        return "not_verified"
    if judge_enabled:
        if judge is None:
            return "not_verified"
        if not judge.entailed:
            return "unsupported"
        return "entailed"
    return "checks_passed"

unsupported means a check failed. not_verified means a check abstained, or the judge was on but did not answer. entailed only appears when an optional LLM judge agrees the evidence supports the claim. And checks_passed means exactly that: no deterministic issue was found. It does not mean the sentence is true, and the README says so. I named it that way on purpose, so the UI never promises more than the code checked.

Step 6: prune without another model call

The "Remove flagged claims" button runs prune_flagged() in src/jaa/truth/prune.py. It drops flagged sentences locally, keeps framing lines, and warns you when something ends up empty:

src/jaa/truth/prune.py:

    cover_kept: list[Sentence] = []
    claims_removed = False
    for sent in drafts.cover_note.sentences:
        if sent.kind == "framing" or sent.id not in flagged:
            cover_kept.append(sent)
        else:
            claims_removed = True

    if claims_removed and not any(s.kind == "claim" for s in cover_kept):
        warnings.append("Cover note has no supported claims - add manually")

No second LLM call means no second chance to invent something while "fixing" the first one.

What the mock really does

The mock provider in src/jaa/providers/mock.py builds sentences from templates and the same BM25 ranking, and by default (inject_unsupported: bool = True) appends one false line to the cover note:

src/jaa/providers/mock.py:

INJECTED_CLAIM = "At Initech, I was part of a 15-person platform team."

The first version of this demo had an embarrassing gap. In all three sample jobs, the planted Initech line was the only claim in the mock cover note. Prune it and you were left with a greeting, two framing lines and the warning Cover note has no supported claims - add manually. The checker was working, but the demo was not showing a real cover note.

The fix was in the mock itself. It now picks cover note evidence with the same eligible() rule as retrieval and writes up to three cited claims before the planted one. In the golden runs for all three sample jobs, the cover note now has three My background includes: ... claims plus the Initech line, and only the Initech line is flagged. When I ran prune_flagged() on each golden run, the cover note kept its opening line, all three cited claims and the closing line, with no warnings.

The mock is still a template engine, though, and it shows. For the data quality question on the first sample job, two of the cited sentences are Data Engineering Specialty - Example Board 2023 and Mentored two junior engineers on code review and incident response. Both are cited and pass the checks, but neither really answers the question. The requirement check makes sure the evidence fits the requirement a sentence claims to address, not the question being asked. That is why a human still reads the draft.

Tests

Running uv run pytest --ignore=tests/e2e -q at this commit gave me 256 passed, including new tests for the framing templates, the requirement check and a guard that keeps Unicode dashes out of the README. A socket guard in tests/conftest.py blocks network access during tests. The suite includes golden runs, adversarial and prompt injection fixtures, privacy and logging tests, and fail closed contract tests for the Anthropic, OpenAI, Google and xAI adapters.

Limits

Skills come from a lexicon, it is English only, extractors are regex based, and an LLM judge can be wrong. There is no authentication, so keep it on 127.0.0.1. With a real provider, the selected evidence and the job post go to that vendor (OpenAI and xAI requests use store=false).

If you like the idea of giving a model a narrow, checkable job, my MCP server for Jira and GitHub tutorial is a good companion, and LoRA fine tuning a small LLM shows the training side. Clone it, edit the sample CV, and try to sneak a number past the checker. It is a fun way to see how each check works.