How I built a side by side LLM comparer with a blind judge, and why the mock does the heavy lifting

I kept doing the same thing by hand: paste one prompt into four chat tabs, squint at four answers, and guess which one was best. So I built multi-model-compare. It sends one prompt to up to eight models at once (Claude, GPT, Gemini and Grok in the live profile), streams the answers side by side, records time to first token, total time and cost, and then asks a separate model to judge the answers blind.

Image: multi-model-compare repo, docs/screenshots/run.png at commit d9e396b.

First, the honest part. This is a portfolio demo, and the default profile is mock. With no API keys, nothing is sent to any provider. Every number in this post comes from the mock, the repo's config files, or the test suite, not from real models. I have not benchmarked any real provider here, and you should not read the mock scores as a ranking of anything. Everything below is from the code at commit d9e396b (full SHA d9e396bf38fbc8661929ede7f17d769a771c89b6).

Try it before reading the code

git clone https://github.com/applycoding/multi-model-compare && cd multi-model-compare
uv sync
uv run mmc run "Explain CAP theorem in 3 sentences" --seed 1
uv run mmc serve

The first command line printed is Mock mode: nothing is sent to any provider. On my box the run printed four rows, one per mock contestant, each with ttft_ms=12, total_ms=37 and cost_usd=$0 (example pricing (mock)). Then came a judge_scores_mock line saying the scores are deterministic mock scores, not a live judge, and four judge_score rows: overall=2.15 for mock Claude, 2.25 for mock Gemini, 1.30 for mock GPT and 3.85 for mock Grok. The answers themselves are strings of Greek letter names like phi kappa upsilon alpha. That is on purpose, and I will come back to why. mmc serve gives you the same thing in a browser at http://127.0.0.1:8000.

One worker per model, all at once

The core lives in src/mmc/core/runner.py. Each contestant gets its own async worker, and the fan out is a single TaskGroup:

src/mmc/core/runner.py:

    async with asyncio.TaskGroup() as tg:
        for i, spec in enumerate(specs):
            tg.create_task(worker(i, spec))

I like TaskGroup here because cancelling the run cancels every worker, so nothing keeps streaming in the background. Each worker also gets its own timeout, timeout_at = min(start + req.timeout_s, deadline), so one slow model cannot outlive the run deadline. In src/mmc/defaults/models.yaml the defaults are timeout_s: 60 per model and run_deadline_s: 900 for the whole run, with max_prompt_chars: 32000 and max_concurrent_runs: 2.

How do you prove the workers really run at the same time? Timing tests are flaky, so I used a barrier. The mock has a gate behaviour: each stream bumps a shared in flight counter and then waits on an asyncio.Event until the test opens it. In tests/unit/test_runner.py, test_gate_max_in_flight starts four gated contestants and waits until all four are inside the barrier at once:

tests/unit/test_runner.py:

    for _ in range(200):
        if mock_mod.mock_in_flight_counter and max(mock_mod.mock_in_flight_counter) >= n:
            break
        await asyncio.sleep(0)
    assert max(mock_mod.mock_in_flight_counter) == n

If the runner were secretly sequential, the counter would never reach n and the test would fail. A sibling test, test_gate_max_in_flight_eight, does the same with eight.

Measuring time to first token honestly

TTFT sounds simple until a provider sends an empty chunk first, or a role marker, or reasoning metadata. I only start the clock on the first non empty text delta:

src/mmc/core/runner.py:

                async for event in stream:
                    if isinstance(event, TextDelta):
                        if event.text:
                            if ttft is None:
                                ttft = (clock.perf() - start) * 1000
                            text_parts.append(event.text)
                    elif isinstance(event, StreamEnd):
                        end_event = event
                        finish_reason = event.finish_reason
                        request_id = event.request_id
                        if event.usage:
                            billing_estimated = event.usage.billing_estimated

Every adapter translates its SDK stream into the same small set of events, so this loop never needs to know the vendor. The StreamEnd event carries the finish reason, request id and usage.

There is one rule I am strict about: a stream must end with a terminal event. If the iterator just stops, I do not call that success:

src/mmc/core/runner.py:

        if end_event is None:
            status = "error"
            error_info = ErrorInfo(kind="protocol", message="stream ended without terminal event")

That is a protocol error, and any text already received is kept and marked partial. Long answers are also capped at 100,000 characters and flagged as truncated, so one runaway model cannot fill the database.

Cost with Decimal, and labels that admit uncertainty

Floats and money do not mix, so src/mmc/core/pricing.py does everything in Decimal and rounds half up to six decimal places:

src/mmc/core/pricing.py:

QUANT = Decimal("0.000001")


def _q(value: Decimal) -> Decimal:
    return value.quantize(QUANT, rounding=ROUND_HALF_UP)

Prices live in src/mmc/defaults/pricing.yaml, per model, with an effective_from date, a source_url and a retrieved_on date. The rates in that file are illustrative placeholders, and the README says to verify each one against its source before using the live profile. The live model_id values in models.yaml are placeholders too. So I am not quoting any prices here.

The part I am proudest of is that a cost is not just a number. It has a status: known, estimated or unknown. It is unknown when there is no pricing entry for that model on that date. It becomes estimated if token counts had to be estimated because a provider sent no usage, if the price is a tiered "from" price, if xAI reasoning billing is unverified, or if Grok's long context tier above 200k input tokens applies:

src/mmc/core/pricing.py:

    status = "known"
    if token_estimated or billing_estimated or entry.basis == "from" or long_context_tier:
        status = "estimated"

Mock mode taught me a lesson. Each mock model has a made up rate of 1.00 USD per million input tokens and 2.00 per million output tokens, so the cost code gets exercised. In the first version, the mock run printed cost_usd=0.000129 per answer, which looks exactly like a real bill for a run that cost nothing. Now apply_profile_honesty() in src/mmc/honesty.py rewrites mock records before they reach the CLI, UI or exports:

src/mmc/honesty.py:

    total = CostTotal(subtotal_usd=Decimal(0), estimated=False, unknown_count=record.total.unknown_count)
    return record.model_copy(
        update={
            "answers": answers,
            "judge": judge,
            "total": total,
            "pricing_disclosure": "example_pricing_mock",
        }
    )

It also zeroes every answer and judge cost. The UI shows an example pricing (mock) badge, and the disclosure is saved in SQLite (schema version 3, with a migration for older databases) and in exports.

The judge: blind, shuffled and not allowed to grade its own family

LLM as judge is easy to get wrong. Judges can prefer the first answer, prefer their own vendor, or be swayed by an answer that says "As Claude, I...". I tackled each of those in src/mmc/judge/.

First, answers get letter labels, shuffled with a seeded RNG so a run is reproducible but the order is not just the order I listed models in:

src/mmc/judge/anonymise.py:

def assign_labels(model_keys: Sequence[str], seed: int) -> dict[str, str]:
    keys = list(model_keys)
    rng = random.Random(seed)
    rng.shuffle(keys)
    labels = [chr(ord("A") + i) for i in range(len(keys))]
    return {label: key for label, key in zip(labels, keys, strict=True)}

Second, redact_identity in the same file replaces vendor names and model ids with [MODEL] using word boundary regexes. The default redact_terms list in models.yaml covers Claude, Anthropic, GPT, ChatGPT, OpenAI, Gemini, Google, Grok and xAI.

Third, the judge does not score answers from its own vendor unless you opt in. In src/mmc/judge/judge.py:

src/mmc/judge/judge.py:

        if ans.model.vendor == judge_spec.vendor and not settings.allow_self_judging:
            excluded[ans.model.key] = "same vendor as judge"
            continue

The live judge is a Gemini model, so by default the Gemini contestant is excluded from judging and the UI shows a "not judged: same vendor as judge" badge. The judge runs at temperature 0.0, is allowed up to 2 retries (max_retries: 2), and must return a score from 1 to 5 for each rubric criterion. The rubric in src/mmc/defaults/rubric.yaml weights correctness at 0.40, completeness 0.25, clarity 0.20 and safety 0.15, and the schema rejects weights that do not sum to 1.

You can check the mock output against those weights. Mock Grok scored correctness 5, completeness 4, clarity 2 and safety 3: 0.40 x 5 + 0.25 x 4 + 0.20 x 2 + 0.15 x 3 = 3.85, which matches the CLI line.

The README is upfront about limits, and I agree with it: prompt injection and position or style bias are reduced, not eliminated. A judge is another model's opinion, not ground truth.

Why the mock is the most important provider

A deterministic mock is what lets me test all of this offline. In src/mmc/providers/mock.py, the answer text comes from a SHA-256 of the model id and the prompt:

src/mmc/providers/mock.py:

def _answer_text(model_id: str, prompt: str, words: int) -> str:
    if words == 0:
        return ""
    seed = hashlib.sha256(f"{model_id}\x00{prompt}".encode()).digest()
    parts: list[str] = []
    for i in range(words):
        parts.append(_VOCAB[(seed[i % len(seed)] + i) % len(_VOCAB)])
    return " ".join(parts)

Same prompt, same words, every time. The mock judge does the same trick for scores:

src/mmc/providers/mock.py:

                h = hashlib.sha256(f"{label}\x00{crit}\x00{answer}".encode()).hexdigest()
                score = int(h[:8], 16) % 5 + 1

So the mock scores are a hash mod 5, plus 1. They are stable enough for golden tests and meaningless as quality judgments, which is exactly what I want from a test double. Early on, nothing on screen said so. Now any run judged by the mock is marked scores_mock: true, and the UI shows a mock scores (deterministic, not a real judge) badge next to the judge name. A 12 ms per chunk delay (_MOCK_CHUNK_DELAY_S = 0.012) exists only so the UI shows nonzero timings.

The mock also has misbehaviours you can switch on per model in config: stall, error, error_midstream, missing_terminal, no_usage, empty, refused, blocked, incomplete, and judge modes like judge_bad_json_once and judge_bad_json_always. Each one drives a branch of the runner or judge that would be hard to trigger on purpose against a real API.

Tests, and what they cover

Running uv run pytest --ignore=tests/e2e -q at this commit gave me 161 passed, including new honesty tests and schema migration tests. Besides unit tests, there are contract tests that replay synthetic provider streams from tests/fixtures/recorded/ for anthropic, openai, google and xai, and a scenario test, tests/scenario/test_long_session.py, that drives eighteen sequential steps and compares the final state with a golden file. The README lists five CI jobs on ubuntu-24.04, including Playwright end to end tests, which I did not run locally. Docker publishes only on 127.0.0.1:8000.

What it does not do

It is a local tool that binds to loopback, with a Host allowlist, Origin checks, CSRF on mutations and a strict CSP, and it is not meant to be exposed to a network. A run in progress is lost if the process dies. Live mode needs your own API keys and real model ids from the vendors' docs, and I have not published any live comparison results. If you want to go further, my MCP server for Jira and GitHub tutorial shows another way to put an LLM behind a careful interface, and LoRA fine tuning a small LLM is a good next step once you have a way to compare outputs.

Clone it, run the mock, and then break it on purpose: flip a contestant to missing_terminal and watch the protocol error show up. That is the fastest way to see how each part works.