THE RELIABILITY LAYER · JURY LIVE · JUDGE IN TRAINING

⚖️ Meerada Judge
A calibrated verdict on every AI answer.

Judging is easier than generating. A small model trained only to judge — is this claim supported by the sources? did the code do what was asked? how sure are you? — beats a giant general model at exactly that, at a tenth of the cost. Meerada is building that judge from the verdicts its jury already produces every day.
Use the jury today The plan, in detail

One verdict shape, forever

Every judge — the jury of foreign models today, our own model tomorrow — speaks the same frozen record. That is what makes verdicts storable, comparable and trainable.
{ "dimension": "faithfulness", // faithfulness · factuality · instruction_following · code_correctness · pairwise "score": 0.31, // 0 = wrong / unsupported · 1 = fully right "calibrated_prob": 0.88, // how sure the judge is that its score is right "flagged_spans": [{"start": 41, "end": 78, "reason": "figure not in the source"}], "reason": "two of three claims are unsupported by the context", "judge_id": "claude-sonnet-5", "judge_lab": "anthropic", "judge_grade": "derived" }

The road, in five steps

STEP 1 · DONE

The jury

Three jurors from three labs, no kin judging, both orders, Fleiss' kappa, hard budget. Every verdict stored in the tenant.

STEP 2 · SHIPPED

Cross-check in the app

A reliability chip beside every answer; the neutral panel replaces the single-judge button; thin panels labelled.

STEP 3 · NEXT

Known-truth data

Controlled hallucination injection on the public battery: swap a number, a date, an entity — the label is known by construction. A frozen eval set per dimension.

STEP 4

Train the judge

A small open model fine-tuned to emit the verdict record, then reinforced with a verified reward and a calibration penalty. Benchmarked against the jury and the best public judges.

STEP 5

In your tenant

The judge replaces the jury, runs inside your boundary (no content leaves), retrains monthly on new models' failure modes. The reliability ledger becomes a product.

Why this is Meerada's to build

We already have the ground truth

The exchange's battery is checked by programs — tests, SQL, exact answers. Every hour it produces labelled outputs from dozens of models. That is the training data most judge projects have to buy.

We are neutral, and we can stay neutral

A lab's own judge cannot fairly grade its rivals. An independent judge that never sells tokens can — and can publish, per model, how often it hallucinates and where.

The judge renews when models renew

Every new model has new failure patterns. We collect its outputs the day it launches, label them, retrain monthly. A frozen judge from a big lab does not know last week's model. Ours does.

It unlocks the Handshake for everyone

Migrating a workload with no programmatic checks used to come back mostly "unknown". A calibrated judge scores it — so the Handshake works for the majority of real work, not just code and schemas.

Known limit, stated up front: a small judge does not know facts a frontier model knows. We start with faithfulness — the knowledge is in the source — and extend to factuality only with a retrieval layer.