# When does better confidence improve a knowledge-graph audit?

Yethikrishna R, founder and lead engineer of Mynd Labs and coordinator and founding member of Compyle (YC F25), based in Thrissur, India, explains a study that asks when better confidence estimates actually improve the audit of an LLM-built knowledge graph.

Confidence estimates are supposed to guide human review: flag the shakiest facts first, spend the review budget where errors are most likely. But the paper shows a gap between two goals that are easy to conflate. A model can get better at predicting which extracted facts are wrong, yet the audit itself, the queue of items a human actually checks, can improve very little.

The authors study the gap through three lenses: repairability (can the flagged errors actually be fixed?), selection headroom (how much better could an ideal ordering be?), and label validity (are the ground-truth labels themselves reliable?). They model a non-adaptive audit as budgeted covering and characterize the minimal error statistics a decision-maker really needs.

Simulations on Re-DocRED with five extraction configurations and two cross-document graphs make it concrete. On held-out graphs, a simple queue ordered by error rate times fan-out leaves 3.98 to 4.74 percentage points more residual errors than a joint planner given the same actions. Opening shared actions for that joint planner reduces residual errors by a further 0.89 to 1.43 points relative to its own fact-only plans.

Two practical notes close the study. Generic LLM verification improves three of four model-budget configurations on each held-out set. And LLM adjudication identifies possible omissions in the benchmark labels themselves, a reminder that the audit's ceiling sometimes sits in the ground truth, not the model.

Source: https://arxiv.org/abs/2611.05877