PACE / Study guide / Verification

Verification

Catching confident-but-wrong before it ships · 200 of 800 points

Verification is the discipline of not taking the agent’s word for it: deterministic checks, real diff review, knowing what a green test suite does and does not prove, and catching the confident-but-wrong answer — the signature failure mode of capable models.

By WholeTech · updated 2026-08-02

Q. Why is “all tests pass” not the same as “it works”?

A green suite proves the code satisfies the tests that exist — nothing about behavior the tests never covered. After an agent refactor, the highest-value check is re-running the suite yourself and reading the diff for changes in behavior the tests do not constrain.

Q. What does confident-but-wrong look like in practice?

It looks like a clean explanation, a plausible diff, and a wrong result. Models present their best guess with the same fluency as their best knowledge. The tell is unverifiable specificity — exact numbers, file paths, or API names you did not give it and it did not read.

Q. How much of an agent's diff should you actually read?

All of it — but with triage. Mechanical, repeated edits can be sampled once the pattern is proven; anything touching auth, money, data deletion, or public behavior gets read line by line. The expensive habit is skipping review because the agent “usually gets it right.”

Q. What are deterministic checks, and why prefer them?

A deterministic check gives the same verdict every run: a test, a type-checker, a curl returning 200, a byte-for-byte comparison. They convert “I think it’s fine” into evidence, and they let agents verify their own work mid-task instead of at the end.

Sample question — released item (not in the live bank)

An agent reports “all 42 tests pass” after a large refactor. What is the strongest next step?

  1. Accept it — green is green.
  2. Re-run the suite yourself and read the diff for behavior changes the tests don’t cover.
  3. Ask the agent whether it is sure.
  4. Have the agent write 58 more tests so there are 100.
Show the answer

B. B independently reproduces the claim and covers the gap green suites leave. A trusts an unverified report; C invites the same confidence that produced the claim; D adds volume, not verification.

Sit the beta exam — Verification is 10 of your 40 questions →