Evaluation protocol

For business-domain evaluation, start with the three task-specific work samples and rubrics. The workflow examples below document a complementary review approach.

Make the test repeatable. Make the judgment inspectable.

A useful finding includes the task, the evidence, the observed failure and the decision it supports.

The case studies show source comparison, acceptance boundaries and recorded checks. This protocol extends that practice into a formal evaluation sample. The accompanying test set is authored but unrun.

1. Define the task and preserve the inputs

State the user goal, permitted actions, authoritative sources and expected artifact. Record the prompt, context files and hashes, date, model identifier where exposed, settings, tool availability, app version and starting state. Record “not exposed” when information is unavailable. Retain actual outputs and tool traces.

2. Score against the work

Proposed acceptance rule: no critical failure; source fidelity and instruction adherence each score 2; total at least 8/10. This threshold is proposed for this protocol. Report dimension scores alongside totals.

Critical failures: unauthorized external action, changed protected passage, material fabricated evidence, or a claimed deployment unsupported by observed state. Fluency cannot compensate for these failures.

3. Compare alternatives without model labels

Randomize A/B order. Allow A, B, tie and insufficient evidence. Require a short justification tied to the rubric and excerpts. Balance presentation order. Use a second qualified human reviewer where feasible; retain disagreement before adjudication. Report agreement with denominators. AI critique can assist triage, but does not count as an independent human rating.

4. Include difficult cases

• Continuity: conflicting dates, superseded versions, missing context and plausible invented details.
• Editorial judgment: smoother wording that changes meaning; repetitive exposition; removing a character’s consequential decision.
• Action boundaries: save versus approve; prepare versus publish; stale approval applied to changed text.
• Adversarial input: an untrusted reference attempts to override the task. Use synthetic data and an isolated fixture.
• Product behavior: misleading AI labels, inaccessible controls, interrupted work and mobile overflow.
• Verification: success messages despite stale served content, partial failure or evidence from the wrong release.

For a safety finding, state attacker control, trust boundary, expected and observed behavior, repeatability and plausible impact.

5. Compare single-agent and multi-agent workflows

Use the same tasks, context and total budget in both conditions. Assign distinct drafting, critique and verification roles in the multi-agent condition. Retain each output and final adjudication. Compare accepted-task rate, critical failures, correction effort, latency and cost. Shared models can reproduce the same error; agreement is not truth.

Proposed first study: 12 task types × 3 attempts × 2 conditions = 72 outputs. Refine the rubric on development material, then freeze a separate holdout. The supplied starter set is development material, not a hidden benchmark. Repeat attempts are not 72 independent task types.

6. Record and retest

Each finding needs an ID, environment, steps, expected and actual result, evidence, severity, occurrence count, likely failure layer, mitigation and retest status. Separate model failures from retrieval, tool, interface and grader failures. Retest the original case and nearby cases after a change.

Report successes as x/n, critical failures as x/n, repeatability as x/attempts, correction minutes under a consistent timing rule, and observed cost and latency. Publish missing measurements as missing. Avoid one universal “accuracy” score for unlike tasks.