Benchmark evidence assessment · Documentary pilot v0.1
What Public Agent Benchmarks Establish About Authorization and Policy Compliance
Source cutoff: 18 October 2026. Published documentary review. No benchmark runs or model evaluations were performed.
The finding · Documentary analysis
The reviewed sources inform different parts of an agent policy. They do not establish end-to-end authorization compliance as a common measure.
Original τ-bench explicitly distinguishes outcome reward from full policy compliance: a required confirmation can be omitted despite a passing outcome. T1
We use that source limitation to propose a separately testable approval control. The control is untested. Benchmark quality, documentation completeness and model performance remain separate questions.
Three source-linked cards
arXiv:2406.13352v3 (2024-11-24)
Useful evidence about the boundary between user intent and attacker-controlled content. A low attack-success figure alone cannot establish general approval compliance.
Limit: Exhaustive authorization lifecycle coverage, live-service protection, or universal safety.
Repository identity verified; immutable commit not captured. Current documentation is supplementary, not assumed identical to paper.
arXiv:2410.09024v3 (2025-04-18)
Misuse resistance is a distinct requirement from respecting a legitimate user’s approval. Authorization cannot make a prohibited action safe.
Limit: Real-world harmful capability, compliant benign approval handling, or recovery after external damage.
Official implementation identity verified; commit not captured. Viewed changelog includes 6-B as recommended version alongside other entries; no version was installed or selected for execution.
arXiv:2406.12045v1 (2024-06-17)
The clearest policy-evidence gap in this pilot is that an outcome metric need not verify the approval procedure that produced it.
Limit: Universal policy adherence from pass^k or final database correctness, or equivalence with successor benchmarks.
Original repository identity verified; commit not captured. Later τ²-bench and τ³-bench are excluded, including later task fixes.
11 documentary dimensions · 33 source-linked cells
Labels describe evidence available for each dimension, not scores or a common scale. In particular, repeatability concerns experimental rerun support; pass^k instead measures consistency of task success across trials. “Not reported” means not established in this source set; it does not prove absence.
Documented: Cited material directly specifies the stated feature; not evidence of comprehensive effectiveness.
Partly documented: Related feature or limited implementation is documented, but coverage or inference remains incomplete.
Not reported: Not established in the reviewed cited material; absence of documentation is not absence of capability.
Not applicable: Outside the stated construct for this comparison; not a quality judgment.
| Dimension | AgentDojo | AgentHarm | Original τ-bench |
|---|---|---|---|
| Construct measured | |||
| Explicit authorization | Partly documentedUser-task intent is explicit; separate action-approval coverage is unspecified. §3.1 User tasks · D1 | DocumentedRetail policy requires detailed action disclosure and affirmative confirmation before consequential database updates. Appendix B.1, Retail policies, opening rules · T1 | |
| Changed authorization | Not reportedDynamic dialogue alone does not establish changed-approval tests; dedicated coverage was not identified in the reviewed design. §3 User simulation · T1 | ||
| Revoked authorization | |||
| Untrusted instructions | Not applicableIndirect third-party injection is outside this paper’s stated direct-misuse setting. §2 · H1 | Not reportedReviewed original design does not specify an adversarial tool-output injection suite. §3 · T1 | |
| Consequential tool side effects | DocumentedTools mutate simulated application state; this is not live-service execution evidence. §3.1 Environments and Tools · D1 | DocumentedAPIs mutate simulated databases deterministically; live-service effects are not established. §3 Databases and APIs · T1 | |
| Final state versus trajectory checks | Partly documentedCurrent API supports optional trace-based security checks, otherwise state/output checks; per-task adoption was not audited. security; security_from_traces · D5 | Partly documentedRubrics inspect calls/outputs; narrow semantic and refusal judges supplement them. §3.1.3 · H1 | Partly documentedFinal database/output reward can pass despite omitted confirmation; authors explicitly acknowledge this. §3 Reward · T1 |
| Utility preservation | DocumentedBenign utility, utility under attack, and targeted attack success are separate measures. results column definitions; see also paper §3.4 · D2 | DocumentedCurrent documentation exposes benign counterpart evaluation separately from harmful tasks. agentharm_benign · H2 | |
| Repeatability | Partly documentedRepository exposes model, suite, attack and defense selections; repeatability was not independently tested. Running the benchmark · D3 | Partly documentedSplit, behavior filters, judges and agent settings are configurable; repeatability was not tested here. Parameters · H2 | Partly documentedpass^k measures all-k-trials task-success reliability, not reproduction of an experiment; repeatability was not tested here. §3 Pass^k · T1 |
| Recovery | Not reportedReviewed scoring API does not specify a separate recovery-after-violation metric. utility/security methods · D5 | Partly documentedTools support self-correction after errors; a separate post-violation recovery metric is not established, nor reversal of real harm. §3.1.2 · H1 | Not reportedReviewed primary reward does not separately score recovery from an authorization violation. §3 Reward · T1 |
| Configuration and scoring comparability | Partly documentedResults page warns against fair cross-model ranking because attack/defense coverage differs. Note · D2 | Partly documentedCurrent changelog alters executed-call credit and refusal-derived metrics; these revisions are not the paper configuration. 7-B and 5-B · H3 | Partly documentedCurrent original-repository defaults allow different user models; its tasks are explicitly flagged outdated. warning; User simulators · T2 |
MR-POL-001 · Proposed and untested
A synthetic customer-support agent cancelling or modifying one customer’s retail order. This is an illustrative policy design, not legal advice, certification, or a deployment authorization.
Before each consequential write, the execution gateway must verify an unexpired, unrevoked approval bound to the authenticated customer, delegated agent identity, exact action, order ID and canonical payload. Material changes require a fresh approval. Retrieved content cannot grant approval. Approval is necessary but does not override other safety or business constraints.
This illustrative retail policy is distinct from the helper-authorization case. Neither design has produced a model-evaluation result.
Evidence gap: Original τ-bench acknowledges that outcome success can omit confirmation (T1). AgentDojo motivates testing third-party attempts to redirect actions (D1). Neither fact proves the proposed control works. T1 · D1
Enforcement proposal: An execution gateway outside the model checks ownership, allowed action, payload digest, expiry and revocation at commit time. Reject mismatches before side effects; write tamper-evident decision and commit events. Use idempotency keys. Fail closed if approval state is unavailable. This is a proposed architecture, not an implemented control.
One intended write succeeds; approval, decision and commit events correlate; task utility succeeds.
No write; request valid approval or deny. Log reason without exposing secrets.
No write under old approval; require fresh displayed details and approval.
Gateway serialization gives revocation precedence if recorded before the commit decision; no write. Record ordering evidence.
No duplicate write; reconcile by idempotency key; never infer new approval.
Trajectory compliance fails despite matching final state; report containment and recovery separately.
All mandatory negative cases must produce zero unauthorized committed writes; positive controls must complete the approved operation. Check gateway events and database diff independently. Report per-case outcomes and repeated-trial distributions, not only an aggregate. No tests have been executed.
Owner: Proposed accountable role: deploying organization’s agent product owner; security engineering owns enforcement and independent evaluation reviews evidence. No named owner has accepted this assignment.
Review trigger: Review before release and whenever tools, policy, identity/approval mechanisms, model/scaffold, scorer or benchmark revision changes; review immediately after any authorization incident.
Method and limitations
Purposive three-benchmark document analysis supporting evidence-backed AI policies; not a systematic literature review, benchmark ranking, safety certification, or conformity assessment.
Read relevant paper design/scoring/limitations sections and official repository/documentation pages, including displayed API excerpts. Code each dimension against the cited locator. This is a document-level review, not an exhaustive task-by-task or code-level audit. not_reported means not established in this reviewed source set, never proven absence. All status assignments and policy implications are MEKreflect analysis, not author endorsements.
BenchmarkCards is prior work for standardized benchmark documentation. B1 Inspect provides evaluation infrastructure; it is not an evaluation result or certification. I1
Artifacts and source register
Original source editions retain their prepared-for-review status as historical provenance. The separate publication record tracks actual release. Papers are versioned; repository commits and page snapshots were not frozen, which limits reproducibility. The 33 cells count documentary coding, not independent experiments.
Original Markdown SHA-256: 0d464b6a0f617a393e0d764e01a9cbd9c17e93fb702bed4e16896493c9f45042
Original JSON SHA-256: 74eaa512d94f8d11206567290ff8e9d8382c74d1855741ab541c8a168a733b7b