boocode/openspec/changes/boocontrol/artifacts/p4-p5-audit.md

P4+P5 Audit: Combined Validation + Code Review

Date: 2026-06-12 Change: boocontrol Phases: P4 (per-consumer attribution) + P5 (quality evals + sandbox) Mode: Implementation (all 8 tasks checked)

---

Build/Test Gates

Gate	Result
pnpm -C apps/server build	PASS
pnpm -C apps/server test	PASS (580 passed, 11 skipped, 51 files)
pnpm -C apps/coder build	PASS
pnpm -C apps/coder test	PASS (587 passed, 32 skipped, 51 files)
pnpm -C apps/control build	PASS
pnpm -C apps/control test	PASS (116 passed, 15 files)
npx tsc -p apps/web/tsconfig.app.json --noEmit	PASS

---

Validation: boocontrol P4+P5 (implementation mode)

Verdict

PASS-WITH-FINDINGS -- all 8 tasks have implementing code; one design-specified behavior (judge temperature=0) is not implemented.

Traceability

Task	Claim	Evidence	Status
P4.1	X-Boo-Source on AI-SDK streaming path	stream-phase-adapter.ts:309 passes 'boochat' to upstreamModel; provider.ts:19-44 getSwapProvider wraps fetch with header, cache keyed baseURL||source	PASS
P4.1	includeUsage: true preserved	provider.ts:38 explicitly set on createOpenAICompatible	PASS
P4.1	compaction.ts + task-model.ts headers	compaction.ts:359 and task-model.ts:27 both inject X-Boo-Source: 'boochat' in direct fetch headers	PASS
P4.2	local-gateway.ts forwards x-boo-source	local-gateway.ts:67 reads inbound header, defaults 'boocoder'; local-gateway.ts:76 forwards as X-Boo-Source	PASS
P4.2	arena-model-call.ts source	arena-model-call.ts:51 sets X-Boo-Source: 'arena'	PASS
P4.3	control_requests.source migration	schema.sql:48 ALTER TABLE ADD COLUMN IF NOT EXISTS source TEXT (idempotent); INSERT at index.ts:182-183 includes source column; index.ts:81 maps source: null for ring data (design S7 deviation documented)	PASS
P4.4	Tests: header present + rows attribute	pipeline.test.ts:248 asserts source=NULL for ring data; import/export tests for all three paths	PARTIAL
P5.1	Suite format + YAML loading + DB schema	eval-suites.ts:67-120 loads YAML from data/; schema.sql:161-222 defines eval_suites (UNIQUE on name+version), eval_runs, eval_results; 4 YAML suite files present	PASS
P5.2	Judge runner temperature=0	judge-runner.ts:239 scoreWithRubric uses temperature: 0 (correct); judge-runner.ts:182 generateResponse uses temperature: 0.7 (NOT 0)	FAIL
P5.2	Judge model+version pinned per run	judge-runner.ts:59 constructs judgeModelVersion string; eval_runs table stores judge_model + judge_model_version	PASS
P5.2	Rationale captured	judge-runner.ts:97-98 stores rationale from scoreWithRubric	PASS
P5.2	X-Boo-Source control-eval	judge-runner.ts:177,237 both set X-Boo-Source: 'control-eval'	PASS
P5.3	Sandbox hardening flags	sandbox-runner.ts:258-273 docker args array: --network none, --user 1000:1000, --memory, --cpus, --pids-limit, --tmpfs /workspace:rw,noexec,size=64m, --rm, --label boocontrol-eval, --security-opt no-new-privileges, --cap-drop ALL	PASS
P5.3	No volume mounts, no docker socket	Verified in docker args array at sandbox-runner.ts:258-273 -- no -v or socket reference	PASS
P5.3	Orphan prune at engine start	sandbox-runner.ts:73 calls pruneOrphanContainers() at start of runCodeEval	PASS
P5.3	Bounded concurrency + allSettled + finally cleanup	sandbox-runner.ts:81-83 batch loop; sandbox-runner.ts:86 Promise.allSettled; sandbox-runner.ts:162-165 finally block with cleanupContainer	PASS
P5.3	SANDBOX_TIMEOUT_MS type	sandbox-runner.ts:37 typed as number but process.env is string -- setTimeout and spawn timeout receive string	ADVISORY
P5.4	Leaderboard UI + scatter	EvalsTab.tsx renders scatter (echarts.init with buildEChartsTheme) + bar chart + run table + launcher	PASS

Findings

Blocking

V1: judge-runner.ts generateResponse uses temperature 0.7 instead of 0

• Location: apps/control/src/services/judge-runner.ts:182
• Evidence: body: JSON.stringify({ model, messages: [{ role: 'user', content: prompt }], temperature: 0.7, max_tokens: 2048 }) -- the generateResponse function (which generates the target model's response to be scored) uses temperature 0.7. The design at design.md:195 specifies "temperature 0, judge model+version pinned per run." The scoreWithRubric function at line 239 correctly uses temperature: 0, but the response generation step does not.
• Impact: The target model's response is generated with non-deterministic sampling. For a reproducible eval framework this undermines the "temperature 0" claim in the task description. The judge scoring is deterministic (temp=0) but the input it scores is not.
• Fix sketch: Change line 182 from temperature: 0.7 to temperature: 0.

Advisory

A1: sandbox-runner.ts SANDBOX_TIMEOUT_MS is string, not number

• Location: apps/control/src/services/sandbox-runner.ts:37
• Evidence: const SANDBOX_TIMEOUT_MS = (process.env.SANDBOX_TIMEOUT_MS ?? '30000') as unknown as number; -- process.env values are string | undefined. The as unknown as number cast silences tsc but the runtime value is '30000' (string). This string flows to spawn(..., { timeout: SANDBOX_TIMEOUT_MS }) at line 277 and setTimeout(..., SANDBOX_TIMEOUT_MS) at line 308. Node's child_process.spawn timeout accepts number | undefined and setTimeout accepts number | string | undefined (string is parsed). The timeout will likely work due to JS coercion, but the type lie masks future bugs (e.g. SANDBOX_TIMEOUT_MS - 1000 would produce NaN).
• Impact: Low immediate risk (JS coercion makes it work), but the incorrect type annotation prevents catching arithmetic bugs. SANDBOX_CONCURRENCY at line 38 has the same issue.
• Fix sketch: const SANDBOX_TIMEOUT_MS = Number(process.env.SANDBOX_TIMEOUT_MS ?? '30000');

A2: judge-runner tests exercise imports, not judge logic

• Location: apps/control/src/services/__tests__/judge-runner.test.ts
• Evidence: Test 1 imports the module and checks typeof mod.runJudgeEval === 'function'. Test 2 calls runJudgeEval with a nonexistent provider and asserts the error message. Neither test exercises the actual judge request flow, rubric scoring, temperature setting, or rationale capture. The temperature=0.7 bug (V1) would not be caught by these tests.
• Impact: Regressions in judge scoring logic, temperature, or X-Boo-Source injection would not be caught by the test suite.
• Reopen trigger: Any bug where judge scoring produces wrong results or wrong temperature.

A3: sandbox-runner tests exercise Promise patterns, not Docker flags

• Location: apps/control/src/services/__tests__/sandbox-runner.test.ts
• Evidence: Tests verify runCodeEval is importable, that Promise.allSettled isolates failures, and that SIGKILL works. None of the tests verify the actual Docker arguments (security flags, label, resource caps), orphan pruning, or container cleanup. The test at line 19 (bounded fan-out) reimplements the pattern inline rather than calling runCodeEval.
• Impact: A regression in the Docker security flags (e.g. removing --cap-drop ALL) would pass all existing tests.
• Reopen trigger: Any sandbox escape or flag regression.

A4: arena dispatch sites not fully traced

• Location: apps/coder/src/services/arena-model-call.ts:51
• Evidence: arenaModelCall sets X-Boo-Source: 'arena'. However, the full arena dispatch chain (battle start, contestant model calls, cross-examination) was not traced end-to-end. The direct arenaModelCall path is verified; whether all arena sub-calls route through this function rather than making their own fetches was not checked.
• Impact: Low -- if arena uses arenaModelCall for all model calls, attribution is correct. If any arena path makes a direct fetch without X-Boo-Source, those requests would show as NULL in the activity feed.
• Reopen trigger: Arena requests showing as NULL in activity feed despite having a source.

Claims I did not verify

• Whether the includeUsage: true survives AI-SDK v6's internal handling (this was verified in prior P1 review -- load-bearing per apps/server/CLAUDE.md)
• Whether the sql.json(value as never) pattern in eval-suites.ts:170 correctly serializes the tasks array as JSONB (pattern is established and used elsewhere in the codebase)
• Whether the ECharts bundle tree-shaking works correctly in the production build (the echarts/core + per-chart imports pattern is established from P1)
• Whether the eval_runs.judge_model_version column is actually populated at run creation time (the createEvalRun function at eval-suites.ts:258 receives judgeModelVersion as a parameter; whether callers pass it was not traced)
• Whether the leaderboard API endpoint exists and returns the correct shape (the frontend fetches from /api/control/eval/leaderboard; the backend route handler was not traced)

---

Review: boocontrol P4+P5

Scope

apps/server/src/services/inference/provider.ts, apps/server/src/services/inference/stream-phase-adapter.ts, apps/server/src/services/compaction.ts, apps/server/src/services/task-model.ts, apps/coder/src/services/local-gateway.ts, apps/coder/src/services/arena-model-call.ts, apps/control/src/services/judge-runner.ts, apps/control/src/services/sandbox-runner.ts, apps/control/src/services/eval-suites.ts, apps/control/src/schema.sql, apps/web/src/components/control/EvalsTab.tsx, apps/web/src/pages/Control.tsx, P4+P5 tests.

Size

Large -- 12 source files across 3 apps + contracts, touches inference streaming path, SSE ingestion, Docker container spawning, DB schema, and ECharts UI.

Summary

P4 (attribution) is correctly implemented end-to-end. All three paths (server streaming, coder gateway, arena) inject the correct X-Boo-Source header. The migration is idempotent and NULL-for-ring-data is documented. P5 (evals) has correct schema, YAML loading, and UI wiring, but the judge runner's response generation temperature (0.7) contradicts the design spec (0). Sandbox hardening is thorough.

Classification	Count
Blocking	1
Advisory	4
Nit	1

Findings

Blocking

B1: Judge response generation temperature is 0.7, not 0

• Location: apps/control/src/services/judge-runner.ts:182
• Evidence: temperature: 0.7 in the generateResponse request body. The design at design.md:195 specifies "temperature 0, judge model+version pinned per run." The scoreWithRubric function at line 239 correctly uses temperature: 0.
• Standard violated: Design spec S8 ("temperature 0, judge model+version pinned per run").
• Risk: Non-deterministic eval inputs undermine reproducibility claims. A reviewer or auditor checking the design vs code will find this discrepancy.
• Fix sketch: temperature: 0 on line 182.

Advisory

A1: SANDBOX_TIMEOUT_MS type mismatch

• Location: apps/control/src/services/sandbox-runner.ts:37
• Evidence: as unknown as number cast on a string from process.env. Works at runtime due to JS coercion, but the type lie prevents catching arithmetic bugs.
• YAGNI gate: No known incident. Defer unless the sandbox timeout needs arithmetic (e.g. grace period).

A2: Judge tests do not exercise scoring logic

• Location: apps/control/src/services/__tests__/judge-runner.test.ts
• Evidence: Tests check import and error-on-bad-provider only. Rubric scoring, temperature, X-Boo-Source injection, and rationale capture are untested.
• YAGNI gate: No known scoring bug. Defer until judge scoring produces real evals.

A3: Sandbox tests do not verify Docker flags

• Location: apps/control/src/services/__tests__/sandbox-runner.test.ts
• Evidence: Tests exercise Promise.allSettled and SIGKILL patterns, not the actual Docker args construction. Security flags (network, caps, user, label) are untested.
• YAGNI gate: No known sandbox escape. Defer until sandbox runner processes untrusted code.

A4: Arena dispatch chain not fully traced

• Location: apps/coder/src/services/arena-model-call.ts:51
• Evidence: arenaModelCall sets X-Boo-Source: 'arena'. Whether all arena sub-calls (battle start, cross-examination) route through this function rather than making direct fetches was not verified.
• YAGNI gate: No known arena attribution bug. Defer until arena requests show NULL source.

Nits

N1: eval_suites UNIQUE on (name, version) uses ON CONFLICT DO NOTHING in seed, but upsertEvalSuite uses ON CONFLICT DO UPDATE

• Location: apps/control/src/services/eval-suites.ts:175 vs eval-suites.ts:230
• Evidence: seedEvalSuites uses ON CONFLICT (id) DO NOTHING (by primary key). upsertEvalSuite uses ON CONFLICT (id) DO UPDATE. The schema also has UNIQUE (name, version) at schema.sql:170 which is NOT the conflict target in either function. If two suites share a name+version, the UNIQUE constraint would reject the second. This is the correct behavior (versioning is explicit), but the UNIQUE constraint and the ON CONFLICT target differ.
• Note: Style -- not a bug.

Verdict

APPROVE-WITH-NITS

One blocking finding (B1: judge temperature 0.7 should be 0). Four advisory findings deferred per YAGNI gates. One nit on UNIQUE constraint targeting.

---

Claims I did not verify

• Whether the AI-SDK createOpenAICompatible internal fetch wrapper correctly merges the custom fetch headers (established pattern from P1, not re-verified)
• Whether the eval_runs.judge_model_version column is populated by callers of createEvalRun (the function accepts it; caller trace was not performed)
• Whether the leaderboard API backend route exists and returns the correct shape
• Whether the ECharts tree-shaking in EvalsTab.tsx produces correct bundle sizes
• Whether arena battle start / cross-examination model calls all go through arenaModelCall
• Whether the control_requests INSERT at index.ts:258 (the non-reconcile path) also correctly sets source: null