diff --git a/docs/superpowers/specs/2026-05-04-pz-deterministic-classifier-design.md b/docs/superpowers/specs/2026-05-04-pz-deterministic-classifier-design.md new file mode 100644 index 0000000..c8754b4 --- /dev/null +++ b/docs/superpowers/specs/2026-05-04-pz-deterministic-classifier-design.md @@ -0,0 +1,244 @@ +# PZ deterministic classifier — design spec + +> Drafted 2026-05-04. Status: design-approved, awaiting implementation plan. +> Sibling tool to the existing pre-production Qwen analyzer (`pz_error_analysis.py`), which is unaffected by this work. + +## Summary + +A new deterministic-only Project Zomboid log classifier that lives alongside the existing Qwen-based analyzer in `tools/pz-analyzer/`. Walks redacted `DebugLog-server*.txt` files, extracts errors/warnings, attributes each to a mod where evidence allows, classifies by kind, and emits a structured JSON report. **Zero AI dependency**: this is the artefact that informs the future PHP / iblogs production path. + +The patterns it implements are inspired by `paraxaQQ/pzmm`'s `core/inspector.py` — Lua mod-marker attribution, multi-fallback file:line extraction, bidirectional stack collection, cause-chain unwinding, engine-noise tagging. Reimplemented originally; no code copied verbatim. + +## Why a separate tool, not an edit of `pz_error_analysis.py` + +Two artefacts, two purposes: + +- `pz_error_analysis.py` (existing, untouched) — pre-production discovery tool. Sends residual log content to Qwen so the developer can see what categories the deterministic side hasn't yet captured. +- `pz_classify.py` (new) — production-bound deterministic classifier. Output is what an iblogs PHP port would eventually emit. Runs in seconds, no API dependency, no PII-going-to-LLM consideration. + +Coexisting them lets the developer compare outputs and treat the LLM's residual output as the "deterministic to-do list." + +## Scope + +**In scope:** +- Two new files: `tools/pz-analyzer/pz_parser.py` (pure module) and `tools/pz-analyzer/pz_classify.py` (CLI orchestrator). +- Tests under `tools/pz-analyzer/tests/` with synthetic fixtures. +- Operates exclusively on the already-redacted directory produced by `pz_redact_all.sh` (`.scratch/pz/Logs.redacted/`). + +**Out of scope:** +- Any modification to `pz_error_analysis.py`, `pz_redact_all.sh`, or PHP codex source. +- Filesystem-based mod-scan reattribution (pzmm's symbol-index, vehicle-index, file-path-ownership reattribution requires an actual mod folder we don't have on the server side). +- iblogs / bosslogs integration. The output schema is designed with that future port in mind, but no PHP code is written here. +- Generic AI tab patterns from pzmm's `core/ai.py`. Explicitly excluded. + +## Architecture + +``` + redacted .txt files + | + v + +---------------------------+ + | pz_classify.py | argparse · directory walk · aggregate · JSON write + | (orchestrator) | + +-------------+-------------+ + | + v + +---------------------------+ + | pz_parser.py | regexes · parse · classify · sign + | (pure module, no I/O | + | beyond reading the path | + | it is handed) | + +---------------------------+ +``` + +Two files inside `tools/pz-analyzer/`: + +- **`pz_parser.py`** — stateless. All regex constants, `parse_file(path) -> list[Entry]`, attribution helpers, file:line extractors, cause-chain extractor, signature computation. No `argparse`, no JSON writing, no directory walking. Unit-testable in isolation. +- **`pz_classify.py`** — entry point. CLI args, walks the redacted directory, calls `pz_parser`, aggregates records by signature, writes JSON, prints a one-line stats summary. + +The split is deliberate: `pz_parser.py` is the module that eventually wants to be ported to PHP codex (separate spec). Keeping it pure makes that port mechanical and Python-side tests trivial. + +## Parser pipeline phases + +For each `*DebugLog-server*.txt`, the parser walks lines once and emits records via the following phases. + +### 1. Severity-prefix recognition + +Regex: `^\s*(ERROR|SEVERE|WARN)\s*[:\s]`. Broader than the existing `pz_error_analysis.py` regex — adds `SEVERE` (Java util-logging convention; appears in some PZ Java exception blocks). `LOG`/`INFO` is ignored at this layer. + +### 2. Stack collection — bidirectional + +Pzmm's contribution: PZ emits stack frames *before* the ERROR/WARN line as often as after. + +- **Pre-stack**: walk up to 25 lines back from the severity line. Stop at another severity line or 8 collected. Only keep the block if at least one line looks stack-shaped (`at `, `[string ...]`, `function:`, `file:`, `.lua` markers). +- **Post-stack**: walk forward up to 25 lines, gated by engine-noise detection. Stop at another severity line or 8 collected. +- Merge deduped, preserving order; cap at 8 frames per record. + +### 3. Mod attribution — three buckets + +| Bucket | Trigger | Confidence | +|---|---|---| +| `direct` | Line itself matches `Lua\(\(MOD:([^)]+)\)\)` (or the `require("X") failed` shape, or an explicit `needed by ` hint elsewhere in the entry) | `high` | +| `inferred` | No marker on this line, but body is Lua-shaped (see below) *and* a `Lua((MOD:Y))` was emitted within the previous 40 lines | `medium` | +| `unattributed` | Neither of the above | `low`; `mod_id = "__unattributed__"` | + +"Lua-shaped" means the body matches at least one of (case-insensitive): `luamanager.getfunctionobject`, `no such function`, `exception thrown`, `runtimeexception`, `illegalstateexception`, or contains the bare token `lua`. This filter prevents inferred attribution from latching onto unrelated severity lines that happened to fall within the lookback window. + +`mod_id` derives from the marker's raw name with a `_norm_mod_key` transform: lowercase, strip spaces / apostrophes / hyphens. `mod_name` preserves the human-readable form. + +We do **not** attempt pzmm's filesystem-based reattribution. + +### 4. File:line extraction — five fallbacks + +Tried in order against the entry body and stack frames: + +1. `at .lua:` +2. `function: ... file: .lua line #` (or `: `) +3. `[string ".lua"]:` +4. quoted path ending in `.lua` / `.txt` / `.xml` / `.json` / `.ini` / `.cfg` / `.bin` +5. unquoted path segment beginning with `media/`, `maps/`, `lua/`, `scripts/` + +Returns `(file, line)`; `line=0` if the matched form had no line number. + +### 5. Cause-chain extraction + +`Caused by: ` chains plus standalone exception lines (`(\w+\.)+\w+(Exception|Error): `) are normalised to `: ` tokens and joined with ` -> `. Up to 6 chain levels, deduped. Captures both Java exception nesting and Lua-wrapped exception chains. + +### 6. Java exception kind detection + +DebugLog-server has both Lua and Java exceptions; pzmm targets `console.txt` which is Lua-dominant. Extension here: + +- `kind = "java_exception"` when the entry body or stack contains `(\w+\.)+\w+(Exception|Error)` AND no `Lua((MOD:X))` marker is present anywhere in the entry. +- These typically resolve to `mod_id: __unattributed__` because Java code in PZ is engine, not mod. The exception class name becomes part of the message skeleton so similar Java exceptions dedup tightly. + +### 7. Engine-noise tagging + +`kind = "engine_noise"` when the body contains `kahluathread.flusherrormessage` or `dumping lua stack trace`. These severity-ERROR lines are PZ's own diagnostic chatter about its error reporting, not actual errors. They stay in the output (consumer can filter on `kind`). + +### 8. Signature computation + +Two-level deterministic identity, both stored on every record: + +``` +pattern_id = sha256(level + normalized_first_line)[:16] +signature = sha256(pattern_id + mod_id)[:16] +``` + +Normalization for `pattern_id`: +- Strip session metadata prefix (`General f:N, t:N, st:N,N,N,N>` shape) +- Flatten double- and single-quoted strings to `""` / `''` +- Flatten ≥2-digit numeric runs to `` +- Collapse whitespace +- Truncate to 200 chars + +Both fields ride on every record. Two consumer views, neither requires LLM: + +- **Per-mod view** (signature is the dedup key): one record per `(mod_id, error_shape)` pair. +- **Pattern fan-out view** (group records by `pattern_id`): see all mods that hit the same shape. + +### 9. Aggregation + +Records dedup on `signature`. On second-and-subsequent occurrences: `occurrence_count++`, `files` set-extends, attribution-confidence promotes (direct beats inferred beats unattributed), stack and `cause_chain` merge. + +## Output schema + +```json +{ + "meta": { + "input_dir": "/opt/ik-codex/.scratch/pz/Logs.redacted", + "files_scanned": 6, + "log_lines_total": 78654, + "error_lines_total": 30984, + "unique_signatures": N, + "unique_patterns": M, + "redacted": true, + "started": "ISO8601", + "finished": "ISO8601" + }, + "signatures": [ + { + "signature": "sha256:...", + "pattern_id": "sha256:...", + "level": "ERROR", + "kind": "lua_runtime|require_failed|java_exception|engine_noise|runtime", + "mod_id": "spongies_clothing", + "mod_name": "Spongie's Clothing", + "attribution": "direct|inferred|unattributed", + "confidence": "high|medium|low", + "attribution_reason": "...", + "file": "media/lua/client/X.lua", + "line": 42, + "cause_chain": "ExceptionA: msg -> ExceptionB: msg", + "stack": ["at A.lua:12", "at B.lua:34"], + "first_seen": {"file": "...", "line": 1234, "timestamp": "26-04-26 17:14:35.128"}, + "occurrence_count": 47, + "files": ["..."], + "excerpt": "..." + } + ], + "summary": { + "errors": N, + "warnings": N, + "by_kind": {"lua_runtime": ..., "java_exception": ..., "require_failed": ..., "engine_noise": ..., "runtime": ...}, + "by_attribution": {"direct": ..., "inferred": ..., "unattributed": ...}, + "by_confidence": {"high": ..., "medium": ..., "low": ...}, + "top_mods": [{"mod_id": "...", "mod_name": "...", "occurrence_count": N}, ...] + } +} +``` + +Default output path: `/opt/ik-codex/.scratch/pz/classify.json` (gitignored under `.scratch/`). + +## CLI + +``` +pz_classify.py [--input ] [--out ] [--quiet] +``` + +- `--input` defaults to `/.scratch/pz/Logs.redacted` +- `--out` defaults to `/.scratch/pz/classify.json` +- `--quiet` suppresses the trailing summary line + +No `--limit`, `--resume`, or `--checkpoint-every`. Runs in seconds; nothing to throttle or resume. + +## Tests + +New directory `tools/pz-analyzer/tests/`. Stdlib `unittest`. Three files, ~18 tests total. + +- **`test_parser.py`** (~10 tests) — one fixture per scenario in `tests/fixtures/` (synthetic, tracked in git): pure-Lua-attributed, pure-Java-exception, inferred-from-context, unattributed-engine-noise, multi-cause-chain, pre-stack-collection, post-stack-collection, severity-variants, file-line-extraction-fallbacks. All synthetic identifiers (placeholder Steam IDs / mod names) per the existing PHP-side `test/src/Games/ProjectZomboid/fixtures/` convention. +- **`test_attribution.py`** (~5 tests) — three confidence buckets, the 40-line lookback boundary, "needed by X" extraction, and the rejection of inferred attribution when the message isn't Lua-shaped. +- **`test_signatures.py`** (~3 tests) — `pattern_id` stability across formatting variations (whitespace, numeric values, quoted strings) and `signature` uniqueness across mods. + +Invocation: `python -m unittest discover tools/pz-analyzer/tests/`. No external deps. + +## Verification + +End-to-end smoke against the redacted real-data directory: + +``` +bash /opt/ik-codex/tools/pz-analyzer/pz_redact_all.sh # one-time, already done +python /opt/ik-codex/tools/pz-analyzer/pz_classify.py +``` + +Expect: +- 6 files scanned, ~30,984 error lines processed. +- A meaningful number of unique signatures and patterns (likely in the low hundreds for signatures; fewer patterns). +- `top_mods` lists the highest-occurrence mods. +- PII audit: no real Steam IDs, IPs, or coordinates in the output JSON (input is already redacted; classifier doesn't introduce PII). + +Test invocation: `python -m unittest discover tools/pz-analyzer/tests/` should be all-green. + +## Risks and open questions + +- **Inferred attribution accuracy.** The 40-line lookback is pzmm's heuristic; it's correct for tightly-paced server bursts but can mis-attribute when an unrelated mod logs in the gap. Surface as `confidence: medium` so consumers can choose to treat them differently. Acceptable for v1; tunable via a constant in `pz_parser.py`. +- **Pzmm targets `console.txt`, we target `DebugLog-server.txt`.** Format overlap is high (both share `Lua((MOD:X))` markers, Caused-by chains, Java exception shapes), but some patterns may be `console.txt`-specific. Tests use `DebugLog-server`-shaped fixtures only. +- **Future PHP port.** `pz_parser.py` is structured for mechanical translation to a `LuaErrorAnalyser` / `ModAttributionAnalyser` pair under `src/Analyser/ProjectZomboid/` in a separate spec. Output schema chosen to be PHP-codex-compatible (Insight subclasses with typed fields). +- **Licence.** The `paraxaQQ/pzmm` zip we reviewed has no top-level LICENSE; this spec mandates rewriting the patterns originally rather than copying code. Regex shapes and heuristics are general programming patterns and not author-specific, but no code blocks are lifted verbatim. + +## Out of scope (explicit) + +- Editing `pz_error_analysis.py` or `pz_redact_all.sh`. +- Modifying any file in `/opt/ik-codex/src/`, `/opt/ik-codex/test/`, or `/opt/iblogs/`. +- AI / LLM integration of any kind in the new tool. +- iblogs front-end rendering of the classification output. +- Filesystem mod-scan reattribution (pzmm's symbol/vehicle indexes).