docs: add design spec for deterministic PZ log classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,244 @@
|
||||
# PZ deterministic classifier — design spec
|
||||
|
||||
> Drafted 2026-05-04. Status: design-approved, awaiting implementation plan.
|
||||
> Sibling tool to the existing pre-production Qwen analyzer (`pz_error_analysis.py`), which is unaffected by this work.
|
||||
|
||||
## Summary
|
||||
|
||||
A new deterministic-only Project Zomboid log classifier that lives alongside the existing Qwen-based analyzer in `tools/pz-analyzer/`. Walks redacted `DebugLog-server*.txt` files, extracts errors/warnings, attributes each to a mod where evidence allows, classifies by kind, and emits a structured JSON report. **Zero AI dependency**: this is the artefact that informs the future PHP / iblogs production path.
|
||||
|
||||
The patterns it implements are inspired by `paraxaQQ/pzmm`'s `core/inspector.py` — Lua mod-marker attribution, multi-fallback file:line extraction, bidirectional stack collection, cause-chain unwinding, engine-noise tagging. Reimplemented originally; no code copied verbatim.
|
||||
|
||||
## Why a separate tool, not an edit of `pz_error_analysis.py`
|
||||
|
||||
Two artefacts, two purposes:
|
||||
|
||||
- `pz_error_analysis.py` (existing, untouched) — pre-production discovery tool. Sends residual log content to Qwen so the developer can see what categories the deterministic side hasn't yet captured.
|
||||
- `pz_classify.py` (new) — production-bound deterministic classifier. Output is what an iblogs PHP port would eventually emit. Runs in seconds, no API dependency, no PII-going-to-LLM consideration.
|
||||
|
||||
Coexisting them lets the developer compare outputs and treat the LLM's residual output as the "deterministic to-do list."
|
||||
|
||||
## Scope
|
||||
|
||||
**In scope:**
|
||||
- Two new files: `tools/pz-analyzer/pz_parser.py` (pure module) and `tools/pz-analyzer/pz_classify.py` (CLI orchestrator).
|
||||
- Tests under `tools/pz-analyzer/tests/` with synthetic fixtures.
|
||||
- Operates exclusively on the already-redacted directory produced by `pz_redact_all.sh` (`.scratch/pz/Logs.redacted/`).
|
||||
|
||||
**Out of scope:**
|
||||
- Any modification to `pz_error_analysis.py`, `pz_redact_all.sh`, or PHP codex source.
|
||||
- Filesystem-based mod-scan reattribution (pzmm's symbol-index, vehicle-index, file-path-ownership reattribution requires an actual mod folder we don't have on the server side).
|
||||
- iblogs / bosslogs integration. The output schema is designed with that future port in mind, but no PHP code is written here.
|
||||
- Generic AI tab patterns from pzmm's `core/ai.py`. Explicitly excluded.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
redacted .txt files
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| pz_classify.py | argparse · directory walk · aggregate · JSON write
|
||||
| (orchestrator) |
|
||||
+-------------+-------------+
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| pz_parser.py | regexes · parse · classify · sign
|
||||
| (pure module, no I/O |
|
||||
| beyond reading the path |
|
||||
| it is handed) |
|
||||
+---------------------------+
|
||||
```
|
||||
|
||||
Two files inside `tools/pz-analyzer/`:
|
||||
|
||||
- **`pz_parser.py`** — stateless. All regex constants, `parse_file(path) -> list[Entry]`, attribution helpers, file:line extractors, cause-chain extractor, signature computation. No `argparse`, no JSON writing, no directory walking. Unit-testable in isolation.
|
||||
- **`pz_classify.py`** — entry point. CLI args, walks the redacted directory, calls `pz_parser`, aggregates records by signature, writes JSON, prints a one-line stats summary.
|
||||
|
||||
The split is deliberate: `pz_parser.py` is the module that eventually wants to be ported to PHP codex (separate spec). Keeping it pure makes that port mechanical and Python-side tests trivial.
|
||||
|
||||
## Parser pipeline phases
|
||||
|
||||
For each `*DebugLog-server*.txt`, the parser walks lines once and emits records via the following phases.
|
||||
|
||||
### 1. Severity-prefix recognition
|
||||
|
||||
Regex: `^\s*(ERROR|SEVERE|WARN)\s*[:\s]`. Broader than the existing `pz_error_analysis.py` regex — adds `SEVERE` (Java util-logging convention; appears in some PZ Java exception blocks). `LOG`/`INFO` is ignored at this layer.
|
||||
|
||||
### 2. Stack collection — bidirectional
|
||||
|
||||
Pzmm's contribution: PZ emits stack frames *before* the ERROR/WARN line as often as after.
|
||||
|
||||
- **Pre-stack**: walk up to 25 lines back from the severity line. Stop at another severity line or 8 collected. Only keep the block if at least one line looks stack-shaped (`at `, `[string ...]`, `function:`, `file:`, `.lua` markers).
|
||||
- **Post-stack**: walk forward up to 25 lines, gated by engine-noise detection. Stop at another severity line or 8 collected.
|
||||
- Merge deduped, preserving order; cap at 8 frames per record.
|
||||
|
||||
### 3. Mod attribution — three buckets
|
||||
|
||||
| Bucket | Trigger | Confidence |
|
||||
|---|---|---|
|
||||
| `direct` | Line itself matches `Lua\(\(MOD:([^)]+)\)\)` (or the `require("X") failed` shape, or an explicit `needed by <mod>` hint elsewhere in the entry) | `high` |
|
||||
| `inferred` | No marker on this line, but body is Lua-shaped (see below) *and* a `Lua((MOD:Y))` was emitted within the previous 40 lines | `medium` |
|
||||
| `unattributed` | Neither of the above | `low`; `mod_id = "__unattributed__"` |
|
||||
|
||||
"Lua-shaped" means the body matches at least one of (case-insensitive): `luamanager.getfunctionobject`, `no such function`, `exception thrown`, `runtimeexception`, `illegalstateexception`, or contains the bare token `lua`. This filter prevents inferred attribution from latching onto unrelated severity lines that happened to fall within the lookback window.
|
||||
|
||||
`mod_id` derives from the marker's raw name with a `_norm_mod_key` transform: lowercase, strip spaces / apostrophes / hyphens. `mod_name` preserves the human-readable form.
|
||||
|
||||
We do **not** attempt pzmm's filesystem-based reattribution.
|
||||
|
||||
### 4. File:line extraction — five fallbacks
|
||||
|
||||
Tried in order against the entry body and stack frames:
|
||||
|
||||
1. `at <path>.lua:<n>`
|
||||
2. `function: ... file: <path>.lua line #<n>` (or `: <n>`)
|
||||
3. `[string "<path>.lua"]:<n>`
|
||||
4. quoted path ending in `.lua` / `.txt` / `.xml` / `.json` / `.ini` / `.cfg` / `.bin`
|
||||
5. unquoted path segment beginning with `media/`, `maps/`, `lua/`, `scripts/`
|
||||
|
||||
Returns `(file, line)`; `line=0` if the matched form had no line number.
|
||||
|
||||
### 5. Cause-chain extraction
|
||||
|
||||
`Caused by: <X>` chains plus standalone exception lines (`(\w+\.)+\w+(Exception|Error): <msg>`) are normalised to `<ExceptionClass>: <msg>` tokens and joined with ` -> `. Up to 6 chain levels, deduped. Captures both Java exception nesting and Lua-wrapped exception chains.
|
||||
|
||||
### 6. Java exception kind detection
|
||||
|
||||
DebugLog-server has both Lua and Java exceptions; pzmm targets `console.txt` which is Lua-dominant. Extension here:
|
||||
|
||||
- `kind = "java_exception"` when the entry body or stack contains `(\w+\.)+\w+(Exception|Error)` AND no `Lua((MOD:X))` marker is present anywhere in the entry.
|
||||
- These typically resolve to `mod_id: __unattributed__` because Java code in PZ is engine, not mod. The exception class name becomes part of the message skeleton so similar Java exceptions dedup tightly.
|
||||
|
||||
### 7. Engine-noise tagging
|
||||
|
||||
`kind = "engine_noise"` when the body contains `kahluathread.flusherrormessage` or `dumping lua stack trace`. These severity-ERROR lines are PZ's own diagnostic chatter about its error reporting, not actual errors. They stay in the output (consumer can filter on `kind`).
|
||||
|
||||
### 8. Signature computation
|
||||
|
||||
Two-level deterministic identity, both stored on every record:
|
||||
|
||||
```
|
||||
pattern_id = sha256(level + normalized_first_line)[:16]
|
||||
signature = sha256(pattern_id + mod_id)[:16]
|
||||
```
|
||||
|
||||
Normalization for `pattern_id`:
|
||||
- Strip session metadata prefix (`General f:N, t:N, st:N,N,N,N>` shape)
|
||||
- Flatten double- and single-quoted strings to `"<S>"` / `'<S>'`
|
||||
- Flatten ≥2-digit numeric runs to `<N>`
|
||||
- Collapse whitespace
|
||||
- Truncate to 200 chars
|
||||
|
||||
Both fields ride on every record. Two consumer views, neither requires LLM:
|
||||
|
||||
- **Per-mod view** (signature is the dedup key): one record per `(mod_id, error_shape)` pair.
|
||||
- **Pattern fan-out view** (group records by `pattern_id`): see all mods that hit the same shape.
|
||||
|
||||
### 9. Aggregation
|
||||
|
||||
Records dedup on `signature`. On second-and-subsequent occurrences: `occurrence_count++`, `files` set-extends, attribution-confidence promotes (direct beats inferred beats unattributed), stack and `cause_chain` merge.
|
||||
|
||||
## Output schema
|
||||
|
||||
```json
|
||||
{
|
||||
"meta": {
|
||||
"input_dir": "/opt/ik-codex/.scratch/pz/Logs.redacted",
|
||||
"files_scanned": 6,
|
||||
"log_lines_total": 78654,
|
||||
"error_lines_total": 30984,
|
||||
"unique_signatures": N,
|
||||
"unique_patterns": M,
|
||||
"redacted": true,
|
||||
"started": "ISO8601",
|
||||
"finished": "ISO8601"
|
||||
},
|
||||
"signatures": [
|
||||
{
|
||||
"signature": "sha256:...",
|
||||
"pattern_id": "sha256:...",
|
||||
"level": "ERROR",
|
||||
"kind": "lua_runtime|require_failed|java_exception|engine_noise|runtime",
|
||||
"mod_id": "spongies_clothing",
|
||||
"mod_name": "Spongie's Clothing",
|
||||
"attribution": "direct|inferred|unattributed",
|
||||
"confidence": "high|medium|low",
|
||||
"attribution_reason": "...",
|
||||
"file": "media/lua/client/X.lua",
|
||||
"line": 42,
|
||||
"cause_chain": "ExceptionA: msg -> ExceptionB: msg",
|
||||
"stack": ["at A.lua:12", "at B.lua:34"],
|
||||
"first_seen": {"file": "...", "line": 1234, "timestamp": "26-04-26 17:14:35.128"},
|
||||
"occurrence_count": 47,
|
||||
"files": ["..."],
|
||||
"excerpt": "..."
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"errors": N,
|
||||
"warnings": N,
|
||||
"by_kind": {"lua_runtime": ..., "java_exception": ..., "require_failed": ..., "engine_noise": ..., "runtime": ...},
|
||||
"by_attribution": {"direct": ..., "inferred": ..., "unattributed": ...},
|
||||
"by_confidence": {"high": ..., "medium": ..., "low": ...},
|
||||
"top_mods": [{"mod_id": "...", "mod_name": "...", "occurrence_count": N}, ...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Default output path: `/opt/ik-codex/.scratch/pz/classify.json` (gitignored under `.scratch/`).
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
pz_classify.py [--input <dir>] [--out <path>] [--quiet]
|
||||
```
|
||||
|
||||
- `--input` defaults to `<repo>/.scratch/pz/Logs.redacted`
|
||||
- `--out` defaults to `<repo>/.scratch/pz/classify.json`
|
||||
- `--quiet` suppresses the trailing summary line
|
||||
|
||||
No `--limit`, `--resume`, or `--checkpoint-every`. Runs in seconds; nothing to throttle or resume.
|
||||
|
||||
## Tests
|
||||
|
||||
New directory `tools/pz-analyzer/tests/`. Stdlib `unittest`. Three files, ~18 tests total.
|
||||
|
||||
- **`test_parser.py`** (~10 tests) — one fixture per scenario in `tests/fixtures/` (synthetic, tracked in git): pure-Lua-attributed, pure-Java-exception, inferred-from-context, unattributed-engine-noise, multi-cause-chain, pre-stack-collection, post-stack-collection, severity-variants, file-line-extraction-fallbacks. All synthetic identifiers (placeholder Steam IDs / mod names) per the existing PHP-side `test/src/Games/ProjectZomboid/fixtures/` convention.
|
||||
- **`test_attribution.py`** (~5 tests) — three confidence buckets, the 40-line lookback boundary, "needed by X" extraction, and the rejection of inferred attribution when the message isn't Lua-shaped.
|
||||
- **`test_signatures.py`** (~3 tests) — `pattern_id` stability across formatting variations (whitespace, numeric values, quoted strings) and `signature` uniqueness across mods.
|
||||
|
||||
Invocation: `python -m unittest discover tools/pz-analyzer/tests/`. No external deps.
|
||||
|
||||
## Verification
|
||||
|
||||
End-to-end smoke against the redacted real-data directory:
|
||||
|
||||
```
|
||||
bash /opt/ik-codex/tools/pz-analyzer/pz_redact_all.sh # one-time, already done
|
||||
python /opt/ik-codex/tools/pz-analyzer/pz_classify.py
|
||||
```
|
||||
|
||||
Expect:
|
||||
- 6 files scanned, ~30,984 error lines processed.
|
||||
- A meaningful number of unique signatures and patterns (likely in the low hundreds for signatures; fewer patterns).
|
||||
- `top_mods` lists the highest-occurrence mods.
|
||||
- PII audit: no real Steam IDs, IPs, or coordinates in the output JSON (input is already redacted; classifier doesn't introduce PII).
|
||||
|
||||
Test invocation: `python -m unittest discover tools/pz-analyzer/tests/` should be all-green.
|
||||
|
||||
## Risks and open questions
|
||||
|
||||
- **Inferred attribution accuracy.** The 40-line lookback is pzmm's heuristic; it's correct for tightly-paced server bursts but can mis-attribute when an unrelated mod logs in the gap. Surface as `confidence: medium` so consumers can choose to treat them differently. Acceptable for v1; tunable via a constant in `pz_parser.py`.
|
||||
- **Pzmm targets `console.txt`, we target `DebugLog-server.txt`.** Format overlap is high (both share `Lua((MOD:X))` markers, Caused-by chains, Java exception shapes), but some patterns may be `console.txt`-specific. Tests use `DebugLog-server`-shaped fixtures only.
|
||||
- **Future PHP port.** `pz_parser.py` is structured for mechanical translation to a `LuaErrorAnalyser` / `ModAttributionAnalyser` pair under `src/Analyser/ProjectZomboid/` in a separate spec. Output schema chosen to be PHP-codex-compatible (Insight subclasses with typed fields).
|
||||
- **Licence.** The `paraxaQQ/pzmm` zip we reviewed has no top-level LICENSE; this spec mandates rewriting the patterns originally rather than copying code. Regex shapes and heuristics are general programming patterns and not author-specific, but no code blocks are lifted verbatim.
|
||||
|
||||
## Out of scope (explicit)
|
||||
|
||||
- Editing `pz_error_analysis.py` or `pz_redact_all.sh`.
|
||||
- Modifying any file in `/opt/ik-codex/src/`, `/opt/ik-codex/test/`, or `/opt/iblogs/`.
|
||||
- AI / LLM integration of any kind in the new tool.
|
||||
- iblogs front-end rendering of the classification output.
|
||||
- Filesystem mod-scan reattribution (pzmm's symbol/vehicle indexes).
|
||||
Reference in New Issue
Block a user