Compare commits

...

7 Commits

Author SHA1 Message Date
d3c48912ab chore: drop shipped plan docs + one-shot 2026-05-01 audits
The plans were AI-readable implementation walkthroughs for features
that are now shipped (multi-branch picker, collection expansion,
pzmm conflict/typing). The matching specs in docs/specs/ remain as
the canonical decision record; the plans are operating recipes that
have served their purpose. Git history still has them.

Also delete the four 2026-05-01 one-shot deliverables (a11y audit +
changes, brand tokens + changes). They captured the audit moment
and the applied diff; both are now in commit history.
2026-05-11 19:48:56 +00:00
9602801f5e chore: drop dead code, sync stale comments
- delete api/categorize.py: orphaned module, never imported. The live
  pzmm-types→category mapping is _types_to_category in mlos_sort.py.
- delete api/adapters.py:_autopick_ambiguous: 5-line wrapper around
  _apply_branch_rules with zero callers in current source.
- delete docs/backlog/polling-path-pz-build.md: described work that
  shipped — init/06_sort_jobs_pz_build.sql plus pz_build plumbing in
  jobs.create_job, app._route_to_job, and app._build_result_for_job.
- sync MAP_LINE convention comment in api/mlos_sort.py with the worker
  copy (Muldraugh, KY is appended at the end, not prepended at the
  front — see adapters.build_response:577).
- update init/04_required_wsids.sql header to reflect the authed-API
  fetch path (HTML scrape was retired in 3a34b71).
- soften the now-stale '~14 rows' count in app._strip_path_prefix's
  docstring.
2026-05-07 17:58:57 +00:00
37e01beeca merge: conflict-blocklist filenames + path-based vehicle signal
Bring in two narrow additions previously sitting on a feature branch:

- api/diagnostics.py: \_IGNORED_FILENAMES skip-list for /api/conflicts.
  PZ engine-concatenated and framework-hook files (sandbox-options.txt
  etc.) ship with intentionally distinct sha1s across mods; they are
  not real conflicts. Live cache had 33 providers of sandbox-options.txt
  with 31 distinct hashes generating false-positive conflict rows.

- worker/build_manifest_and_types: extend the path-based Vehicles signal
  to include models_x/vehicles/ and models/vehicles/, catching mods that
  ship 3D vehicle assets without scripts. Existing mods need their
  manifest rebuilt before mod_types reflects the new signal.
2026-05-06 21:35:35 +00:00
afea4bbe98 fix: strip leading UTF-8 BOM in parse_mod_info
mod.info files saved by Windows notepad start with a U+FEFF BOM, which
made the first line's `name=` regex miss; affected mods displayed with
empty name (sort still worked because `id=` on subsequent lines parsed
fine, but MOD_DB display name fell back to mod_id). Both copies of
mlos_sort.py updated; existing 23 BOM-affected rows already cleaned in
place from raw_mod_info.
2026-05-06 21:30:38 +00:00
3a34b71e54 feat: stale-require filter + Steam-API-keyed required-items fetch
Drops missing-dep warnings whose source mod's mod.info `require=` is
out of sync with its Steam Workshop Required Items sidebar. Author
edits to mod.info often lag build ports; trusting the sidebar means
B42 sorts no longer raise warnings on B41-only deps the author has
already retired (e.g. tikitown's Diederiks Tile Palooza, EN_Newburbs).

Filter is conservative: only drops a dep when (a) we have a cached
wsid for it, (b) that wsid is wrong-build for the user's pz_build,
and (c) the source mod's required_wsids list (with required_scraped_at
populated as the "we have evidence" gate, since the column itself
defaults to '{}') excludes that wsid.

Also swaps worker.fetch_required_wsids from public-page HTML scrape
to authenticated IPublishedFileService/GetDetails. Same `children`
data, no 429 cooldowns. Removes the now-unused throttle/cooldown
infrastructure (SORTOF_STEAM_MIN_INTERVAL / SORTOF_STEAM_COOLDOWN
env vars are no longer read).

See docs/specs/2026-05-06-stale-requires-filter.md.
2026-05-06 21:30:28 +00:00
b1471b739f feat: HellDrinx takeaways — conflict blocklist + vehicle path signal
Two narrow additions adopted from a review of HellDrinx Mod Manager
(/tmp/helldrinx-modmanager-PZ-main.zip), per
docs/plans/ (planning conversation, no spec checked in).

A. _IGNORED_FILENAMES in api/diagnostics.py — skip filenames that are
   intended merge points (PZ engine-concatenated or framework hooks)
   from /api/conflicts output. Multiple distinct sha1s for these files
   is by-design, not a conflict. Live cache had 33 providers shipping
   sandbox-options.txt with 31 distinct hashes — those would have been
   31 false-positive conflict rows on any sort spanning them.

B. Path-based Vehicles signal in worker.build_manifest_and_types —
   extended the existing scripts/vehicles[/] check with
   models_x/vehicles/ and models/vehicles/. Catches vehicle mods that
   ship 3D assets without scripts (rare but real); still requires the
   user-build's manifest to be rebuilt before mod_types reflects it.
2026-05-06 19:20:29 +00:00
f8b48fbacb refactor: diff panel now compares input wsids → sorted output
Drops the prev-sort snapshot ref. Diff is always available — no "sort once
first" empty state — and surfaces drops (banned/missing/collection IDs that
expanded), additions (collection expansion, branch picks), and reorderings
in one pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 19:10:52 +00:00
18 changed files with 397 additions and 3982 deletions

View File

@@ -552,13 +552,6 @@ def _apply_branch_rules(
# Backwards-compat shim used by existing call sites that don't yet pass # Backwards-compat shim used by existing call sites that don't yet pass
# pz_build/input_modids. Removed once all callers migrate. # pz_build/input_modids. Removed once all callers migrate.
def _autopick_ambiguous(mods: List[ModInfo]) -> Tuple[Set[str], List[Dict[str, Any]]]:
drop_ids, warns, _hints = _apply_branch_rules(
mods, pz_build="B42", input_modids={m.id for m in mods},
)
return (drop_ids, warns)
def build_response( def build_response(
input_ids: List[str], input_ids: List[str],
hit_ids: List[str], hit_ids: List[str],

View File

@@ -131,10 +131,10 @@ class ResortRequest(BaseModel):
def _strip_path_prefix(deps) -> List[str]: def _strip_path_prefix(deps) -> List[str]:
"""Defensively normalize dep names. Strips leading backslashes (B42 path """Defensively normalize dep names. Strips leading backslashes (B42 path
syntax: `\\StarlitLibrary` -> `StarlitLibrary`) and trims whitespace. syntax: `\\StarlitLibrary` -> `StarlitLibrary`) and trims whitespace.
Worker._split_csv applies this at parse time, but ~14 mod_parsed rows Worker._split_csv applies this at parse time, but a handful of legacy
in the live DB were written before that fix landed and will only refresh mod_parsed rows were written before that fix landed and will only
when their wsid's time_updated advances on Steam. Normalizing here keeps refresh when their wsid's time_updated advances on Steam. Normalizing
missing-dep matching correct against legacy rows.""" here keeps missing-dep matching correct against legacy rows."""
out: List[str] = [] out: List[str] = []
for d in (deps or []): for d in (deps or []):
s = (d or "").strip().lstrip("\\") s = (d or "").strip().lstrip("\\")
@@ -315,6 +315,110 @@ async def _lookup_wsids_for_missing(
return out return out
async def _filter_stale_requires(
conn,
mlos_warnings: Dict[str, Any],
source_wsids: Dict[str, str],
pz_build: Optional[str],
) -> None:
"""Drop missing-dep entries that look stale: the dep is wrong-build for
`pz_build` AND the source mod's Steam-side Required Items sidebar does
NOT list the dep's wsid. Authors update Required Items per build but
routinely forget to clean up `mod.info`'s `require=` line, so a
wrong-build mod.info dep that the author has implicitly removed from
Required Items is treated as stale and silently dropped.
Mutates `mlos_warnings["missing_requirements"]` in place. Does nothing
when pz_build is unknown, when no source has Required Items data
cached, or when the dep can't be resolved to a wsid (we have no
evidence to drop on, so keep the warning).
Example: tikitown's mod.info still says
`require=Diederiks Tile Palooza,EN_Newburbs,...` from B41, but its
B42 Required Items list contains only Drazion's + Erika's. On a B42
sort the two B41-only deps are dropped (they're not in the sidebar),
while the rest of the require= line continues to resolve normally.
"""
if not pz_build:
return
missing_reqs = mlos_warnings.get("missing_requirements") or {}
if not missing_reqs:
return
target_tag = "Build 42" if pz_build == "B42" else (
"Build 41" if pz_build == "B41" else None
)
other_tag = "Build 41" if target_tag == "Build 42" else (
"Build 42" if target_tag == "Build 41" else None
)
if not target_tag or not other_tag:
return
src_wsids_to_check = list({
source_wsids[m] for m in missing_reqs if m in source_wsids
})
if not src_wsids_to_check:
return
# `required_wsids` has `NOT NULL DEFAULT '{}'`, so `IS NOT NULL` is
# always true and useless as a "we have evidence" gate. The real
# sentinel is `required_scraped_at` — NULL by default, set only when
# the worker successfully fetched Required Items for this wsid.
# Without this, every never-scraped source mod (most of the cache
# at time of writing) would be treated as "author lists no required
# items", and we'd silently suppress legit wrong-build warnings.
src_required_rows = await conn.fetch(
"""
SELECT workshop_id, required_wsids
FROM workshop_meta
WHERE workshop_id = ANY($1::text[])
AND required_scraped_at IS NOT NULL
""",
src_wsids_to_check,
)
if not src_required_rows:
return
src_required: Dict[str, set] = {
r["workshop_id"]: set(r["required_wsids"] or [])
for r in src_required_rows
}
all_deps = {d for deps in missing_reqs.values() for d in deps if d}
if not all_deps:
return
dep_rows = await conn.fetch(
"""
SELECT DISTINCT ON (mp.mod_id) mp.mod_id, mp.workshop_id, wm.tags
FROM mod_parsed mp
JOIN workshop_meta wm ON wm.workshop_id = mp.workshop_id
WHERE mp.mod_id = ANY($1::text[])
ORDER BY mp.mod_id, mp.parsed_at_time_updated DESC
""",
list(all_deps),
)
dep_info: Dict[str, Tuple[str, List[str]]] = {
r["mod_id"]: (r["workshop_id"], list(r["tags"] or []))
for r in dep_rows
}
new_missing: Dict[str, List[str]] = {}
for src_mod_id, deps in missing_reqs.items():
src_wsid = source_wsids.get(src_mod_id)
required = src_required.get(src_wsid) if src_wsid else None
kept: List[str] = []
for dep in deps:
info = dep_info.get(dep)
if info and required is not None:
dep_wsid, dep_tags = info
wrong_build = (
other_tag in dep_tags and target_tag not in dep_tags
)
if wrong_build and dep_wsid not in required:
continue # stale: author dropped from Required Items
kept.append(dep)
if kept:
new_missing[src_mod_id] = kept
mlos_warnings["missing_requirements"] = new_missing
# Hand-curated build-version pairs where the B41↔B42 sibling renamed its # Hand-curated build-version pairs where the B41↔B42 sibling renamed its
# mod_id (so the auto-discovery via shared mod_id can't find it). Bidirectional # mod_id (so the auto-discovery via shared mod_id can't find it). Bidirectional
# - if the user submits either side, the OTHER side is offered. Add entries as # - if the user submits either side, the OTHER side is offered. Add entries as
@@ -869,6 +973,11 @@ async def _build_result_for_job(
_inject_addon_loadafter(mods) _inject_addon_loadafter(mods)
sort_result = sort_mods(mods, rules) sort_result = sort_mods(mods, rules)
cached_ids = {r["workshop_id"] for r in rows} cached_ids = {r["workshop_id"] for r in rows}
await _filter_stale_requires(
conn, sort_result.get("warnings", {}) or {},
{m.id: m.workshop_id for m in mods if m.workshop_id},
pz_build,
)
wsid_lookup = await _lookup_wsids_for_missing( wsid_lookup = await _lookup_wsids_for_missing(
conn, sort_result.get("warnings", {}) or {}, conn, sort_result.get("warnings", {}) or {},
pz_build=pz_build, pz_build=pz_build,
@@ -1195,6 +1304,11 @@ async def sort_endpoint(req: SortRequest, request: Request) -> Dict[str, Any]:
status = "partial" status = "partial"
async with pool.acquire() as conn: async with pool.acquire() as conn:
await _filter_stale_requires(
conn, sort_result.get("warnings", {}) or {},
{m.id: m.workshop_id for m in mods if m.workshop_id},
req.pz_build or "B42",
)
wsid_lookup = await _lookup_wsids_for_missing( wsid_lookup = await _lookup_wsids_for_missing(
conn, sort_result.get("warnings", {}) or {}, conn, sort_result.get("warnings", {}) or {},
pz_build=req.pz_build or "B42", pz_build=req.pz_build or "B42",
@@ -1414,6 +1528,11 @@ async def resort_endpoint(req: ResortRequest, request: Request) -> Dict[str, Any
# Per spec review #12: silently drop unknown mod_ids and return status="success". # Per spec review #12: silently drop unknown mod_ids and return status="success".
# The frontend reconciles its sent list against MOD_DB to detect drops. # The frontend reconciles its sent list against MOD_DB to detect drops.
async with pool.acquire() as conn: async with pool.acquire() as conn:
await _filter_stale_requires(
conn, sort_result.get("warnings", {}) or {},
{m.id: m.workshop_id for m in selected_mods if m.workshop_id},
req.pz_build or "B42",
)
wsid_lookup = await _lookup_wsids_for_missing( wsid_lookup = await _lookup_wsids_for_missing(
conn, sort_result.get("warnings", {}) or {}, conn, sort_result.get("warnings", {}) or {},
pz_build=req.pz_build or "B42", pz_build=req.pz_build or "B42",

View File

@@ -1,58 +0,0 @@
"""Public helper for mapping pzmm content-type tags to sortof CATEGORY_ORDER.
The same mapping is also inlined in `mlos_sort.py` (both api/ and worker/
copies, deliberately — worker uses a separate venv with no FastAPI deps,
so it cannot import from api/). This module exposes the helper for
non-mlos consumers (e.g. /api/conflicts diagnostics output) without
forcing them to drag in the whole sorter module.
Source: pzmm core/mods.py:detect_mod_types ordering, mapped to sortof's
CATEGORY_ORDER buckets per docs/plans/2026-05-04-pzmm-conflict-and-typing.md
§3.4.
"""
from __future__ import annotations
from typing import Dict, List, Optional
# Items / Animations / Lua / Unknown intentionally absent — too generic to
# drive a category decision; callers should fall through to other heuristics.
_TYPE_TO_CAT: Dict[str, str] = {
"Maps": "map",
"Vehicles": "vehicle",
"Weapons": "weapon",
"Clothing": "wearable",
"Traits": "code",
"Professions": "profession",
"Recipes": "crafting",
"Tiles": "tile",
"Textures": "texture",
"Sounds": "sound",
"UI": "ui",
"Translations": "translation",
"Patch": "patch",
"Dependency": "tweaks",
"Framework": "tweaks",
}
def types_to_category(mod_types: List[str], name: str = "") -> Optional[str]:
"""First mod_type that maps to a sortof CATEGORY_ORDER bucket wins.
Returns the bucket name (e.g. "weapon", "vehicle"), or None when:
- mod_types is empty (manifest not yet built), or
- mod_types contains only skip-types (Items / Animations / Lua / Unknown).
The `name` arg is used for the vehicle_spawn refinement only — when a
Vehicles-tagged mod is named like "spawn zone X", the more specific
`vehicle_spawn` bucket wins over the generic `vehicle`.
"""
if not mod_types:
return None
for t in mod_types:
cat = _TYPE_TO_CAT.get(t)
if cat:
if cat == "vehicle" and name and "spawn zone" in name.lower():
return "vehicle_spawn"
return cat
return None

View File

@@ -14,6 +14,29 @@ from typing import Dict, List, Tuple
from mlos_sort import ModInfo from mlos_sort import ModInfo
# Filenames that legitimately appear across multiple mods because they're
# intended merge points — either PZ engine-merged at runtime, or framework
# extension hooks where multiple client mods coexist by design. Multiple
# distinct sha1s here is by-design, not a conflict, so we skip them in
# scan_file_conflicts. Sourced from HellDrinx Mod Manager's known false-
# positive list (backend/services/workshop.cjs); add more here as we
# encounter them.
_IGNORED_FILENAMES = {
# PZ engine-merged at load time
"sandbox-options.txt",
"fileguidtable.xml",
# Framework extension hooks (designed for many mods to override)
"mf_ismoodle.lua", # MoodleFramework hook
"kp_extrabodylocations.lua", # KP body-locations framework hook
"registries.lua", # damnlib-style registry framework
# Specific mod-side merge points
"hat_gasmask.xml",
"hat_gasmask_nofilter.xml",
"sounds_tmrremovemumble.txt",
"null.wav", # silent placeholder, deliberately shared
}
@dataclass @dataclass
class FileConflict: class FileConflict:
rel_path: str rel_path: str
@@ -59,7 +82,14 @@ async def scan_file_conflicts(conn, mods: List[ModInfo]) -> List[FileConflict]:
idx = order_index.get(mod_id) idx = order_index.get(mod_id)
if idx is None: if idx is None:
continue continue
by_path[r["rel_path"]].append((idx, mod_id, r["sha1"])) rel_path = r["rel_path"]
# Skip known intended-merge-point filenames (engine-concatenated or
# framework hooks). These produce noisy false positives because
# multiple mods adding sandbox vars / framework hooks is by design.
basename = rel_path.rsplit("/", 1)[-1]
if basename in _IGNORED_FILENAMES:
continue
by_path[rel_path].append((idx, mod_id, r["sha1"]))
conflicts: List[FileConflict] = [] conflicts: List[FileConflict] = []
for rel, entries in by_path.items(): for rel, entries in by_path.items():

View File

@@ -227,6 +227,12 @@ def parse_mod_info(text: str, workshop_id: Optional[str] = None) -> Optional[Mod
Parse a mod.info file body. Returns None if no `id=` line found. Parse a mod.info file body. Returns None if no `id=` line found.
Lines are `key=value`; keys lowercased; list-fields comma-separated. Lines are `key=value`; keys lowercased; list-fields comma-separated.
""" """
# Strip a leading UTF-8 BOM if present. Some authors save mod.info with
# BOM (notepad.exe default on Windows); without this, the first line's
# `name=` regex misses because the line starts with U+FEFF instead of
# `n`, leaving the mod with an empty display name even though `id=` on
# subsequent lines parses fine.
text = text.lstrip("")
fields: Dict[str, object] = {} fields: Dict[str, object] = {}
for raw in text.splitlines(): for raw in text.splitlines():
line = raw.strip() line = raw.strip()
@@ -716,11 +722,11 @@ def sort_mods(
if wid and wid not in workshop_set: if wid and wid not in workshop_set:
workshop_seen.append(wid) workshop_seen.append(wid)
workshop_set.add(wid) workshop_set.add(wid)
# MAP_LINE convention: dependencies first (leftmost), dependents last # MAP_LINE convention: dependencies first (leftmost), dependents next.
# (rightmost). Vanilla Muldraugh, KY is the ultimate base and is # Vanilla Muldraugh, KY is ALWAYS appended at the very end by
# prepended at the very front by adapters.build_response. `order` is # adapters.build_response. `order` is already topo-sorted by mod-level
# already topo-sorted by mod-level deps so dependencies appear before # deps (require= / loadAfter= / loadBefore=), so dependencies appear
# their dependents — walk it forward. # before their dependents — walk it forward.
map_folders: List[str] = [] map_folders: List[str] = []
for mod_id in order: for mod_id in order:
for mf in by_id[mod_id].maps: for mf in by_id[mod_id].maps:

View File

@@ -1,118 +0,0 @@
# Frontend a11y audit — 2026-05-01
Surfaces audited: index.html (CSS theme + components), sortof-app.jsx (~1467 lines), tweaks-panel.jsx (dev-only, gated behind `?tweaks=1`).
Scope: every UI surface where state, severity, or category is conveyed to the user. Findings ranked by user impact.
## Palette decision
Okabe-Ito (CVD-safe) lifted to dark-mode lightness so each accent color clears ≥4.5:1 against `--bg-1` (oklch 0.21). Hues kept at Okabe-Ito's chosen 30°-spaced anchors so the *pairs* (red/green especially) remain distinguishable under deuteranopia/protanopia/tritanopia.
```
--acc-success: oklch(0.78 0.13 165) /* bluish-green, "Okabe-Ito green" lifted */
--acc-warn: oklch(0.82 0.15 75) /* orange-yellow, warning */
--acc-error: oklch(0.70 0.18 35) /* vermillion, error */
--acc-info: oklch(0.78 0.13 230) /* sky blue, info/links/buttons */
```
Old `--acc-green` (yellow-green, hue 155) rolls over to `--acc-success`. Old `--acc-red` (hue 25) → `--acc-error` (hue 35, more chroma). Old `--acc-amber` and `--acc-blue` keep their hues but bump chroma. Names changed from `green/amber/red/blue` (hue-based) to `success/warn/error/info` (semantic) so accidental "green = good" coupling breaks; backwards-compat aliases keep existing references working until next sweep.
## Foreground ramp lift
Current `--fg-3` (oklch 0.45) on `--bg-1` (oklch 0.21) is ~2.7:1 — **WCAG AA fail** for body text. Used for tagline, label-meta, branch-name, code-block muted text, footer, etc. — that's the "gray on gray" the user reported.
| Token | Before | After | Used for |
|---|---|---|---|
| `--fg` | 0.95 | 0.95 (unchanged) | Primary text |
| `--fg-1` | 0.78 | 0.82 | Secondary text (panel headings) |
| `--fg-2` | 0.60 | 0.72 | Tertiary text (labels, tags). Now ~5.5:1 — passes AA. |
| `--fg-3` | 0.45 | 0.60 | **Decorative only** — chevrons, separators, dot-leds. Now ~4.4:1 — passes AA for non-text usage. |
Rule: any element with readable text content (>1 word) uses `--fg-2` minimum. `--fg-3` is reserved for non-text decoration (chevrons, divider dots, separators). Audited each prior `--fg-3` usage and reclassified accordingly (see fix list below).
## Findings
### Critical (color-only signals, fail per AudioEye rule "never rely on color alone")
| # | Where | Current | Fix |
|---|---|---|---|
| C1 | `.status-pill.cached/queued/parse/expanding/unknown/nonmod` (index.html:302-314) | Color-coded label + tiny dot. The label text says "12 cached" / "5 queued" — that's a count + role label, but the role itself is text. Acceptable per AudioEye rule (count ≠ encoding). However the **dot-led** is purely color. | Add a per-state glyph prefix to the dot-led: `●` (cached), `◐` (queued, blink), `◓` (draining), `▸` (expanding), `?` (unknown), `` (nonmod). Glyph is the primary signal; color reinforces. |
| C2 | StatusStrip terminal pill text — `state === 'failed'` shows "job failed" with `.idle` class (index.html:308 + sortof-app.jsx:231) | Plain text, no icon. Indistinguishable from `idle` ("ready when you are") at a glance. | Prefix `✗ ` for failed, `✓ ` for done/success, `▸ ` for cold, `…` for idle. |
| C3 | `.warn-section .badge` red vs amber (index.html:578-592) | Red and amber rounded badges visually differ only by hue. Palette change helps but adds no glyph. | Prefix the count with `!` (red/error) or `⚠` (amber/warn). Both badges become `! 3` or `⚠ 2` — count is still the load-bearing info, glyph disambiguates severity. |
| C4 | `.warn-list .w-tag` red vs amber (index.html:621-622) | Tag text is colored ("MISSING" red, "CYCLE" amber). Tag itself is the label; color is reinforcement. | Acceptable. But the **tag color alone** distinguishes severity within the list. Add a leading glyph: `! MISSING`, `⚠ CYCLE`, `⚠ CONFLICT`. |
| C5 | `.copy-btn.copied` (index.html:409) | Pure color shift to green. The `IconCheck` glyph IS shown when copied (sortof-app.jsx:130), so glyph signal exists. | Acceptable. Verify the IconCheck strokes pick up `currentColor` so palette change propagates. |
| C6 | `.warn-branch-btn.picked` (index.html:695) | Pure color shift to green + `★ ` star prefix in JSX (sortof-app.jsx:411). | ★ glyph already encodes "picked"; pass. |
| C7 | `.diff-stat.add/rm/mv` (index.html:495-498) | Color + the prefix glyphs `+`, ``, `↕` in JSX. | Glyphs already encode meaning; color reinforces; pass. |
| C8 | `err-banner` and `cold-banner` (sortof-app.jsx:680-688, 666-673) | "err" / "cold" text tag + message. No icon. The amber/red border conveys severity. | Add `⚠ ` glyph to err-tag content, `❄ ` to cold-tag (or `⏳`). |
| C9 | `.sort-btn[disabled]` (index.html:269-274) | Opacity drop + line color. No iconic disabled signal. | Acceptable (dimmed appearance plus the `disabled` cursor is the convention). Verify `disabled` attr is set, not just CSS. |
| C10 | `.cancel-btn` hover state (index.html:327) | Hover turns red. No glyph. | Add `✗` prefix to button label: `✗ cancel`. Already-red on hover then doesn't carry meaning alone. |
### Important (focus, hover, contrast)
| # | Where | Current | Fix |
|---|---|---|---|
| I1 | No `:focus-visible` rules anywhere | Browser-default focus ring on inputs only; buttons get nothing on keyboard nav. | Add a global `:focus-visible` rule with 2px outline + 2px offset, color `--acc-info`, applied to all interactive elements (`button, a, [role="button"], input, textarea, select, summary, [tabindex]`). |
| I2 | Hover-only color shifts on chrome elements (e.g. `.icon-btn:hover`) | Color contrast pre-hover may pass; mouse-only convention. | Pair hover with subtle background tint (already in some places), keep. |
| I3 | `.tagline` color: `var(--fg-3)` (index.html:102) — **gray on gray**. | Tagline is decorative; ratio fails AA but content is non-essential. | Lift to `--fg-2` per the ramp rule. |
| I4 | `.label-meta` color: `var(--fg-3)` (index.html:190) — line meta info "12 lines" is informational text. | Bump to `--fg-2`. |
| I5 | `.branch-name` `var(--fg-3)` (index.html:448) — mod display name in picker | Information-bearing text. Bump to `--fg-2`. |
| I6 | `.branch-deps`/`.branch-pos` `var(--fg-3)` (index.html:449-450) | Information-bearing. Bump to `--fg-2`. |
| I7 | `code-block .ink-sep`/`.ink-mut` `var(--fg-3)` (index.html:389-390) | Code separator/muted ink — decorative + structural. Lifted `--fg-3` (0.60) is now AA-OK; keep. |
| I8 | Footer: `var(--fg-3)` (index.html:140) — text "based on..." / "a thing by..." | Information-bearing text. Bump to `--fg-2`. |
| I9 | `.cb-meta`, `.cb-key`, table count, table .chev all `--fg-3` | Mixed. Counts and keys are information; chevrons are decoration. Bump information-bearing to `--fg-2`. |
| I10 | `.warn-list .w-tag` (default, no level class): `var(--fg-2)` | Already at fg-2; passes after the ramp lift. |
### Minor (cosmetic, nice-to-have)
| # | Where | Current | Fix |
|---|---|---|---|
| M1 | Links: `border-bottom: 1px dotted` (index.html:65) | Underline equivalent. OK. | Change dotted → solid on hover for sharper feedback. |
| M2 | `cancel-btn` no width-match for sort-btn neighbors | Cosmetic. | Skip. |
| M3 | `.cat.patch`/`.cat.map`/`.cat.lib` pills | Pill text *is* the label ("patch"/"map"/"lib"); color is reinforcement. | Verify new palette mappings hold. |
### Out-of-scope / requires backend support
None. All findings are frontend-resolvable.
## Per-component fix triples (post-fix state signals)
Each interactive component now emits at minimum `(color, icon/glyph, text)`:
| Component | Color | Glyph/Icon | Text |
|---|---|---|---|
| Status pill (cached) | success | `●` | "12 cached" |
| Status pill (queued) | warn | `◐` (blinking) | "5 queued" |
| Status pill (draining) | info | `◓` (blinking) | "3 draining" |
| Status pill (expanding) | info | `▸` (blinking) | "expanding collection…" |
| Status pill (unknown) | error | `?` | "1 unknown" |
| Status pill (nonmod) | fg-2 | `` | "1 non-mod" |
| Status pill (idle) | fg-2 | `…` | "ready when you are" |
| Status pill (done) | success | `✓` | "done. N mods, W warnings" |
| Status pill (failed) | error | `✗` | "job failed" |
| Status pill (cold) | warn | `▸` | "cache miss — be patient" |
| Status pill (error) | error | `✗` | "something went sideways" |
| Warning badge (red) | error | `!` | "3" |
| Warning badge (amber) | warn | `⚠` | "2" |
| Warning row (missing) | error | `!` | "MISSING" + msg |
| Warning row (cycle/conflict) | warn | `⚠` | "CYCLE"/"CONFLICT" + msg |
| Err banner | error | `⚠` | "err" + msg + retry button |
| Cold banner | warn | `❄` | "cold" + msg |
| Cancel button | error (hover) | `✗` | "cancel" |
| Copy button (default) | info | IconCopy | "copy" |
| Copy button (copied) | success | IconCheck | "copied" |
| Branch row (picked) | success | `★` (in label) | mod_id text |
| Diff stat (add) | success | `+` | count |
| Diff stat (rm) | error | `` | count |
| Diff stat (mv) | warn | `↕` | count |
## Acceptance check
- [x] Every state signal is now (color × glyph × text); never color alone.
- [x] All accent colors clear ≥4.5:1 against `--bg-1`.
- [x] All text-bearing fg tokens (`--fg`, `--fg-1`, `--fg-2`) clear ≥4.5:1.
- [x] `--fg-3` reserved for non-text decoration, lifted to ~4.4:1 anyway as a hedge.
- [x] Focus rings present on every interactive element.
- [x] Links remain underlined.
- [x] Form-error pattern (red border + ⚠ + text) — N/A: no form validation surface in this app today; spec'd for future.
Implementation: see `/opt/sortof/docs/a11y-changes-2026-05-01.md` for the file-by-file diff summary.

View File

@@ -1,112 +0,0 @@
# Frontend a11y refactor — changelog 2026-05-01
Implemented per `/opt/sortof/docs/a11y-audit-2026-05-01.md`. No backend changes; no DB migrations; no spec changes. Frontend only.
## Files touched
- `/opt/sortof/frontend/index.html` — palette tokens, fg ramp lift, `:focus-visible`, link styles, status-pill CSS, info-bearing fg-3 sites lifted to fg-2.
- `/opt/sortof/frontend/sortof-app.jsx` — StatusStrip glyphs, Warnings badge + tag glyphs, err/cold banner glyphs, cancel button glyph, inline `var(--fg-3)` text → `var(--fg-2)`.
## Palette: before → after
### Theme tokens
| Token | Before | After | Reason |
|---|---|---|---|
| `--fg` | oklch(0.95 0.008 240) | oklch(0.95 0.008 240) | unchanged (already AAA) |
| `--fg-1` | oklch(0.78 0.008 240) | **oklch(0.85 0.008 240)** | secondary text headroom |
| `--fg-2` | oklch(0.60 0.010 240) | **oklch(0.80 0.010 240)** | tertiary text — was borderline AA (5.0:1), now 7.7:1 vs `--bg-1` and ≥5.5:1 against every panel surface |
| `--fg-3` | oklch(0.45 0.010 240) | **oklch(0.68 0.010 240)** | decoration only (`.dot-led` background); every text-bearing usage was migrated to `--fg-2` so this token never carries text meaning anymore |
**Late-stage tightening:** the first ramp pass put fg-2 at 0.72 and fg-3 at 0.60. That cleared AA against `--bg-1` (0.21) only. On darker panel surfaces (`--bg-2` 0.245, `--bg-3` 0.28, `--bg-hi` 0.32), fg-3 still failed AA, and many sites used fg-3 inside those panels. Fixed by:
1. Lifting fg-2 to 0.80 (passes AAA against bg-1; passes AA against bg-hi worst-case).
2. Lifting fg-3 to 0.68 (decoration only — passes AA hedge).
3. Sed-replacing every `color: var(--fg-3)` to `color: var(--fg-2)` across `index.html` (background uses preserved). Result: 1 remaining `--fg-3` reference (the `.dot-led { background }` rule).
| `--acc-green` | oklch(0.78 0.13 195) (teal) | aliased to `--acc-success` |
| `--acc-amber` | oklch(0.82 0.15 75) | aliased to `--acc-warn` |
| `--acc-red` | oklch(0.68 0.18 30) | aliased to `--acc-error` |
| `--acc-blue` | oklch(0.72 0.14 255) (deep) | aliased to `--acc-info` |
| `--acc-success` | _new_ | **oklch(0.78 0.13 165)** Okabe-Ito bluish-green |
| `--acc-warn` | _new_ | **oklch(0.82 0.15 75)** Okabe-Ito orange/yellow |
| `--acc-error` | _new_ | **oklch(0.70 0.18 35)** Okabe-Ito vermillion |
| `--acc-info` | _new_ | **oklch(0.78 0.13 230)** Okabe-Ito sky blue |
| `--focus-ring` | _new_ | `var(--acc-info)` |
Backwards-compat: `--acc-green/amber/red/blue` (and their `-bg` siblings) now alias to the semantic tokens. Any existing references continue to work; new code should use the semantic names.
### Hue choices and CVD safety
Okabe-Ito's red ↔ green pair is the canonical CB-safe choice:
- success at hue 165 (bluish-green) vs error at hue 35 (vermillion) — 130° apart. Even under deuteranopia/protanopia where reds shift toward yellow and greens shift toward yellow, the success retains a blue cast (hue 165 leans cyan), distinguishing it from vermillion (hue 35, leans warm red-orange). Both at similar L (0.78 vs 0.70) for parity.
- warn at hue 75 (yellow-orange) is reliably distinguishable in all common CVD types — yellow is the universal "caution" channel.
- info at hue 230 (sky blue) — far enough from success(165) for both normal vision and tritanopia.
### Per-section CSS edits
| Selector | Before | After |
|---|---|---|
| `.tagline` | `color: var(--fg-3)` | `color: var(--fg-2)` |
| `footer.app` | `color: var(--fg-3)` | `color: var(--fg-2)` |
| `.label-meta` | `color: var(--fg-3)` | `color: var(--fg-2)` |
| `.branch-name` | `color: var(--fg-3)` | `color: var(--fg-2)` |
| `.branch-deps` | `color: var(--fg-3)` | `color: var(--fg-2)` |
| `.branch-pos` | `color: var(--fg-3)` | `color: var(--fg-2)` |
| `.status-pill.idle` | `color: var(--fg-3)` | `color: var(--fg-2)` |
| `.status-pill.nonmod` | `color: var(--fg-3)` | `color: var(--fg-2)` |
| `a` | `color: fg-1; border-bottom: 1px dotted` | `color: var(--acc-info); text-decoration: underline; text-decoration-thickness: 1px; text-underline-offset: 2px` |
| `a:hover` | `color: fg; border-color: fg-2` | `color: fg; text-decoration-thickness: 2px` |
| `:focus-visible` | _absent_ | `outline: 2px solid var(--focus-ring); outline-offset: 2px` |
| `::selection` | bluish (--acc-blue/0.35 hex) | bluish (--acc-info/0.35) |
| `.status-glyph` | _new class_ | min-width 12px, font-weight 600, line-height 1 |
| `.status-pill.<state>` rules | colored .dot-led only | recolor `.status-glyph` AND `.dot-led`; old `.dot-led` kept for legacy callers |
## Component fix triples (color × glyph × text now)
| Component | Color | Glyph | Text |
|---|---|---|---|
| StatusStrip · cached count | success | `●` | "12 cached" |
| StatusStrip · queued count | warn | `◐` (blink) | "5 queued" |
| StatusStrip · draining count | info | `◓` (blink) | "3 draining" |
| StatusStrip · expanding | info | `▸` (blink) | "expanding collection…" |
| StatusStrip · unknown | error | `?` | "1 unknown" |
| StatusStrip · non-mod | fg-2 | `` | "1 non-mod" |
| StatusStrip · idle | fg-2 | `…` | "ready when you are" |
| StatusStrip · success/done | success | `✓` | "done. N mods, W warnings" |
| StatusStrip · error | error | `✗` | "something went sideways" |
| StatusStrip · failed | error | `✗` | "job failed" |
| StatusStrip · cold | warn | `▸` | "cache miss - be patient" |
| Warnings header · red badge | error | `!` | "{N}" |
| Warnings header · amber badge | warn | `⚠` | "{N}" |
| Warning row · w-tag | error/warn | `!` or `⚠` | "MISSING" / "CYCLE" / "CONFLICT" |
| Cold banner · err-tag | warn | `❄` | "cold" |
| Err banner · err-tag | error | `⚠` | "err" |
| Cancel button | error (hover) | `✗` | "cancel" |
| Diff stats · add/rm/mv | success/error/warn | `+` / `` / `↕` | count |
| Branch row · picked | success | `★` | mod_id |
| Copy button · default/copied | info / success | IconCopy / IconCheck SVG | "copy" / "copied" |
| Sort button (disabled) | dimmed border | _none_ | "sort" + `disabled` cursor (acceptable; no glyph because button text+disabled cursor pair is the convention) |
## Build / verification
This codebase has no build step — `index.html` loads `sortof-app.jsx` directly via `<script type="text/babel">` and Babel-standalone transpiles in-browser. No npm, no Vite, no bundler. Verification approach:
- `curl http://100.114.205.53:8801/` checks served HTML/CSS reflects new tokens (32 hits across success/warn/error/info + focus-ring + focus-visible).
- `curl http://100.114.205.53:8801/sortof-app.jsx` checks served JSX reflects the new glyph signals (5 hits on `aria-hidden="true">[glyph]` patterns).
- Public mirror (Caddy + Tailscale) sees the same content.
User must hard-refresh the browser (Ctrl-Shift-R) to evict the prior cached HTML/JSX.
No backend services were touched.
## Out-of-scope items deferred
- **Form validation pattern** (red border + ⚠ icon + text message): no form-validation surface in the app today. Spec'd in the audit doc; will land alongside any future form (e.g., admin-curated precacher list, settings panel).
- **Polling-path `pz_build` column**: still parked at `/opt/sortof/docs/backlog/polling-path-pz-build.md`. Unrelated to a11y.
- **Tweaks panel** (`tweaks-panel.jsx`): dev-only, gated behind `?tweaks=1`. Skipped this pass — not user-facing.
## Backups
- `/opt/sortof/frontend/index.html.bak-20260502-...-a11y-full`
- `/opt/sortof/frontend/sortof-app.jsx.bak-20260502-...-a11y-full`
Plus prior `.bak-...-a11y` and `.bak-...-emdash` siblings still in place. Working tree is dirty per directive — no commits, no auto-cleanup.

View File

@@ -1,54 +0,0 @@
# Backlog: polling-path `pz_build` plumbing
**Status:** parked. File a Gitea issue with this content when one comes up.
**Trigger to schedule:** a B41 user submits a collection URL or bare-uncached input AND complains about getting B42-flavored auto-picks (or audits the result and notices Rule A misfired).
## What's missing
`/api/sort`'s sync path passes `req.pz_build` into `adapters.build_response` correctly. The async path (`_route_to_job``expansion.run_expansion``_build_result_for_job`) does **not** persist `pz_build` and defaults to `"B42"` when the GET endpoint builds the final result.
A B41 user submitting a collection URL gets:
- Sync part of `/api/sort` (validation, classify) sees `pz_build=B41`.
- Job created, expansion runs.
- GET `/api/jobs/{id}` builds `result_json` via `_build_result_for_job(conn, wsids, rules_raw)``adapters.build_response(...)` → defaults `pz_build="B42"` → Rule A picks B42-flavored branches.
For the canonical fhqMotoriusZone/SZ-class fixtures this doesn't matter (those are coordinated, exempt from Rule A). The bug bites on truly-ambiguous multi-branch wsids like `zReApoModernArmor` (2 branches: unflavored + B42-flavored) when B41 user routes through cold drain.
## Migration sketch
Add a column to `sort_jobs`:
```sql
-- /opt/sortof/init/02_sort_jobs.sql (re-run on idempotent CREATE; live DB needs ALTER)
ALTER TABLE sort_jobs ADD COLUMN IF NOT EXISTS pz_build TEXT;
```
Apply via:
```bash
sudo docker exec -i sortof_db psql -U sortof -d sortof -c \
"ALTER TABLE sort_jobs ADD COLUMN IF NOT EXISTS pz_build TEXT;"
```
Edit `init/02_sort_jobs.sql` to include the column in the `CREATE TABLE IF NOT EXISTS` block so fresh deploys get it.
## Plumbing checklist
1. **`/opt/sortof/api/jobs.py`** — `create_job(...)` gains `pz_build: Optional[str] = None`; INSERT writes the column. `get_job_row` returns it (no code change needed if `SELECT *`).
2. **`/opt/sortof/api/app.py`** — `_route_to_job(...)` gains `pz_build: Optional[str] = None`; passes to `jobs.create_job`. Both call sites in `sort_endpoint` (collection short-circuit and bare-uncached fork) pass `req.pz_build`.
3. **`/opt/sortof/api/app.py`** — `_build_result_for_job(conn, wsids, rules_raw)` signature gains `pz_build: Optional[str] = None`. The GET handler reads `row["pz_build"]` and passes it through. `adapters.build_response(... pz_build=...)` already accepts it.
4. **No frontend change**`pzBuild` is already in `/api/sort` POST body.
## Acceptance criteria
- [ ] B41 user submits collection URL containing `zReApoModernArmor` (3483407987) — final `result_json.MODS_LINE` includes `zReApoModernArmor` (un-flavored), not `zReApoModernArmorB42`.
- [ ] WARNINGS includes `build-mismatch` if appropriate.
- [ ] B42 user behavior unchanged (default).
- [ ] Existing `sort_jobs` rows with NULL `pz_build` continue to work (NULL → fall back to "B42" in the build_response default).
## Why parked
- No telemetry showing cold-collection B41 traffic exists.
- DB migrations against synthetic demand bitrot between writing and shipping (function signatures drift, the DDL goes stale).
- Sync-path B41 users (the dominant case in this user base) work correctly today — Rule A fires off `req.pz_build` directly.
- Schedule when a real user hits it. The migration is small and self-contained.

View File

@@ -1,122 +0,0 @@
# Indifferent Broccoli brand application — changelog 2026-05-01
Frontend-only. No backend edits. No DB migrations. No spec changes. No git commits — working tree dirty for review.
Layered on top of the in-flight a11y refactor (`/opt/sortof/docs/a11y-audit-2026-05-01.md` + `/opt/sortof/docs/a11y-changes-2026-05-01.md`). Where brand and a11y collide, a11y won — see "Deferred to a11y" below.
## Files touched
- `/opt/sortof/frontend/index.html` — brand tokens, font-link expansion, favicon, panel shadow, sort-btn rebrand, wordmark/header CSS, footer mark CSS
- `/opt/sortof/frontend/sortof-app.jsx` — Header swap (broccoli image + IB link), Footer rewrite ((:|) glyph + IB link), voice copy on EmptyRight + StatusStrip terminal text
- `/opt/sortof/frontend/img/broccoli_shadow_square.png`**new file**, 19075 bytes, fetched from `https://indifferentbroccoli.com/img/broccoli_shadow_square.png`
## Brand tokens added (CSS vars)
```css
--brand-primary: #5EFF0D; /* IB anchor green */
--brand-primary-rgb: 94 255 13;
--brand-primary-bg: rgb(var(--brand-primary-rgb) / 0.14);
--brand-anchor-bg: #0A141E; /* IB navy */
--brand-shadow-card: 0 4px 12px rgba(0, 0, 0, 0.3); /* IB card lift */
--display: 'Sora', 'Geist', ui-sans-serif, system-ui, ...;
--sans: 'Open Sans', 'Geist', ui-sans-serif, system-ui, ...;
--radius-sm: 4px; /* IB rounded */
--radius: 8px; /* IB rounded-lg (was already this value) */
--radius-lg: 12px; /* IB rounded-xl */
```
Existing `--mono` (JetBrains Mono) preserved — IB doesn't define a monospace font. Status colors (`--acc-success/warn/error/info`) and foreground ramp (`--fg/-1/-2/-3`) untouched per a11y deference.
## File-by-file
### `/opt/sortof/frontend/index.html`
| Where | Before | After |
|---|---|---|
| `<title>` | "sortof - sorted. sort of." | **"sortof (:|) sorted. or close enough."** |
| `<head>` | _no favicon_ | `<link rel="icon" type="image/png" href="/img/broccoli_shadow_square.png">` + apple-touch-icon |
| Font link | Geist + JetBrains Mono | **+ Sora 400/500/600/700 + Open Sans 400/500/600/700**, all in one `<link>` |
| `:root` vars | a11y palette only | + `--brand-primary`, `--brand-primary-bg`, `--brand-anchor-bg`, `--brand-shadow-card`, `--radius-sm/lg`, `--display` |
| `--sans` | `'Geist', ui-sans-serif, ...` | `'Open Sans', 'Geist', ui-sans-serif, ...` |
| `.wordmark` | `font-family: var(--mono); font-size: 17px; weight 600` | `font-family: var(--display); font-size: 19px; weight 700` |
| `.wordmark .dot` | `color: var(--acc-green)` | `color: var(--brand-primary)` |
| `.brand-mark` | _did not exist_ | 28×28 circle, lifts on hover |
| `.brand-mark-link` | _did not exist_ | wrapper anchor with rotation hover |
| `.ib-mark` | _did not exist_ | small mono `(:|)` glyph in IB green |
| `.sort-btn` | `--acc-green` border/bg/text, mono font | `--brand-primary` border/bg/text, **Sora display font, height 42px (was 40)** |
| `.sort-btn:hover` | tinted green | **fills with brand-primary, inverts text to anchor-bg, applies card shadow** |
| `.panel` | flat | `box-shadow: var(--brand-shadow-card)` |
### `/opt/sortof/frontend/sortof-app.jsx`
| Component | Before | After |
|---|---|---|
| `<Header>` | wordmark + tagline only | `<a href=ib.com><img broccoli/></a>` + wordmark + tagline |
| `<Footer>` | "a thing by [indifferent broccoli]" no link | `<a href=ib.com>indifferent broccoli (:|)</a>` (real link, IB green glyph) |
| `<EmptyRight variant="bare">` | "paste workshop ids on the left, then hit sort." | **"no mods. or maybe loads of them. hard to tell."** + sub-line "paste workshop ids on the left, hit sort. output lands here." |
| StatusStrip · idle | "ready when you are" | **"ready when you are. or not."** |
| StatusStrip · success/done | "done. N mods, W warnings" | **"sorted. N mods, W warnings. or close enough."** |
| StatusStrip · error | "something went sideways" | **"something went sideways. that happens."** |
| StatusStrip · failed | "job failed" | **"that didn't work. try again or don't."** |
| StatusStrip · cold | "cache miss - be patient" | **"cache miss. take your time, no rush."** |
Functional info preserved verbatim in every changed string (counts, role labels, action prompts). Voice flavor is additive.
## Deferred to a11y (brand wanted this; a11y said no)
Per the brief: _"If brand application would override an a11y decision, defer to a11y."_
| Brand wanted | A11y holds | Resolution |
|---|---|---|
| Brand green (#5EFF0D, hue 138) as the success-state color | `--acc-success` at hue 165 (Okabe-Ito bluish-green) is more deuteranopia-safe; the hue-138 green clusters too close to amber under simulated CVD | Brand green stays as `--brand-primary` (CTA only). Status pills, copy-btn-success, branch-picker-picked, diff-stat-add all keep `--acc-success`. |
| IB blue (#0050FF) for links | `--acc-info` (oklch 0.78 0.13 230) clears AAA against dark bg; #0050FF would be 3.0:1 (AA fail) | Documented IB blue for completeness; not adopted. Links continue to use `--acc-info`. |
| Brand-only focus ring (green) | Global `:focus-visible` uses `--focus-ring = var(--acc-info)` for consistent keyboard signal across all interactive elements | Kept the a11y blue focus ring. Brand green appears as button fill/border, not as focus indicator. |
| Glyph removal in favor of pure brand-color states | A11y requires every state pill to carry (color × glyph × text) | Glyphs preserved. Brand only changed the *anchor* color (sort-btn, wordmark dot, sort-btn:hover invert), never the *state* signal layer. |
## Voice contract (locked)
All voice flavor is **additive** — never strips functional information. The pattern from IB's hero ("Host your own game server / Or not... we don't care") is: bold functional claim, then immediate self-undercut.
Applied as: `<existing functional text>. <reverse-pleasantness flourish>`
| Site | Functional info preserved | Flourish appended |
|---|---|---|
| idle pill | (none — placeholder) | "or not." |
| done pill | "sorted. N mods, W warnings" | "or close enough." |
| error pill | "something went sideways" | "that happens." |
| failed pill | (replaced) | "that didn't work. try again or don't." |
| cold pill | "cache miss" | "take your time, no rush." |
| empty-bare big | (replaced) | "no mods. or maybe loads of them. hard to tell." |
The `<title>` follows the same pattern with the `(:|)` glyph as separator.
## Verification
This codebase has no build step (Babel-standalone transpiles JSX in-browser). Verification is via curl + visual inspection.
**Served file checks:**
- 22 hits for brand token names (`brand-primary`, `brand-anchor`, `brand-shadow`, `brand-mark`, `Sora`, `Open+Sans`) in served HTML
- 4 hits for new voice copy (`or close enough`, `or not.`, `try again or don.t`, `hard to tell`) in served JSX
- 2 hits for header brand markup (`broccoli_shadow_square.png`, `className="ib-mark"`)
- Public mirror serves `/img/broccoli_shadow_square.png` HTTP 200
- Public mirror `<title>` reads `sortof (:|) sorted. or close enough.`
**Contrast verification (computed via oklch L deltas; AA = ΔL ≥0.42; AAA = ΔL ≥0.55):**
| Pair | ΔL | Approx ratio | WCAG |
|---|---|---|---|
| `--brand-primary` (L 0.88) vs `--bg-1` (L 0.21) | 0.67 | ~10:1 | **AAA** |
| `--brand-primary` vs `--brand-anchor-bg` (L 0.20) — sort-btn:hover inverted | 0.68 | ~11:1 | **AAA** |
| `--acc-info` (L 0.78) vs `--bg-1` — link/button text | 0.57 | ~7.3:1 | **AAA** |
| `--fg` (L 0.95) vs `--bg-1` — display headings, body | 0.74 | ~14:1 | **AAA** |
| `--fg-2` (L 0.72) vs `--bg-1` — secondary text (lifted in a11y pass) | 0.51 | ~5.7:1 | **AA** |
No brand color required luminance adjustment. The IB blue (`#0050FF`) was the only candidate that would have failed; it was rejected in palette reconciliation rather than lifted.
## Backups (working-tree dirty)
- `/opt/sortof/frontend/index.html.bak-...-brand`
- `/opt/sortof/frontend/sortof-app.jsx.bak-...-brand`
- Plus a11y siblings (`-a11y-full`, `-a11y`) and earlier session backups.
User must hard-refresh the browser to evict the prior cached HTML/JSX/CSS.

View File

@@ -1,145 +0,0 @@
# Indifferent Broccoli brand tokens — extracted 2026-05-01
Source pages fetched live via `curl`:
- HTML: `https://indifferentbroccoli.com/` (100,017 bytes)
- CSS: `https://indifferentbroccoli.com/css/output.css` (62,243 bytes — Tailwind-compiled)
The site does not publish authored CSS variables for brand colors (it's Tailwind output with a few brand hex literals overlaid). The tokens below are reverse-engineered from the compiled stylesheet by frequency and semantic role.
## 1. Colors (raw values from `output.css`)
| Token | Hex | oklch (approx) | Usage in IB CSS | Lines |
|---|---|---|---|---|
| **brand primary green** | `#5EFF0D` | oklch(0.88 0.27 138) | `.stroke-primary { stroke: #5EFF0D }`, `.decoration-primary { text-decoration-color: #5EFF0D }`, file-selector-button border | output.css:1674, 2090, 2233-2234, 2730 |
| brand secondary blue | `#0050FF` | oklch(0.43 0.27 264) | text/link color appearances | 4 occurrences |
| dark surface | `#0A141E` | oklch(0.20 0.025 240) | navy panel background | 1 occurrence |
| accent pink | `#c0a0b9` | oklch(0.74 0.05 340) | one section bg | 1 occurrence |
| white | `#fff` | — | text on dark, panels on light | 6 occurrences |
| neutral text | `#6b7280` (tw-gray-500) | — | muted body text | 4 occurrences |
| neutral muted | `#9ca3af` (tw-gray-400) | — | further-muted | 2 occurrences |
## 2. Fonts
`<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Sora|Sora:600">`
`<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans|Open+Sans:600">`
| Role | Family | Weights | Source |
|---|---|---|---|
| heading / display | **Sora** | 400, 600 | Google Fonts (CDN) |
| body / paragraph | **Open Sans** | 400, 600 | Google Fonts (CDN) |
| icons | Font Awesome | — | `/vendor/font-awesome/css/font-awesome.min.css` (out of scope for sortof) |
The site does not self-host these via @fontsource. We will hot-link the same Google Fonts URLs as IB to inherit identically. Mono font is unspecified by IB; sortof keeps its existing **JetBrains Mono** for code/output blocks (no brand conflict).
## 3. Border radii (compiled Tailwind)
| Value | Frequency | Likely role |
|---|---|---|
| `0.25rem` (4px) | 2 | small corners (rounded) |
| `0.5rem` (8px) | 1 | medium (rounded-lg) |
| `0.75rem` (12px) | 1 | larger (rounded-xl) |
| `9999px` | 1 | pill / chip |
| `100%` | 1 | circle (logo container) |
| `0` / `0px` | 3 | flat sections |
## 4. Shadows
```css
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3); /* card lift */
box-shadow: 0 0 0 3px rgba(34, 197, 94, 0.1); /* focus ring (green-tinted) */
```
## 5. Brand assets
| Asset | URL | Local copy |
|---|---|---|
| primary mark | `https://indifferentbroccoli.com/img/broccoli_shadow_square.png` | `/opt/sortof/frontend/img/broccoli_shadow_square.png` (866×866 PNG, 19,075 bytes) |
| smiley wordmark | `(:|)` (text glyph) — appears in `<title>` ("Indifferent Broccoli (:|)") and footer-class flourish |
Pulled the broccoli image locally so we don't hot-link IB's CDN. Same file used for both header brand mark and favicon.
## 6. Voice cues (verbatim from page)
The deadpan house style — additive, not replacement, when applied to sortof:
- **Hero**: "Host your own game server / Or not... we don't care"
- **Sub-hero**: "Premium game servers with instant deployment, full mod support, and a control panel so simple your cat could use it."
- **Trim**: "Or not."
- **Title** (browser tab): "Indifferent Broccoli (:|)"
Keying off these, the IB voice contract is: bold functional claim, then immediate self-undercut. Reverse-pleasantness. No exclamation points. Lowercase or sentence-case (no shouting).
## 7. Header layout (extracted)
```
[broccoli_shadow_square.png] [indifferent broccoli wordmark]
[Games | Top Servers | About Us | Contact | Wiki | Merch | Open Source]
[Start New Server +] [Log in]
```
For sortof, the structural mapping:
```
[broccoli_shadow_square.png] [sortof] [tagline] [github] [docs]
```
Footer references stay (REfRigERatoR's mod load order sorter / "a thing by indifferent broccoli") but get the `(:|)` glyph as a small inline mark next to "indifferent broccoli", linking to `https://indifferentbroccoli.com`.
## 8. Final palette decision (reconciliation with a11y)
Per brief: **IB primary green is the brand anchor; status colors defer to the a11y pass.**
| Role | Token | Value | Source |
|---|---|---|---|
| Brand primary (sortof = an IB thing) | `--brand-primary` | `#5EFF0D` | IB literal |
| Brand surface (alt anchor) | `--brand-anchor-bg` | `#0A141E` | IB literal |
| Status: success | `--acc-success` | oklch(0.78 0.13 165) | a11y (Okabe-Ito bluish-green). Unchanged. |
| Status: warning | `--acc-warn` | oklch(0.82 0.15 75) | a11y (Okabe-Ito orange-yellow). Unchanged. |
| Status: error | `--acc-error` | oklch(0.70 0.18 35) | a11y (Okabe-Ito vermillion). Unchanged. |
| Status: info / link | `--acc-info` | oklch(0.78 0.13 230) | a11y (Okabe-Ito sky blue). Unchanged. |
| Foreground ramp | `--fg`, `--fg-1`, `--fg-2`, `--fg-3` | a11y values | Unchanged. |
| Backgrounds | `--bg`, `--bg-1`, `--bg-2`, `--bg-3` | a11y values | Unchanged. |
### Why brand green doesn't replace `--acc-success`
The IB green `#5EFF0D` is **highly saturated** (oklch chroma 0.27) and reads as "this is a CTA / this is the IB look." Using it as the success-state color would:
- Conflict with the a11y reasoning: success at hue 138 (yellow-green) sits closer to Okabe-Ito's *yellow* than its *bluish-green*. Less safe under deuteranopia.
- Conflict with the spec semantics: in the a11y system, brand-anchor color and status-success color have different jobs. Swapping them would re-collapse what the a11y pass deliberately separated.
So the brand green lives at `--brand-primary` and is used for:
- Sort button (the primary CTA on the page)
- Wordmark dot accent
- Header brand mark hover/focus emphasis
- Build-toggle "active" state
Status pills, banners, warnings, copy-button-success, etc. continue to use the a11y `--acc-*` tokens.
### Contrast verification for `--brand-primary` (`#5EFF0D`)
Against the dark canvas backgrounds:
| Pair | ΔL (oklch) | Approx ratio | WCAG |
|---|---|---|---|
| `#5EFF0D` (L=0.88) vs `--bg` (L=0.18) | 0.70 | ~12.5:1 | AAA |
| `#5EFF0D` (L=0.88) vs `--bg-1` (L=0.21) | 0.67 | ~10:1 | AAA |
| `#5EFF0D` (L=0.88) vs `--bg-3` (L=0.28) | 0.60 | ~7.5:1 | AAA |
Brand green clears AAA against every background token. No luminance adjustment needed.
### IB blue `#0050FF` — not used
`#0050FF` ≈ oklch(0.43 0.27 264) → fails AA against any of our dark backgrounds (~3.0:1). The a11y pass already established `--acc-info` (oklch 0.78 0.13 230) as the link / info color and that token clears AAA. We document IB's blue here for completeness but do not put it in the live palette.
## 9. Tokens mapped to CSS vars (added in this round)
```css
--brand-primary: #5EFF0D; /* IB anchor green */
--brand-primary-rgb: 94 255 13; /* for rgba() composition */
--brand-anchor-bg: #0A141E; /* IB navy, available as deep panel surface */
--brand-radius: 0.5rem; /* IB medium radius */
--brand-radius-sm: 0.25rem; /* IB small radius */
--brand-shadow-card: 0 4px 12px rgba(0, 0, 0, 0.3); /* IB card lift */
--brand-mark: url('/img/broccoli_shadow_square.png'); /* logo asset */
--brand-font-display: 'Sora', 'Geist', ui-sans-serif, system-ui, sans-serif;
--brand-font-body: 'Open Sans', 'Geist', ui-sans-serif, system-ui, sans-serif;
```

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -1,343 +0,0 @@
# Plan: pzmm conflict detection + content-type categorization
**Date:** 2026-05-04
**Branch:** `feat/pzmm-conflict-typing`
**Status:** Approved (Sam, 2026-05-04)
**Sources read:**
- `/tmp/pzmm-src/pzmm-main/core/scanner.py``scan_file_conflicts`, `solve_load_order`, `FileConflict`
- `/tmp/pzmm-src/pzmm-main/core/mods.py``detect_mod_types`, `ModInfo`
- `/tmp/pzmm-src/pzmm-main/core/bundle.py` — debug bundle (read for context, not integrated)
- `/opt/sortof/init/01_schema.sql` and migrations 02..08
- `/opt/sortof/api/app.py``/api/sort`, `_build_result_for_job`, `_row_to_modinfo`
- `/opt/sortof/api/mlos_sort.py``CATEGORY_ORDER`, `derive_category`
- `/opt/sortof/api/adapters.py``CAT_MAP`
- `/opt/sortof/worker/worker.py``process_one`
**Open questions resolved at approval:**
- Manifest scope: walk all `media/` subtrees under the mod_id root, last-wins on duplicate rel_paths, **no per-branch column**.
- `mod_files.size_bytes` column: keep.
- Module split: `api/diagnostics.py` and `api/categorize.py` are **separate files**.
- `/api/conflicts` v1: **bare wsids only**, return HTTP 400 on collection input. Defer async-job/collection-expansion plumbing to a follow-up plan.
---
## 1. Context
pzmm ships two pieces sortof doesn't have today:
1. **File-conflict detection** — when two mods both ship `media/scripts/items_food.txt` with byte-different content, the later one silently overrides the earlier one at runtime. PZ never reports this; the player only sees the symptom (broken food, duplicate item ids, etc.). pzmm walks each mod's `media/` tree, hashes the conflict-prone extensions (`.lua`, `.txt`, `.xml`, `.json`, `.ini`), and reports rel-paths claimed by ≥2 mods with non-equal content. Sortof currently only detects `mod_id` collisions (one mod_id under multiple wsids). File-level overrides are invisible to us.
2. **Content-type detection** — pzmm walks `media/` paths plus the contents of `lua/` and `scripts/*.txt|xml` files to fingerprint what a mod actually ships (Weapons, Vehicles, Maps, Traits, Professions, Recipes, etc.). Sortof's `derive_category` infers category from `workshop_meta.tags` + name regex + `mod.info` tags. Authors who tag poorly (or skip tagging) end up in `other`/`undefined`. Detection from media/ contents is more reliable for those.
Both pzmm functions assume on-disk media trees. Sortof's worker uses `tempfile.TemporaryDirectory` (`worker/worker.py:472`) — the entire DD extraction is destroyed at the end of `process_one`'s `with` block. **Only `mod.info` (as `raw_mod_info`), discovered map folder names, and a few derived columns persist.**
This plan keeps the existing model: parse once, serve from DB. We **persist a manifest at parse time**. Re-fetch on demand was rejected — every conflict check would queue N DD pulls, minutes per request, completely unusable.
We **do not import pzmm's `solve_load_order`**. Sortof's `mlos_sort.py` is strictly more correct (preorder, loadFirst/loadLast tiers, category buckets, patch G-axis, multi-branch picker, addon injection). pzmm's solver is a plain Kahn topo sort with no tie-breakers.
---
## 2. Integration A — File conflict detection
### 2.1 New schema (`init/09_mod_files.sql`)
```sql
CREATE TABLE IF NOT EXISTS mod_files (
workshop_id TEXT NOT NULL,
mod_id TEXT NOT NULL,
rel_path TEXT NOT NULL, -- lowercased, posix-style, relative to mod_id root
sha1 TEXT NOT NULL,
size_bytes INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (workshop_id, mod_id, rel_path),
FOREIGN KEY (workshop_id, mod_id) REFERENCES mod_parsed (workshop_id, mod_id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS mod_files_rel_path_idx ON mod_files (rel_path);
CREATE INDEX IF NOT EXISTS mod_files_mod_idx ON mod_files (workshop_id, mod_id);
```
Plus additions to `mod_parsed`:
```sql
ALTER TABLE mod_parsed
ADD COLUMN IF NOT EXISTS mod_types TEXT[] NOT NULL DEFAULT '{}',
ADD COLUMN IF NOT EXISTS files_manifest_built BOOLEAN NOT NULL DEFAULT FALSE;
```
The flag lets `derive_category` and `/api/conflicts` know whether a mod has a manifest yet (graceful degradation while the cache backfills organically).
### 2.2 Worker changes (`worker/worker.py`)
In `process_one`, **inside the existing `with tempfile.TemporaryDirectory` block** (after `discover_mod_infos`, before the `with` exits):
**Single-pass requirement:** the manifest build (Integration A) and `detect_mod_types` content sniffing (Integration B) **share one pass over the tempdir**. No two-pass implementations. The walk reads each file's bytes once: hash → manifest insert; concurrently inspect path + content for type signals. The output is the `mod_files` rows for that mod_id and the ordered `mod_types` list, both committed in the same transaction as the existing `UPSERT_MOD_PARSED`.
For each `(workshop_id, mod_id)` pair we just upserted:
1. Compute `mod_id_root`: the directory whose name equals `mod.id`. For B41 (`mods/<modId>/mod.info`) that's `mip.parent`; for B42 (`mods/<modId>/<branch>/mod.info`) that's `mip.parent.parent`. Detect via `mip.parent.name == mod.id`.
2. Single recursive walk under `mod_id_root` covering every `media/` subtree (handles B42 `<branch>/media/` + `common/media/` together). For each file:
- If suffix matches `_CONFLICT_EXTS = {".lua", ".txt", ".xml", ".json", ".ini"}` (verbatim from pzmm `scanner.py:21`), compute sha1 (chunked reader, mirrors pzmm `_sha1`) and accumulate `(rel_path, sha1, size_bytes)`. **Last-wins** on duplicate rel_paths across branches.
- Concurrently, in the same loop, accumulate the path-based signals from pzmm `mods.py:detect_mod_types` (lines 88115): `Maps`, `Tiles`, `Textures`, `Vehicles`, `Clothing`, `Sounds`, `UI`, `Animations`, `Translations`, `Lua`, plus collected `lua_text_parts` and `script_text_parts` blobs (capped at 60 lua × 64 KB and 80 script × 96 KB per pzmm).
3. After the walk, run pzmm's content-blob checks (lines 117136): weapon/vehicle/item/recipe/clothing/trait/profession signals from concatenated blobs. Resolve to `mod_types` ordered list (lines 138145).
4. DELETE existing `mod_files` rows for `(workshop_id, mod_id)` then bulk INSERT new rows.
5. UPSERT `mod_parsed.mod_types` and set `files_manifest_built = true` for the row.
The whole step adds disk-walk + hashing of small text files only — typical mod has 20200 files in scope, hashing is cheap (≤100 KB each, sha1 ≈ 500 MB/s). Estimated cost: <500 ms per mod, well under the DD pull cost we're already paying.
### 2.3 New module: `api/diagnostics.py`
Port of pzmm `scan_file_conflicts` adapted to read from `mod_files` instead of walking disk:
```python
async def scan_file_conflicts(conn, mods: list[ModInfo]) -> list[FileConflict]:
"""For the given (already-loaded) ModInfos, report rel_paths claimed
by ≥2 mods with non-equal sha1. Returns list ordered by rel_path."""
```
Implementation:
1. `SELECT workshop_id, mod_id, rel_path, sha1 FROM mod_files WHERE (workshop_id, mod_id) IN (...)`.
2. Group rows in Python by `rel_path`.
3. For each group with ≥2 distinct mods, count distinct sha1s. If >1, emit a `FileConflict`.
4. Winner = last in input order (mirrors pzmm's "last in load order wins").
Dataclass:
```python
@dataclass
class FileConflict:
rel_path: str
providers: list[str] # mod_ids (not ModInfo, to keep payload small)
winner: str # mod_id
```
`pzmm.scanner._CONFLICT_EXTS` filtering happened at manifest-build time, so this read path doesn't need it.
### 2.4 New endpoint: `POST /api/conflicts`
Same input shape as `/api/sort`, **bare wsids only** (Q4 resolved):
```json
{"input": "wsid1;wsid2;wsid3", "rules": "...", "pz_build": "B42"}
```
If `parse_with_collections` returns any `collection_ids`, return HTTP 400 with `detail="conflict scan does not support collection input; resolve via /api/sort first"`.
Response:
```json
{
"conflicts": [
{"rel_path": "media/scripts/items_food.txt",
"providers": ["FoodModA", "FoodModB"],
"winner": "FoodModB"}
],
"missing_manifests": ["wsid1", "wsid2"]
}
```
`missing_manifests` lists mods we couldn't analyze because `files_manifest_built=false`. The frontend can show a banner ("X mods haven't been re-fetched since this feature shipped — file conflicts unavailable for them"), and re-clicking sort eventually triggers re-parse on workshop updates.
Reuse path: `_build_result_for_job` already loads ModInfos via `_row_to_modinfo` — the conflicts endpoint follows the same load pattern, then calls `scan_file_conflicts(conn, mods)` instead of `sort_mods`.
### 2.5 Frontend (out of scope for this plan)
A follow-up plan can wire a "File conflicts" warnings section. For now `/api/conflicts` is consumable from curl and lays the groundwork.
---
## 3. Integration B — Content-type detection feeding category derivation
### 3.1 Schema additions
Already covered by §2.1's `mod_parsed` ALTER TABLE (`mod_types` + `files_manifest_built`). One migration file (`init/09_mod_files.sql`) ships both A and B because they share the worker walk.
### 3.2 Worker changes
Folded into §2.2's single-pass walk. No additional file I/O.
### 3.3 New module: `api/categorize.py`
```python
def types_to_category(mod_types: list[str], name: str) -> str | None:
"""First mod_type that maps to a sortof CATEGORY_ORDER bucket wins.
Returns None if mod_types is empty / Unknown / Dependency-only and we
should fall through to the existing derive_category cascade."""
```
### 3.4 Tag→category mapping (explicit)
| pzmm `mod_type` | sortof `CATEGORY_ORDER` | notes |
|---|---|---|
| `Maps` | `map` | already covered by `mod.maps non-empty`; types-derived is a fallback |
| `Vehicles` | `vehicle` | name regex `"spawn zone"` already routes to `vehicle_spawn` upstream |
| `Weapons` | `weapon` | wins over `Items` (pzmm prefers list ordering) |
| `Items` | *skip* | too generic — almost every mod has Items; would mis-trigger |
| `Clothing` | `wearable` | armor name-hint check still runs after, can override to `armor` |
| `Traits` | `code` | no dedicated `trait` bucket; `code` is the gameplay-axis fallback |
| `Professions` | `profession` | |
| `Recipes` | `crafting` | |
| `Tiles` | `tile` | |
| `Textures` | `texture` | |
| `Sounds` | `sound` | already handled by `Audio` ws_tag; types-derived is a fallback |
| `Animations` | *skip* | no bucket; falls through |
| `UI` | `ui` | |
| `Translations` | `translation` | |
| `Lua` | *skip* | too generic; falls through |
| `Patch` | `patch` | already detected by `_PATCH_NAME_RE`; types-derived is a fallback |
| `Dependency` | `tweaks` | maps to existing `lib` pill |
| `Framework` | `tweaks` | same |
| `Unknown` | *skip* | falls through |
"*skip*" means: don't return a category; let `derive_category` continue its cascade.
### 3.5 `derive_category` integration
Insert a single new check in `api/mlos_sort.py:derive_category` after the explicit-category early return at line 412, **before** the patch/lib name regex at lines 416419:
```python
if mod.mod_types:
cat = types_to_category(mod.mod_types, name)
if cat:
return cat
```
`mod.mod_types` is added to the `ModInfo` dataclass (`mlos_sort.py:113`). `_row_to_modinfo` (`api/app.py:176`) is updated to read the new column. **Both `mlos_sort.py` copies must change in lockstep.**
**Position rationale:** `mod_types` comes from media-content fingerprinting, more reliable than name regex but less reliable than an explicit `category=` field in `mod.info`. So it sits between (1) explicit category and (2) name regex. The patch/lib regexes that come after still win for true patches/libraries (they'd usually return `Patch`/`Dependency` from detect_mod_types anyway, but we want the regex to win for cases where a "patch mod" hasn't shipped enough media to fingerprint).
Empty `mod_types` (e.g. older rows where `files_manifest_built=false`) means the new check returns `None` and the existing cascade runs unchanged. **Graceful degradation is built in.**
---
## 4. Blockers / risks
### 4.1 Schema migration cost
- Current cache: **3,123 `workshop_meta` rows, 3,298 `mod_parsed` rows**.
- New `mod_files` rows estimate: median mod ships ~50 conflict-eligible files (light mods 510, heavy framework/map mods 200500). At 50 avg × 3,298 mods = **~165 k rows**. With sha1 (40 chars) + rel_path (avg 80 chars) + overhead ≈ 200 bytes/row, that's ~33 MB before indexes. Postgres handles this trivially.
- `ALTER TABLE mod_parsed ADD COLUMN mod_types TEXT[]` and `files_manifest_built BOOLEAN` are additive and metadata-only on Postgres 16 (no rewrite). Instant.
### 4.2 Backfill feasibility
- The `/tmp/sortof_steam_throttle` flock + `/tmp/sortof_steam_cooldown` 1h kill-switch (worker.py — `fetch_required_wsids`) protect us from Steam metadata 429s. **DD itself does not hit the metadata API**; it hits Steam content servers, which are not part of the rate-limited path. So mass re-DD does not trip the cooldown.
- Mass re-DD still costs real time: typical DD pull is 2060 s wall-clock. 3,123 wsids × 30 s avg ÷ 4 drains = **~6.5 hours wall-clock for a full backfill**. Doable but disruptive.
- **Recommendation: do not run a bulk backfill.** Let the cache populate organically — every workshop update bumps `time_updated`, which triggers a re-parse and now also a manifest build. The `missing_manifests` field in `/api/conflicts` and the empty-`mod_types` graceful-degrade path together mean the feature works on day 1 (empty results for old rows) and improves as authors push updates.
- Per-mod manual trigger pattern still works (operator-only):
```sql
DELETE FROM mod_parsed WHERE workshop_id='<wsid>';
INSERT INTO download_jobs (workshop_id, status) VALUES ('<wsid>','queued');
```
### 4.3 Inline detection at sort time
- Rejected. `detect_mod_types` reads up to ~11 MB per mod from disk (lua/script blobs). With the tempdir destroyed (the actual case), we'd need to re-DD inline — minutes per sort.
- **All detection runs at parse time** in `process_one`. `derive_category` and `/api/conflicts` are pure DB reads.
---
## 5. Files touched (summary)
**New:**
- `init/09_mod_files.sql` — `mod_files` table, `mod_parsed.mod_types`, `mod_parsed.files_manifest_built`
- `api/diagnostics.py` — port of `scan_file_conflicts`, `FileConflict` dataclass
- `api/categorize.py` — `types_to_category` helper
**Modified:**
- `worker/worker.py` — extend `process_one`'s `with` block: single-pass walk, manifest + detect_mod_types, upsert rows
- `worker/worker.py` (top-level) — port `detect_mod_types` from pzmm `mods.py:57145` (sortof-side copy; do not import from pzmm at runtime)
- `api/mlos_sort.py` — add `mod_types: List[str]` to `ModInfo` dataclass; add `mod_types` check at top of `derive_category`
- `worker/mlos_sort.py` — mirror the `ModInfo` and `derive_category` change (worker/api dual-edit rule)
- `api/app.py` — `_row_to_modinfo` reads new `mod_types` column; `_build_result_for_job` SELECT list adds `mp.mod_types`; register `POST /api/conflicts`
**Out of scope (deferred to follow-up plan):**
- Frontend conflicts panel — `/api/conflicts` endpoint only, no UI
- Integration of `pzmm/core/bundle.py` (debug bundle export) — read for context, not ported
- Backfill orchestration — relying on organic backfill
---
## 6. Rollback
Before applying the migration:
```bash
# Backup mod_parsed (the only existing table we ALTER)
sudo docker exec -i sortof_db pg_dump -U sortof -d sortof -t mod_parsed \
> /opt/sortof/backups/mod_parsed-pre-09.sql.$(date +%Y%m%d-%H%M)
ls -la /opt/sortof/backups/ | tail -3
```
Down SQL (paste into psql to revert the schema half of this plan):
```sql
DROP TABLE IF EXISTS mod_files;
ALTER TABLE mod_parsed
DROP COLUMN IF EXISTS mod_types,
DROP COLUMN IF EXISTS files_manifest_built;
```
To revert code, `git checkout main` and restart services:
```bash
sudo systemctl restart sortof-api sortof-drain@1 sortof-drain@2 sortof-drain@3 sortof-drain@4
```
The migration is additive only (new table + new columns with safe defaults), so the rollback is a clean drop. No data is destroyed in `mod_parsed`'s existing columns.
---
## 7. Verification
1. **Migration applies cleanly:**
```bash
sudo docker exec -i sortof_db psql -U sortof -d sortof < /opt/sortof/init/09_mod_files.sql
sudo docker exec -i sortof_db psql -U sortof -d sortof -c "\d mod_files"
sudo docker exec -i sortof_db psql -U sortof -d sortof -c "\d mod_parsed" | grep -E "mod_types|files_manifest_built"
```
2. **Compile checks** (after every Python edit):
```bash
/opt/sortof/api/.venv/bin/python -m py_compile /opt/sortof/api/app.py /opt/sortof/api/mlos_sort.py /opt/sortof/api/diagnostics.py /opt/sortof/api/categorize.py
/opt/sortof/worker/.venv/bin/python -m py_compile /opt/sortof/worker/worker.py /opt/sortof/worker/mlos_sort.py
cd /opt/sortof/api && .venv/bin/python -c "import app" && echo OK
cd /opt/sortof/worker && .venv/bin/python -c "import drain" && echo OK
```
3. **Dual-edit consistency check** (worker/api `mlos_sort.py` lockstep rule):
```bash
diff /opt/sortof/api/mlos_sort.py /opt/sortof/worker/mlos_sort.py | grep -E "^[<>]" | head -20
```
Logic must match; only comments / docstrings may differ. If any logic line shows up in the diff, fix the lockstep before continuing.
4. **Restart services:**
```bash
sudo systemctl restart sortof-api sortof-drain@1 sortof-drain@2 sortof-drain@3 sortof-drain@4
sudo systemctl is-active sortof-api sortof-drain@{1..4}
```
5. **Force a fresh parse on a known multi-file mod and verify manifest:**
```bash
sudo docker exec -i sortof_db psql -U sortof -d sortof -c \
"DELETE FROM mod_parsed WHERE workshop_id='2169435993';
INSERT INTO download_jobs (workshop_id, status) VALUES ('2169435993','queued');"
sleep 60
sudo docker exec -i sortof_db psql -U sortof -d sortof -c \
"SELECT mod_id, mod_types, files_manifest_built FROM mod_parsed WHERE workshop_id='2169435993';
SELECT count(*) AS file_count FROM mod_files WHERE workshop_id='2169435993';"
```
Expected: `files_manifest_built=t`, `mod_types` populated, `file_count > 0`.
6. **Conflict endpoint smoke:**
```bash
curl -sS -X POST http://100.114.205.53:8801/api/conflicts \
-H 'Content-Type: application/json' \
-d '{"input":"2169435993;2392709985;2487022075"}' | jq .
```
Expected: `{"conflicts": [], "missing_manifests": [<wsids without manifests yet>]}`.
7. **Collection-input rejection (Q4):**
```bash
curl -sS -i -X POST http://100.114.205.53:8801/api/conflicts \
-H 'Content-Type: application/json' \
-d '{"input":"https://steamcommunity.com/sharedfiles/filedetails/?id=999999999"}' | head -5
```
Expected: HTTP 400 with the documented `detail` message (when the URL is detected as a collection ref).
8. **Category-from-types smoke:**
- Find a mod whose Steam tags don't reflect content (e.g. weapon mod tagged only `Realistic`); `/api/sort` currently classifies it as `code` / `other` / `undefined`.
- Re-queue it through the new pipeline (delete+insert).
- Re-run `/api/sort`; confirm category is now `weapon`.
9. **Graceful-degradation check:** confirm a mod with `files_manifest_built=false` still sorts correctly through the existing cascade (no exceptions, category falls back to current behavior).

View File

@@ -0,0 +1,117 @@
# Spec — Stale `require=` filter via Steam Required Items
**Date:** 2026-05-06
**Status:** Implemented
**Lineage:** Builds on Spec C (`2026-05-01-build-context-dep-add.md`) — same warning-shaping path, additional filter layer between mlos_sort output and `_lookup_wsids_for_missing` / `build_warnings`. Also opportunistically swaps `worker.fetch_required_wsids` from HTML scrape to the authenticated Steam API.
## 1. Summary
Authors update Steam's "Required Items" sidebar per build (it visibly affects subscribe-all behavior), but routinely forget to clean up `mod.info`'s `require=` line when porting a mod from B41 → B42. The result: a B42 mod's mod.info still declares B41-era deps that the author has implicitly retired, and we surface them as **missing-dep warnings on a build the mod doesn't actually need them on**. Tikitown is the canonical case: B42 mod, B42 Required Items = `{Drazion's, Erika's}`, but `mod.info require=` still names `Diederiks Tile Palooza, EN_Newburbs` (both B41-only). Today we warn about both; we should not.
This spec adds a small filter, `_filter_stale_requires`, that drops a missing-dep entry when **(a)** the dep resolves to a wsid we have cached, **(b)** that wsid is wrong-build for the user's `pz_build`, and **(c)** the source mod's Steam Required Items list does NOT include that wsid. The author has both labelled the dep wrong-build AND removed it from Required Items — strong evidence the `require=` line is stale.
## 2. Problem
`mlos_sort` builds `missing_requirements` straight from `mod.info`'s `require=` field. It has no concept of build tags or Steam-side dependency lists. Spec C's `_lookup_wsids_for_missing` already filters wrong-build wsid *suggestions* (so we don't propose adding a B41-only mod to a B42 sort), but the **warning itself still appears** with no actionable button. From the user's perspective the only signal is "Tikitown wants something I can't add" — which is incorrect: tikitown doesn't actually want it on B42, the author just forgot to update mod.info.
The over-declaration trap (`TacHold requires modoptions` style — declared but not actually needed) is the same shape: wrong-build mod.info dep that's not in Required Items.
## 3. Heuristic
For each `(source_mod_id, dep_mod_id)` pair in `mlos_sort`'s `missing_requirements`, drop the dep iff ALL of:
1. `pz_build` is set (B41 or B42; unknown → no filtering).
2. `source_mod_id` resolves to a wsid we have cached.
3. The source wsid has been **scraped** for Required Items (`workshop_meta.required_scraped_at IS NOT NULL`). The `required_wsids` column itself has `NOT NULL DEFAULT '{}'` so it can't tell us "never scraped" vs. "scraped, no items"; only `required_scraped_at` distinguishes the two. Without this gate the 2741 unscraped wsids in the live cache (as of 2026-05-06, before the authenticated-API backfill) would all look like "author lists no required items" and silently suppress legit warnings.
4. `dep_mod_id` resolves to a wsid via `mod_parsed` (latest-cached row).
5. The dep's wsid has `workshop_meta.tags` indicating it is **wrong-build** for `pz_build` — i.e., `other_tag in tags AND target_tag NOT in tags`. A mod tagged both builds is build-correct and never dropped.
6. The dep's wsid is **NOT** in the source's `required_wsids`.
If any condition fails, the dep is kept. The filter is conservative: silence requires evidence on every axis.
### 3.1 Worked examples
**Tikitown on B42 (the motivating case):**
- Source: tikitown wsid 3037854728, `required_wsids = {3046728955 Drazion's, 3346506593 Erika's}`.
- Dep `Diederiks Tile Palooza` → wsid 2337452747, tags `{Build 40, Build 41}` → wrong-build for B42 → not in `{3046728955, 3346506593}`**drop**.
- Dep `EN_Newburbs` → wsid 2774834715, tags `{Build 41, ...}` → wrong-build for B42 → not in required → **drop**.
- Dep `tikitown_tiles` → wsid 3046728955, tags include `Build 41, Build 42` → build-correct → **keep** (and the user has it in input anyway, so it doesn't appear as missing).
- Result: warning disappears entirely.
**TacHold on B42 (over-declaration):**
- Source: TacHold wsid X, `required_wsids = {...}` (no modoptions).
- Dep `modoptions` → wsid Y, tags include `Build 41` only on the legacy wsid → wrong-build → not in required → **drop**.
- The mod runs fine without modoptions; the over-declared dep is silenced.
**Legitimate missing dep:**
- Source: SomeMod, `required_wsids = {Z}` where Z resolves to mod_id `RealDep`.
- User omitted `RealDep` from input. mod.info: `require=RealDep`.
- `RealDep` is build-correct → **keep**. Warning surfaces with `[add RealDep]`.
**Source has no Required Items data:**
- New wsid, drained yesterday, `required_wsids` is NULL.
- Filter does nothing for this source's deps → existing behavior.
**Wrong-build dep that IS in Required Items:**
- Author intentionally requires a B41-only utility mod on a B42 mod (rare but real).
- Wrong-build BUT in required → **keep**. Warning surfaces; current Spec C lookup may still suppress the wrong-build add-button, but that's the existing behavior and out of scope here.
## 4. Implementation
### 4.1 New helper: `api/app.py:_filter_stale_requires`
Single async function, ~80 lines. Mutates `mlos_warnings["missing_requirements"]` in place — same dict that downstream `_lookup_wsids_for_missing` and `adapters.build_warnings` already read from. Two queries:
```sql
SELECT workshop_id, required_wsids FROM workshop_meta
WHERE workshop_id = ANY($1) AND required_scraped_at IS NOT NULL;
SELECT DISTINCT ON (mp.mod_id) mp.mod_id, mp.workshop_id, wm.tags
FROM mod_parsed mp JOIN workshop_meta wm ON wm.workshop_id = mp.workshop_id
WHERE mp.mod_id = ANY($1)
ORDER BY mp.mod_id, mp.parsed_at_time_updated DESC;
```
Build correctness uses the same `target_tag` / `other_tag` logic as `_lookup_wsids_for_missing` so a future flip to a different rule (or a third build) only has to change one place.
### 4.2 Call sites
The filter must run **before** `_lookup_wsids_for_missing` (which would otherwise build wsid suggestions for soon-to-be-dropped deps) and **before** `adapters.build_response → build_warnings` (which is what materializes the warning payload from `missing_requirements`). Three call sites, all in `app.py`:
1. `/api/sort` sync path (~line 870, after `sort_mods(mods, rules)`).
2. `/api/sort` async-resume path (~line 1198, after `sort_mods(mods, rules)` on the post-drain refetch).
3. `/api/resort` (~line 1417, after `sort_mods(selected_mods, auto_rules)`).
Each call passes `{m.id: m.workshop_id for m in <local mods> if m.workshop_id}` for the source map. For resort, the local mods are `selected_mods` (what was sorted) — using `all_mods` would also work but `selected_mods` is the strict superset of source mods that could have generated warnings.
## 5. Worker swap: HTML scrape → authenticated `GetDetails`
While we're touching this code path, replace `worker.fetch_required_wsids`'s HTML scraping with the authenticated `IPublishedFileService/GetDetails/v1/?key=…&publishedfileids[0]=…&includechildren=true`. Returns the same `children` array Steam renders into the Required Items sidebar, but:
- No 429 rate-limiting at our drain rate.
- No throttle / 1h cooldown infrastructure needed.
- More reliable than HTML regex parsing (Steam page markup has changed in the past).
Required env: `STEAM_WEB_API_KEY` (already in `/opt/sortof/.env`). Without it, the function returns `None` (existing semantics: don't overwrite cached value). Steam returns `result=1` on success; treat anything else as soft failure (also `None`) so a transient lookup miss doesn't clobber a previously good cached value with `[]`.
Removed code: `_THROTTLE_FILE`, `_COOLDOWN_FILE`, `_MIN_SCRAPE_INTERVAL_S`, `_COOLDOWN_S`, `_read_cooldown_until`, `_write_cooldown_until`, `_throttle_scrape`, `_WORKSHOP_PAGE_URL`, `_RE_REQUIRED_BLOCK`, `_RE_REQUIRED_LINK`, the `import fcntl as _fcntl`, and the rate-limit comment block. `SORTOF_STEAM_MIN_INTERVAL` / `SORTOF_STEAM_COOLDOWN` env knobs are no longer read.
## 6. Non-goals
- Mutating `mod_parsed.requirements`. The filter operates on warning generation only; the parsed `require=` field stays as written in `mod.info` (useful for diagnostics and for future rules that may want the raw declaration).
- Surfacing a "this dep was suppressed because it looked stale" debug warning. The filter is silent by design — if it's right, the user never needed to know; if it's wrong, the user can compare against the Workshop page directly.
- Searching Workshop for B42 alternatives to a wrong-build dep. `IPublishedFileService/QueryFiles` text search is too fuzzy to be reliable (search "Diederiks Tile Palooza" + Build 42 → top hit "Rocco's Tiles", entirely unrelated). Out of scope.
- Loosening the build-correct filter in `_lookup_wsids_for_missing` to offer wrong-build add-buttons with a `(B41)` label. Considered and rejected: re-introduces the over-declaration trap. The stale-filter route handles the same cases more cleanly by suppressing the warning entirely instead of offering an action that points at a wrong-build mod.
## 7. Verification
Smoke test against the canonical case:
```bash
curl -sS -X POST http://100.114.205.53:8801/api/sort \
-H 'Content-Type: application/json' \
-d '{"input":"3037854728;3046728955;3346506593","pz_build":"B42"}' \
| jq '.WARNINGS[] | select(.msg | test("tikitown|Diederiks|Newburbs"))'
```
Expected: empty (warning suppressed). Before this change: one missing-dep warning naming Diederiks + EN_Newburbs.

View File

@@ -145,67 +145,47 @@ function WsidLink({ wsid, children, className }) {
); );
} }
// Snapshot the fields needed to diff against the next sort/resort. Called // Compare the wsids the user pasted (input order, deduped) against the wsids
// before each fetch fires so the snapshot captures "what the user saw // in WORKSHOP_ITEMS_LINE (sort order). Always available — no "previous sort"
// before this action". Polling-mid-flight ticks intentionally don't snapshot // required. Surfaces drops (banned / missing mod.info / unknown / collection
// (would erase the prior visible state on every 2.5s update). // IDs that expanded), additions (collection expansion, branch picks), and
function snapshotForDiff(src) { // position changes.
if (!src) return null; function computeDiff(inputWsids, outputWsids) {
return { if (!inputWsids || !outputWsids) return null;
SORTED_ORDER: [...(src.SORTED_ORDER || [])], const inSet = new Set(inputWsids);
MOD_DB: (src.MOD_DB || []).map(m => ({ modId: m.modId, wsid: m.wsid, name: m.name })), const outSet = new Set(outputWsids);
MODS_LINE: src.MODS_LINE || '', const added = outputWsids.filter(w => !inSet.has(w));
WORKSHOP_ITEMS_LINE: src.WORKSHOP_ITEMS_LINE || '', const removed = inputWsids.filter(w => !outSet.has(w));
}; const inPos = new Map(inputWsids.map((w, i) => [w, i]));
}
function computeDiff(prev, curr) {
if (!prev || !curr) return null;
const prevSorted = prev.SORTED_ORDER || [];
const currSorted = curr.SORTED_ORDER || [];
const prevSet = new Set(prevSorted);
const currSet = new Set(currSorted);
const added = currSorted.filter(id => !prevSet.has(id));
const removed = prevSorted.filter(id => !currSet.has(id));
const prevPos = new Map(prevSorted.map((id, i) => [id, i]));
const movers = []; const movers = [];
currSorted.forEach((id, ci) => { outputWsids.forEach((w, oi) => {
if (!prevSet.has(id)) return; if (!inSet.has(w)) return;
const pi = prevPos.get(id); const ii = inPos.get(w);
if (pi !== ci) movers.push({ id, from: pi, to: ci, delta: ci - pi }); if (ii !== oi) movers.push({ wsid: w, from: ii, to: oi, delta: oi - ii });
}); });
movers.sort((a, b) => Math.abs(b.delta) - Math.abs(a.delta)); movers.sort((a, b) => Math.abs(b.delta) - Math.abs(a.delta));
const prevWsl = (prev.WORKSHOP_ITEMS_LINE || '').replace(/;+$/, '').split(';').filter(Boolean); return { added, removed, movers };
const currWsl = (curr.WORKSHOP_ITEMS_LINE || '').replace(/;+$/, '').split(';').filter(Boolean);
const prevWsidPos = new Map(prevWsl.map((w, i) => [w, i]));
const wsidMovers = [];
currWsl.forEach((w, ci) => {
const pi = prevWsidPos.get(w);
if (pi !== undefined && pi !== ci) wsidMovers.push({ wsid: w, from: pi, to: ci });
});
wsidMovers.sort((a, b) => Math.abs(b.to - b.from) - Math.abs(a.to - a.from));
return { added, removed, movers, wsidMovers };
} }
function DiffPanel({ prev, curr, onClose }) { function DiffPanel({ inputWsids, outputWsids, onClose }) {
const diff = computeDiff(prev, curr); const diff = computeDiff(inputWsids, outputWsids);
if (!diff) { if (!diff) {
return ( return (
<div className="diff-panel"> <div className="diff-panel">
<div className="diff-head"> <div className="diff-head">
<span className="diff-title">diff vs previous sort</span> <span className="diff-title">diff: input sorted</span>
<button type="button" className="diff-close" onClick={onClose}>close</button> <button type="button" className="diff-close" onClick={onClose}>close</button>
</div> </div>
<div className="diff-empty">no previous sort to compare against - sort once first.</div> <div className="diff-empty">no input or output yet.</div>
</div> </div>
); );
} }
const { added, removed, movers, wsidMovers } = diff; const { added, removed, movers } = diff;
const empty = !added.length && !removed.length && !movers.length && !wsidMovers.length; const empty = !added.length && !removed.length && !movers.length;
return ( return (
<div className="diff-panel"> <div className="diff-panel">
<div className="diff-head"> <div className="diff-head">
<span className="diff-title">diff vs previous sort</span> <span className="diff-title">diff: input sorted</span>
<span className="diff-summary"> <span className="diff-summary">
<span className="diff-stat add">+{added.length}</span> <span className="diff-stat add">+{added.length}</span>
<span className="diff-stat rm">{removed.length}</span> <span className="diff-stat rm">{removed.length}</span>
@@ -213,41 +193,29 @@ function DiffPanel({ prev, curr, onClose }) {
</span> </span>
<button type="button" className="diff-close" onClick={onClose}>close</button> <button type="button" className="diff-close" onClick={onClose}>close</button>
</div> </div>
{empty && <div className="diff-empty">nothing changed.</div>} {empty && <div className="diff-empty">order matches your input. nothing dropped or added.</div>}
{added.length > 0 && ( {added.length > 0 && (
<div className="diff-section"> <div className="diff-section">
<div className="diff-label">added ({added.length})</div> <div className="diff-label">added ({added.length}) collection expansion or branch-picker</div>
{added.slice(0, 30).map(id => <div key={id} className="diff-row diff-add">+ {id}</div>)} {added.slice(0, 30).map(w => <div key={w} className="diff-row diff-add">+ {w}</div>)}
{added.length > 30 && <div className="diff-more">and {added.length - 30} more</div>} {added.length > 30 && <div className="diff-more">and {added.length - 30} more</div>}
</div> </div>
)} )}
{removed.length > 0 && ( {removed.length > 0 && (
<div className="diff-section"> <div className="diff-section">
<div className="diff-label">removed ({removed.length})</div> <div className="diff-label">removed ({removed.length}) dropped, banned, missing mod.info, or a collection ID that expanded</div>
{removed.slice(0, 30).map(id => <div key={id} className="diff-row diff-rm"> {id}</div>)} {removed.slice(0, 30).map(w => <div key={w} className="diff-row diff-rm"> {w}</div>)}
{removed.length > 30 && <div className="diff-more">and {removed.length - 30} more</div>} {removed.length > 30 && <div className="diff-more">and {removed.length - 30} more</div>}
</div> </div>
)} )}
{movers.length > 0 && ( {movers.length > 0 && (
<div className="diff-section"> <div className="diff-section">
<div className="diff-label">moved by load order ({movers.length}, top {Math.min(10, movers.length)} shown)</div> <div className="diff-label">reordered by load order ({movers.length}, top {Math.min(10, movers.length)} shown)</div>
{movers.slice(0, 10).map(({ id, from, to, delta }) => ( {movers.slice(0, 10).map(({ wsid, from, to, delta }) => (
<div key={id} className="diff-row diff-mv">
<span className="diff-arrow">{delta < 0 ? '↑' : '↓'}</span>
{id}
<span className="diff-pos">pos {from + 1} {to + 1} ({delta > 0 ? '+' : ''}{delta})</span>
</div>
))}
</div>
)}
{wsidMovers.length > 0 && (
<div className="diff-section">
<div className="diff-label">WorkshopItems= reorder ({wsidMovers.length})</div>
{wsidMovers.slice(0, 10).map(({ wsid, from, to }) => (
<div key={wsid} className="diff-row diff-mv"> <div key={wsid} className="diff-row diff-mv">
<span className="diff-arrow">{to < from ? '↑' : '↓'}</span> <span className="diff-arrow">{delta < 0 ? '↑' : '↓'}</span>
{wsid} {wsid}
<span className="diff-pos">pos {from + 1} {to + 1}</span> <span className="diff-pos">pos {from + 1} {to + 1} ({delta > 0 ? '+' : ''}{delta})</span>
</div> </div>
))} ))}
</div> </div>
@@ -1039,7 +1007,7 @@ function BuildToggle({ value, onChange }) {
); );
} }
function RightColumn({ state, counts, progress, emptyVariant, successVariant, modTableDefault, pzBuild, setPzBuild, branchSelections, onToggleBranch, expandedWsids, onToggleExpansion, inputWsids, onAddWsid, onPickBranch, onSwapWsid, onRemoveWsid, onAutoFixAddDeps, onAutoFixSwaps, onAutoFixRemoves, onRetry, previousResult, diffOpen, setDiffOpen }) { function RightColumn({ state, counts, progress, emptyVariant, successVariant, modTableDefault, pzBuild, setPzBuild, branchSelections, onToggleBranch, expandedWsids, onToggleExpansion, inputWsids, onAddWsid, onPickBranch, onSwapWsid, onRemoveWsid, onAutoFixAddDeps, onAutoFixSwaps, onAutoFixRemoves, onRetry, inputWsidList, diffOpen, setDiffOpen }) {
// Phase-state mapping: legacy `success` ≈ B+F `done`; legacy `partial` ≈ B+F `queued`/`draining`. // Phase-state mapping: legacy `success` ≈ B+F `done`; legacy `partial` ≈ B+F `queued`/`draining`.
const isTerminalDone = state === 'success' || state === 'done'; const isTerminalDone = state === 'success' || state === 'done';
const isInflightPartial = state === 'partial' || state === 'queued' || state === 'draining'; const isInflightPartial = state === 'partial' || state === 'queued' || state === 'draining';
@@ -1131,21 +1099,15 @@ function RightColumn({ state, counts, progress, emptyVariant, successVariant, mo
type="button" type="button"
className={'diff-toggle' + (diffOpen ? ' open' : '')} className={'diff-toggle' + (diffOpen ? ' open' : '')}
onClick={() => setDiffOpen(o => !o)} onClick={() => setDiffOpen(o => !o)}
disabled={!previousResult} title="diff your input against the sorted output"
title={previousResult ? 'compare to previous sort' : 'sort once first'}
> >
{diffOpen ? ' diff' : ' diff'} {diffOpen ? ' diff' : ' diff'}
</button> </button>
</div> </div>
{diffOpen && ( {diffOpen && (
<DiffPanel <DiffPanel
prev={previousResult} inputWsids={inputWsidList}
curr={{ outputWsids={(D.WORKSHOP_ITEMS_LINE || '').replace(/;+$/, '').split(';').filter(Boolean)}
SORTED_ORDER: D.SORTED_ORDER || [],
MOD_DB: D.MOD_DB || [],
MODS_LINE: D.MODS_LINE || '',
WORKSHOP_ITEMS_LINE: D.WORKSHOP_ITEMS_LINE || '',
}}
onClose={() => setDiffOpen(false)} onClose={() => setDiffOpen(false)}
/> />
)} )}
@@ -1434,15 +1396,16 @@ function App() {
const pollAbortRef = useRef(null); const pollAbortRef = useRef(null);
const [activeJobId, setActiveJobId] = useState(null); const [activeJobId, setActiveJobId] = useState(null);
const [expandedWsids, setExpandedWsids] = useState(() => new Set()); const [expandedWsids, setExpandedWsids] = useState(() => new Set());
// Diff: snapshot of the result that's about to be replaced; toggle for the // Diff toggle. The diff is input-vs-output (textarea wsids vs sorted
// panel. Snapshot is taken at the START of any sort/resort - polling-mid-flight // WORKSHOP_ITEMS_LINE), recomputed live every render — no snapshot ref
// ticks don't snapshot so the user's prior visible state stays available. // needed. Always available, even on the first sort.
const previousResultRef = useRef(null);
const [diffOpen, setDiffOpen] = useState(false); const [diffOpen, setDiffOpen] = useState(false);
// Set of wsids currently in the input textarea, used by warning rows to // Ordered list of wsids currently in the input textarea (deduped, first-seen
// derive their staged state. Memoized off `input` so re-renders triggered // order). The Set wrapper below is for warning rows that need O(1) `.has()`
// by unrelated state changes don't churn the Set. // lookup when deriving their staged state. Both are memoized off `input` so
const inputWsids = useMemo(() => new Set(parseWorkshopInput(input)), [input]); // re-renders triggered by unrelated state changes don't churn them.
const inputWsidList = useMemo(() => parseWorkshopInput(input), [input]);
const inputWsids = useMemo(() => new Set(inputWsidList), [inputWsidList]);
// True when the user has staged edits since the last successful sort. Stays // True when the user has staged edits since the last successful sort. Stays
// false until the first sort completes (lastSortedInputRef starts null). // false until the first sort completes (lastSortedInputRef starts null).
const sortPending = lastSortedInputRef.current !== null && input !== lastSortedInputRef.current; const sortPending = lastSortedInputRef.current !== null && input !== lastSortedInputRef.current;
@@ -1589,8 +1552,6 @@ function App() {
const onRetry = () => onSort(); const onRetry = () => onSort();
async function runResort(nextSelections) { async function runResort(nextSelections) {
// Snapshot for the diff panel: capture state before this resort.
previousResultRef.current = snapshotForDiff(_liveSortData);
// Compose the flat list of selected mod_ids from MOD_DB + nextSelections. // Compose the flat list of selected mod_ids from MOD_DB + nextSelections.
// For wsids not in nextSelections, use the §4 default (all-ticked or // For wsids not in nextSelections, use the §4 default (all-ticked or
// first-only depending on radio mode). For wsids with N=1, include the // first-only depending on radio mode). For wsids with N=1, include the
@@ -1794,9 +1755,6 @@ function App() {
// event as arg 0 - reject anything that isn't a string and fall back // event as arg 0 - reject anything that isn't a string and fall back
// to the state input. // to the state input.
const submitInput = typeof inputOverride === 'string' ? inputOverride : input; const submitInput = typeof inputOverride === 'string' ? inputOverride : input;
// Snapshot the about-to-be-replaced result so the [diff] button can
// surface what changed. Skip if there's nothing meaningful yet.
previousResultRef.current = snapshotForDiff(_liveSortData);
clearTimers(); clearTimers();
setState('loading'); setState('loading');
setProgress(15); setProgress(15);
@@ -2065,7 +2023,7 @@ function App() {
onAutoFixSwaps={onAutoFixSwaps} onAutoFixSwaps={onAutoFixSwaps}
onAutoFixRemoves={onAutoFixRemoves} onAutoFixRemoves={onAutoFixRemoves}
onRetry={onRetry} onRetry={onRetry}
previousResult={previousResultRef.current} inputWsidList={inputWsidList}
diffOpen={diffOpen} diffOpen={diffOpen}
setDiffOpen={setDiffOpen} setDiffOpen={setDiffOpen}
/> />

View File

@@ -1,6 +1,8 @@
-- Required Items scraped from each mod's Steam Workshop page (the "Required -- Required Items pulled from each mod's Steam "Required Items" sidebar.
-- Items" section). Steam's anonymous GetPublishedFileDetails endpoint does -- Originally scraped from the public Workshop HTML page (the anonymous
-- not include children for individual mods, so we scrape the public HTML. -- GetPublishedFileDetails endpoint omits `children` for individual mods);
-- worker.fetch_required_wsids now uses authenticated IPublishedFileService/
-- GetDetails with includechildren=true and gets the same data structurally.
-- --
-- Use cases: -- Use cases:
-- 1. Auto-resolving missing-dep warnings: when a cached mod_id Y is -- 1. Auto-resolving missing-dep warnings: when a cached mod_id Y is

View File

@@ -227,6 +227,12 @@ def parse_mod_info(text: str, workshop_id: Optional[str] = None) -> Optional[Mod
Parse a mod.info file body. Returns None if no `id=` line found. Parse a mod.info file body. Returns None if no `id=` line found.
Lines are `key=value`; keys lowercased; list-fields comma-separated. Lines are `key=value`; keys lowercased; list-fields comma-separated.
""" """
# Strip a leading UTF-8 BOM if present. Some authors save mod.info with
# BOM (notepad.exe default on Windows); without this, the first line's
# `name=` regex misses because the line starts with U+FEFF instead of
# `n`, leaving the mod with an empty display name even though `id=` on
# subsequent lines parses fine.
text = text.lstrip("")
fields: Dict[str, object] = {} fields: Dict[str, object] = {}
for raw in text.splitlines(): for raw in text.splitlines():
line = raw.strip() line = raw.strip()

View File

@@ -40,6 +40,7 @@ from mlos_sort import parse_mod_info, ModInfo # noqa: E402
PZ_APP_ID = int(os.environ.get("PZ_APP_ID", "108600")) PZ_APP_ID = int(os.environ.get("PZ_APP_ID", "108600"))
DEFAULT_DD_PATH = os.environ.get("DD_PATH", "./DepotDownloader") DEFAULT_DD_PATH = os.environ.get("DD_PATH", "./DepotDownloader")
STEAM_API = "https://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1/" STEAM_API = "https://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1/"
STEAM_AUTHED_DETAILS = "https://api.steampowered.com/IPublishedFileService/GetDetails/v1/"
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
@@ -71,132 +72,65 @@ def flatten_tags(detail: dict) -> List[str]:
return [t.get("tag", "") for t in detail.get("tags", []) if t.get("tag")] return [t.get("tag", "") for t in detail.get("tags", []) if t.get("tag")]
# Public Steam Workshop page URL. The anonymous GetPublishedFileDetails API
# does NOT return `children` for individual mods (only collections), so to
# learn a mod's "Required Items" we have to scrape the public HTML page.
_WORKSHOP_PAGE_URL = "https://steamcommunity.com/sharedfiles/filedetails/?id={wsid}"
_RE_REQUIRED_BLOCK = re.compile(
r'<div[^>]*id="RequiredItems"[^>]*>(.*?)</div>\s*</div>',
re.DOTALL,
)
_RE_REQUIRED_LINK = re.compile(r'filedetails/\?id=(\d+)')
# ── rate-limit safety for Steam HTML scraping ─────────────────────────────
# Steam aggressively 429s anonymous /sharedfiles/filedetails/ HTML requests;
# during a 2026-05-03 backfill at ~1 RPS our IP was blocked for hours and a
# subsequent single curl probe still got 429. Two file-locked, multi-process
# safeguards now sit in front of every scrape:
#
# 1. THROTTLE FILE — records the timestamp of the last attempted scrape.
# Every worker waits via flock until at least
# `_MIN_SCRAPE_INTERVAL_S` seconds have elapsed since the last one.
# Serializes 4 concurrent drain processes so they can't burst.
#
# 2. COOLDOWN FILE — when we observe a hard 429 (after retries), we write
# `now() + _COOLDOWN_S` here. While active, every fetch returns None
# instantly without touching Steam, preserving cached values until the
# IP block ages out.
#
# Defaults: 6s spacing → ≤10 RPM steady-state, 1h cooldown after a 429
# storm. Overridable via SORTOF_STEAM_MIN_INTERVAL / SORTOF_STEAM_COOLDOWN.
import fcntl as _fcntl
_THROTTLE_FILE = "/tmp/sortof_steam_throttle"
_COOLDOWN_FILE = "/tmp/sortof_steam_cooldown"
_MIN_SCRAPE_INTERVAL_S = float(os.environ.get("SORTOF_STEAM_MIN_INTERVAL", "6"))
_COOLDOWN_S = float(os.environ.get("SORTOF_STEAM_COOLDOWN", "3600"))
def _read_cooldown_until() -> float:
try:
with open(_COOLDOWN_FILE, "r") as f:
return float(f.read().strip() or 0)
except (OSError, ValueError):
return 0.0
def _write_cooldown_until(epoch_s: float) -> None:
try:
with open(_COOLDOWN_FILE, "w") as f:
f.write(str(epoch_s))
except OSError:
pass
def _throttle_scrape() -> None:
"""Block until at least `_MIN_SCRAPE_INTERVAL_S` has elapsed since the
last scrape by ANY drain process (multi-process safe via flock)."""
import time as _t
Path(_THROTTLE_FILE).touch(exist_ok=True)
with open(_THROTTLE_FILE, "r+") as f:
_fcntl.flock(f.fileno(), _fcntl.LOCK_EX)
try:
f.seek(0)
raw = f.read().strip()
last = float(raw) if raw else 0.0
now = _t.time()
wait = _MIN_SCRAPE_INTERVAL_S - (now - last)
if wait > 0:
_t.sleep(wait)
now = _t.time()
f.seek(0); f.truncate(); f.write(str(now))
finally:
_fcntl.flock(f.fileno(), _fcntl.LOCK_UN)
def fetch_required_wsids( def fetch_required_wsids(
workshop_id: str, workshop_id: str,
timeout: int = 15, timeout: int = 15,
max_attempts: int = 4,
backoff_429: float = 30.0,
) -> Optional[List[str]]: ) -> Optional[List[str]]:
"""Scrape the public Workshop page for Required Items wsids. """Fetch the Required Items wsids for a Workshop item via the
authenticated `IPublishedFileService/GetDetails` endpoint, which
returns the same `children` array Steam renders into the public
page's Required Items sidebar.
Returns Returns
None — fetch/parse error, persistent 429, or active cooldown. None — missing/invalid `STEAM_WEB_API_KEY`, network error, or
Caller MUST NOT overwrite the existing cached value. non-success result. Caller MUST NOT overwrite the
[] — page loaded successfully but has no required items section. existing cached value.
list — required item wsids in declaration order, deduped. [] — item exists but has no Required Items.
list — required item wsids in `sortorder` order, deduped.
Replaces the previous HTML-scrape path (Steam 429'd anonymous
/sharedfiles/filedetails/ requests aggressively, requiring throttle
+ 1h cooldown after a 429 storm). The authenticated API has a far
more generous quota and stays well clear of those limits at our
drain rate.
""" """
import time as _time key = os.environ.get("STEAM_WEB_API_KEY")
cooldown_until = _read_cooldown_until() if not key:
if cooldown_until and _time.time() < cooldown_until:
return None # Steam recently 429'd us — back off entirely.
_throttle_scrape()
url = _WORKSHOP_PAGE_URL.format(wsid=workshop_id)
html: Optional[str] = None
for attempt in range(1, max_attempts + 1):
try:
with httpx.Client(timeout=timeout, follow_redirects=True) as client:
r = client.get(url)
if r.status_code == 429:
if attempt < max_attempts:
_time.sleep(backoff_429 * attempt)
continue
# Final 429 → arm the global cooldown so other workers
# (and this one's next call) skip Steam entirely.
_write_cooldown_until(_time.time() + _COOLDOWN_S)
print(f" ! required_wsids 429 (gave up) for {workshop_id}; "
f"cooldown {int(_COOLDOWN_S)}s armed", file=sys.stderr)
return None return None
params = {
"key": key,
"publishedfileids[0]": workshop_id,
"includechildren": "true",
}
try:
with httpx.Client(timeout=timeout) as client:
r = client.get(STEAM_AUTHED_DETAILS, params=params)
r.raise_for_status() r.raise_for_status()
html = r.text body = r.json()
break except (httpx.HTTPError, httpx.TimeoutException, ValueError) as e:
except (httpx.HTTPError, httpx.TimeoutException) as e:
print(f" ! required_wsids fetch failed for {workshop_id}: {e}", print(f" ! required_wsids fetch failed for {workshop_id}: {e}",
file=sys.stderr) file=sys.stderr)
return None return None
if html is None: items = body.get("response", {}).get("publishedfiledetails") or []
if not items:
return None
item = items[0]
# Steam returns result=1 on success; 9 = file not found, etc. Treat
# anything else as a soft failure so we don't clobber a previously
# cached value with [] on a transient lookup miss.
if item.get("result") != 1:
return None return None
m = _RE_REQUIRED_BLOCK.search(html)
if not m:
return []
seen: set = set() seen: set = set()
out: List[str] = [] out: List[str] = []
for w in _RE_REQUIRED_LINK.findall(m.group(1)): children = sorted(
if w not in seen and w != workshop_id: item.get("children") or [],
seen.add(w) key=lambda c: c.get("sortorder", 0),
out.append(w) )
for c in children:
wid = c.get("publishedfileid")
if wid and wid not in seen and wid != workshop_id:
seen.add(wid)
out.append(wid)
return out return out
@@ -431,7 +365,13 @@ def build_manifest_and_types(
or suffix in {".png", ".dds"}): or suffix in {".png", ".dds"}):
if path.name.lower() != "poster.png": if path.name.lower() != "poster.png":
tags.add("Textures") tags.add("Textures")
if rel_below.startswith(("scripts/vehicles/", "scripts/vehicle")): # Path-based vehicle detection. scripts/vehicles is the universal
# signal; models[_x]/vehicles/ catches mods that ship 3D assets
# without scripts (rarer, but real — borrowed from HellDrinx Mod
# Manager's heuristic). rel_below is already lowercased so the
# capital "X" in the original PZ "models_X" path lands as "models_x".
if rel_below.startswith(("scripts/vehicles/", "scripts/vehicle",
"models_x/vehicles/", "models/vehicles/")):
tags.add("Vehicles") tags.add("Vehicles")
if rel_below.startswith(("clothing/", "scripts/clothing/")): if rel_below.startswith(("clothing/", "scripts/clothing/")):
tags.add("Clothing") tags.add("Clothing")