test

2026-04-21 14:31:59 +00:00
parent c6edc5c0bf
commit 74d7f49c8d
12 changed files with 125 additions and 28 deletions
--- a/.env.example
+++ b/.env.example
@@ -65,7 +65,7 @@ DISCORD_ONLY_PORT=5000            # Port for healthcheck / Discord-only server
 # BOSSCORD_CORS_ORIGIN=*          # Optional; default * (set to bOSScord origin in production)

 # --- Database ---
-MONGODB_URI=                      # MongoDB connection string (e.g. mongodb+srv://user:pass@cluster/dbname)
+MONGODB_URI=                      # MongoDB connection string (e.g. mongodb://broccoli_bot:CHANGE_ME@localhost:27017/broccoli_db?authSource=broccoli_db)
 # MONGODB_DATABASE=               # Optional; DB name usually in MONGODB_URI path (not read by app currently)

 # --- Branding & copy ---
--- a/.env.test.example
+++ b/.env.test.example
@@ -64,7 +64,7 @@ DISCORD_ONLY_PORT=5000
 # BOSSCORD_CORS_ORIGIN=*

 # --- Database (test cluster or local) ---
-MONGODB_URI=
+MONGODB_URI=                      # e.g. mongodb://broccoli_bot:CHANGE_ME@localhost:27017/broccoli_db_test?authSource=broccoli_db_test
 # MONGODB_DATABASE=

 # --- Branding & copy ---
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -21,7 +21,7 @@ When the user asks for direct fixes, make them — but still avoid unsolicited r
 ## Project
 - **broccolini-bot** — Discord ticketing + support bot for Indifferent Broccoli (game hosting).
 - **Repo:** `/opt/broccolini-bot/` · **Gitea:** `ssh://git@100.114.205.53:2222/indifferentketchup/broccolini-bot.git`
- **DB:** MongoDB Atlas, database `broccoli_db`.
+- **DB:** Self-hosted MongoDB on same host as bot, database `broccoli_db`. Dedicated user per DB.
 - **Host port 8892 → container port 5000** (`CONFIG.PORT`, env `DISCORD_ONLY_PORT`).
 - **Deploy:** `cd /opt/broccolini-bot && git pull && docker compose up --build -d`  ·  tail: `docker logs broccolini-bot --tail 50 -f`

@@ -36,7 +36,7 @@ When the user asks for direct fixes, make them — but still avoid unsolicited r
 Many files under `scripts/` are one-shot maintenance utilities (backups, user lookups, transcript mapping). They are not wired into CI or into the bot's runtime.

 ## Stack
-Node.js **CommonJS** · Discord.js 14 · Express 5 · Mongoose 6 (MongoDB Atlas) · googleapis · express-rate-limit · p-queue · dotenv/dotenv-expand.
+Node.js **CommonJS** · Discord.js 14 · Express 5 · Mongoose 6 · googleapis · express-rate-limit · p-queue · dotenv/dotenv-expand.

 ## Hard Rules

@@ -76,7 +76,7 @@ Every `interactionCreate` branch runs through `runHandler(name, interaction, fn)
 ### Tickets (`services/tickets.js`, `models.js`)
 - `Ticket` schema has indexes on `{gmailThreadId}` (unique), `{status, lastActivity}`, `{senderEmail, status}`, `{discordThreadId}`.
 - **Discord-originated tickets** use `gmailThreadId` with prefix `discord-` / `discord-msg-` — skip the Gmail reply path entirely.
- Renames route through `utils/renamer.js` (RENAMER_BOT secondary token). On 401/403/429 from the secondary, `services/channelQueue.js` falls back to the primary bot via `channel.setName`. `canRename()` is retained as an always-ok shim for back-compat. `Ticket.renameCount` / `Ticket.renameWindowStart` remain in the schema but are now unread/unwritten orphan fields.
+- Renames route through `utils/renamer.js` (RENAMER_BOT secondary token). On 401/403/429 from the secondary, `services/channelQueue.js` falls back to the primary bot via `channel.setName`. `canRename()` in `services/tickets.js` is retained as an always-ok shim for back-compat. `Ticket.renameCount` / `Ticket.renameWindowStart` remain in the schema but are now unread/unwritten orphan fields.
 - `getOrCreateTicketCategory()` handles Discord's 50-channels-per-category ceiling by creating `"<name> (Overflow N)"` categories; `cleanupEmptyOverflowCategory()` removes empties. The primary category is never deleted.
 - Scheduled jobs in `ready`: `checkAutoClose`, `checkAutoUnclaim`, `reconcileDeletedTicketChannels`, plus `services/staffNotifications.js#notifyAllStaffUnclaimed` and the pattern/surge/chat checkers.

--- a/README.md
+++ b/README.md
@@ -120,7 +120,7 @@ See [Staff notification channels](#staff-notification-channels--reply-alerts) an
 |-------------|--------|
 | **Node.js** | **18+**; Docker image uses **20** (`Dockerfile`). |
 | **npm** | `npm install` locally; `npm ci --omit=dev` in Docker. |
-| **MongoDB** | Atlas or self-hosted; `MONGODB_URI` required at startup. |
+| **MongoDB** | Self-hosted; `MONGODB_URI` required at startup. |
 | **Discord application** | Bot token, application ID; intents: **Message Content**, **Server Members**; also Guilds + Guild Messages. |
 | **Google Cloud** | Gmail API enabled; OAuth2 client ID/secret + refresh token for the support mailbox. |

@@ -498,7 +498,7 @@ This repo includes [`.gitlab-ci.yml`](.gitlab-ci.yml) with GitLab **SAST** and *
 |---------|--------|
 | **Commands missing** | Correct `DISCORD_APPLICATION_ID` + `DISCORD_GUILD_ID`; **restart** bot; Discord can take time to sync. |
 | **Gmail not ingesting** | `REFRESH_TOKEN`, API enablement, inbox auth; regenerate token if revoked. |
-| **MongoDB errors** | `MONGODB_URI`, Atlas IP allowlist, `npm run test-mongodb`. |
+| **MongoDB errors** | `MONGODB_URI`, `npm run test-mongodb`. |
 | **Channels not creating** | Bot **Manage Channels** in ticket categories; category not full (50) unless overflow set. |
 | **Modal / button no response** | Intents + permissions; bot online; check `DEBUGGING_CHANNEL_ID` / console. |
 | **Renames “too quickly”** | Discord rename cooldown; wait for channel queue / timestamp in bot message. |
--- a/broccolini_bot_context.md
+++ b/broccolini_bot_context.md
@@ -8,7 +8,7 @@ Node.js (CommonJS) Discord ticketing bot for Indifferent Broccoli. Single proces

 - A discord.js v14 client (ticket lifecycle, slash/button/modal handlers, context menus)
 - A Gmail bridge (~30s polling → Discord channels; staff replies → Gmail)
- A Mongoose/MongoDB Atlas layer (`broccoli_db`) for tickets + settings
+- A Mongoose/self-hosted MongoDB layer (`broccoli_db`) for tickets + settings
 - Two Express servers: healthcheck + bOSScord API (`PORT`, default 5000 → host 8892), and an internal settings API (`INTERNAL_API_PORT`)
 - Background jobs: auto-close, unclaimed reminders, auto-unclaim, pattern detection, surge detection, chat monitoring, orphan reconciliation

@@ -335,7 +335,7 @@ ticketSchema.index({ ticketNumber: 1 });
 `discordThreadId` should be `unique, sparse` because Discord-only tickets set it immediately, email tickets may briefly lack it during creation, and no two tickets should share a channel. Confirm the sparse-unique behavior doesn't conflict with existing data before enabling (see Verify).
 **Verify:**
 - Before deploy, run `db.tickets.aggregate([{$group: {_id: "$discordThreadId", c: {$sum: 1}}}, {$match: {c: {$gt: 1}}}])` against `broccoli_db` to confirm no duplicate `discordThreadId` values exist. If any do, investigate (they indicate prior orphaning bugs) before adding the unique index.
- After redeploy, run `db.tickets.getIndexes()` in Atlas and confirm all five new indexes exist.
+- After redeploy, run `db.tickets.getIndexes()` via mongosh and confirm all five new indexes exist.
 - Spot-check with `db.tickets.find({discordThreadId: "<some id>"}).explain("executionStats")` — should show `IXSCAN`, not `COLLSCAN`.

 ---
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,4 +1,21 @@
 services:
+  mongo:
+    image: mongo:7
+    container_name: broccoli-mongo
+    restart: unless-stopped
+    environment:
+      MONGO_INITDB_ROOT_USERNAME: ${MONGO_ROOT_USERNAME}
+      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_ROOT_PASSWORD}
+    volumes:
+      - broccoli-mongo-data:/data/db
+    networks:
+      - broccoli-net
+    healthcheck:
+      test: ["CMD", "mongosh", "--quiet", "--eval", "db.runCommand({ping:1}).ok"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 20s
  broccolini:
    build: .
    image: broccolini-bot
@@ -11,6 +28,9 @@ services:
      - ./.env:/app/.env:rw
    ports:
      - "100.114.205.53:8892:5000"
+    depends_on:
+      mongo:
+        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:5000/"]
      interval: 30s
@@ -23,3 +43,6 @@ networks:
  broccoli-net:
    name: broccoli-net
    external: true
+volumes:
+  broccoli-mongo-data:
+    name: broccoli-mongo-data
--- a/docs/setup/ENV_AND_SECURITY.md
+++ b/docs/setup/ENV_AND_SECURITY.md
@@ -34,7 +34,7 @@
 - **Secrets:** All secrets live in `.env` (or `.env.test` for test). Never commit them. `.gitignore` excludes `.env` and `.env.*` except `.env.example` and `.env.test.example`.
 - **Code:** No `eval()` or `new Function()` of user input. No hardcoded tokens, passwords, or API keys in source.
 - **Config:** Credentials are read from `process.env` via `config.js`; config is loaded once at startup from the file specified by `ENV_FILE` or default `.env`.
- **MongoDB:** Use a dedicated user and database; restrict network access (Atlas IP allowlist or VPC). For test, use a separate DB or cluster.
+- **MongoDB:** Use a dedicated user and database; bind Mongo to loopback or docker network only; firewall 27017 from public interfaces. For test, use a separate DB or cluster.
 - **Discord / Google:** Use tokens with minimal required scopes; rotate if compromised.
 - **HTML in emails:** `LOGO_URL`, `EMAIL_SIGNATURE`, and closure messages are escaped in outbound HTML to prevent injection.
 - **Healthcheck:** Optional `HEALTHCHECK_HOST=127.0.0.1` in `.env` binds the healthcheck server to localhost only; omit to listen on all interfaces.
--- a/docs/setup/MONGODB_SETUP.md
+++ b/docs/setup/MONGODB_SETUP.md
@@ -20,10 +20,23 @@ Broccolini Bot uses **MongoDB only** for persistent storage (tickets, transcript
 Add to your `.env` file:

 ```env
-MONGODB_URI=mongodb://localhost:27018/broccolini_bot
+MONGODB_URI=mongodb://broccoli_bot:CHANGE_ME@localhost:27017/broccoli_db?authSource=broccoli_db
 ```

-**Note:** Uses port `27018` to match your existing setup (as defined in docker-compose.yml).
+**Note:** Mongo runs self-hosted on the same host as the bot. A **dedicated user per database** is required — create `broccoli_bot` with `readWrite` on `broccoli_db` only (no admin/root, no cross-DB access). For test, create a separate user with `readWrite` on `broccoli_db_test` only.
+
+Example mongosh setup:
+
+```javascript
+use broccoli_db
+db.createUser({
+  user: "broccoli_bot",
+  pwd: "CHANGE_ME",
+  roles: [ { role: "readWrite", db: "broccoli_db" } ]
+})
+```
+
+Bind Mongo to loopback (`bindIp: 127.0.0.1`) or the internal docker network only; firewall `27017` from public interfaces.

 ### 2. Install Dependencies

@@ -141,11 +154,12 @@ process.on('SIGINT', async () => {

 ### Connection refused
 - Check MongoDB is running: `docker ps` or `systemctl status mongodb`
- Verify port 27018 is correct in `.env`
+- Verify port 27017 is correct in `.env` (or whatever port your mongod is bound to)
 - Check MongoDB logs for errors

 ### Authentication failed
- If MongoDB requires auth, update URI: `mongodb://username:password@localhost:27018/broccolini_bot`
+- Verify the user exists in the correct DB's `authSource` (URI must include `?authSource=broccoli_db`)
+- Confirm the user has `readWrite` on `broccoli_db`: `db.getUser("broccoli_bot")` in mongosh

 ### Schema validation errors
 - Check required fields are provided when creating documents
--- a/gmail-poll.js
+++ b/gmail-poll.js
@@ -33,7 +33,10 @@ let authErrorNotified = false;
 let pollSuspended = false;
 let pollCount = 0, totalProcessed = 0, totalSkipped = 0, totalErrors = 0;

-function setPollSuspended(val) { pollSuspended = !!val; }
+function setPollSuspended(val) {
+  pollSuspended = !!val;
+  if (!pollSuspended) authErrorNotified = false;
+}
 function isPollSuspended() { return pollSuspended; }

 /**
@@ -362,18 +365,16 @@ async function poll(client) {
    }
    authErrorNotified = false;
  } catch (e) {
-    const isAuthError =
-      (e.message && (
-        e.message.includes('invalid_grant') ||
-        e.message.includes('unauthorized') ||
-        e.message.includes('Invalid Credentials')
-      )) ||
-      e.status === 401 ||
-      e.code === 401;
+    // Only treat Google-reported permanent-grant failures as reasons to suspend
+    // the loop. Transient 401/403/429/5xx/network errors fall through to the
+    // next interval tick naturally. The OAuth error codes come back on the
+    // response body, not the message string.
+    const oauthError = e && e.response && e.response.data && e.response.data.error;
+    const isPermanentAuth = oauthError === 'invalid_grant' || oauthError === 'invalid_client';

-    if (isAuthError) {
+    if (isPermanentAuth) {
      pollSuspended = true;
-      const suspendMsg = 'Gmail OAuth token invalid or expired. Polling SUSPENDED — will not retry automatically. Re-authenticate to resume.';
+      const suspendMsg = `Gmail OAuth ${oauthError}. Polling SUSPENDED — will not retry automatically. Re-authenticate and POST /internal/gmail/reload to resume.`;
      console.error('[gmail-poll]', suspendMsg);
      logError('Gmail OAuth', { message: suspendMsg, stack: e.stack || e.message || String(e) }, null, client).catch(() => {});
      try { require('./broccolini-discord').clearGmailPollInterval?.(); } catch (_) {}
--- a/routes/internalApi.js
+++ b/routes/internalApi.js
@@ -228,6 +228,35 @@ router.post('/notifications/toggle', express.json(), async (req, res) => {
  res.json({ state: getNotificationState() });
 });

+// POST /gmail/reload — hot-swap Gmail OAuth creds after weekly reauth without
+// restarting the process. Reads REFRESH_TOKEN from .env via configPersistence,
+// probes Google with users.getProfile, and on success clears pollSuspended and
+// re-installs the poll interval. On failure returns 400 with Google's error so
+// the operator can see why (e.g. still invalid_grant).
+router.post('/gmail/reload', express.json(), async (req, res) => {
+  const { reloadGmailClient } = require('../services/gmail');
+  const { setPollSuspended } = require('../gmail-poll');
+  try {
+    const { emailAddress } = await reloadGmailClient();
+    setPollSuspended(false);
+    // Lazy require — same reason as /restart above (module scope cycle).
+    const parent = require('../broccolini-discord');
+    if (parent.setGmailPollInterval) {
+      parent.setGmailPollInterval(CONFIG.GMAIL_POLL_INTERVAL_MS);
+    }
+    await logSystem('Gmail OAuth reloaded', [
+      { name: 'Account', value: emailAddress, inline: false }
+    ]).catch(() => {});
+    res.json({ ok: true, email: emailAddress });
+  } catch (err) {
+    const oauthError = err && err.response && err.response.data && err.response.data.error;
+    res.status(400).json({
+      ok: false,
+      error: oauthError || err.code || err.message || 'reload failed'
+    });
+  }
+});
+
 // Expose the allowlist for the Phase 8 schema smoke test. Attached to the
 // router function object; doesn't show up as a route.
 router._allowedKeys = Array.from(ALLOWED_CONFIG_KEYS);
--- a/services/gmail.js
+++ b/services/gmail.js
@@ -6,6 +6,7 @@ const { CONFIG } = require('../config');
 const { extractRawEmail, escapeHtml } = require('../utils');
 const { getStaffSignatureBlocks } = require('./staffSignature');
 const { logError } = require('./debugLog');
+const { readEnvFile } = require('./configPersistence');

 function sanitizeHeaderValue(v) { return String(v || '').replace(/[\r\n]+/g, ' ').trim(); }
 const EMAIL_RE = /^[^@\s]+@[^@\s]+$/;
@@ -19,6 +20,31 @@ function getGmailClient() {
  return google.gmail({ version: 'v1', auth });
 }

+/**
+ * Re-read REFRESH_TOKEN from .env, update in-memory config, and probe Google.
+ * Used by the internal /gmail/reload endpoint so the weekly reauth chore does
+ * not require a full container restart.
+ *
+ * Throws if the env file is missing the token, or if the probe call (getProfile)
+ * fails — the caller surfaces the error so the UI can see why.
+ *
+ * @returns {Promise<{emailAddress: string}>}
+ */
+async function reloadGmailClient() {
+  const envMap = readEnvFile();
+  const newToken = envMap.get('REFRESH_TOKEN');
+  if (!newToken) {
+    const err = new Error('REFRESH_TOKEN not set in .env');
+    err.code = 'ENOTOKEN';
+    throw err;
+  }
+  process.env.REFRESH_TOKEN = newToken;
+  CONFIG.REFRESH_TOKEN = newToken;
+  const gmail = getGmailClient();
+  const profile = await gmail.users.getProfile({ userId: 'me' });
+  return { emailAddress: profile.data.emailAddress };
+}
+
 async function sendTicketClosedEmail(ticket, discordDisplayName) {
  try {
    const gmail = getGmailClient();
@@ -318,6 +344,7 @@ async function sendGmailReply(

 module.exports = {
  getGmailClient,
+  reloadGmailClient,
  sendGmailReply,
  sendTicketClosedEmail,
  sendTicketNotificationEmail
--- a/settings-site/CLAUDE.md
+++ b/settings-site/CLAUDE.md
@@ -42,6 +42,9 @@ browser ──► settings server.js (:SETTINGS_PORT, default 12752)
 | `GET /api/discord/guild` | `GET /internal/discord/guild` |
 | `POST /api/restart` | `POST /internal/restart` |
 | `GET /api/restart/status` | `GET /internal/restart/status` |
+| `GET /api/notifications/alerts` | `GET /internal/notifications/alerts` |
+| `GET /api/notifications/state` | `GET /internal/notifications/state` |
+| `POST /api/notifications/toggle` | `POST /internal/notifications/toggle` |

 Every response-shape change in the bot's `/internal/*` handlers (`routes/internalApi.js`) is a breaking change here. The bot also gates `POST /internal/config` on an `ALLOWED_CONFIG_KEYS` allowlist — **adding a new field to the UI requires adding the key to that Set in the bot first**, otherwise the save returns 400 for that key.

@@ -49,11 +52,11 @@ Every response-shape change in the bot's `/internal/*` handlers (`routes/interna
 `server.js:20-26` sets `cookie.secure: true`. Browsers will refuse to persist the session cookie over plain HTTP, so login silently fails when not behind an HTTPS reverse proxy (`SETTINGS_DOMAIN` is the deployed domain). If you're reproducing a login bug, check this first before debugging auth logic. The `session secret` falls back to `'fallback-secret-change-me'` when `INTERNAL_API_SECRET` is unset — don't rely on the fallback in any environment that matters.

 ### Client-side routing
-`public/index.html` is a single page with all sections rendered; `public/js/app.js` toggles `.hidden` on sections based on `location.pathname`. Routes live in the `ROUTES` map (`app.js:425`). The server has `app.get('*', requireAuth, …)` as a catch-all back to `index.html` (`server.js:97`), so any new client route works without server changes as long as it's added to `ROUTES`.
+`public/js/` is split into focused modules (phase 4 refactor): `app.js` (bootstrap), `router.js`, `fields.js`, `notifications.js`, `discord.js`, `login.js`, `util.js` — no bundler, loaded via `<script>` tags. Routes live in the `ROUTES` map (`router.js:4`); the server has a catch-all back to `index.html` (`server.js:202`, Express 5 `'/*splat'` syntax), so adding a client route only requires editing `ROUTES`.

 ### Config field binding (frontend)
 Any form element with `data-key="SOME_CONFIG_KEY"` participates in the editor:
- `populateFields()` (`app.js:102`) fills it from `GET /api/config` and wires change listeners.
+- `populateFields()` (`fields.js:11`) fills it from `GET /api/config` and wires change listeners.
 - Checkboxes serialize to the strings `'true'` / `'false'`, and `<input type="color">` serializes to `0xRRGGBB` — this matches how the bot stores these values.
 - `pendingChanges` accumulates diffs; `saveConfig()` POSTs the whole diff at once.
 - `data-smart="channel|category|role|member|multi-member"` swaps the bare `<input>` for a searchable Discord picker backed by `GET /api/discord/guild` (see `public/js/discord.js`).
@@ -61,7 +64,7 @@ Any form element with `data-key="SOME_CONFIG_KEY"` participates in the editor:
 **To add a new editable config field:** (1) add the key to the bot's `ALLOWED_CONFIG_KEYS`, (2) add a `<input data-key="NEW_KEY">` (optionally `data-smart=…`) inside the appropriate `.section` in `public/index.html`. No JS changes needed.

 ### Notification thresholds editor
-The Notifications section is **not** a simple `data-key` field — it's a custom editor (`app.js:239-423`) that serializes into a single hidden `NOTIFICATION_THRESHOLDS_JSON` field. Alert keys are hard-coded in `NOTIFICATION_TAB_KEYS` (surge / patterns / unclaimed / chat) and described in `NOTIFICATION_ALERT_DESCRIPTIONS`. **Adding a new alert key requires editing both of those objects** — otherwise it won't show up in any tab. Threshold values accept whole numbers or duration strings matching `^(\d+[mhd])+$` (e.g. `15m`, `1h`, `1d6h`).
+The Notifications section is **not** a simple `data-key` field — it's a custom editor in `notifications.js` that serializes into a single hidden `NOTIFICATION_THRESHOLDS_JSON` field. Alert metadata is now a **dynamic registry** (phase 5): the bot is canonical and serves it via `GET /api/notifications/alerts`; `notifications.js` uses `FALLBACK_TAB_KEYS` only if the fetch fails. **To add a new alert key, register it in the bot** (not in this codebase) — the UI picks it up automatically on next load. Threshold values accept whole numbers or duration strings matching `^(\d+[mhd])+$` (e.g. `15m`, `1h`, `1d6h`).

 ## Gotchas