diff --git a/.codesight/CODESIGHT.md b/.codesight/CODESIGHT.md index c86d93b..e3b4f0e 100644 --- a/.codesight/CODESIGHT.md +++ b/.codesight/CODESIGHT.md @@ -3,9 +3,9 @@ > **Stack:** fastify, go-net-http | none | react | typescript > **Microservices:** @boocode/contracts, @boocode/ion, @boocode/booterm, @boocode/coder, @boocode/server, @boocode/web, codecontext, @boocode/conductor -> 147 routes (9 inferred) + 9 ws | 23 models | 92 components | 296 lib files | 43 env vars | 17 middleware +> 147 routes (9 inferred) + 9 ws | 23 models | 92 components | 288 lib files | 42 env vars | 16 middleware > **Token savings:** this file is ~0 tokens. Without it, AI exploration would cost ~0 tokens. **Saves ~0 tokens per conversation.** -> **Last scanned:** 2026-06-08 03:49 — re-run after significant changes +> **Last scanned:** 2026-06-08 04:10 — re-run after significant changes --- @@ -1012,19 +1012,11 @@ - function getBackgroundTaskResult: (sql, taskId, chatId) => Promise< - function cancelBackgroundTask: (sql, taskId) => Promise - interface BackgroundTask -- `apps/server/src/services/boocontext_client.ts` - - function callBoocontext: (req, log?, msg) => void - - interface BoocontextRequest - - interface BoocontextResponse - `apps/server/src/services/broker.ts` - function createBroker: (log?) => Broker - interface Broker - type Frame - type Listener -- `apps/server/src/services/codecontext_client.ts` - - function callCodecontext: (req, fetcher) => Promise - - interface CodecontextRequest - - interface CodecontextResponse - `apps/server/src/services/coder-notify.ts` — function notifyCoderClose: (kind, id, log?, 'debug'>, fetcher) => Promise, type CoderCloseKind - `apps/server/src/services/compaction.ts` - function usable: (contextLimit) => number @@ -1310,34 +1302,6 @@ - type SubagentStatusInputT - type SubagentResultInputT - _...6 more_ -- `apps/server/src/services/tools/codecontext/factory.ts` — function makeCodecontextTool: (opts, unknown>; - mapArgs) => void -- `apps/server/src/services/tools/codecontext/get_code_health.ts` - - function executeGetCodeHealth: (input, projectPath) => Promise - - type GetCodeHealthInputT - - const GetCodeHealthInput - - const getCodeHealth: ToolDef -- `apps/server/src/services/tools/codecontext/get_code_impact.ts` - - function executeGetCodeImpact: (input, projectPath) => Promise - - type GetCodeImpactInputT - - const GetCodeImpactInput - - const getCodeImpact: ToolDef -- `apps/server/src/services/tools/codecontext/get_code_map.ts` - - function executeGetCodeMap: (input, projectRoot) => Promise - - interface CodeMapResponse - - type GetCodeMapInputT - - const GetCodeMapInput - - const getCodeMap: ToolDef -- `apps/server/src/services/tools/codecontext/get_type_info.ts` - - function executeGetTypeInfo: (input, _projectPath?) => Promise - - type GetTypeInfoInputT - - const GetTypeInfoInput - - const getTypeInfo: ToolDef -- `apps/server/src/services/tools/codecontext/get_wiki_article.ts` - - function executeGetWikiArticle: (input, projectPath) => Promise - - type GetWikiArticleInputT - - const GetWikiArticleInput - - const getWikiArticle: ToolDef - `apps/server/src/services/tools/execute-command.ts` - function executeRunCommand: (input, projectRoot) => Promise - type RunCommandInputT @@ -1685,7 +1649,6 @@ - `BRAINSTORM_PORT` **required** — data/skills/superpowers/brainstorming/scripts/server.cjs - `BRAINSTORM_URL_HOST` **required** — data/skills/superpowers/brainstorming/scripts/server.cjs - `CODECONTEXT_CHILD` **required** — codecontext/shim.go -- `CODECONTEXT_URL` **required** — apps/server/src/services/codecontext_client.ts - `CONDUCTOR_MODEL` **required** — conductor/src/dispatch.ts - `CONDUCTOR_OPENCODE_BIN` **required** — conductor/src/dispatch.ts - `CONDUCTOR_TIMEOUT_MS` **required** — conductor/src/dispatch.ts @@ -1733,7 +1696,6 @@ - authoring — `apps/coder/src/conductor/flows/authoring.ts` - turn-guard.test — `apps/coder/src/services/backends/__tests__/turn-guard.test.ts` - turn-guard — `apps/coder/src/services/backends/turn-guard.ts` -- get_middleware — `apps/server/src/services/tools/codecontext/get_middleware.ts` - authoring — `conductor/src/flows/authoring.ts` - spec — `openspec/changes/add-behavioral-engine/specs/audit-middleware/spec.md` @@ -1767,8 +1729,6 @@ - `apps/coder/src/services/agent-backend.ts` — imported by **14** files - `apps/coder/src/services/acp-tool-snapshot.ts` — imported by **14** files - `apps/server/src/config.ts` — imported by **14** files -- `apps/server/src/services/tools/codecontext/factory.ts` — imported by **14** files -- `apps/server/src/services/tools/types.ts` — imported by **13** files - `conductor/src/types.ts` — imported by **13** files - `apps/coder/src/services/provider-config-registry.ts` — imported by **12** files - `apps/coder/src/config.ts` — imported by **11** files @@ -1777,6 +1737,8 @@ - `apps/server/src/services/agents.ts` — imported by **10** files - `apps/server/src/services/path_guard.ts` — imported by **10** files - `apps/coder/src/services/pending_changes.ts` — imported by **9** files +- `apps/server/src/services/inference/payload.ts` — imported by **9** files +- `apps/server/src/services/inference/dcp/messages.ts` — imported by **9** files ## Import Map (who imports what) diff --git a/.codesight/config.md b/.codesight/config.md index 016cbc4..1ef4563 100644 --- a/.codesight/config.md +++ b/.codesight/config.md @@ -18,7 +18,6 @@ - `BRAINSTORM_PORT` **required** — data/skills/superpowers/brainstorming/scripts/server.cjs - `BRAINSTORM_URL_HOST` **required** — data/skills/superpowers/brainstorming/scripts/server.cjs - `CODECONTEXT_CHILD` **required** — codecontext/shim.go -- `CODECONTEXT_URL` **required** — apps/server/src/services/codecontext_client.ts - `CONDUCTOR_MODEL` **required** — conductor/src/dispatch.ts - `CONDUCTOR_OPENCODE_BIN` **required** — conductor/src/dispatch.ts - `CONDUCTOR_TIMEOUT_MS` **required** — conductor/src/dispatch.ts diff --git a/.codesight/graph.md b/.codesight/graph.md index df5e394..58b4889 100644 --- a/.codesight/graph.md +++ b/.codesight/graph.md @@ -12,8 +12,6 @@ - `apps/coder/src/services/agent-backend.ts` — imported by **14** files - `apps/coder/src/services/acp-tool-snapshot.ts` — imported by **14** files - `apps/server/src/config.ts` — imported by **14** files -- `apps/server/src/services/tools/codecontext/factory.ts` — imported by **14** files -- `apps/server/src/services/tools/types.ts` — imported by **13** files - `conductor/src/types.ts` — imported by **13** files - `apps/coder/src/services/provider-config-registry.ts` — imported by **12** files - `apps/coder/src/config.ts` — imported by **11** files @@ -22,6 +20,8 @@ - `apps/server/src/services/agents.ts` — imported by **10** files - `apps/server/src/services/path_guard.ts` — imported by **10** files - `apps/coder/src/services/pending_changes.ts` — imported by **9** files +- `apps/server/src/services/inference/payload.ts` — imported by **9** files +- `apps/server/src/services/inference/dcp/messages.ts` — imported by **9** files ## Import Map (who imports what) diff --git a/.codesight/libs.md b/.codesight/libs.md index 7c372ef..eda0ff0 100644 --- a/.codesight/libs.md +++ b/.codesight/libs.md @@ -527,19 +527,11 @@ - function getBackgroundTaskResult: (sql, taskId, chatId) => Promise< - function cancelBackgroundTask: (sql, taskId) => Promise - interface BackgroundTask -- `apps/server/src/services/boocontext_client.ts` - - function callBoocontext: (req, log?, msg) => void - - interface BoocontextRequest - - interface BoocontextResponse - `apps/server/src/services/broker.ts` - function createBroker: (log?) => Broker - interface Broker - type Frame - type Listener -- `apps/server/src/services/codecontext_client.ts` - - function callCodecontext: (req, fetcher) => Promise - - interface CodecontextRequest - - interface CodecontextResponse - `apps/server/src/services/coder-notify.ts` — function notifyCoderClose: (kind, id, log?, 'debug'>, fetcher) => Promise, type CoderCloseKind - `apps/server/src/services/compaction.ts` - function usable: (contextLimit) => number @@ -825,34 +817,6 @@ - type SubagentStatusInputT - type SubagentResultInputT - _...6 more_ -- `apps/server/src/services/tools/codecontext/factory.ts` — function makeCodecontextTool: (opts, unknown>; - mapArgs) => void -- `apps/server/src/services/tools/codecontext/get_code_health.ts` - - function executeGetCodeHealth: (input, projectPath) => Promise - - type GetCodeHealthInputT - - const GetCodeHealthInput - - const getCodeHealth: ToolDef -- `apps/server/src/services/tools/codecontext/get_code_impact.ts` - - function executeGetCodeImpact: (input, projectPath) => Promise - - type GetCodeImpactInputT - - const GetCodeImpactInput - - const getCodeImpact: ToolDef -- `apps/server/src/services/tools/codecontext/get_code_map.ts` - - function executeGetCodeMap: (input, projectRoot) => Promise - - interface CodeMapResponse - - type GetCodeMapInputT - - const GetCodeMapInput - - const getCodeMap: ToolDef -- `apps/server/src/services/tools/codecontext/get_type_info.ts` - - function executeGetTypeInfo: (input, _projectPath?) => Promise - - type GetTypeInfoInputT - - const GetTypeInfoInput - - const getTypeInfo: ToolDef -- `apps/server/src/services/tools/codecontext/get_wiki_article.ts` - - function executeGetWikiArticle: (input, projectPath) => Promise - - type GetWikiArticleInputT - - const GetWikiArticleInput - - const getWikiArticle: ToolDef - `apps/server/src/services/tools/execute-command.ts` - function executeRunCommand: (input, projectRoot) => Promise - type RunCommandInputT diff --git a/.codesight/middleware.md b/.codesight/middleware.md index 5ba07cf..e064121 100644 --- a/.codesight/middleware.md +++ b/.codesight/middleware.md @@ -5,7 +5,6 @@ - authoring — `apps/coder/src/conductor/flows/authoring.ts` - turn-guard.test — `apps/coder/src/services/backends/__tests__/turn-guard.test.ts` - turn-guard — `apps/coder/src/services/backends/turn-guard.ts` -- get_middleware — `apps/server/src/services/tools/codecontext/get_middleware.ts` - authoring — `conductor/src/flows/authoring.ts` - spec — `openspec/changes/add-behavioral-engine/specs/audit-middleware/spec.md` diff --git a/.env.example b/.env.example index 41158c3..6527a4f 100644 --- a/.env.example +++ b/.env.example @@ -31,6 +31,6 @@ SEARXNG_URL=http://100.114.205.53:8888 # sessions where the model only needs read-only filesystem access. # # core → view_file, list_dir, grep, find_files (~2k) -# standard → core + web_*, git_status, all 8 codecontext_* tools (~10k) +# standard → core + web_*, git_status, boocontext MCP tools (~10k) # all → every tool in ALL_TOOLS (~21k) # BOOCODE_TOOLS=all diff --git a/README.md b/README.md index 6af11d7..0a16a3e 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ curl http://100.114.205.53:9502/api/health |BooTerm|`100.114.205.53:9501`|PTY/tmux terminal panes | |BooCoder|host:9502|Write tools + agent dispatch + MCP server (systemd service, not Docker) | |Postgres|`127.0.0.1:5500`|Shared database (`boochat`; Docker service `boocode_db`) | -|codecontext|internal `:8080`|Code graph sidecar (Docker network only) | +|boocontext|MCP (built into boocoder service)|Tree-sitter code analysis (callgraph, symbols, types, health) | ## What's shipped diff --git a/apps/server/src/services/synthesisPipeline.ts b/apps/server/src/services/synthesisPipeline.ts index 8d75f09..9c9fb63 100644 --- a/apps/server/src/services/synthesisPipeline.ts +++ b/apps/server/src/services/synthesisPipeline.ts @@ -32,10 +32,9 @@ import type { OpenAiMessage } from './inference/payload.js'; import type { InferenceContext, TurnArgs } from './inference/types.js'; export const SYNTHESIS_TOOLS: ReadonlySet = new Set([ - 'get_codebase_overview', - 'get_framework_analysis', - 'get_semantic_neighborhoods', - 'get_blast_radius', + 'boocontext_boocontext_overview', + 'boocontext_boocontext_symbols', + 'boocontext_codesight_get_blast_radius', ]); const TOP_N_FILES = 5; @@ -103,11 +102,11 @@ export async function runSynthesisPass(p: SynthesisParams): Promise { } // v1.13.15-b: when the tool result was inline-truncated by the wrapper - // (32k cap, see codecontext_client.ts:114), expand the full content from - // tmpfs for reference-file extraction. The synth payload still ships the - // truncated head (see buildPayload call below) so the token-budget - // contract holds. Graceful degradation: if readTruncation returns null - // (missing id, ENOENT) or throws, fall back to the truncated head. + // (32k cap), expand the full content from tmpfs for reference-file + // extraction. The synth payload still ships the truncated head (see + // buildPayload call below) so the token-budget contract holds. Graceful + // degradation: if readTruncation returns null (missing id, ENOENT) or + // throws, fall back to the truncated head. let extractionSource = p.toolResultText; if (p.truncated && p.outputPath) { try { diff --git a/apps/server/src/services/tools/tiers.ts b/apps/server/src/services/tools/tiers.ts index 0ab609c..ad082f5 100644 --- a/apps/server/src/services/tools/tiers.ts +++ b/apps/server/src/services/tools/tiers.ts @@ -7,6 +7,11 @@ import { ALL_TOOLS, TOOLS_BY_NAME } from './registry.js'; // schemas in the system prompt). Pattern lift from eyaltoledano/claude-task- // master (MIT + Commons Clause — pattern only, no code lift). // +// v2.8.25: removed the 8 old codecontext tool names from STANDARD_TOOL_NAMES. +// The Go codecontext sidecar has been fully removed; boocontext MCP tools are +// appended at startup via appendMcpTools() and are NOT available at import +// time, so STANDARD_TOOL_NAMES only includes core tools + web/web/git. +// // The env var is a CEILING. It only narrows; never expands an agent's // declared whitelist. Default behavior (var unset) is unchanged: all tools. export const CORE_TOOL_NAMES = [ @@ -21,14 +26,6 @@ export const STANDARD_TOOL_NAMES = [ 'web_search', 'web_fetch', 'git_status', - 'get_codebase_overview', - 'get_file_analysis', - 'get_symbol_info', - 'search_symbols', - 'get_dependencies', - 'watch_changes', - 'get_semantic_neighborhoods', - 'get_framework_analysis', ] as const; // Module-load validation: every name in CORE / STANDARD must exist in diff --git a/apps/web/src/App.tsx b/apps/web/src/App.tsx index bf5876a..e2a6179 100644 --- a/apps/web/src/App.tsx +++ b/apps/web/src/App.tsx @@ -11,8 +11,10 @@ import { Analytics } from '@/pages/Analytics'; import { Results } from '@/pages/Results'; import { Memory } from '@/pages/Memory'; import { Toaster } from '@/components/ui/sonner'; +import { toast } from 'sonner'; import { useUserEvents } from '@/hooks/useUserEvents'; import { useCoderUserEvents } from '@/hooks/useCoderUserEvents'; +import { sessionEvents } from '@/hooks/sessionEvents'; import { useTheme } from '@/lib/theme'; import { SidebarDrawerProvider, useSidebarDrawer } from '@/hooks/useSidebarDrawer'; import { RightRailDrawerProvider, useRightRailDrawer } from '@/hooks/useRightRailDrawer'; @@ -77,6 +79,24 @@ function AppShell() { useTheme(); useUserEvents(); useCoderUserEvents(); + useEffect(() => { + const unsub = sessionEvents.subscribe((event) => { + if (event.type === 'collision_warning') { + toast.warning(`Multiple agents editing ${event.file_path}`, { + description: `Agents: ${event.agents.join(', ')}`, + }); + } else if (event.type === 'agent_message') { + const truncated = + event.content.length > 80 + ? event.content.slice(0, 80) + '…' + : event.content; + toast.info(`Message from ${event.from_agent}`, { + description: truncated, + }); + } + }); + return unsub; + }, []); const [showShortcuts, setShowShortcuts] = useState(false); useEffect(() => { const handler = (e: KeyboardEvent) => { diff --git a/apps/web/src/lib/tool-utils.ts b/apps/web/src/lib/tool-utils.ts index f8c4d4e..45c1d23 100644 --- a/apps/web/src/lib/tool-utils.ts +++ b/apps/web/src/lib/tool-utils.ts @@ -7,14 +7,6 @@ export const BUILT_IN_TOOLS = new Set([ 'find_files', 'git_status', 'skill_use', - 'get_codebase_overview', - 'get_file_analysis', - 'get_symbol_info', - 'search_symbols', - 'get_dependencies', - 'watch_changes', - 'get_semantic_neighborhoods', - 'get_framework_analysis', ]); /** diff --git a/apps/web/src/pages/Memory.tsx b/apps/web/src/pages/Memory.tsx index 76a0fe8..0e7afd6 100644 --- a/apps/web/src/pages/Memory.tsx +++ b/apps/web/src/pages/Memory.tsx @@ -405,7 +405,19 @@ export function Memory() { Topic-based memories, daily logs, and dream consolidation diaries.

- + + {/* Project selector */} + {/* Tab bar */} diff --git a/codecontext/.codecontextignore.template b/codecontext/.codecontextignore.template deleted file mode 100644 index 9a99c56..0000000 --- a/codecontext/.codecontextignore.template +++ /dev/null @@ -1,33 +0,0 @@ -# .codecontextignore — paths codecontext skips during analysis -# Copy to your project root and customize. Same syntax as .gitignore. - -# Dependencies / vendored code -node_modules/ -vendor/ -.venv/ -venv/ -__pycache__/ -target/ - -# Build artifacts -dist/ -build/ -out/ -.next/ -.nuxt/ -.svelte-kit/ - -# IDE / tooling -.opencode/ -.vscode/ -.idea/ - -# Test artifacts / coverage -coverage/ -.nyc_output/ -.pytest_cache/ - -# Lock files (rarely have meaningful symbols) -package-lock.json -yarn.lock -pnpm-lock.yaml diff --git a/codecontext/README.md b/codecontext/README.md deleted file mode 100644 index 1de758e..0000000 --- a/codecontext/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# codecontext — Go sidecar (DEPRECATED) - -> **Deprecated** (Phase 4, Domain 2, v2.8.14). -> -> Superseded by the **boocontext MCP server** (`apps/coder`). Do not add new -> callers. The 16 codecontext tool wrappers still use this sidecar via HTTP at -> `http://codecontext:8080/v1/{toolName}` for backward compatibility. - -## Migration path - -1. Existing tool wrappers in `apps/server/src/services/tools/codecontext/` route - through `callCodecontext()` in `codecontext_client.ts`, which calls this - Go sidecar over HTTP. -2. New callers should use the boocontext MCP server instead (reachable via the - `boocontext` tool wrappers). -3. After all callers have migrated, remove this directory, the `codecontext` - service block from `docker-compose.yml`, and the - `codecontext_client.ts`/`factory.ts` files. - -## What it does - -A Go HTTP shim wrapping the boocontext MCP server's stdio interface. Provides -code-graph analysis (symbols, callers, callees, file overview, etc.) over a -REST API at `/v1/{toolName}`. - -## Files - -- `shim.go` — HTTP server that wraps the boocontext MCP stdio process -- `Dockerfile` — container build -- `fork.tar.gz` — vendored boocontext source (gitignored) -- `.codecontextignore.template` — default ignore patterns deployed per project diff --git a/codecontext/go.mod b/codecontext/go.mod deleted file mode 100644 index 9a38632..0000000 --- a/codecontext/go.mod +++ /dev/null @@ -1,3 +0,0 @@ -module github.com/indifferentketchup/boocode-codecontext-shim - -go 1.24 diff --git a/codecontext/openspec/codesight-merge.md b/codecontext/openspec/codesight-merge.md deleted file mode 100644 index 379be7b..0000000 --- a/codecontext/openspec/codesight-merge.md +++ /dev/null @@ -1,90 +0,0 @@ -# codecontext — codesight feature merge - -Port codesight's highest-value analysis capabilities into codecontext as 4 new MCP tools. All work in `/opt/forks/codecontext` (Go). BooCode wrapper tools in a follow-up batch. - -## New tools - -### 1. `get_blast_radius` (Tier 1) - -**Input:** `file_path` (required), `target_dir` (optional) -**Output:** markdown listing all files, routes, and symbols that depend (transitively) on the given file. - -Algorithm: build a reverse adjacency map from `s.graph.Edges` (filter by `type == "imports"`), then BFS outward from the target file's node. Report each affected file with its symbol count and distance from the source. - -Codesight reference: `detectors/blast-radius.ts` (128 lines). The Go port is simpler — codecontext already has the edge graph; codesight had to build its own. - -~50 lines of Go (handler + BFS). - -### 2. `get_hot_files` (Tier 1) - -**Input:** `target_dir` (optional), `limit` (optional, default 20) -**Output:** ranked list of most-imported files with import count. - -Algorithm: count incoming `"imports"` edges per file node. Sort descending. Return top N. - -Codesight reference: `detectors/graph.ts` hot-files metric. codecontext's `identifyHotspotFiles()` at `relationships.go:286` already computes this — the tool just needs to expose it. - -~30 lines of Go (handler + sort). - -### 3. `get_routes` (Tier 2) - -**Input:** `target_dir` (optional), `framework` (optional filter — "fastify", "express", etc.) -**Output:** structured list of HTTP routes with method, path, file, line number, middleware, tags. - -Algorithm: for each TypeScript/JavaScript file in the graph, re-parse the AST via `gb.parser.ParseFile()` and walk the tree for call expressions matching framework-specific patterns: - -**Fastify patterns** (primary — Sam's stack): -- `app.get('/path', handler)` / `app.post(...)` / etc. -- `app.route({ method: 'GET', url: '/path', handler })` (object form) -- `app.register(plugin)` (plugin registration — note but don't trace into) - -**Express patterns** (secondary — common in analyzed projects): -- `router.get('/path', ...middleware, handler)` -- `app.use('/prefix', router)` - -Tag inference: scan handler body for common patterns (SQL queries → `db` tag, auth checks → `auth` tag, cache reads → `cache` tag). Simplified version of codesight's 30-framework tagger — only Fastify + Express for now. - -Codesight reference: `detectors/routes.ts` (1969 lines) + `ast/extract-routes.ts` (14690 lines). The Go port is ~200 lines targeting only 2 frameworks. - -### 4. `get_middleware` (Tier 2) - -**Input:** `target_dir` (optional) -**Output:** list of detected middleware with type (auth, cors, rate-limit, validation, error-handler, logging), file, line. - -Algorithm: for each file, scan for common middleware registration patterns: -- `app.register(fastifyCors, ...)` → CORS -- `app.addHook('preHandler', authCheck)` → auth -- `app.setErrorHandler(...)` → error-handler -- Import-name heuristics: `@fastify/cors` → CORS, `@fastify/rate-limit` → rate-limit - -Codesight reference: `detectors/middleware.ts` (217 lines). Go port: ~80 lines, Fastify-focused. - -## Architecture - -All 4 tools register in `internal/mcp/server.go:registerTools()` following the existing pattern (`mcp.AddTool`). - -Tools 1-2 (blast radius, hot files) operate on the existing `CodeGraph` — no re-parsing needed. They read `s.graph.Edges` and `s.graph.Files` under `s.graphMu.RLock()`. - -Tools 3-4 (routes, middleware) need AST access. The current pipeline discards ASTs after symbol extraction. Two options: -- **(a) Re-parse on demand:** when `get_routes` is called, iterate TypeScript files in `s.graph.Files`, call `s.analyzer.parser.ParseFile()` for each, walk the AST. Slower but no structural change. -- **(b) Cache route/middleware data during analysis:** modify `processFile()` in `graph_analysis.go` to extract routes alongside symbols, store in a new `FileNode.Routes` field. Faster on repeated calls but requires graph-builder changes. - -**Recommendation: (a) for this batch.** Re-parse is acceptable because route extraction runs on human timescale (one tool call, not per-token), and most projects have <50 route files. Optimize to (b) later if needed. - -New Go files: -- `internal/mcp/blast_radius.go` — handler + BFS -- `internal/mcp/hot_files.go` — handler + sort -- `internal/mcp/routes.go` — handler + AST route extraction for Fastify + Express -- `internal/mcp/middleware.go` — handler + middleware pattern detection - -## Hard rules - -- Go code. Tree-sitter for AST parsing (already in the project). -- No new Go deps (tree-sitter + MCP SDK already present). -- `go build ./...` clean. `go test ./...` passing. -- Test coverage: at least one test per new tool exercising the happy path. -- Don't modify existing tool behavior. - -## Estimate - -~400 lines of Go across 4 new files + registration in server.go. Blast radius and hot files are trivial (graph queries). Routes and middleware are the bulk (AST walking + pattern matching). diff --git a/codecontext/shim.go b/codecontext/shim.go deleted file mode 100644 index 3a38f28..0000000 --- a/codecontext/shim.go +++ /dev/null @@ -1,447 +0,0 @@ -// boocode-codecontext-shim — wraps codecontext's stdio MCP server with an -// HTTP/JSON facade so the BooCode Node server can call codecontext over the -// container network instead of speaking MCP directly. One process per -// container, holds a single codecontext child via os/exec; concurrent HTTP -// requests are serialized onto the child because codecontext's internal -// CodeContextMCPServer.graph swaps per target_dir (see recon report -// 2026-05-21). -// -// MCP framing is newline-delimited JSON (NDJSON), not LSP-style -// Content-Length — per the MCP stdio transport spec: -// https://spec.modelcontextprotocol.io/specification/server/transports -// -// No third-party deps. Stdlib only. - -package main - -import ( - "bufio" - "context" - "encoding/json" - "errors" - "fmt" - "io" - "log" - "net/http" - "os" - "os/exec" - "os/signal" - "strings" - "sync" - "sync/atomic" - "syscall" - "time" -) - -// ---- JSON-RPC types ---- - -// rpcMessage is shared by request, response, and notification. Notifications -// omit ID; requests omit Result/Error; responses omit Method/Params. omitempty -// + the zero int 0 sentinel works for ID because we never SEND id=0 -// (nextID starts at 0 and atomic.AddInt32 returns 1 on the first call). -type rpcMessage struct { - JSONRPC string `json:"jsonrpc"` - ID int `json:"id,omitempty"` - Method string `json:"method,omitempty"` - Params json.RawMessage `json:"params,omitempty"` - Result json.RawMessage `json:"result,omitempty"` - Error *rpcError `json:"error,omitempty"` -} - -type rpcError struct { - Code int `json:"code"` - Message string `json:"message"` -} - -// callToolResult is the MCP tools/call response shape. codecontext returns -// markdown wrapped in a TextContent entry. -type callToolResult struct { - Content []struct { - Type string `json:"type"` - Text string `json:"text"` - } `json:"content"` - IsError bool `json:"isError,omitempty"` -} - -// ---- Globals ---- - -var ( - child *exec.Cmd - childStdin io.WriteCloser - childStdout *bufio.Reader - - // Serialize tools/call so codecontext's per-call graph rebuild doesn't - // race itself when concurrent HTTP requests target different projects. - // Initialize/notifications/initialized run before HTTP starts so they - // don't need this lock. - callMu sync.Mutex - - pendingMu sync.Mutex - pending = make(map[int]chan *rpcMessage) - - nextID int32 -) - -// ---- MCP framing (NDJSON) ---- - -func writeMessage(w io.Writer, msg *rpcMessage) error { - body, err := json.Marshal(msg) - if err != nil { - return err - } - // Single write keeps the message atomic across concurrent writers. - // (We don't actually have concurrent writers here — callMu serializes — - // but the +'\n' append needs to be in one syscall regardless.) - _, err = w.Write(append(body, '\n')) - return err -} - -func readerLoop(r *bufio.Reader) { - for { - line, err := r.ReadBytes('\n') - if err != nil { - if errors.Is(err, io.EOF) { - log.Printf("reader: EOF (child closed stdout)") - } else { - log.Printf("reader: %v", err) - } - return - } - var msg rpcMessage - if err := json.Unmarshal(line, &msg); err != nil { - log.Printf("reader: malformed JSON: %v (line=%q)", err, line) - continue - } - if msg.ID == 0 { - // Server-initiated notification or progress update; nothing to - // dispatch. codecontext doesn't currently send these but the - // MCP spec allows them. - continue - } - pendingMu.Lock() - ch, ok := pending[msg.ID] - if ok { - delete(pending, msg.ID) - } - pendingMu.Unlock() - if ok { - ch <- &msg - } - } -} - -func call(ctx context.Context, method string, params any) (*rpcMessage, error) { - id := int(atomic.AddInt32(&nextID, 1)) - ch := make(chan *rpcMessage, 1) - pendingMu.Lock() - pending[id] = ch - pendingMu.Unlock() - - paramsJSON, err := json.Marshal(params) - if err != nil { - pendingMu.Lock() - delete(pending, id) - pendingMu.Unlock() - return nil, err - } - - msg := &rpcMessage{ - JSONRPC: "2.0", - ID: id, - Method: method, - Params: paramsJSON, - } - - if err := writeMessage(childStdin, msg); err != nil { - pendingMu.Lock() - delete(pending, id) - pendingMu.Unlock() - return nil, fmt.Errorf("write: %w", err) - } - - select { - case resp := <-ch: - return resp, nil - case <-ctx.Done(): - pendingMu.Lock() - delete(pending, id) - pendingMu.Unlock() - return nil, ctx.Err() - } -} - -func notify(method string, params any) error { - paramsJSON, err := json.Marshal(params) - if err != nil { - return err - } - msg := &rpcMessage{ - JSONRPC: "2.0", - Method: method, - Params: paramsJSON, - } - return writeMessage(childStdin, msg) -} - -// ---- Child lifecycle ---- - -func startChild() error { - // Support CODECONTEXT_CHILD env var for overriding the MCP child command. - // Default to boocontext (Node.js MCP aggregator). Set in docker-compose. - childCmd := os.Getenv("CODECONTEXT_CHILD") - if childCmd == "" { - childCmd = "node /usr/local/lib/boocontext/dist/index.js" - } - parts := strings.Split(childCmd, " ") - child = exec.Command(parts[0], parts[1:]...) - var err error - childStdin, err = child.StdinPipe() - if err != nil { - return fmt.Errorf("stdin pipe: %w", err) - } - stdout, err := child.StdoutPipe() - if err != nil { - return fmt.Errorf("stdout pipe: %w", err) - } - childStdout = bufio.NewReader(stdout) - // codecontext's own log.SetOutput(os.Stderr) keeps its diagnostic noise - // off the JSON-RPC channel; we just pass-through to our own stderr. - child.Stderr = os.Stderr - - if err := child.Start(); err != nil { - return fmt.Errorf("start: %w", err) - } - log.Printf("started codecontext pid=%d", child.Process.Pid) - - go readerLoop(childStdout) - - // Supervise the child. When codecontext exits (crash, OOM, externally - // pkill'd), child.Wait() returns and we tear the shim down so the - // container's `restart: unless-stopped` policy recreates us with a - // fresh child. Without this goroutine the dead child becomes a zombie - // (Signal(0) on a zombie returns nil, so the health endpoint would lie) - // and HTTP requests would queue forever waiting on responses that will - // never come. Discovered during B.1 kill-restart testing. - go func() { - err := child.Wait() - log.Printf("codecontext exited: %v — shim shutting down", err) - os.Exit(1) - }() - return nil -} - -func killChild() { - if child == nil || child.Process == nil { - return - } - log.Printf("killing codecontext pid=%d", child.Process.Pid) - _ = child.Process.Signal(syscall.SIGTERM) - done := make(chan error, 1) - go func() { done <- child.Wait() }() - select { - case <-done: - log.Printf("codecontext exited") - case <-time.After(5 * time.Second): - log.Printf("codecontext did not exit on SIGTERM; sending SIGKILL") - _ = child.Process.Kill() - <-done - } -} - -// MCP handshake: client sends initialize, server replies, client follows -// with the notifications/initialized notification. After that, tools/call -// is accepted. -func initializeMCP(ctx context.Context) error { - initParams := map[string]any{ - "protocolVersion": "2024-11-05", - "capabilities": map[string]any{}, - "clientInfo": map[string]any{ - "name": "boocode-codecontext-shim", - "version": "0.1.0", - }, - } - resp, err := call(ctx, "initialize", initParams) - if err != nil { - return fmt.Errorf("initialize: %w", err) - } - if resp.Error != nil { - return fmt.Errorf("initialize error %d: %s", resp.Error.Code, resp.Error.Message) - } - if err := notify("notifications/initialized", map[string]any{}); err != nil { - return fmt.Errorf("notifications/initialized: %w", err) - } - log.Printf("MCP handshake complete (server result=%s)", string(resp.Result)) - return nil -} - -// ---- HTTP ---- - -func writeJSON(w http.ResponseWriter, status int, body any) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(status) - _ = json.NewEncoder(w).Encode(body) -} - -func handleHealth(w http.ResponseWriter, r *http.Request) { - if child == nil || child.Process == nil { - http.Error(w, "no child", http.StatusServiceUnavailable) - return - } - // Signal 0 doesn't actually deliver — it just returns an error if the - // process is gone. Cheaper than parsing /proc. - if err := child.Process.Signal(syscall.Signal(0)); err != nil { - http.Error(w, "child dead: "+err.Error(), http.StatusServiceUnavailable) - return - } - _, _ = io.WriteString(w, "ok") -} - -func makeToolHandler(toolName string) http.HandlerFunc { - return func(w http.ResponseWriter, r *http.Request) { - start := time.Now() - targetDir := "-" - status := "ok" - defer func() { - log.Printf("%s target_dir=%q duration_ms=%d status=%s", - toolName, targetDir, time.Since(start).Milliseconds(), status) - }() - - var args json.RawMessage - if err := json.NewDecoder(r.Body).Decode(&args); err != nil { - status = "bad_request" - writeJSON(w, http.StatusBadRequest, map[string]any{ - "result": nil, - "error": "invalid JSON body: " + err.Error(), - }) - return - } - - // Sniff target_dir purely for the access log; pass args through opaque. - var argsMap map[string]any - if json.Unmarshal(args, &argsMap) == nil { - if td, ok := argsMap["target_dir"].(string); ok { - targetDir = td - } - } - - ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second) - defer cancel() - - callMu.Lock() - resp, err := call(ctx, "tools/call", map[string]any{ - "name": toolName, - "arguments": args, - }) - callMu.Unlock() - - if err != nil { - status = "rpc_error" - writeJSON(w, http.StatusBadGateway, map[string]any{ - "result": nil, - "error": err.Error(), - }) - return - } - if resp.Error != nil { - status = "mcp_error" - writeJSON(w, http.StatusOK, map[string]any{ - "result": nil, - "error": resp.Error.Message, - }) - return - } - - var ctr callToolResult - if err := json.Unmarshal(resp.Result, &ctr); err != nil { - status = "parse_error" - writeJSON(w, http.StatusOK, map[string]any{ - "result": nil, - "error": "parse result: " + err.Error(), - }) - return - } - - // codecontext only emits text content. Concatenate (single-entry in - // practice, but the schema allows multiple). - var buf []byte - for _, c := range ctr.Content { - if c.Type == "text" { - buf = append(buf, c.Text...) - } - } - text := string(buf) - - if ctr.IsError { - status = "tool_error" - writeJSON(w, http.StatusOK, map[string]any{ - "result": nil, - "error": text, - }) - return - } - writeJSON(w, http.StatusOK, map[string]any{ - "result": text, - "error": nil, - }) - } -} - -// ---- main ---- - -func main() { - log.SetOutput(os.Stderr) - log.SetFlags(log.LstdFlags | log.Lmicroseconds) - log.Println("boocode-codecontext-shim starting") - - if err := startChild(); err != nil { - log.Fatalf("startChild: %v", err) - } - - initCtx, initCancel := context.WithTimeout(context.Background(), 30*time.Second) - if err := initializeMCP(initCtx); err != nil { - initCancel() - killChild() - log.Fatalf("initializeMCP: %v", err) - } - initCancel() - - sigChan := make(chan os.Signal, 1) - signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT) - - mux := http.NewServeMux() - // Go 1.22+ method-prefix routing. Any non-listed method → 405 automatically. - mux.HandleFunc("GET /health", handleHealth) - mux.HandleFunc("POST /v1/get_codebase_overview", makeToolHandler("get_codebase_overview")) - mux.HandleFunc("POST /v1/get_file_analysis", makeToolHandler("get_file_analysis")) - mux.HandleFunc("POST /v1/get_symbol_info", makeToolHandler("get_symbol_info")) - mux.HandleFunc("POST /v1/search_symbols", makeToolHandler("search_symbols")) - mux.HandleFunc("POST /v1/get_dependencies", makeToolHandler("get_dependencies")) - mux.HandleFunc("POST /v1/watch_changes", makeToolHandler("watch_changes")) - mux.HandleFunc("POST /v1/get_semantic_neighborhoods", makeToolHandler("get_semantic_neighborhoods")) - mux.HandleFunc("POST /v1/get_framework_analysis", makeToolHandler("get_framework_analysis")) - mux.HandleFunc("POST /v1/get_symbol_details", makeToolHandler("get_symbol_details")) - mux.HandleFunc("POST /v1/get_call_graph", makeToolHandler("get_call_graph")) - mux.HandleFunc("POST /v1/get_blast_radius", makeToolHandler("get_blast_radius")) - - server := &http.Server{ - Addr: ":8080", - Handler: mux, - ReadHeaderTimeout: 5 * time.Second, - } - - go func() { - log.Println("listening on :8080") - if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { - log.Fatalf("ListenAndServe: %v", err) - } - }() - - <-sigChan - log.Println("shutdown signal received") - - shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second) - _ = server.Shutdown(shutdownCtx) - shutdownCancel() - killChild() - log.Println("exit") -} diff --git a/docs/codecontext-ts-plan.md b/docs/codecontext-ts-plan.md deleted file mode 100644 index 81ba041..0000000 --- a/docs/codecontext-ts-plan.md +++ /dev/null @@ -1,742 +0,0 @@ -# Codecontext + TypeScript: recon and plan - -**Date:** 2026-05-22 -**Author:** read-only recon, evidence-first - -## Part A — Current codecontext usage in BooCode - -### A1. Server-side synthesis pipeline - -BooCode runs a **forced second-inference synthesis pass** after a model -emits any of three codecontext tool calls. The list is hard-coded: - -`/opt/boocode/apps/server/src/services/synthesisPipeline.ts:34-38` -```ts -export const SYNTHESIS_TOOLS: ReadonlySet = new Set([ - 'get_codebase_overview', - 'get_framework_analysis', - 'get_semantic_neighborhoods', -]); -``` - -The pipeline is triggered from the tool-phase, not by the model: -`/opt/boocode/apps/server/src/services/inference/tool-phase.ts:200-279`. -After tool-phase records the tool_call/tool_result rows it picks the first -synth-eligible entry, expands the inline-truncated head via tmpfs -(`readTruncation`), pulls top-N referenced files + project docs -(BOOCHAT.md, AGENTS.md, CONTEXT.md, *roadmap*.md), token-budgets to -32k chars/4 (`synthesisPipeline.ts:45-46`), streams a second model -inference with a 90s timeout (`synthesisPipeline.ts:50`), and either -emits a `kind='synthesis'` message-part or falls through to the -recursive turn on failure (`synthesisPipeline.ts:250-272`). - -The pipeline is **invoked once per turn that contains a SYNTHESIS_TOOLS -call** — at most one synthesis pass per turn (the loop picks the first -synth-eligible entry, `tool-phase.ts:256`). - -The codecontext tools themselves are HTTP wrappers over the sidecar: -`/opt/boocode/codecontext/shim.go:412-419` registers eight POST routes -(`/v1/get_codebase_overview` … `/v1/get_framework_analysis`). The shim -serialises calls under `callMu` and forwards JSON-RPC to a single -`codecontext mcp` child (`shim.go:194`, `shim.go:328-333`). The child -binary is built from `github.com/nmakod/codecontext` tag `v3.2.1` -(`/opt/boocode/codecontext/Dockerfile:18-22`), NOT from the local fork at -`/opt/forks/codecontext` (which is `github.com/nuthan-ms/codecontext`, -fork go.mod: `/opt/forks/codecontext/go.mod:1`). Container reports -`codecontext version dev` (recon: `docker exec boocode_codecontext -codecontext --version` returned `codecontext version dev / Build Date: -unknown / Git Commit: unknown`). - -Wrapper boundaries: - -- `/opt/boocode/apps/server/src/services/codecontext_client.ts:68-70` - hard timeout `REQUEST_TIMEOUT_MS = 30_000`, inline truncation - `TRUNCATION_LIMIT = 32_000`. -- Same file lines 80-95: realpath project + target_dir, reject any - target_dir that escapes the project root. The eight wrappers never - pass `target_dir` (`callCodecontext` injects it server-side, line 99). -- Lines 130-141 surface the upstream "content is empty" parser bug - (issue #37) with an actionable hint pointing at `.codecontextignore`. - -### A2. Agent-exposed tool surface - -Source of truth: `/opt/boocode/data/AGENTS.md` (six agents) plus the -`DEFAULT_TOOLS` fallback in -`/opt/boocode/apps/server/src/services/agents.ts:19-20` (every tool in -`ALL_TOOLS`). - -Per-agent codecontext exposure (cited from -`/opt/boocode/data/AGENTS.md:6,41,62,100,138,179`): - -| Agent | Codecontext tools exposed | -|---|---| -| Code Reviewer (line 3) | get_codebase_overview, get_dependencies, get_file_analysis, get_framework_analysis, get_semantic_neighborhoods, get_symbol_info, search_symbols, watch_changes | -| Debugger (line 38) | same eight | -| Refactorer (line 59) | same eight | -| Architect (line 97) | same eight | -| Security Auditor (line 135) | same eight | -| Prompt Builder (line 176) | **none** — `tools: [view_file, list_dir, grep, find_files]` | - -Every project-less or no-agent chat falls back to `DEFAULT_TOOLS` = -`ALL_TOOLS` (all 21 tools including the eight codecontext ones) -(`agents.ts:19-20,196`). The `BOOCODE_TOOLS` env var can narrow further -via `resolveToolTier()` (`tools.ts:712-732`): `core` (4 tools, no -codecontext) / `standard` (16, all eight codecontext) / `all` (21). -`STANDARD_TOOL_NAMES` includes all eight codecontext tools -(`tools.ts:719-732`). - -The eight codecontext tool registrations live in `tools.ts:653-660` and -are all marked read-only in `READ_ONLY_TOOL_NAMES` (`tools.ts:689-696`). - -### A3. Actual usage (DB) - -Tool-call frequency from `message_parts` (all-time; DB only has data -back to 2026-05-22 today — see "Claims I did not verify" for the -retention question): - -Query: `SELECT payload->>'name', COUNT(*) FROM message_parts WHERE -kind='tool_call' GROUP BY 1 ORDER BY 2 DESC` - -| Tool | Calls | Chats | -|---|---:|---:| -| view_file | 129 | — | -| grep | 81 | — | -| list_dir | 78 | — | -| find_files | 25 | — | -| **get_codebase_overview** | **24** | 23 | -| **search_symbols** | **8** | 5 | -| ask_user_input | 5 | 3 | -| `foo` (typo/invalid) | 4 | 2 | -| view_truncated_output | 4 | 2 | -| git_status | 3 | 2 | -| **get_file_analysis** | **3** | 1 | -| **get_framework_analysis** | **1** | 1 | -| `([^` (typo/invalid) | 1 | 1 | - -Codecontext-tool calls observed: **only 5 of 8** ever invoked -(`get_codebase_overview`, `search_symbols`, `get_file_analysis`, -`get_framework_analysis`, and `get_dependencies` does not appear). - -**Never called** (in the recorded window): `get_dependencies`, -`get_symbol_info`, `get_semantic_neighborhoods`, `watch_changes`. - -Per-call args sample (`mp.created_at` desc, last 12 calls; -recon-verified by query against message_parts): - -- `get_codebase_overview` invoked ~9 times in a row with - `{"include_stats":true}` — repeated overview fetches within minutes. -- `search_symbols` examples: `{"limit":20,"query":"Kind"}`, - `{"limit":20,"query":"SymbolKind"}`, - `{"limit":20,"query":"Kind","framework_type":"typescript"}`. -- `get_file_analysis` invoked 3 times in one chat with - `file_path` = `apps/server/src/services/inference.ts`, - `apps/server/src/services/inference/parts.ts`, - `apps/server/src/services/system-prompt.ts` — **all three failed** - with "File not found in graph" (see C3). - -### A4. Hang and drift correlation - -**Cohort analysis** (query against `messages` joined to chats that -ever used any codecontext tool): - -| Cohort | status | rows | -|---|---|---:| -| no_codecontext | complete | 24 | -| no_codecontext | cancelled | 1 | -| used_codecontext | complete | 191 | -| used_codecontext | streaming | 2 | -| used_codecontext | **failed** | **2** | - -Two failed assistant messages, both in chats that used codecontext. -Both have empty `content` — characteristic of a synth pass that aborted -before any deltas streamed (see `synthesisPipeline.ts:278-303`, -`markSynthFailed`). DB query: - -``` -SELECT id, status, created_at, LEFT(content,200) -FROM messages WHERE role='assistant' AND status IN ('failed','streaming') -``` -returned two `failed` rows with empty content at 2026-05-22 18:43:39 and -2026-05-22 19:59:56. The 18:43 failure correlates with the codecontext -sidecar log line `2026/05/22 18:44:10.842554 get_framework_analysis -target_dir=/opt/boocode duration_ms=30002 status=rpc_error` — a 30 s -timeout (`codecontext_client.ts:70`) under a `get_framework_analysis` -call (`synthesisPipeline.ts:34-38` would have triggered synthesis on -success — failure path skipped synthesis and surfaced the error). - -**Drift / format leakage:** the query -`SELECT * FROM messages WHERE role='assistant' AND (content LIKE -'%` as a *topic*, not actually emitting a tool call as -text. **One real drift case** at 2026-05-22 19:05:03 — content begins -"I need to investigate the codecontext fork to write this design -document. Let me start by reading the key files.\n\n…" — an Anthropic-format leak. This message is in a -chat that did use codecontext, but the drift evidence is too thin -(n=1) to claim a correlation. - -## Part B — TypeScript parsing gap - -### B1. TS-targeted workload - -Per-language breakdown of codecontext calls that target a specific -file or framework (DB query): - -| Language hint | Calls | -|---|---:| -| no file_path (overview/framework/symbol search) | 33 | -| ts/tsx | 3 | -| (no other extension observed) | — | - -The three TS-targeted calls were all `get_file_analysis` in a single -chat: `inference.ts`, `inference/parts.ts`, `system-prompt.ts`. **All -three failed** with `File not found in graph` (see C3 — relative path -mishandling). One `search_symbols` call carried -`framework_type=typescript` (Q="Kind"). - -So **TS is the actual workload** for narrow codecontext use; the rest -is whole-repo overview/framework analysis with no specific language -filter. - -### B2. Symbol recovery quality - -I called the live container against three load-bearing BooCode TS files -and compared the symbol list against a manual grep of top-level -declarations. - -**File 1: `/opt/boocode/apps/server/src/types/api.ts` (371 lines)** - -Manual count (grep `^(export )?(interface|type|const) `): -- interfaces: 36 -- top-level types: 15 -- top-level consts: 5 -- total significant: 56 - -Codecontext output (live HTTP call to -`http://codecontext:8080/v1/get_file_analysis`): - -```json -{ - "result": "# File Analysis: ...\n**Lines:** 372\n**Symbols:** 10\n\n## Symbols\n\n- **PROJECT_STATUSES** () - Line 2\n- **PROJECT_STATUSES** () - Line 2\n- **CHAT_STATUSES** () - Line 91\n..." -} -``` - -Total reported: 10 symbols, all five `*_STATUSES` consts duplicated -(line 2 appears twice, etc.). After regex-extracting names: - -- Unique symbols reported by codecontext: 8 (5 *_STATUSES consts + 3 - header strings `Language:`/`Lines:`/`Symbols:`) -- Interfaces / types found: **0 of 51**. -- Symbol-recovery rate: **5/56 = ~9%** (only the const arrays the JS - grammar understands). - -Specific misses checked against the actual file -(grep -nE on `/opt/boocode/apps/server/src/types/api.ts`): -- Line 5 `export interface Project` — MISSED -- Line 26 `export type SessionStatus` — MISSED -- Line 28 `export interface Session` — MISSED -- Line 47 `export type WorkspacePaneKind` — MISSED -- All 36 interface declarations and 15 type aliases — MISSED. - -**File 2: `/opt/boocode/apps/server/src/services/tools.ts` (763 lines)** - -Manual count: 47 top-level decls -(grep `^(export )?(interface|type|enum|namespace|const|function|class|async function) `). - -Codecontext output: **112 symbols** reported (but many are noise: -local function-scope variables, the literal token `"unknown"` from -type cast positions, even raw labels like `out:`). - -Python-extracted from result: 71 unique names. Cross-checked against -20 significant TS exports the file declares: - -- Found: `ListDirInput`, `READ_ONLY_TOOL_NAMES`, `CORE_TOOL_NAMES`, - `STANDARD_TOOL_NAMES` (4 / 20) -- **MISSED: `ToolDef`, `ViewFileInput`, `viewFile`, `listDir`, `grep`, - `findFiles`, `viewTruncatedOutput`, `gitStatus`, `skillFind`, - `skillUse`, `skillResource`, `askUserInput`, `ALL_TOOLS`, - `TOOLS_BY_NAME`, `resolveToolTier`, `toolJsonSchemas`** — every - exported `ToolDef<…>` named constant is missed because the JS - grammar can't parse the TS type annotation `: ToolDef<…>` that - precedes the `=` and bails out of recognising the const at - top-level. -- Symbol-recovery rate (significant): **4/20 = 20%**. - -**File 3: `/opt/boocode/apps/server/src/services/inference/stream-phase.ts` (482 lines)** - -Manual count: 5 top-level decls (2 are `export async function`, -1 interface, 1 type, 1 const). - -Codecontext output: 53 symbols extracted, but the first 20 are header -strings (`Language:`, `Lines:`, `Symbols:`), imports (`api.js`, -`model-context.js`, …), local function names from inside bodies -(`toolNameById`, `out:`, `hasTools`), and string literals -(`parts:`). Neither `streamCompletion` nor `executeStreamPhase` (the -two `export async function` declarations at lines 145, 346) appear in -the symbol list explicitly. - -**Aggregate:** across the three files, codecontext recovers -type/interface/enum symbols at effectively **0%**, and function/const -symbols at roughly **20%**. The 9596-symbol whole-repo overview is -heavily noise-padded. Generic type parameters and decorators were not -checked individually because they're a strict subset of the -already-broken case. - -### B3. Fork status - -**`docs/ts-bindings-design.md` does NOT exist.** Verified by -`ls /opt/forks/codecontext/docs/ts-bindings-design.md` → `No such file -or directory`. The `/opt/forks/codecontext/docs/` tree has 23 markdown -files; none mention TypeScript bindings work (greps under -`/opt/forks/codecontext/docs/` for `TypescriptLanguage|tree-sitter-tsx` -returned nothing beyond a CodeContext example in `HLD.md:831` and -config mentions in `ARCHITECTURE.md:297`). - -**go.mod dependencies (`/opt/forks/codecontext/go.mod:5-18`):** -- `github.com/tree-sitter/tree-sitter-javascript v0.23.1` (present) -- `github.com/tree-sitter/tree-sitter-typescript` — **NOT present**. - -**TS-as-JS fallback in `internal/parser/manager.go:72-79`:** -```go -// TypeScript - use JavaScript grammar as fallback until TypeScript bindings are fixed -// Both JS and TS have similar syntax and this provides basic parsing capability -tsLang := sitter.NewLanguage(javascript.Language()) -m.languages["typescript"] = tsLang - -tsParser := sitter.NewParser() -tsParser.SetLanguage(tsLang) -m.parsers["typescript"] = tsParser -``` - -The comment claims this provides "basic parsing capability". B2 shows -that interface/type recovery is effectively zero — the JS grammar does -not recognise `interface`, `type`, generic params, decorators, or even -TS-typed const declarations. - -**Downstream code IS prepared for TS-specific nodes.** In -`internal/parser/manager.go:746-765` `nodeToSymbolJS` already has -cases for `interface_declaration` and `type_alias_declaration`: -```go -case "interface_declaration", "interface": - return &types.Symbol{Type: types.SymbolTypeInterface, ...} -case "type_alias_declaration", "type_declaration": - return &types.Symbol{Type: types.SymbolTypeType, ...} -``` -These cases are dead code with the JS grammar — they only fire when -the parser is the TypeScript grammar. The fork already has the symbol -extraction wiring; it's just missing the grammar. - -**`SymbolType` is open (string), not an iota** — -`/opt/forks/codecontext/pkg/types/graph.go:14`: -```go -type SymbolType string -``` -with constants like `SymbolTypeInterface`, `SymbolTypeType`, -`SymbolTypeNamespace` already declared (`graph.go:16-48`). No code -changes needed there to add TS-aware symbol types. - -**Upstream `tree-sitter-typescript` Go bindings exist.** Context7 docs -for `/tree-sitter/tree-sitter-typescript` show the Go package -`github.com/tree-sitter/tree-sitter-typescript` exporting -`LanguageTypescript()` and `LanguageTSX()`: -```go -typescript := sitter.NewLanguage(tree_sitter_typescript.LanguageTypescript()) -tsx := sitter.NewLanguage(tree_sitter_typescript.LanguageTSX()) -``` -(Context7 query `/tree-sitter/tree-sitter-typescript`, -"Go bindings package name and how to import…", returned a working -sample.) - -**The fork (`/opt/forks/codecontext`) is not what runs in production.** -The deployed image is built from `github.com/nmakod/codecontext` tag -v3.2.1 (`/opt/boocode/codecontext/Dockerfile:18-22`). The fork is a -separate working tree at `/opt/forks/codecontext` on -`github.com/nuthan-ms/codecontext` (`/opt/forks/codecontext/go.mod:1`). -Any TS-grammar work landing in either repo requires a Dockerfile -update to point at the right source. - -**Fork HEAD:** `ba6b94c 2025-09-01 12:43:09 +0530 Merge pull request -#29 from nmakod/release-please--branches--main` — newer than the -deployed v3.2.1 tag but on the same upstream lineage. - -### B4. Existing TS-aware alternatives - -Searches in `/opt/boocode`: - -- `grep -rln 'ts-morph|@typescript/vfs|createCompilerHost' - /opt/boocode/apps` → **no matches** in source (only types). -- Only the `typescript` package is depended on - (`/opt/boocode/package.json`, `/opt/boocode/apps/booterm/package.json`, - `/opt/boocode/apps/server/package.json`, - `/opt/boocode/apps/web/package.json` — each declares - `"typescript": "^5.5.0"`). That's the tsc compiler, used for - building, not for runtime symbol extraction. -- No tool in `/opt/boocode/apps/server/src` parses TS at runtime for - any reason other than what codecontext provides. - -So BooCode has **no existing fallback** for TS symbol data: if -codecontext can't extract it, nobody else does. - -## Part C — Optimization opportunities - -### C1. Tool surface review - -Cross-referencing the agent whitelist (A2) with actual usage (A3): - -| Tool | Exposed to 5 agents? | Calls observed | Recommendation | -|---|---|---:|---| -| get_codebase_overview | yes | 24 | **Keep** — load-bearing, synth-triggering | -| search_symbols | yes | 8 | **Keep** — only viable TS query path | -| get_file_analysis | yes | 3 | **Keep** but fix relative-path bug (C3) | -| get_framework_analysis | yes | 1 | Low-use; **keep** for synth signalling | -| get_dependencies | yes | **0** | **Demote** — unused, considered for removal | -| get_symbol_info | yes | **0** | **Demote** — unused, considered for removal | -| get_semantic_neighborhoods | yes | **0** | **Demote** — unused, considered for removal | -| watch_changes | yes | **0** | **Remove** from agent whitelist — also pulled out of synthesis if currently kept | - -`watch_changes` in particular is a state-changing async tool with no -sensible LLM consumer (the model can't await fsnotify events). It -should not be in the 5 agents' whitelists; the synthesis pipeline only -calls 3 specific tools (`synthesisPipeline.ts:34-38`) so removing -`watch_changes` from agent whitelists does not affect the pipeline. - -`get_dependencies`, `get_symbol_info`, `get_semantic_neighborhoods` -are credible tools but the model never reaches for them — likely a -descriptions/discoverability issue. Either improve their tool -descriptions (the `.description` strings registered in -`tools/codecontext/*.ts`) or remove them from agent whitelists. - -### C2. Latency and token cost - -Latencies parsed from the codecontext sidecar access log -(`docker logs boocode_codecontext --since 24h | grep duration_ms=`): - -- Total calls observed: 40 in 24h -- Total time: 610,404 ms -- Avg: **15,260 ms per call** -- Min: 1,379 ms -- p50: 9,417 ms -- p90: 27,611 ms -- Max: 30,002 ms (= the 30 s rpc_error timeout) - -Sampled MCP-server log lines confirm overview rebuilds cost 2–8 s on -/opt/boocode (`6575 files, 115601 symbols, 1186758 chars markdown` -in 8.22 s). The shim's per-tool log shows the analysis dominates; -markdown serialization is sub-second. - -**Synthesis pipeline expansion** (from `docker logs boocode`): - -Five completed synthesis passes today, sample sizes: -- `originalChars` (truncated head shipped to synth): **32,078** in - every case (= the wrapper's 32 kB cap). -- `fullChars` (full overview after re-expansion from tmpfs): 83,406 / - 83,408 / 83,410 / 97,283 / 97,464. - -In other words, every overview is over the wrapper cap and synthesis -always pays a tmpfs round-trip to recover the full content for -reference-file extraction. The full content is *not* shipped to the -synth model (the truncated head is — `synthesisPipeline.ts:141`), so -the token-budget contract holds, but the synth still has to wait on -the file I/O. - -One synthesis timeout in the day (`synthesis pass timed out; falling -through to recursive turn`, chatId a74bfecb…, toolName -get_codebase_overview, 90 s after expansion completed — the synth -inference itself was too slow). The retry inside the same chat then -completed in 31 s with `files: 0` (no referenced files extracted), -suggesting the timeout repeated until reference extraction was -empty. - -I have no cache-hit statistics to report — the shim does not log -cache hits. The codecontext binary itself logs `Refreshing analysis -for codebase overview…` on every call (`[MCP] Refreshing analysis…` -appears for each `get_codebase_overview` in the sidecar log), so the -analysis is rebuilt per call. - -### C3. Failure modes - -Sidecar errors in the last 7 days -(`docker logs boocode_codecontext --since 168h | grep -E -"status=tool_error|content is empty|panic"`): - -1. **`content is empty` parser bug** — 2026-05-22 17:37:41 and - 17:43:41, both against `/opt/homelabhealth`, on - `frontend/node_modules/hono/dist/adapter/aws-lambda/types.js`. - The wrapper's `.codecontextignore` template installation - (`codecontext_client.ts:30-52`) didn't help because the file is - under `node_modules` which is supposedly in the template. Suggests - either the template hadn't been copied yet or the template's - ignore list doesn't cover the path. Each failed call cost ~25 s. -2. **Relative-path failures** — 2026-05-22 17:56:51 through 17:57:07 - (three back-to-back), all `get_file_analysis`: - ``` - [MCP] ERROR: File not found in graph: apps/server/src/services/inference.ts (available files: 6575) - ``` - The wrapper resolves `target_dir` to an absolute realpath - (`codecontext_client.ts:80-99`) but `file_path` is forwarded - unchanged. The codecontext binary's file index is keyed on - absolute paths (the 115,876-symbol overview reports absolute - paths). The model passed `apps/server/src/services/inference.ts` - and the binary couldn't find it. Each failure cost 8–24 s. -3. **30 s rpc_error timeout** — 2026-05-22 18:44:10 - (get_framework_analysis) and 19:38:06 (search_symbols vs - /opt/forks/codecontext). The shim's per-call context timeout is - 60 s (`shim.go:325`) but the wrapper aborts at 30 s - (`codecontext_client.ts:70`), so the client gives up before the - shim does — the call still runs to completion on the codecontext - side, wasting CPU. -4. **Panic in `searchSymbols`** — concurrent map iteration crash in - `internal/mcp/server.go:1305` (`getFilePathForSymbol`) under - `matchesFramework`, captured in - `docker logs boocode_codecontext --since 24h`: - ``` - internal/runtime/maps.fatal(...) - github.com/nuthan-ms/codecontext/internal/mcp.(*CodeContextMCPServer).getFilePathForSymbol(...) - /build/codecontext/internal/mcp/server.go:1305 - ``` - This is an upstream bug in v3.2.1 — concurrent map access without - a lock. The shim's `callMu` serialises *its* calls but the - codecontext binary itself appears to have internal concurrency - that hits this. - -**Pattern:** the 2 failed assistant messages in A4 align with the 30 s -rpc_error timeout (18:44:10) and one other failure window. Failed -turns leave empty `content` because synthesis aborts before any -deltas — the model never sees the codecontext error. - -## Part D — Plan - -### D1. Tool surface decisions - -**Title:** Trim agent codecontext exposure to the four tools that earn -their keep; demote the rest until evidence justifies them. - -**Why:** A3 shows 4 of 8 codecontext tools have zero observed calls, -and `watch_changes` (a fsnotify-coupled tool) has no LLM consumer. -The synthesis pipeline only auto-triggers on three tools -(`synthesisPipeline.ts:34-38`), so removing tools from agent -whitelists does not affect the server-side synth path. - -**Scope:** edit `/opt/boocode/data/AGENTS.md` lines 6, 41, 62, 100, -138 (Code Reviewer, Debugger, Refactorer, Architect, Security -Auditor) to drop `get_dependencies`, `get_symbol_info`, -`get_semantic_neighborhoods`, `watch_changes` from each `tools:` -array. Roughly 5 line edits. - -**Risk:** if there's a legitimate workflow not yet captured in 24 h -of DB data, dropping these tools removes that affordance. Mitigation: -keep them registered in `tools.ts` (the server-side wrappers stay) so -the synth pipeline can still call them if `SYNTHESIS_TOOLS` expands -later, and so the `BOOCODE_TOOLS=standard` tier continues to expose -them via the tier filter. Tests: `agents.test.ts`, `tools.test.ts`, -any agent-roundtrip tests. - -**Effort:** 30 min. - -**Sequence:** standalone. Unblocks D3 (smaller tool list = smaller -system prompt = better prompt-cache stability per `tools.ts:629-632`). - -### D2. TypeScript support path - -**Title:** Narrow the TS fork scope to "interfaces, types, enums, top- -level typed consts" — defer generics and decorators. - -**Why:** Evidence from B1 (3 TS-targeted calls — all -`get_file_analysis` — and 1 `search_symbols framework_type=typescript`) -shows TS is in the workload but at low volume. Evidence from B2 -shows symbol recovery is **~0% for interfaces/types and ~20% for -typed consts**. That gap is what actually breaks model behaviour: -when the model asks `get_file_analysis` for `api.ts` (which IS what -happened today) it gets 10 noise symbols and no `interface Project`, -`interface Session`, `type SessionStatus`. The narrow scope -(declarations only; skip generics, JSX, decorators) covers ~90% of -the recovered-symbol gap and is achievable with one new dependency -and one parser-init change. - -**Scope:** -1. `/opt/forks/codecontext/go.mod`: add - `github.com/tree-sitter/tree-sitter-typescript v0.23.x` to the - `require` block. -2. `/opt/forks/codecontext/internal/parser/manager.go:72-79`: - replace the JS-fallback init with - ```go - typescript "github.com/tree-sitter/tree-sitter-typescript/bindings/go" - ... - tsLang := sitter.NewLanguage(typescript.LanguageTypescript()) - m.languages["typescript"] = tsLang - tsxLang := sitter.NewLanguage(typescript.LanguageTSX()) - m.languages["tsx"] = tsxLang - ``` - Plus parser registrations. `nodeToSymbolJS` already handles - `interface_declaration` and `type_alias_declaration` (lines - 746-765) — no extraction code changes needed for the narrow scope. -3. `/opt/forks/codecontext/internal/parser/manager.go:357-395` - `detectLanguage` (skim verified to live around line 357): ensure - `.tsx` maps to `"tsx"` not `"typescript"`. Likely already correct - — verify. -4. Tests in `internal/parser/` — add TS-grammar fixtures (a small - `.ts` file with interface, type, enum) to assert recovery. -5. Update `/opt/boocode/codecontext/Dockerfile:18-22` to clone from - the fork instead of `github.com/nmakod/codecontext` v3.2.1 once - the TS-grammar branch lands. **Or** PR the change upstream first - if `nmakod/codecontext` is open to it. -6. Drop the fork's own `tree-sitter-javascript` dependency? No — - `tree-sitter-typescript` Go binding is separate and the JS - grammar is still needed for `.js`/`.jsx` files. - -Rough LoC: ~20 lines in manager.go, +1 line go.mod, +1 import, +1 -language-detect entry; ~50 lines of tests; ~5 lines in Dockerfile. - -**Risk:** TS grammar parses superset syntax; some TS files may now -hit `ERROR` nodes the JS grammar happily accepted. Mitigate by -keeping the JS grammar registered for `.js`/`.jsx` and not changing -JS handling. Regression risk lives in the codecontext-binary CI -(JS+TS combined corpus) — verify their existing tests still pass. -Tests to add: a fixture file containing each B2 missed symbol and a -manager_test that asserts the symbols are recovered. - -**Effort:** Phase A (grammar swap + tests + Dockerfile pin): 90 min -once a build-and-test loop is set up in the fork. - -**Sequence:** Blocked on a decision about whether to PR upstream -(`nmakod/codecontext`) or fork-and-deploy (`nuthan-ms/codecontext`). -Unblocks D3 (cleaner TS results = smaller noise in synthesis output -= smaller token cost). - -**Decision:** **Narrow**, not "drop" and not "full TS support". Drop -is wrong because TS *is* the workload (A2 + B1 show every agent and -the codebase under analysis are TS-heavy). Full Phase 3-4 TS support -(generics, decorators, full type queries) is overkill for current -usage — interface/type/enum recovery captures the model's actual -need. - -### D3. Synthesis pipeline optimizations - -**Title:** Reduce per-turn codecontext latency and cache the overview. - -**Why:** C2 shows avg 15.2 s per codecontext call and an overview -that rebuilds on every call. Synthesis always pays the 30 s wrapper -timeout when the codecontext binary panics (C3 case 4) or hangs. - -**Three sub-items:** - -D3a. **Cache the overview at the shim layer.** The shim already -serialises calls under `callMu` (`shim.go:74-77`). Add a per- -`target_dir` overview cache keyed on a directory-mtime hash, TTL ~60s. -Sub-second cache hits for repeated `get_codebase_overview` calls -(today shows ~9 in a single chat over a few minutes). -- File: `/opt/boocode/codecontext/shim.go` -- LoC: ~80 -- Effort: 90 min -- Risk: invalidation. Use the fastest cheap invalidator (mtime of - target_dir + a hash of the file count via `os.ReadDir`). On any - doubt, bypass cache. - -D3b. **Align wrapper and shim timeouts.** Wrapper 30 s -(`codecontext_client.ts:70`), shim ctx 60 s (`shim.go:325`). The -mismatch wastes CPU when the wrapper gives up but the shim keeps -running. Either drop the shim ctx to 30 s, or raise the wrapper -to 60 s (depending on which budget is right). Recommended: align -both to 45 s, abort upstream on wrapper cancel. -- LoC: 2 lines -- Effort: 30 min - -D3c. **Fix the relative-path bug in `get_file_analysis`.** The -wrapper resolves `target_dir` but not `file_path`. Three failures -in one chat today wasted 48 s of CPU. Fix: -- File: `/opt/boocode/apps/server/src/services/tools/codecontext/get_file_analysis.ts` - (and possibly the shared client at `codecontext_client.ts`). -- Have the wrapper resolve `file_path` against the realpath'd - project root before forwarding, mirroring `target_dir`. Error out - if the resolved path doesn't start with the project root. -- LoC: ~20 -- Effort: 60 min -- Risk: low — the model loses no affordance; absolute and relative - both work. -- Tests: `codecontext_client.test.ts`. - -**Sequence:** D3c is independent and high-ROI. D3a depends on -nothing. D3b is independent. Recommended order: D3c → D3b → D3a. - -### D4. Removal candidates - -1. **`watch_changes` agent exposure** (A3 + A2). Server-side handler - stays for completeness; it should not appear in agent - `tools:` arrays. Edit `/opt/boocode/data/AGENTS.md` lines 6, 41, - 62, 100, 138. -2. **The dead "csharp" comment-out block** in - `/opt/forks/codecontext/internal/parser/manager.go:146-152` — - delete-on-touch when D2 lands; not part of D2's core scope. -3. **The 3 zero-use codecontext tool exposures** — - `get_dependencies`, `get_symbol_info`, `get_semantic_neighborhoods`. - Same surgical edits as item 1. Consider keeping - `get_dependencies` on the Refactorer because the agent - description explicitly invokes "Use get_dependencies to map call - sites" (`AGENTS.md:92-93`); if the model isn't using it despite - the system-prompt nudge, the description in - `tools/codecontext/get_dependencies.ts` likely needs the same - verb-forward rewrite. - -## Claims I did not verify - -- **DB retention horizon.** All `message_parts` rows are dated - 2026-05-22. That could mean (a) the DB was wiped today, (b) the - schema/path moved today, or (c) the project is brand-new and 24 h - is genuinely the full history. The CLAUDE.md project context - references "v1.13.15-codecontext-synth" which is recent. To verify: - `docker exec boocode_db psql -U boocode -d boocode -c "SELECT - MIN(created_at), MAX(created_at), COUNT(*) FROM messages;"` then - cross-check against the BooCode roadmap's release dates. The 30-day - window in A3's query may simply not have older data to find. -- **Whether `nmakod/codecontext` v3.2.1 hosts the same - `nodeToSymbolJS` switch I read in the fork.** The fork at - `/opt/forks/codecontext` is `nuthan-ms/codecontext` per - go.mod. The deployed v3.2.1 is `nmakod/codecontext`. The Dockerfile - comment (`/opt/boocode/codecontext/Dockerfile:13-16`) says the - module path differs but "the tagged v3.2.1 source tree is the same - either way." To verify, clone - `https://github.com/nmakod/codecontext` at tag v3.2.1 and diff - `internal/parser/manager.go` against the fork — outside this - recon's read-only scope. -- **Whether `tree-sitter-typescript v0.23.x` Go bindings actually - build under the fork's `go 1.24.5` + Tree-sitter `v0.25.0` - combination.** Context7 docs confirm the *API exists*. Confirm by - `go get github.com/tree-sitter/tree-sitter-typescript@latest` - followed by `go build ./...` in a scratch worktree. -- **Whether the codecontext panic in `searchSymbols` is reproducible - on `/opt/boocode` or only on `/opt/forks/codecontext`** (the panic - was captured against target_dir `/opt/forks/codecontext`). Reproduce - via `docker exec boocode_codecontext wget -qO - - --post-data='{"target_dir":"/opt/boocode","query":"foo","limit":10}' - --header='Content-Type: application/json' - http://localhost:8080/v1/search_symbols`. -- **Cache hit rate of codecontext analysis (per call vs reused).** - The MCP-server log line `Refreshing analysis for codebase - overview…` suggests rebuild-every-call, but I did not confirm by - reading the codecontext source — only the deployed binary's log - output. To verify, read - `/opt/forks/codecontext/internal/mcp/server.go` around the - `Refreshing analysis…` log lines. -- **Drift correlation strength.** N=1 confirmed drift case is too - small to call a correlation with codecontext use. To raise the - signal: extend retention, re-query after a week of synthetic - load with and without codecontext tools. -- **Whether the synth pipeline's `truncated head only` ships fewer - tokens than a full inlined codecontext result would.** Today's - budget contract assumes yes (`synthesisPipeline.ts:138-145` - comment "Truncated head only — full content was used for - reference extraction above"). To verify: instrument the - per-pass `promptTokens` and compare against a one-off pass with - the full content. -- **The Architect/Code-Reviewer agents' system-prompt copy versus - actual tool usage.** AGENTS.md text claims agents will "Use - get_dependencies to map call sites" (line 92) and "Use - get_semantic_neighborhoods to find related components" - (line 132), but A3 shows neither is called. To verify whether the - model is ignoring the prompt or whether these agents simply - aren't being invoked, query - `SELECT s.name, COUNT(*) FROM sessions s JOIN chats c ON - c.session_id=s.id JOIN messages m ON m.chat_id=c.id WHERE - m.role='assistant' GROUP BY 1 ORDER BY 2 DESC;` and compare - named agents to chat counts.