indifferentketchup/boocode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v1.1 batch 1: markdown, message actions, tok/s+ctx, AI naming

Browse Source 
Four features land together on this branch:

1. Markdown rendering — assistant messages go through react-markdown +
   remark-gfm. Fenced code blocks render via existing CodeBlock (with copy
   button); inline `code` is styled inline. User messages stay plain text.
   No raw HTML (no rehype-raw).

2. Per-message Copy + Regenerate. New endpoint
   POST /api/sessions/:id/messages/:message_id/regenerate validates the
   target (404/400/409), atomically deletes the target plus any later
   messages in the session, inserts a fresh streaming assistant row, and
   enqueues a normal inference run. The DELETE bound uses a SQL subquery
   (`created_at >= (SELECT created_at FROM messages WHERE id = $1)`)
   instead of a JS round-trip so postgres TIMESTAMPTZ µs precision is
   preserved — otherwise sub-ms clock_timestamp() differences between the
   user row and the assistant row collapsed to the same JS Date, pulling
   the triggering user message into the >= bound. New `messages_deleted`
   WS frame so already-connected clients prune the stale tail without
   needing a full snapshot resend.

3. tok/s + ctx counter. Five new nullable message columns: tokens_used,
   ctx_used, ctx_max, started_at, finished_at. started_at is set right
   before the OpenAI call in services/inference.ts (not in the route, not
   in the frame handler); finished_at + tokens_used + ctx_used + ctx_max
   are committed in the same UPDATE that flips status to 'complete'. The
   inference request now opts into stream_options.include_usage so the
   final chunk carries usage; defensive parsing also picks up timings.n_ctx
   when llama.cpp emits it (currently absent for our llama-swap models, so
   ctx_max stays NULL and the UI just shows `<used> ctx`). message_complete
   frame extended with tokens_used / ctx_used / ctx_max / started_at /
   finished_at / model. Frontend StatsLine in MessageBubble computes tok/s
   client-side from the timestamps and renders muted mono text below the
   body of completed assistant messages.

4. AI chat naming after the first turn. Backend services/auto_name.ts
   runs via setImmediate after the top-level inference resolves; it
   checks that there is exactly one completed assistant message and that
   the session has not been user-renamed (`name IS NULL OR name = '' OR
   name = 'New session'`), then fires a single non-streaming chat
   completion with the spec prompt. Qwen3 chat templates emit chain-of-
   thought into reasoning_content and burn the entire max_tokens budget
   without producing visible output, so the request includes
   `chat_template_kwargs: { enable_thinking: false }` and max_tokens=30.
   Title is trimmed, quote-stripped, "Title:" prefix dropped, and
   truncated to 60 chars before a guarded UPDATE on sessions.name. New
   `session_renamed` WS frame propagates to the open session view
   directly and to the project's session list via a tiny module-scope
   event bus (apps/web/src/hooks/sessionEvents.ts) — kept dumb: one event
   type, two methods, no library.

Cleanups: dropped the now-unused splitCodeBlocks export from CodeBlock.tsx
(react-markdown supersedes it), and added a long-form NOTE in auto_name.ts
documenting the enable_thinking + max_tokens pattern for any future Qwen-
family non-streaming utility calls (planned: fork-message, agent-routing,
web-search summarization).

Schema bootstrap remains idempotent (ADD COLUMN IF NOT EXISTS). Auth,
broker, clock_timestamp() conventions, and zod validation all unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
indifferentketchup
2026-05-14 22:52:40 +00:00
parent a7f218e182
commit 2464d23bb6
18 changed files with 1559 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

178
apps/server/src/services/inference.ts
View File

@@ -4,6 +4,7 @@ import type { Config } from '../config.js';
import type { Message, Project, Session, ToolCall } from '../types/api.js';
import { ALL_TOOLS, TOOLS_BY_NAME, toolJsonSchemas } from './tools.js';
import { PathScopeError, resolveProjectRoot } from './path_guard.js';
import { maybeAutoNameSession } from './auto_name.js';
const BASE_SYSTEM_PROMPT = (projectPath: string) =>
`You are BooCode Chat, a code investigation assistant. The user is working on a project located at ${projectPath}. Use the file-read tools (view_file, list_dir, grep, find_files) to investigate code when needed. Be concise. Cite file paths and line numbers when discussing code. Do not hallucinate file contents — read the file first. Tool results may be truncated; if so, narrow your query rather than guessing.`;
@@ -12,8 +13,17 @@ const DB_FLUSH_INTERVAL_MS = 500;
const MAX_TOOL_LOOP_DEPTH = 5;
export interface InferenceFrame {
type: 'message_started' | 'delta' | 'tool_call' | 'tool_result' | 'message_complete' | 'error';
type:
| 'message_started'
| 'delta'
| 'tool_call'
| 'tool_result'
| 'message_complete'
| 'messages_deleted'
| 'session_renamed'
| 'error';
message_id?: string;
message_ids?: string[];
tool_message_id?: string;
tool_call_id?: string;
role?: 'assistant' | 'tool' | 'user';
@@ -22,6 +32,14 @@ export interface InferenceFrame {
output?: unknown;
truncated?: boolean;
error?: string;
tokens_used?: number | null;
ctx_used?: number | null;
ctx_max?: number | null;
started_at?: string | null;
finished_at?: string | null;
model?: string;
session_id?: string;
name?: string;
}
export type FramePublisher = (sessionId: string, frame: InferenceFrame) => void;
@@ -49,13 +67,21 @@ interface ChatCompletionDelta {
}
interface ChatCompletionChunk {
choices: Array<{
choices?: Array<{
delta: ChatCompletionDelta;
finish_reason: string | null;
}>;
usage?: {
prompt_tokens?: number;
completion_tokens?: number;
total_tokens?: number;
};
timings?: {
n_ctx?: number;
};
}
interface InferenceContext {
export interface InferenceContext {
sql: Sql;
config: Config;
log: FastifyBaseLogger;
@@ -130,7 +156,8 @@ async function loadContext(
const project = projectRows[0]!;
const history = await sql<Message[]>`
SELECT id, session_id, role, content, tool_calls, tool_results, status, last_seq, created_at
SELECT id, session_id, role, content, tool_calls, tool_results, status, last_seq,
tokens_used, ctx_used, ctx_max, started_at, finished_at, created_at
FROM messages
WHERE session_id = ${sessionId}
ORDER BY created_at ASC, id ASC
@@ -162,14 +189,28 @@ async function* sseLines(stream: ReadableStream<Uint8Array>): AsyncGenerator<str
}
}
interface StreamResult {
finishReason: string | null;
content: string;
toolCalls: ToolCall[];
promptTokens: number | null;
completionTokens: number | null;
nCtx: number | null;
}
async function streamCompletion(
ctx: InferenceContext,
model: string,
messages: OpenAiMessage[],
includeTools: boolean,
onDelta: (content: string) => void
): Promise<{ finishReason: string | null; content: string; toolCalls: ToolCall[] }> {
const body: Record<string, unknown> = { model, messages, stream: true };
): Promise<StreamResult> {
const body: Record<string, unknown> = {
model,
messages,
stream: true,
stream_options: { include_usage: true },
};
if (includeTools) {
body['tools'] = toolJsonSchemas();
body['tool_choice'] = 'auto';
@@ -187,6 +228,9 @@ async function streamCompletion(
let content = '';
let finishReason: string | null = null;
let promptTokens: number | null = null;
let completionTokens: number | null = null;
let nCtx: number | null = null;
const toolCallsBuffer = new Map<number, { id: string; name: string; argsText: string }>();
for await (const line of sseLines(res.body)) {
@@ -199,6 +243,19 @@ async function streamCompletion(
} catch {
continue;
}
if (parsed.usage) {
if (typeof parsed.usage.prompt_tokens === 'number') {
promptTokens = parsed.usage.prompt_tokens;
}
if (typeof parsed.usage.completion_tokens === 'number') {
completionTokens = parsed.usage.completion_tokens;
}
}
if (parsed.timings && typeof parsed.timings.n_ctx === 'number') {
nCtx = parsed.timings.n_ctx;
}
const choice = parsed.choices?.[0];
if (!choice) continue;
const delta = choice.delta ?? {};
@@ -232,7 +289,7 @@ async function streamCompletion(
toolCalls.push({ id: t.id || `call_${toolCalls.length}`, name: t.name, args });
}
return { finishReason, content, toolCalls };
return { finishReason, content, toolCalls, promptTokens, completionTokens, nCtx };
}
async function executeToolCall(
@@ -279,7 +336,9 @@ async function runAssistantTurn(
if (depth > MAX_TOOL_LOOP_DEPTH) {
await ctx.sql`
UPDATE messages
SET status = 'failed', content = ${'tool loop depth exceeded'}
SET status = 'failed',
content = ${'tool loop depth exceeded'},
finished_at = clock_timestamp()
WHERE id = ${assistantMessageId}
`;
ctx.publish(sessionId, {
@@ -299,6 +358,14 @@ async function runAssistantTurn(
const projectRoot = await resolveProjectRoot(project.path);
const messages = buildMessagesPayload(session, project, history);
const startedRow = await ctx.sql<{ started_at: string }[]>`
UPDATE messages
SET started_at = clock_timestamp()
WHERE id = ${assistantMessageId}
RETURNING started_at
`;
const startedAt = startedRow[0]?.started_at ?? null;
ctx.publish(sessionId, {
type: 'message_started',
message_id: assistantMessageId,
@@ -328,12 +395,9 @@ async function runAssistantTurn(
}, DB_FLUSH_INTERVAL_MS);
};
let content = '';
let finishReason: string | null = null;
let toolCalls: ToolCall[] = [];
let result: StreamResult;
try {
const result = await streamCompletion(
result = await streamCompletion(
ctx,
session.model,
messages,
@@ -349,9 +413,6 @@ async function runAssistantTurn(
scheduleFlush();
}
);
content = result.content;
finishReason = result.finishReason;
toolCalls = result.toolCalls;
} catch (err) {
if (pendingFlushTimer) {
clearTimeout(pendingFlushTimer);
@@ -360,7 +421,9 @@ async function runAssistantTurn(
const errMsg = err instanceof Error ? err.message : String(err);
await ctx.sql`
UPDATE messages
SET status = 'failed', content = ${accumulated}
SET status = 'failed',
content = ${accumulated},
finished_at = clock_timestamp()
WHERE id = ${assistantMessageId}
`;
ctx.publish(sessionId, {
@@ -378,12 +441,22 @@ async function runAssistantTurn(
}
await flushPromise;
const { content, finishReason, toolCalls, promptTokens, completionTokens, nCtx } = result;
if (toolCalls.length > 0) {
await ctx.sql`
const [updated] = await ctx.sql<
{ tokens_used: number | null; ctx_used: number | null; ctx_max: number | null; finished_at: string | null }[]
>`
UPDATE messages
SET content = ${content}, status = 'complete',
tool_calls = ${ctx.sql.json(toolCalls as never)}
SET content = ${content},
status = 'complete',
tool_calls = ${ctx.sql.json(toolCalls as never)},
tokens_used = ${completionTokens},
ctx_used = ${promptTokens},
ctx_max = ${nCtx},
finished_at = clock_timestamp()
WHERE id = ${assistantMessageId}
RETURNING tokens_used, ctx_used, ctx_max, finished_at
`;
for (const tc of toolCalls) {
ctx.publish(sessionId, {
@@ -395,6 +468,12 @@ async function runAssistantTurn(
ctx.publish(sessionId, {
type: 'message_complete',
message_id: assistantMessageId,
tokens_used: updated?.tokens_used ?? null,
ctx_used: updated?.ctx_used ?? null,
ctx_max: updated?.ctx_max ?? null,
started_at: startedAt,
finished_at: updated?.finished_at ?? null,
model: session.model,
});
await Promise.all(
@@ -405,12 +484,12 @@ async function runAssistantTurn(
RETURNING id
`;
const toolMessageId = toolRow!.id;
const result = await executeToolCall(projectRoot, tc);
const tres = await executeToolCall(projectRoot, tc);
const stored = {
tool_call_id: tc.id,
output: result.output,
truncated: result.truncated,
...(result.error ? { error: result.error } : {}),
output: tres.output,
truncated: tres.truncated,
...(tres.error ? { error: tres.error } : {}),
};
await ctx.sql`
UPDATE messages
@@ -421,9 +500,9 @@ async function runAssistantTurn(
type: 'tool_result',
tool_message_id: toolMessageId,
tool_call_id: tc.id,
output: result.output,
truncated: result.truncated,
...(result.error ? { error: result.error } : {}),
output: tres.output,
truncated: tres.truncated,
...(tres.error ? { error: tres.error } : {}),
});
})
);
@@ -437,16 +516,40 @@ async function runAssistantTurn(
return;
}
await ctx.sql`
const [updated] = await ctx.sql<
{ tokens_used: number | null; ctx_used: number | null; ctx_max: number | null; finished_at: string | null }[]
>`
UPDATE messages
SET content = ${content}, status = 'complete'
SET content = ${content},
status = 'complete',
tokens_used = ${completionTokens},
ctx_used = ${promptTokens},
ctx_max = ${nCtx},
finished_at = clock_timestamp()
WHERE id = ${assistantMessageId}
RETURNING tokens_used, ctx_used, ctx_max, finished_at
`;
ctx.publish(sessionId, {
type: 'message_complete',
message_id: assistantMessageId,
tokens_used: updated?.tokens_used ?? null,
ctx_used: updated?.ctx_used ?? null,
ctx_max: updated?.ctx_max ?? null,
started_at: startedAt,
finished_at: updated?.finished_at ?? null,
model: session.model,
});
ctx.log.info({ sessionId, assistantMessageId, finishReason, chars: content.length }, 'inference complete');
ctx.log.info(
{
sessionId,
assistantMessageId,
finishReason,
chars: content.length,
tokens_used: updated?.tokens_used,
ctx_used: updated?.ctx_used,
},
'inference complete'
);
}
export async function runInference(
@@ -460,9 +563,18 @@ export async function runInference(
export function createInferenceRunner(ctx: InferenceContext) {
return {
enqueue(sessionId: string, assistantMessageId: string) {
void runInference(ctx, sessionId, assistantMessageId).catch((err) => {
ctx.log.error({ err }, 'unhandled inference error');
});
void (async () => {
try {
await runInference(ctx, sessionId, assistantMessageId);
setImmediate(() => {
void maybeAutoNameSession(ctx, sessionId).catch((err) => {
ctx.log.warn({ err, sessionId }, 'auto-name failed');
});
});
} catch (err) {
ctx.log.error({ err }, 'unhandled inference error');
}
})();
},
};
}