boocode/openspec/changes/archived/2026-06-07-memory-context-engineering/tasks.md

1. Module Scaffold & Data Schemas

• 1.1 Create memory-engine/ directory tree with all subdirectories and __init__.py files
• 1.2 Create config.py with MemoryConfig pydantic model (embedding, chunking, search, tier settings)
• 1.3 Create core/schemas.py with MemoryChunk, SearchResult, Fact, RunningSummary, ExtractedMemory data classes
• 1.4 Create utils/token_counter.py with tiktoken + char-fallback token counting
• 1.5 Create utils/namespace.py with NamespaceTemplate for runtime namespace resolution
• 1.6 Create utils/chunker.py with TextChunker (line-based, overlapping, configurable max_tokens)

2. Core Store: SQLite + FTS5 + Vector

• 2.1 Create core/store.py with MemoryStore — SQLite init with WAL mode, FTS5 tables, integrity checks
• 2.2 Implement create_chunks_table() with embedding BLOB storage, indexes, meta table
• 2.3 Implement create_fts5_tables() with standard unicode61 tokenizer + trigram tokenizer for CJK
• 2.4 Implement FTS5 triggers (AFTER INSERT/UPDATE/DELETE) for auto-sync
• 2.5 Implement save_chunk() / save_chunks_batch() with SQLite UPSERT (INSERT ... ON CONFLICT DO UPDATE)
• 2.6 Implement delete_by_path(), get_file_hash(), update_file_metadata()
• 2.7 Implement FTS5 self-healing: _fts5_state_inconsistent(), _fts5_shadow_corrupt(), reset_fts5()
• 2.8 Implement embedding encode/decode (float32 BLOB via numpy, struct fallback, legacy JSON fallback)
• 2.9 Implement get_stats() and close() methods

3. Hybrid Search

• 3.1 Implement search_vector() — numpy matrix cosine similarity with argpartition top-K (pure-Python fallback)
• 3.2 Implement FTS5 keyword search with BM25 scoring: _search_fts5(), _search_fts5_trigram()
• 3.3 Implement _search_like() — CJK (1+ chars) + ASCII word (3+ chars) with dynamic scoring
• 3.4 Implement search_keyword() — three-tier strategy (FTS5 → trigram FTS5 → LIKE)
• 3.5 Implement BM25 rank to score conversion (0.3 + 0.69 * abs(r)/(1+abs(r)))
• 3.6 Create core/hybrid_search.py with weighted merge (vector_weight, keyword_weight) + temporal decay
• 3.7 Implement _compute_temporal_decay(path, half_life=30) — exponential decay for dated files

4. LLM Memory Extraction

• 4.1 Create extraction/prompts.py with memory update system prompt (structured JSON output)
• 4.2 Create extraction/manager.py with MemoryUpdater — LLM fact extraction from conversation
• 4.3 Implement _prepare_update_prompt() — loads current memory, formats conversation, builds prompt
• 4.4 Implement _parse_memory_update_response() — JSON extraction from LLM response (handles fences/thinking)
• 4.5 Implement _apply_updates() — update user/history sections, add/remove facts, enforce max_facts
• 4.6 Implement create_fact(), update_fact(), delete_memory_fact() CRUD operations
• 4.7 Implement content deduplication (casefold comparison) and confidence threshold filtering
• 4.8 Implement upload-mention scrubbing from memory data

5. Tiered Consolidation

• 5.1 Create tiers/daily.py with DailyTier — lazy file creation, append-only writes with timestamped headers
• 5.2 Create tiers/context.py with ContextTier — short-term context window management with RunningSummary
• 5.3 Create tiers/core.py with CoreTier — wraps MemoryStore, manages MEMORY.md file
• 5.4 Create tiers/__init__.py with flush_messages() — context summarization + daily file append
• 5.5 Implement incremental summarization (initial summary, extend existing, RunningSummary tracking)
• 5.6 Create background/deep_dream.py with DeepDream — LLM-based MEMORY.md consolidation
• 5.7 Implement Deep Dream dedup (content-hash check), dream diary writing, empty-output guard

6. Background Processing Queue

• 6.1 Create background/queue.py with MemoryUpdateQueue — thread-safe, debounced, keyed by (thread, user, agent)
• 6.2 Implement add() with debounce timer reset, add_nowait() for immediate processing
• 6.3 Implement timer-triggered processing with rate limiting between updates
• 6.4 Implement signal detection: detect_correction(), detect_reinforcement() with pattern matching
• 6.5 Create background/__init__.py with flush_messages() — dedup + background thread LLM summarization
• 6.6 Support context_summary_callback for in-context injection of summaries

7. Agent Tools & Public API

• 7.1 Create tools/manage.py with manage_memory() — create/update/delete facts with namespace isolation
• 7.2 Create tools/search.py with search_memory() — hybrid search with query/filter/limit/offset
• 7.3 Implement __init__.py with MemoryEngine unified class: manage(), search(), flush(), dream(), format_for_injection()
• 7.4 Implement format_for_injection() — token-budgeted memory string for system prompts
• 7.5 Thread-safe singleton pattern for MemoryUpdateQueue and MemoryStore

8. Embedding Provider Interface

• 8.1 Create embedding/base.py with EmbeddingProvider ABC — embed_query(), embed_batch()
• 8.2 Create embedding/openai.py with OpenAIEmbeddingProvider implementation
• 8.3 Implement EmbeddingCache — per-session cache keyed by (provider, model, text_hash)
• 8.4 Create embedding/__init__.py with create_embedding_provider() factory

9. Integration Tests

• 9.1 Test short-term context summarization with token budget enforcement
• 9.2 Test long-term fact extraction with LLM mock
• 9.3 Test hybrid search: vector-only, keyword-only, and combined
• 9.4 Test tiered consolidation: flush → daily file → Deep Dream → MEMORY.md rewrite
• 9.5 Test background queue: debounce, dedup, async execution
• 9.6 Test namespace isolation: scoped searches across tenants
• 9.7 Test graceful degradation: no embeddings → keyword-only, no numpy → Python fallback
• 9.8 Test memory tools: create/update/delete/search round-trip