WeKnora/migrations/versioned/000041_task_queue_and_wiki_...

17 lines
793 B
MySQL
Raw Normal View History

perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter The wiki ingest post-process pipeline OOM'd and ran for hours on KBs with ~40k documents. The dominant tail was a per-batch ListAllPages that pulled every page (multi-MB content blobs) into Go memory, plus a 24h-TTL Redis pending list whose data could be evicted before a long serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go on every batch. Lint loaded the full graph. The index page kept the entire wiki directory in its content column and rewrote the TOAST chunk on every ingest. None of these survive at 4w docs. This change reworks the write path end to end and pulls the durable- queue + dead-letter primitives out of wiki and into shared infrastructure that every asynq task type now benefits from. - migration 000041: task_pending_ops + task_dead_letters tables, plus three GIN indexes on wiki_pages (source_refs jsonb_path_ops, source_refs text fallback, lower(title) trgm). - TaskPendingOpsRepository + TaskDeadLetterRepository in internal/application/repository/task_queue.go: cursor-paginated list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped delete that refuses empty keys so a buggy caller can't wipe a KB's queue. - internal/middleware/asynqdl: writes a task_dead_letters row when an asynq task exhausts its retry budget. Payload-agnostic — a small probe struct extracts TenantID + scope hints across every existing payload type, so summary:generation / image:multimodal / faq_import / kb:clone / etc. all dead-letter without per-handler code. Best-effort: an insert failure never masks the underlying task error. Installed in router/task.go before the langfuse mw so it sees raw errors. - Pending queue moved Redis → PG (no TTL, restart-durable). Redis keeps just the active-batch lock and the delete tombstone. - In-batch retry budget (wikiMaxFailRetries=5) tracked via pendingRepo.IncrFailCount; over the cap the row is moved to task_dead_letters with task_type=wiki:ingest, related_id=knowledge id. Asynq retries (10) are handled by the new middleware. - Removed the per-batch ListAllPages. WikiBatchContext now carries lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with mutex-protected caches; reduce reaches for titles / summaries on demand instead of pre-loading the whole KB. - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef, which the new GIN index on source_refs serves as a Bitmap Index Scan instead of a sequential text LIKE. - Dedup pre-filter uses pg_trgm via FindSimilarPages (idx_wiki_pages_title_trgm). For each new entity/concept (and each of its aliases) we ask the DB for the top-K trigram-similar existing pages and union the results — bounded prompt size, no Go- side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter. - cleanDeadLinks / injectCrossLinks are scoped to the batch's affected slugs (a few dozen) instead of every page in the KB. Both use the new lite ListBySlugs / ExistsSlugs repo methods so they pull only slug + title + outlinks, not full content. - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits [[bad-slug|display]] the cleanup path now tries display-text reverse lookup → hyphen/case normalized equality → char-bigram Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word- break drift case ("shang-hai-tower" vs "shanghai-tower") in place instead of replacing the link with plain text. - rebuildIndexPage uses ListByTypeRecent(200) for the first-time intro and drops the full DocumentSummaries blob from the incremental update prompt, so its context stays bounded regardless of KB size. - Concurrency tunables surfaced in WikiConfig: IngestBatchSize / IngestMapParallel / IngestReduceParallel, with sensible defaults via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so follow-ups don't waste asynq retry slots bouncing on the active lock. - RunLint walks pages via the new ListPagesCursor in 200-page windows and computes the live-slug set with a one-column ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized every node + edge. Memory is now bounded; 40k pages walks in constant ~4MB. - 14 GORM tests for both repos against an in-memory SQLite mirror of the production DDL (task_queue_test.go). - 6 tests for the dead-letter middleware covering retry budget detection, payload-agnostic scope inference, error truncation, and repo-failure isolation. - 7 tests for the slug fuzzy resolve helper covering the three resolution stages, display-text priority over normalized equality, bigram fallback acceptance, and rejection of unrelated slugs. - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the new fetcher / cursor APIs.
2026-05-09 15:42:53 +00:00
-- Migration: 000041_task_queue_and_wiki_indexes (rollback)
-- Description: Drop the generic task queue tables and the new wiki_pages
-- GIN indexes. Pending rows are NOT migrated back to Redis on
-- rollback; operators are expected to drain the queue first
-- (or accept the loss of un-consumed ops).
DO $$ BEGIN RAISE NOTICE '[Migration 000041 DOWN] Reverting task queue + wiki indexes schema'; END $$;
DROP INDEX IF EXISTS idx_wiki_pages_title_trgm;
DROP INDEX IF EXISTS idx_wiki_pages_source_refs_text;
DROP INDEX IF EXISTS idx_wiki_pages_source_refs;
DROP TABLE IF EXISTS task_dead_letters;
DROP TABLE IF EXISTS task_pending_ops;
DO $$ BEGIN RAISE NOTICE '[Migration 000041 DOWN] task queue + wiki indexes schema reverted successfully'; END $$;