WeKnora/migrations/versioned/000056_knowledge_pending_su...

2 lines
69 B
MySQL
Raw Normal View History

feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
ALTER TABLE knowledges DROP COLUMN IF EXISTS pending_subtasks_count;