WeKnora/internal/application/service/knowledge_post_process.go

608 lines
24 KiB
Go
Raw Normal View History

package service
import (
"context"
"encoding/json"
"errors"
"fmt"
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
"sort"
"time"
"github.com/Tencent/WeKnora/internal/logger"
feat(observability): extend Langfuse tracing across asynq pipeline Previously the Langfuse integration only traced in-process HTTP requests (chat / search / eval), so file uploads and every downstream asynq task (document parse, chunk embedding, OCR/VLM, summary / question gen, wiki ingest, datasource sync, etc.) produced either disconnected shallow traces or no observation at all. This change threads one trace end-to-end: - tracer: add SPAN observation type and StartSpan; add ResumeTrace so a worker can attach to an upstream trace without emitting a duplicate trace-create; StartGeneration now auto-picks parentObservationId from ctx so nested trace -> span -> generation trees render correctly. - types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq payloads so trace_id / parent_obs_id / user_id / session_id serialise into every job. - langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal so the HTTP-layer trace survives the Redis hop. - langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either resumes the upstream trace or opens a standalone asynq.<type> trace for scheduled jobs, and wraps the handler in a SPAN with task metadata (id / queue / retry / payload_bytes) plus ERROR level on failure. - GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation / FAQ / wiki / datasource endpoints so the root trace actually starts. - Tests: tracer_test.go covers span nesting, error status, and ResumeTrace no-trace-create guarantee; asynq_test.go covers InjectTracing round-trip, middleware resume path, and standalone trace fallback. - Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types and documents the cross-process propagation model. No behavioural change when Langfuse is disabled (all new code paths are no-ops and carriers serialise to empty strings with omitempty).
2026-04-24 04:23:30 +00:00
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
"github.com/redis/go-redis/v9"
)
// KnowledgePostProcessService acts as an orchestrator for all post-processing tasks
// after a document has been parsed and split into chunks (including multimodal OCR/Caption).
type KnowledgePostProcessService struct {
knowledgeRepo interfaces.KnowledgeRepository
kbService interfaces.KnowledgeBaseService
chunkService interfaces.ChunkService
taskEnqueuer interfaces.TaskEnqueuer
pendingRepo interfaces.TaskPendingOpsRepository
redisClient *redis.Client
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
spanTracker SpanTracker
}
func NewKnowledgePostProcessService(
knowledgeRepo interfaces.KnowledgeRepository,
kbService interfaces.KnowledgeBaseService,
chunkService interfaces.ChunkService,
taskEnqueuer interfaces.TaskEnqueuer,
perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter The wiki ingest post-process pipeline OOM'd and ran for hours on KBs with ~40k documents. The dominant tail was a per-batch ListAllPages that pulled every page (multi-MB content blobs) into Go memory, plus a 24h-TTL Redis pending list whose data could be evicted before a long serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go on every batch. Lint loaded the full graph. The index page kept the entire wiki directory in its content column and rewrote the TOAST chunk on every ingest. None of these survive at 4w docs. This change reworks the write path end to end and pulls the durable- queue + dead-letter primitives out of wiki and into shared infrastructure that every asynq task type now benefits from. - migration 000041: task_pending_ops + task_dead_letters tables, plus three GIN indexes on wiki_pages (source_refs jsonb_path_ops, source_refs text fallback, lower(title) trgm). - TaskPendingOpsRepository + TaskDeadLetterRepository in internal/application/repository/task_queue.go: cursor-paginated list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped delete that refuses empty keys so a buggy caller can't wipe a KB's queue. - internal/middleware/asynqdl: writes a task_dead_letters row when an asynq task exhausts its retry budget. Payload-agnostic — a small probe struct extracts TenantID + scope hints across every existing payload type, so summary:generation / image:multimodal / faq_import / kb:clone / etc. all dead-letter without per-handler code. Best-effort: an insert failure never masks the underlying task error. Installed in router/task.go before the langfuse mw so it sees raw errors. - Pending queue moved Redis → PG (no TTL, restart-durable). Redis keeps just the active-batch lock and the delete tombstone. - In-batch retry budget (wikiMaxFailRetries=5) tracked via pendingRepo.IncrFailCount; over the cap the row is moved to task_dead_letters with task_type=wiki:ingest, related_id=knowledge id. Asynq retries (10) are handled by the new middleware. - Removed the per-batch ListAllPages. WikiBatchContext now carries lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with mutex-protected caches; reduce reaches for titles / summaries on demand instead of pre-loading the whole KB. - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef, which the new GIN index on source_refs serves as a Bitmap Index Scan instead of a sequential text LIKE. - Dedup pre-filter uses pg_trgm via FindSimilarPages (idx_wiki_pages_title_trgm). For each new entity/concept (and each of its aliases) we ask the DB for the top-K trigram-similar existing pages and union the results — bounded prompt size, no Go- side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter. - cleanDeadLinks / injectCrossLinks are scoped to the batch's affected slugs (a few dozen) instead of every page in the KB. Both use the new lite ListBySlugs / ExistsSlugs repo methods so they pull only slug + title + outlinks, not full content. - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits [[bad-slug|display]] the cleanup path now tries display-text reverse lookup → hyphen/case normalized equality → char-bigram Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word- break drift case ("shang-hai-tower" vs "shanghai-tower") in place instead of replacing the link with plain text. - rebuildIndexPage uses ListByTypeRecent(200) for the first-time intro and drops the full DocumentSummaries blob from the incremental update prompt, so its context stays bounded regardless of KB size. - Concurrency tunables surfaced in WikiConfig: IngestBatchSize / IngestMapParallel / IngestReduceParallel, with sensible defaults via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so follow-ups don't waste asynq retry slots bouncing on the active lock. - RunLint walks pages via the new ListPagesCursor in 200-page windows and computes the live-slug set with a one-column ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized every node + edge. Memory is now bounded; 40k pages walks in constant ~4MB. - 14 GORM tests for both repos against an in-memory SQLite mirror of the production DDL (task_queue_test.go). - 6 tests for the dead-letter middleware covering retry budget detection, payload-agnostic scope inference, error truncation, and repo-failure isolation. - 7 tests for the slug fuzzy resolve helper covering the three resolution stages, display-text priority over normalized equality, bigram fallback acceptance, and rejection of unrelated slugs. - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the new fetcher / cursor APIs.
2026-05-09 15:42:53 +00:00
pendingRepo interfaces.TaskPendingOpsRepository,
redisClient *redis.Client,
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
spanTracker SpanTracker,
) interfaces.TaskHandler {
return &KnowledgePostProcessService{
knowledgeRepo: knowledgeRepo,
kbService: kbService,
chunkService: chunkService,
taskEnqueuer: taskEnqueuer,
perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter The wiki ingest post-process pipeline OOM'd and ran for hours on KBs with ~40k documents. The dominant tail was a per-batch ListAllPages that pulled every page (multi-MB content blobs) into Go memory, plus a 24h-TTL Redis pending list whose data could be evicted before a long serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go on every batch. Lint loaded the full graph. The index page kept the entire wiki directory in its content column and rewrote the TOAST chunk on every ingest. None of these survive at 4w docs. This change reworks the write path end to end and pulls the durable- queue + dead-letter primitives out of wiki and into shared infrastructure that every asynq task type now benefits from. - migration 000041: task_pending_ops + task_dead_letters tables, plus three GIN indexes on wiki_pages (source_refs jsonb_path_ops, source_refs text fallback, lower(title) trgm). - TaskPendingOpsRepository + TaskDeadLetterRepository in internal/application/repository/task_queue.go: cursor-paginated list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped delete that refuses empty keys so a buggy caller can't wipe a KB's queue. - internal/middleware/asynqdl: writes a task_dead_letters row when an asynq task exhausts its retry budget. Payload-agnostic — a small probe struct extracts TenantID + scope hints across every existing payload type, so summary:generation / image:multimodal / faq_import / kb:clone / etc. all dead-letter without per-handler code. Best-effort: an insert failure never masks the underlying task error. Installed in router/task.go before the langfuse mw so it sees raw errors. - Pending queue moved Redis → PG (no TTL, restart-durable). Redis keeps just the active-batch lock and the delete tombstone. - In-batch retry budget (wikiMaxFailRetries=5) tracked via pendingRepo.IncrFailCount; over the cap the row is moved to task_dead_letters with task_type=wiki:ingest, related_id=knowledge id. Asynq retries (10) are handled by the new middleware. - Removed the per-batch ListAllPages. WikiBatchContext now carries lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with mutex-protected caches; reduce reaches for titles / summaries on demand instead of pre-loading the whole KB. - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef, which the new GIN index on source_refs serves as a Bitmap Index Scan instead of a sequential text LIKE. - Dedup pre-filter uses pg_trgm via FindSimilarPages (idx_wiki_pages_title_trgm). For each new entity/concept (and each of its aliases) we ask the DB for the top-K trigram-similar existing pages and union the results — bounded prompt size, no Go- side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter. - cleanDeadLinks / injectCrossLinks are scoped to the batch's affected slugs (a few dozen) instead of every page in the KB. Both use the new lite ListBySlugs / ExistsSlugs repo methods so they pull only slug + title + outlinks, not full content. - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits [[bad-slug|display]] the cleanup path now tries display-text reverse lookup → hyphen/case normalized equality → char-bigram Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word- break drift case ("shang-hai-tower" vs "shanghai-tower") in place instead of replacing the link with plain text. - rebuildIndexPage uses ListByTypeRecent(200) for the first-time intro and drops the full DocumentSummaries blob from the incremental update prompt, so its context stays bounded regardless of KB size. - Concurrency tunables surfaced in WikiConfig: IngestBatchSize / IngestMapParallel / IngestReduceParallel, with sensible defaults via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so follow-ups don't waste asynq retry slots bouncing on the active lock. - RunLint walks pages via the new ListPagesCursor in 200-page windows and computes the live-slug set with a one-column ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized every node + edge. Memory is now bounded; 40k pages walks in constant ~4MB. - 14 GORM tests for both repos against an in-memory SQLite mirror of the production DDL (task_queue_test.go). - 6 tests for the dead-letter middleware covering retry budget detection, payload-agnostic scope inference, error truncation, and repo-failure isolation. - 7 tests for the slug fuzzy resolve helper covering the three resolution stages, display-text priority over normalized equality, bigram fallback acceptance, and rejection of unrelated slugs. - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the new fetcher / cursor APIs.
2026-05-09 15:42:53 +00:00
pendingRepo: pendingRepo,
redisClient: redisClient,
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
spanTracker: spanTracker,
}
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
func (s *KnowledgePostProcessService) tracker() SpanTracker {
if s.spanTracker == nil {
return noopSpanTracker{}
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
return s.spanTracker
}
// finishRunningMultimodalStage closes the multimodal stage only when image
// work really ran and is still open. The canonical stage also exists when
// multimodal processing is disabled, but that row is already "skipped" and
// must not be rewritten to "done" with the postprocess queueing delay as its
// duration.
func (s *KnowledgePostProcessService) finishRunningMultimodalStage(
ctx context.Context,
knowledgeID string,
attempt int,
) {
mm := s.tracker().LookupStage(ctx, knowledgeID, attempt, types.StageMultimodal)
if mm == nil ||
mm.Kind != types.SpanKindStage ||
mm.Status != types.SpanStatusRunning {
return
}
s.tracker().EndSpan(ctx, mm, nil)
}
// Handle implements asynq handler for TypeKnowledgePostProcess.
func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Task) error {
var payload types.KnowledgePostProcessPayload
if err := json.Unmarshal(task.Payload(), &payload); err != nil {
return fmt.Errorf("unmarshal knowledge post process payload: %w", err)
}
logger.Infof(ctx, "[KnowledgePostProcess] Orchestrating post processing for knowledge: %s", payload.KnowledgeID)
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
if payload.Language != "" {
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
// Resolve attempt: payload carries it from the upstream stage, but
// fall back to the latest known attempt for compatibility with
// in-flight tasks queued before this code shipped.
attempt := payload.Attempt
if attempt <= 0 {
attempt = s.tracker().LatestAttempt(ctx, payload.KnowledgeID)
}
// Close the multimodal stage span (parent enqueued it as "running"
// and we never see the per-image fan-in here other than by reaching
// post-process). If the parent skipped multimodal entirely, the
// stage row will already be in "skipped" state and must remain so.
// Per-image success/failure counts are NOT
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
// aggregated here — the frontend already walks the children when
// rendering the multimodal stage detail and counts them itself,
// avoiding an extra query path.
s.finishRunningMultimodalStage(ctx, payload.KnowledgeID, attempt)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
postSpan := s.tracker().BeginStage(ctx, payload.KnowledgeID, attempt, types.StagePostProcess, nil)
// 1. Fetch Knowledge and KB
knowledge, err := s.knowledgeRepo.GetKnowledgeByIDOnly(ctx, payload.KnowledgeID)
if err != nil {
return fmt.Errorf("get knowledge %s: %w", payload.KnowledgeID, err)
}
if knowledge == nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Knowledge %s not found, aborting.", payload.KnowledgeID)
return nil
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
// Skip post-processing entirely when the knowledge has been cancelled
// by the user or marked for deletion. We must NOT enqueue summary /
// question / graph / wiki child tasks for an aborted knowledge. We
// MUST also close postSpan before returning, otherwise it stays in
// running state forever and the trace viewer shows an orange bar
// long after the user cancelled (the AbortAttempt sweep ran before
// we opened postSpan, so the sweep didn't catch this row).
switch knowledge.ParseStatus {
case types.ParseStatusCancelled, types.ParseStatusDeleting:
logger.Infof(ctx,
"[KnowledgePostProcess] Knowledge %s aborted (%s), skipping post-processing.",
payload.KnowledgeID, knowledge.ParseStatus,
)
s.tracker().SkipSpan(ctx, postSpan,
"knowledge "+knowledge.ParseStatus+" before postprocess started")
return nil
}
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, payload.KnowledgeBaseID)
if err != nil || kb == nil {
return fmt.Errorf("get knowledge base %s: %w", payload.KnowledgeBaseID, err)
}
processOverrides, _ := knowledge.ProcessOverrides()
eff := ResolveProcessConfig(kb, processOverrides)
// 2. Fetch all chunks
chunks, err := s.chunkService.ListChunksByKnowledgeID(ctx, payload.KnowledgeID)
if err != nil {
return fmt.Errorf("list chunks for knowledge %s: %w", payload.KnowledgeID, err)
}
// Gather all text-like chunks (including newly added OCR and Caption from multimodal tasks)
var textChunks []*types.Chunk
for _, c := range chunks {
if c.ChunkType == types.ChunkTypeText || c.ChunkType == types.ChunkTypeImageOCR || c.ChunkType == types.ChunkTypeImageCaption {
textChunks = append(textChunks, c)
}
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
// 3. Compute the enrichment subtask count up front so we can flip to
// "finalizing" with the right counter BEFORE spawning any subtasks.
// Each subtask handler atomically decrements pending_subtasks_count
// on its terminal exit; the row promotes itself to "completed" when
// the counter hits zero (see knowledgeRepository.FinalizeSubtask).
//
// Wiki ingest IS counted (as a single subtask): although it's a
// KB-scoped debounced batch, each upload enqueues exactly one
// per-knowledge op, and the batch worker calls FinalizeSubtask once
// when that op reaches a terminal state (mapped successfully or
// dead-lettered after exhausting retries). Counting it keeps the row
// in "finalizing" — i.e. shown as in-progress and still cancellable —
// until wiki generation actually finishes instead of flipping to
// completed while wiki runs minutes later. A wiki op that never
// drains is bounded by the housekeeping finalizing sweep.
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
willSpawnSummary := len(textChunks) > 0
willSpawnQuestion := willSpawnSummary && kb.NeedsEmbeddingModel() &&
eff.QuestionGenerationConfig.Enabled
willSpawnWiki := kb.IndexingStrategy.WikiEnabled && len(textChunks) > 0
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
// Question generation now fans out one subtask per plain text chunk
// (mirroring the graph-extract per-chunk pattern) so each chunk's LLM
// call retries / cancels / traces independently. We only target
// ChunkTypeText here — OCR / Caption chunks were never fed to question
// generation in the legacy whole-knowledge loop, so excluding them
// keeps behavior identical. Sorted by StartAt so the per-chunk
// context (prev / next) matches the legacy ordering.
var questionChunks []*types.Chunk
if willSpawnQuestion {
for _, c := range textChunks {
if c.ChunkType == types.ChunkTypeText {
questionChunks = append(questionChunks, c)
}
}
sort.Slice(questionChunks, func(i, j int) bool {
return questionChunks[i].StartAt < questionChunks[j].StartAt
})
}
// Question generation is batched: one subtask per window of
// questionGenChunkBatchSize text chunks (not one per chunk), so a
// huge document doesn't spawn thousands of tiny tasks. The counter
// must match exactly how many batch tasks we enqueue below.
questionBatchCount := (len(questionChunks) + questionGenChunkBatchSize - 1) / questionGenChunkBatchSize
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
graphChunkCount := 0
if eff.GraphEnabled {
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
graphChunkCount = len(textChunks)
}
expectedSubtasks := 0
if willSpawnSummary {
expectedSubtasks++
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
expectedSubtasks += questionBatchCount
if willSpawnWiki {
expectedSubtasks++
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
expectedSubtasks += graphChunkCount
feat: Implement comprehensive wiki feature for knowledge bases This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes: **Backend Changes:** - Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log) - Database schema: wiki_pages table with full migration support - WikiPageService: CRUD operations and page management - WikiPageRepository: GORM-based persistence layer - Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents * Summary page generation using LLM * Entity and concept extraction in a single LLM call * Synthesis opportunity detection * Index page rebuilding * Log page maintenance - Wiki boost feature: Enhance chat retrieval with wiki context - Wiki linting: Maintenance and validation utilities - Agent wiki tools: Enable agents to query and interact with wiki pages - Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks - Language support: Reuse existing middleware language infrastructure for LLM prompts **Frontend Changes:** - Wiki browser UI: View all wiki pages with filtering and search - Wiki API client: Knowledge base wiki management endpoints - Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model) - i18n updates: Support for English, Korean, Russian, and Chinese interfaces **Configuration:** - Container DI: Wire up all wiki services - Router: Register wiki API endpoints - Task handling: Support async wiki ingest tasks **Testing:** - Unit tests for wiki page types - Service layer tests - Endpoint tests for wiki operations - Integration tests with LLM mocking **Documentation:** - Language refactoring analysis and guides - Implementation completion reports - Quick reference guides for developers **Key Features:** ✅ LLM-powered wiki page generation from documents ✅ Multi-language support (9+ languages) ✅ Automatic extraction of entities and concepts ✅ Synthesis opportunity detection ✅ Index and log page maintenance ✅ Progressive wiki building across multiple documents ✅ Agent-based wiki interaction ✅ Chat retrieval enhancement with wiki context ✅ Full frontend UI for wiki browsing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 05:56:13 +00:00
// enteredFinalizing is set only when the processing-to-finalizing handoff
// actually seeded the counter. For Wiki-enabled knowledge, that handoff
// also persists the pending Wiki op in the same transaction.
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
enteredFinalizing := false
wikiSlotOwned := false
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
switch {
case knowledge.ParseStatus == types.ParseStatusFinalizing && kb.IndexingStrategy.WikiEnabled:
// A previous delivery may have persisted the Wiki op but failed to
// enqueue its KB-scoped trigger. Retry only the trigger: appending a
// second pending op would duplicate durable work and its finalizer.
if err := enqueueWikiIngestTrigger(ctx, s.taskEnqueuer, payload.TenantID, payload.KnowledgeBaseID); err != nil {
s.tracker().FailSpan(ctx, postSpan, "WIKI_TRIGGER_ENQUEUE_FAILED", err.Error(), err)
return fmt.Errorf("retry wiki ingest trigger: %w", err)
}
logger.Infof(ctx, "[KnowledgePostProcess] Re-enqueued wiki ingest trigger for %s", payload.KnowledgeID)
retryOutput := types.JSONMap{"retried_wiki_trigger": true}
s.tracker().EndSpan(ctx, postSpan, retryOutput)
s.tracker().FinalizeAttempt(ctx, payload.KnowledgeID, attempt,
types.SpanStatusDone, retryOutput, "", "")
return nil
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
case knowledge.ParseStatus != types.ParseStatusProcessing:
// The row was already in some other state (deleting / cancelled /
// failed / completed) when we arrived. Don't touch parse_status
// and don't spawn enrichment — the upstream that put the row in
// that state has already decided this attempt is over.
logger.Infof(ctx, "[KnowledgePostProcess] Knowledge %s is in %s, skipping enrichment fan-out.",
payload.KnowledgeID, knowledge.ParseStatus)
s.tracker().EndSpan(ctx, postSpan, types.JSONMap{
"skipped": "non_processing_status",
"observed_status": knowledge.ParseStatus,
})
s.tracker().FinalizeAttempt(ctx, payload.KnowledgeID, attempt,
types.SpanStatusDone, types.JSONMap{
"skipped": "non_processing_status",
"observed_status": knowledge.ParseStatus,
}, "", "")
return nil
case expectedSubtasks == 0:
// Nothing to enrich — fast path keeps the previous behavior so
// users without summary/question/graph see 'completed' immediately.
updates := map[string]interface{}{
"parse_status": types.ParseStatusCompleted,
"updated_at": time.Now(),
}
if len(textChunks) > 0 {
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
updates["summary_status"] = types.SummaryStatusNone
}
if err := s.knowledgeRepo.UpdateKnowledgeColumns(ctx, payload.KnowledgeID, updates); err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to mark %s completed (no subtasks): %v",
payload.KnowledgeID, err)
} else {
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
logger.Infof(ctx, "[KnowledgePostProcess] Knowledge %s marked completed (no enrichment subtasks).",
payload.KnowledgeID)
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
default:
// Flip processing to finalizing before fan-out so a parallel
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
// cancel/delete cannot race us into completed.
var promoted bool
var err error
if willSpawnWiki {
seeder, ok := s.pendingRepo.(interfaces.TaskPendingOpsFinalizingSeeder)
if !ok {
return errors.New("wiki post-process requires atomic finalizing handoff")
}
pendingOp, buildErr := newWikiIngestPendingOp(
ctx, payload.TenantID, payload.KnowledgeBaseID, payload.KnowledgeID,
)
if buildErr != nil {
return buildErr
}
promoted, err = seeder.SeedKnowledgeFinalizingWithPendingOp(
ctx, payload.KnowledgeID, expectedSubtasks, pendingOp,
)
wikiSlotOwned = promoted
} else {
promoted, err = s.knowledgeRepo.SetFinalizing(ctx, payload.KnowledgeID, expectedSubtasks)
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
if err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] SetFinalizing failed for %s: %v",
payload.KnowledgeID, err)
if willSpawnWiki {
return fmt.Errorf("seed finalizing with wiki pending op: %w", err)
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
}
if promoted {
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
enteredFinalizing = true
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
// Reflect summary status separately so the UI shows the
// summary as queued for users who already had it visible.
summaryStatus := types.SummaryStatusNone
if willSpawnSummary {
summaryStatus = types.SummaryStatusPending
}
if err := s.knowledgeRepo.UpdateKnowledgeColumn(ctx,
payload.KnowledgeID, "summary_status", summaryStatus); err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to update summary_status for %s: %v",
payload.KnowledgeID, err)
}
logger.Infof(ctx,
"[KnowledgePostProcess] Knowledge %s entered finalizing (pending_subtasks=%d).",
payload.KnowledgeID, expectedSubtasks)
} else {
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
// Row was no longer 'processing' (cancel / delete won the race).
// Skip enrichment entirely so we don't waste LLM quota on a row
// the user already abandoned.
logger.Infof(ctx,
"[KnowledgePostProcess] Knowledge %s no longer in processing, skipping enrichment fan-out.",
payload.KnowledgeID)
s.tracker().EndSpan(ctx, postSpan, types.JSONMap{
"skipped": "knowledge_no_longer_processing",
})
s.tracker().FinalizeAttempt(ctx, payload.KnowledgeID, attempt,
types.SpanStatusDone, types.JSONMap{
"skipped": "knowledge_no_longer_processing",
}, "", "")
return nil
}
}
// 4. Spawn Summary and Question Tasks
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
enqueuedSummary := false
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
enqueuedQuestionCount := 0
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
if willSpawnSummary {
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
enqueuedSummary = s.enqueueSummaryGenerationTask(ctx, payload, attempt)
if !enqueuedSummary {
_ = s.knowledgeRepo.UpdateKnowledgeColumn(
ctx, payload.KnowledgeID, "summary_status", types.SummaryStatusFailed,
)
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
if willSpawnQuestion {
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
// Create the postprocess.question grouping span up front so the
// per-batch subspans (enqueued just below, run later in their own
// workers) have a parent to nest under. It's begun and ended right
// here as a structural container — the batches extend past it,
// which the timeline renders with the wrapping outline bar.
if grp := s.tracker().BeginSubSpan(ctx, postSpan, postprocessQuestionGroupSpanName,
types.SpanKindSubSpan, types.JSONMap{
"batch_count": questionBatchCount,
"chunk_count": len(questionChunks),
"batch_size": questionGenChunkBatchSize,
}); grp != nil {
s.tracker().EndSpan(ctx, grp, types.JSONMap{
"batch_count": questionBatchCount,
"chunk_count": len(questionChunks),
})
}
enqueuedQuestionCount = s.enqueueQuestionGenerationTasks(ctx, payload, eff.QuestionGenerationConfig, attempt, questionChunks)
}
}
// 5. Spawn Graph RAG Tasks — only when graph indexing is enabled in IndexingStrategy
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
enqueuedGraphCount := 0
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
if graphChunkCount > 0 {
logger.Infof(ctx, "[KnowledgePostProcess] Spawning Graph RAG extract tasks for %d text-like chunks", len(textChunks))
for i, chunk := range textChunks {
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
ok, err := NewChunkExtractTask(ctx, s.taskEnqueuer, payload.TenantID, chunk.ID, kb.SummaryModelID,
payload.KnowledgeID, attempt, i)
if err != nil {
logger.Errorf(ctx, "[KnowledgePostProcess] Failed to create chunk extract task for %s: %v", chunk.ID, err)
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
if ok {
enqueuedGraphCount++
}
}
}
// 6. Schedule the Wiki trigger. The durable per-knowledge op already owns
// its finalizing slot because it was committed atomically with the state
// transition above. A trigger failure is returned so the post-process
// task retries only the trigger without double-accounting.
var wikiEnqueueErr error
if willSpawnWiki {
wikiEnqueueErr = enqueueWikiIngestTrigger(
ctx, s.taskEnqueuer, payload.TenantID, payload.KnowledgeBaseID,
)
if wikiEnqueueErr != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to enqueue wiki ingest for %s: %v",
payload.KnowledgeID, wikiEnqueueErr)
} else if wikiSlotOwned {
logger.Infof(ctx, "[KnowledgePostProcess] Enqueued wiki ingest task for %s", payload.KnowledgeID)
}
feat: Implement comprehensive wiki feature for knowledge bases This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes: **Backend Changes:** - Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log) - Database schema: wiki_pages table with full migration support - WikiPageService: CRUD operations and page management - WikiPageRepository: GORM-based persistence layer - Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents * Summary page generation using LLM * Entity and concept extraction in a single LLM call * Synthesis opportunity detection * Index page rebuilding * Log page maintenance - Wiki boost feature: Enhance chat retrieval with wiki context - Wiki linting: Maintenance and validation utilities - Agent wiki tools: Enable agents to query and interact with wiki pages - Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks - Language support: Reuse existing middleware language infrastructure for LLM prompts **Frontend Changes:** - Wiki browser UI: View all wiki pages with filtering and search - Wiki API client: Knowledge base wiki management endpoints - Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model) - i18n updates: Support for English, Korean, Russian, and Chinese interfaces **Configuration:** - Container DI: Wire up all wiki services - Router: Register wiki API endpoints - Task handling: Support async wiki ingest tasks **Testing:** - Unit tests for wiki page types - Service layer tests - Endpoint tests for wiki operations - Integration tests with LLM mocking **Documentation:** - Language refactoring analysis and guides - Implementation completion reports - Quick reference guides for developers **Key Features:** ✅ LLM-powered wiki page generation from documents ✅ Multi-language support (9+ languages) ✅ Automatic extraction of entities and concepts ✅ Synthesis opportunity detection ✅ Index and log page maintenance ✅ Progressive wiki building across multiple documents ✅ Agent-based wiki interaction ✅ Chat retrieval enhancement with wiki context ✅ Full frontend UI for wiki browsing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 05:56:13 +00:00
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
// Reconcile the seeded counter against what was actually enqueued.
// summary/question/graph each own a counted slot that ONLY their own
// task drains; a slot whose task was never enqueued (graph with NEO4J
// off, a transient enqueue/marshal failure, a nil enqueuer) has no owner
// and would otherwise strand the row in "finalizing". Release exactly the
// shortfall — each release is a clamped decrement that promotes the row to
// "completed" if it brings the counter to zero. Safe against fast workers:
// shortfall slots have no draining
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
// task, so total drains == seeded count regardless of ordering.
//
// Detached ctx: the same reasoning that motivates finalizeSubtaskDetached
// for terminal worker drains applies here. If the postprocess handler's
// ctx is cancelled (graceful shutdown, preempted worker) between SetFinalizing
// and this point, the seeded slots have NO other path to drain — every
// owning task either failed to enqueue or was never created. Riding a
// cancelled ctx would silently abort the releases and strand the row in
// "finalizing". The bound is per-call (matches the helper) so a wedged
// connection can't pin the goroutine for the whole serial loop.
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
if enteredFinalizing {
plannedOwned := questionBatchCount + graphChunkCount
if willSpawnSummary {
plannedOwned++
}
if willSpawnWiki {
plannedOwned++
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
actualOwned := enqueuedQuestionCount + enqueuedGraphCount
if enqueuedSummary {
actualOwned++
}
if wikiSlotOwned {
actualOwned++
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
if shortfall := plannedOwned - actualOwned; shortfall > 0 {
logger.Warnf(ctx,
"[KnowledgePostProcess] Releasing %d un-enqueued subtask slot(s) for %s (planned=%d actual=%d)",
shortfall, payload.KnowledgeID, plannedOwned, actualOwned)
for i := 0; i < shortfall; i++ {
rctx, cancel := context.WithTimeout(
context.WithoutCancel(ctx), finalizeSubtaskDetachedTimeout)
_, _, err := s.knowledgeRepo.FinalizeSubtask(rctx, payload.KnowledgeID)
cancel()
if err != nil {
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to release subtask slot for %s: %v",
payload.KnowledgeID, err)
break
}
}
}
}
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
postOutput := types.JSONMap{
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
"chunks_total": len(textChunks),
"enqueued_summary": enqueuedSummary,
"enqueued_question": enqueuedQuestionCount > 0,
"enqueued_question_count": enqueuedQuestionCount,
"enqueued_wiki": wikiSlotOwned && wikiEnqueueErr == nil,
"wiki_slot_owned": wikiSlotOwned,
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
"enqueued_graph": enqueuedGraphCount > 0,
"enqueued_graph_count": enqueuedGraphCount,
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
}
s.tracker().EndSpan(ctx, postSpan, postOutput)
if wikiSlotOwned && wikiEnqueueErr != nil {
return fmt.Errorf("enqueue wiki ingest trigger: %w", wikiEnqueueErr)
}
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
// Close the root span — the parse pipeline is done. Async
// downstream stages (summary/question/wiki/graph) record their
// own spans independently; their finishing extends the trace's
// end-time but does not reopen the root. A late failure in one
// of those stages does not poison the parse result.
s.tracker().FinalizeAttempt(ctx, payload.KnowledgeID, attempt,
types.SpanStatusDone, postOutput, "", "")
return nil
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
// enqueueSummaryGenerationTask enqueues the summary task. Returns true only
// when a task was actually placed on the queue, so the caller can release the
// seeded pending-subtask slot when enqueue is skipped or fails.
func (s *KnowledgePostProcessService) enqueueSummaryGenerationTask(ctx context.Context, payload types.KnowledgePostProcessPayload, attempt int) bool {
if s.taskEnqueuer == nil {
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
return false
}
taskPayload := types.SummaryGenerationPayload{
TenantID: payload.TenantID,
KnowledgeBaseID: payload.KnowledgeBaseID,
KnowledgeID: payload.KnowledgeID,
Language: payload.Language,
Attempt: attempt,
}
feat(observability): extend Langfuse tracing across asynq pipeline Previously the Langfuse integration only traced in-process HTTP requests (chat / search / eval), so file uploads and every downstream asynq task (document parse, chunk embedding, OCR/VLM, summary / question gen, wiki ingest, datasource sync, etc.) produced either disconnected shallow traces or no observation at all. This change threads one trace end-to-end: - tracer: add SPAN observation type and StartSpan; add ResumeTrace so a worker can attach to an upstream trace without emitting a duplicate trace-create; StartGeneration now auto-picks parentObservationId from ctx so nested trace -> span -> generation trees render correctly. - types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq payloads so trace_id / parent_obs_id / user_id / session_id serialise into every job. - langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal so the HTTP-layer trace survives the Redis hop. - langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either resumes the upstream trace or opens a standalone asynq.<type> trace for scheduled jobs, and wraps the handler in a SPAN with task metadata (id / queue / retry / payload_bytes) plus ERROR level on failure. - GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation / FAQ / wiki / datasource endpoints so the root trace actually starts. - Tests: tracer_test.go covers span nesting, error status, and ResumeTrace no-trace-create guarantee; asynq_test.go covers InjectTracing round-trip, middleware resume path, and standalone trace fallback. - Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types and documents the cross-process propagation model. No behavioural change when Langfuse is disabled (all new code paths are no-ops and carriers serialise to empty strings with omitempty).
2026-04-24 04:23:30 +00:00
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to marshal summary generation payload: %v", err)
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
return false
}
task := asynq.NewTask(types.TypeSummaryGeneration, payloadBytes,
asynq.Queue(types.QueueSummary), asynq.MaxRetry(3), asynq.Timeout(30*time.Minute))
if _, err := s.taskEnqueuer.Enqueue(task); err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to enqueue summary generation for %s: %v", payload.KnowledgeID, err)
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
return false
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
logger.Infof(ctx, "[KnowledgePostProcess] Enqueued summary generation task for %s", payload.KnowledgeID)
return true
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
// questionGenChunkBatchSize is the number of text chunks handled by a single
// question-generation task. Batching keeps the task count bounded for very
// large documents (a 5k-chunk doc becomes ~250 tasks instead of 5k) while
// preserving per-batch retry / cancellation granularity and letting each task
// do one embedding BatchIndex over the whole batch.
const questionGenChunkBatchSize = 20
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
// postprocessQuestionGroupSpanName is the grouping span the per-batch
// question subspans (postprocess.question.batch[i]) nest under, so the trace
// viewer shows one "postprocess.question" node instead of dozens of siblings
// directly beneath the postprocess stage.
const postprocessQuestionGroupSpanName = "postprocess.question"
// enqueueQuestionGenerationTasks fans out one TypeQuestionGeneration task per
// batch of questionGenChunkBatchSize text chunks. Each task carries only chunk
// ids (+ the adjacent boundary ids for context) — never the chunk content — so
// the payload stays small and the worker reads fresh content at run time,
// matching the ExtractChunkPayload precedent.
//
// Returns the number of batch tasks successfully enqueued. A failed
// marshal/enqueue is logged and skipped; the caller's reconciliation
// step (the shortfall-release loop in Handle) compares this count
// against questionBatchCount and releases any unowned slots so a
// half-fanned-out batch can't strand the row in "finalizing".
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
func (s *KnowledgePostProcessService) enqueueQuestionGenerationTasks(
ctx context.Context,
payload types.KnowledgePostProcessPayload,
qg types.QuestionGenerationConfig,
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
attempt int,
questionChunks []*types.Chunk,
) int {
if s.taskEnqueuer == nil || len(questionChunks) == 0 {
return 0
}
if !qg.Enabled {
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
return 0
}
questionCount := qg.QuestionCount
if questionCount <= 0 {
questionCount = 3
}
if questionCount > 10 {
questionCount = 10
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
total := len(questionChunks)
enqueued := 0
batchIndex := 0
for start := 0; start < total; start += questionGenChunkBatchSize {
end := start + questionGenChunkBatchSize
if end > total {
end = total
}
batch := questionChunks[start:end]
chunkIDs := make([]string, len(batch))
for i, c := range batch {
chunkIDs[i] = c.ID
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
taskPayload := types.QuestionGenerationPayload{
TenantID: payload.TenantID,
KnowledgeBaseID: payload.KnowledgeBaseID,
KnowledgeID: payload.KnowledgeID,
QuestionCount: questionCount,
Language: payload.Language,
Attempt: attempt,
ChunkIDs: chunkIDs,
BatchIndex: batchIndex,
}
// Boundary context: the text chunk just before / after this window.
if start > 0 {
taskPayload.PrevChunkID = questionChunks[start-1].ID
}
if end < total {
taskPayload.NextChunkID = questionChunks[end].ID
}
batchIndex++
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to marshal question generation payload for batch %d: %v", batchIndex-1, err)
continue
}
task := asynq.NewTask(types.TypeQuestionGeneration, payloadBytes,
asynq.Queue(types.QueueQuestion), asynq.MaxRetry(3), asynq.Timeout(30*time.Minute))
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
if _, err := s.taskEnqueuer.Enqueue(task); err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to enqueue question generation batch %d for %s: %v", batchIndex-1, payload.KnowledgeID, err)
continue
}
enqueued++
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
logger.Infof(ctx, "[KnowledgePostProcess] Enqueued %d question generation batch tasks (%d chunks, batch_size=%d) for %s (count=%d)",
enqueued, total, questionGenChunkBatchSize, payload.KnowledgeID, questionCount)
return enqueued
}