WeKnora/internal/application/service/knowledge_span_tracker.go

842 lines
30 KiB
Go
Raw Normal View History

refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
// Package service: span tracker.
//
// SpanTracker is the pipeline-facing facade for recording per-attempt
// progress trees. It mirrors Langfuse's vocabulary (root / span /
// generation) so the UI's mental model matches what operators already use
// for LLM call observability.
//
// Lifecycle:
//
// attempt := tracker.OpenAttempt(ctx, knowledgeID, langfuseTraceID)
// // creates the root span; every subsequent Begin* call uses this attempt
//
// stage := tracker.BeginStage(ctx, knowledgeID, attempt, types.StageDocReader, input)
// // ...do work...
// tracker.EndSpan(ctx, stage, output) // success
// tracker.FailSpan(ctx, stage, code, msg, err) // error
// tracker.SkipSpan(ctx, stage, reason) // intentionally not run
//
// sub := tracker.BeginSubSpan(ctx, parentSpan, "multimodal.image[0]", types.SpanKindGeneration, input)
// // ...
//
// All operations are best-effort: a DB error is logged and swallowed so a
// tracker hiccup never breaks the parsing pipeline. Knowledge.parse_status
// remains the authoritative source of truth for completion.
package service
import (
"context"
"strings"
"sync"
"time"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/google/uuid"
test(knowledge): add coverage and protect housekeeping from false-killing long stages Addresses three review concerns from the prior PR: 1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5). This commit adds coverage for the four most regression-prone surfaces: span repo upsert/cascade/attempt isolation, span tracker cascade-cancel and cross-process LookupStage, housekeeping false-kill protection, and handler tree assembly. 2. Housekeeping was using only knowledge.updated_at as its staleness signal, but knowledge.updated_at advances only at parse_status transitions — a long DocReader call (or large embedding batch) can run for an hour with no updated_at change, so a tight DocumentProcessTimeout setting would falsely flip an actively running parse to "failed". The sweep now does a two-stage check: candidates by knowledge updated_at, then filtered by MAX(spans.updated_at). Every SpanTracker Begin/End/Fail/Skip now also pokes knowledge.updated_at as a side-channel heartbeat, so the filter sees recent activity even when no parse_status transition has fired. 3. parseHeartbeatTime accepts the timestamp formats both Postgres and SQLite emit for an aggregated MAX() column (the SQLite driver doesn't auto-cast aggregates to time.Time the way Postgres does), so the same code path works in Lite mode. The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at combined with a 2-minute-fresh span row must NOT be killed.
2026-05-27 07:01:54 +00:00
"gorm.io/gorm"
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
)
// Span is the in-memory handle the pipeline holds while a stage / subspan
// is executing. It carries enough context for End/Fail/Skip to write back
// without re-querying the DB. Returned (and required) from every Begin*.
type Span struct {
KnowledgeID string
Attempt int
SpanID string
ParentSpanID string
Name string
Kind string
StartedAt time.Time
}
// SpanTracker is the only public surface — kept as an interface so tests
// can swap in a no-op without spinning up a database.
type SpanTracker interface {
// OpenAttempt creates a new root span for (knowledgeID,
// nextAttempt) and returns its number plus the root *Span. Call
// at the start of a parse / reparse, before any other Begin*.
OpenAttempt(ctx context.Context, knowledgeID, langfuseTraceID string) (root *Span, attempt int, err error)
// LatestAttempt returns the highest attempt number recorded for
// the knowledge, or 0 if it's never been parsed. Used by the API
// layer to default to "show me the most recent run".
LatestAttempt(ctx context.Context, knowledgeID string) int
// BeginStage starts one of the canonical stages. Looks up the
// root span for (kid, attempt) — caller passes attempt to make
// the wiring explicit and let cross-process workers join an
// existing attempt without new repo lookups.
BeginStage(ctx context.Context, knowledgeID string, attempt int, stage string, input types.JSONMap) *Span
// BeginSubSpan creates a child span under parent. parent may be a
// stage span (for multimodal.image[i] / embedding.batch[i]) or
// another subspan. kind is "subspan" or "generation" — generations
// will be stitched to a Langfuse generation by trace_id.
BeginSubSpan(ctx context.Context, parent *Span, name, kind string, input types.JSONMap) *Span
// EndSpan marks span as done with optional output. Safe with nil.
EndSpan(ctx context.Context, span *Span, output types.JSONMap)
// FailSpan marks span as failed and cascade-cancels its
// descendants. errorDetail (a Go error) is recorded verbatim in
// error_detail (truncated to 8 KB) for admin views.
FailSpan(ctx context.Context, span *Span, errorCode, errorMessage string, errorDetail error)
// SkipSpan marks an intentionally not-run span (e.g. multimodal
// on a text-only document). Distinct from cancelled — skipped is
// "we chose not to" while cancelled is "an upstream broke".
SkipSpan(ctx context.Context, span *Span, reason string)
// LookupStage returns the stage's *Span for an in-flight attempt
// — the cross-process bridge that lets an asynq worker (e.g.
// image_multimodal) attach subspans to the parent stage span
// created by the upstream pipeline.
LookupStage(ctx context.Context, knowledgeID string, attempt int, stage string) *Span
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
// LookupSpanByName returns the first span of any kind matching name
// for (knowledgeID, attempt) — the cross-process bridge that lets a
// fan-out worker (e.g. a question-generation batch) attach its subspan
// under a grouping span created earlier by the orchestrator. Returns
// nil when no such span exists (caller should fall back to the stage).
LookupSpanByName(ctx context.Context, knowledgeID string, attempt int, name string) *Span
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
// FinalizeAttempt closes the root span for (knowledgeID, attempt)
// with the given terminal status (done | failed). Idempotent:
// re-closing an already-terminal root is a no-op so callers from
// multiple paths (success orchestrator, dead-letter handler,
// housekeeping) can fire without coordination. status defaults to
// done; output/error are written verbatim.
FinalizeAttempt(ctx context.Context, knowledgeID string, attempt int, status string,
output types.JSONMap, errorCode, errorMessage string)
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
// AbortAttempt cascade-cancels every still-running descendant of the
// attempt's root span and then closes the root as cancelled. Used by
// the user-initiated cancel path so the trace viewer doesn't leave
// stranded subspans (multimodal images, postprocess subtasks)
// looking like they're still in flight forever after the user
// stopped the parse. Idempotent.
AbortAttempt(ctx context.Context, knowledgeID string, attempt int, errorCode, errorMessage, reason string)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
}
type spanTracker struct {
repo repository.KnowledgeSpanRepository
test(knowledge): add coverage and protect housekeeping from false-killing long stages Addresses three review concerns from the prior PR: 1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5). This commit adds coverage for the four most regression-prone surfaces: span repo upsert/cascade/attempt isolation, span tracker cascade-cancel and cross-process LookupStage, housekeeping false-kill protection, and handler tree assembly. 2. Housekeeping was using only knowledge.updated_at as its staleness signal, but knowledge.updated_at advances only at parse_status transitions — a long DocReader call (or large embedding batch) can run for an hour with no updated_at change, so a tight DocumentProcessTimeout setting would falsely flip an actively running parse to "failed". The sweep now does a two-stage check: candidates by knowledge updated_at, then filtered by MAX(spans.updated_at). Every SpanTracker Begin/End/Fail/Skip now also pokes knowledge.updated_at as a side-channel heartbeat, so the filter sees recent activity even when no parse_status transition has fired. 3. parseHeartbeatTime accepts the timestamp formats both Postgres and SQLite emit for an aggregated MAX() column (the SQLite driver doesn't auto-cast aggregates to time.Time the way Postgres does), so the same code path works in Lite mode. The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at combined with a 2-minute-fresh span row must NOT be killed.
2026-05-27 07:01:54 +00:00
// db is held purely for the heartbeat side-channel: every span
// state transition pokes knowledge.updated_at so the housekeeping
// sweep can tell "actively running long stage" from "abandoned".
// nil-safe — when missing (test harness) the heartbeat is skipped.
db *gorm.DB
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
// startsMu guards the in-process duration cache. Cross-process
// workers won't find their parent's start here — that's fine,
// duration_ms falls back to (FinishedAt - row.StartedAt) computed
// at write time when the cache misses.
startsMu sync.Mutex
starts map[string]time.Time // span_id → started_at
}
// NewSpanTracker constructs the GORM-backed tracker. A nil repo collapses
test(knowledge): add coverage and protect housekeeping from false-killing long stages Addresses three review concerns from the prior PR: 1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5). This commit adds coverage for the four most regression-prone surfaces: span repo upsert/cascade/attempt isolation, span tracker cascade-cancel and cross-process LookupStage, housekeeping false-kill protection, and handler tree assembly. 2. Housekeeping was using only knowledge.updated_at as its staleness signal, but knowledge.updated_at advances only at parse_status transitions — a long DocReader call (or large embedding batch) can run for an hour with no updated_at change, so a tight DocumentProcessTimeout setting would falsely flip an actively running parse to "failed". The sweep now does a two-stage check: candidates by knowledge updated_at, then filtered by MAX(spans.updated_at). Every SpanTracker Begin/End/Fail/Skip now also pokes knowledge.updated_at as a side-channel heartbeat, so the filter sees recent activity even when no parse_status transition has fired. 3. parseHeartbeatTime accepts the timestamp formats both Postgres and SQLite emit for an aggregated MAX() column (the SQLite driver doesn't auto-cast aggregates to time.Time the way Postgres does), so the same code path works in Lite mode. The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at combined with a 2-minute-fresh span row must NOT be killed.
2026-05-27 07:01:54 +00:00
// to a no-op so test harnesses don't need to spin up a database. The db
// is optional too: it's used only for the housekeeping heartbeat (see
// touchKnowledgeHeartbeat) and a nil db just disables that side-channel.
func NewSpanTracker(repo repository.KnowledgeSpanRepository, db *gorm.DB) SpanTracker {
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
if repo == nil {
return noopSpanTracker{}
}
return &spanTracker{
repo: repo,
test(knowledge): add coverage and protect housekeeping from false-killing long stages Addresses three review concerns from the prior PR: 1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5). This commit adds coverage for the four most regression-prone surfaces: span repo upsert/cascade/attempt isolation, span tracker cascade-cancel and cross-process LookupStage, housekeeping false-kill protection, and handler tree assembly. 2. Housekeeping was using only knowledge.updated_at as its staleness signal, but knowledge.updated_at advances only at parse_status transitions — a long DocReader call (or large embedding batch) can run for an hour with no updated_at change, so a tight DocumentProcessTimeout setting would falsely flip an actively running parse to "failed". The sweep now does a two-stage check: candidates by knowledge updated_at, then filtered by MAX(spans.updated_at). Every SpanTracker Begin/End/Fail/Skip now also pokes knowledge.updated_at as a side-channel heartbeat, so the filter sees recent activity even when no parse_status transition has fired. 3. parseHeartbeatTime accepts the timestamp formats both Postgres and SQLite emit for an aggregated MAX() column (the SQLite driver doesn't auto-cast aggregates to time.Time the way Postgres does), so the same code path works in Lite mode. The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at combined with a 2-minute-fresh span row must NOT be killed.
2026-05-27 07:01:54 +00:00
db: db,
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
starts: make(map[string]time.Time),
}
}
test(knowledge): add coverage and protect housekeeping from false-killing long stages Addresses three review concerns from the prior PR: 1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5). This commit adds coverage for the four most regression-prone surfaces: span repo upsert/cascade/attempt isolation, span tracker cascade-cancel and cross-process LookupStage, housekeeping false-kill protection, and handler tree assembly. 2. Housekeeping was using only knowledge.updated_at as its staleness signal, but knowledge.updated_at advances only at parse_status transitions — a long DocReader call (or large embedding batch) can run for an hour with no updated_at change, so a tight DocumentProcessTimeout setting would falsely flip an actively running parse to "failed". The sweep now does a two-stage check: candidates by knowledge updated_at, then filtered by MAX(spans.updated_at). Every SpanTracker Begin/End/Fail/Skip now also pokes knowledge.updated_at as a side-channel heartbeat, so the filter sees recent activity even when no parse_status transition has fired. 3. parseHeartbeatTime accepts the timestamp formats both Postgres and SQLite emit for an aggregated MAX() column (the SQLite driver doesn't auto-cast aggregates to time.Time the way Postgres does), so the same code path works in Lite mode. The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at combined with a 2-minute-fresh span row must NOT be killed.
2026-05-27 07:01:54 +00:00
// touchKnowledgeHeartbeat advances knowledge.updated_at to the current
// wall-clock so the housekeeping sweep treats this row as actively
// progressing. Called on every span Begin/End/Fail/Skip — the cost is
// one indexed UPDATE per transition (≤ a few dozen per knowledge), which
// is dwarfed by the work the stages themselves do.
//
// touchKnowledgeHeartbeat advances knowledge.updated_at to the current
// wall-clock so the housekeeping sweep treats this row as actively
// progressing. Called on root / stage span transitions only — subspan and
// generation transitions skip this side-channel because:
//
// - The spans table itself is updated on every transition, and the
// housekeeping sweep already reads MAX(spans.updated_at) per
// knowledge, so subspan progress is observable without poking the
// parent row.
// - A multimodal stage with N images would produce 2*N+ extra UPDATEs
// on the same hot row (Begin+End per image plus retries), which we
// observed contributing to row-level contention under bursty
// uploads.
//
test(knowledge): add coverage and protect housekeeping from false-killing long stages Addresses three review concerns from the prior PR: 1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5). This commit adds coverage for the four most regression-prone surfaces: span repo upsert/cascade/attempt isolation, span tracker cascade-cancel and cross-process LookupStage, housekeeping false-kill protection, and handler tree assembly. 2. Housekeeping was using only knowledge.updated_at as its staleness signal, but knowledge.updated_at advances only at parse_status transitions — a long DocReader call (or large embedding batch) can run for an hour with no updated_at change, so a tight DocumentProcessTimeout setting would falsely flip an actively running parse to "failed". The sweep now does a two-stage check: candidates by knowledge updated_at, then filtered by MAX(spans.updated_at). Every SpanTracker Begin/End/Fail/Skip now also pokes knowledge.updated_at as a side-channel heartbeat, so the filter sees recent activity even when no parse_status transition has fired. 3. parseHeartbeatTime accepts the timestamp formats both Postgres and SQLite emit for an aggregated MAX() column (the SQLite driver doesn't auto-cast aggregates to time.Time the way Postgres does), so the same code path works in Lite mode. The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at combined with a 2-minute-fresh span row must NOT be killed.
2026-05-27 07:01:54 +00:00
// Best-effort. We deliberately do NOT bump status here: the parse_status
// column remains under the pipeline's control. Only the timestamp gets
// nudged, which is exactly what housekeeping reads as the fallback.
func (t *spanTracker) touchKnowledgeHeartbeat(ctx context.Context, knowledgeID, kind string) {
test(knowledge): add coverage and protect housekeeping from false-killing long stages Addresses three review concerns from the prior PR: 1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5). This commit adds coverage for the four most regression-prone surfaces: span repo upsert/cascade/attempt isolation, span tracker cascade-cancel and cross-process LookupStage, housekeeping false-kill protection, and handler tree assembly. 2. Housekeeping was using only knowledge.updated_at as its staleness signal, but knowledge.updated_at advances only at parse_status transitions — a long DocReader call (or large embedding batch) can run for an hour with no updated_at change, so a tight DocumentProcessTimeout setting would falsely flip an actively running parse to "failed". The sweep now does a two-stage check: candidates by knowledge updated_at, then filtered by MAX(spans.updated_at). Every SpanTracker Begin/End/Fail/Skip now also pokes knowledge.updated_at as a side-channel heartbeat, so the filter sees recent activity even when no parse_status transition has fired. 3. parseHeartbeatTime accepts the timestamp formats both Postgres and SQLite emit for an aggregated MAX() column (the SQLite driver doesn't auto-cast aggregates to time.Time the way Postgres does), so the same code path works in Lite mode. The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at combined with a 2-minute-fresh span row must NOT be killed.
2026-05-27 07:01:54 +00:00
if t.db == nil || knowledgeID == "" {
return
}
// Subspan / generation transitions are observable through the spans
// table directly; skip the parent-row UPDATE to avoid write
// amplification on fan-out workloads.
if kind != types.SpanKindRoot && kind != types.SpanKindStage {
return
}
test(knowledge): add coverage and protect housekeeping from false-killing long stages Addresses three review concerns from the prior PR: 1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5). This commit adds coverage for the four most regression-prone surfaces: span repo upsert/cascade/attempt isolation, span tracker cascade-cancel and cross-process LookupStage, housekeeping false-kill protection, and handler tree assembly. 2. Housekeeping was using only knowledge.updated_at as its staleness signal, but knowledge.updated_at advances only at parse_status transitions — a long DocReader call (or large embedding batch) can run for an hour with no updated_at change, so a tight DocumentProcessTimeout setting would falsely flip an actively running parse to "failed". The sweep now does a two-stage check: candidates by knowledge updated_at, then filtered by MAX(spans.updated_at). Every SpanTracker Begin/End/Fail/Skip now also pokes knowledge.updated_at as a side-channel heartbeat, so the filter sees recent activity even when no parse_status transition has fired. 3. parseHeartbeatTime accepts the timestamp formats both Postgres and SQLite emit for an aggregated MAX() column (the SQLite driver doesn't auto-cast aggregates to time.Time the way Postgres does), so the same code path works in Lite mode. The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at combined with a 2-minute-fresh span row must NOT be killed.
2026-05-27 07:01:54 +00:00
if err := t.db.WithContext(ctx).Model(&types.Knowledge{}).
Where("id = ?", knowledgeID).
Update("updated_at", time.Now()).Error; err != nil {
// Don't log every failure — heartbeat is best-effort and
// noisy logs would drown out real errors. Single line at
// warn level is enough for ops to spot a chronic outage.
logger.Warnf(ctx, "[SpanTracker] heartbeat update failed kid=%s: %v", knowledgeID, err)
}
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
func newSpanID() string {
// Stripping the dashes saves 4 bytes per row — JSON parsers don't
// care, and operators paste these into queries / Langfuse where a
// hex-only ID is friendlier.
return strings.ReplaceAll(uuid.NewString(), "-", "")
}
func (t *spanTracker) recordStart(spanID string, at time.Time) {
t.startsMu.Lock()
t.starts[spanID] = at
t.startsMu.Unlock()
}
func (t *spanTracker) takeStart(spanID string) (time.Time, bool) {
t.startsMu.Lock()
defer t.startsMu.Unlock()
v, ok := t.starts[spanID]
if ok {
delete(t.starts, spanID)
}
return v, ok
}
func (t *spanTracker) OpenAttempt(ctx context.Context, knowledgeID, langfuseTraceID string) (*Span, int, error) {
attempt, err := t.repo.NextAttempt(ctx, knowledgeID)
if err != nil {
return nil, 0, err
}
now := time.Now()
rootID := newSpanID()
meta := types.JSONMap{}
if langfuseTraceID != "" {
// The frontend renders a "open in Langfuse" link from this.
meta["langfuse_trace_id"] = langfuseTraceID
}
row := &types.KnowledgeProcessingSpan{
KnowledgeID: knowledgeID,
Attempt: attempt,
SpanID: rootID,
Name: "knowledge_processing",
Kind: types.SpanKindRoot,
Status: types.SpanStatusRunning,
Metadata: meta,
StartedAt: &now,
}
if err := t.repo.Upsert(ctx, row); err != nil {
logger.Warnf(ctx, "[SpanTracker] OpenAttempt failed kid=%s: %v", knowledgeID, err)
return nil, attempt, err
}
t.recordStart(rootID, now)
t.touchKnowledgeHeartbeat(ctx, knowledgeID, types.SpanKindRoot)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
return &Span{
KnowledgeID: knowledgeID,
Attempt: attempt,
SpanID: rootID,
Name: "knowledge_processing",
Kind: types.SpanKindRoot,
StartedAt: now,
}, attempt, nil
}
func (t *spanTracker) LatestAttempt(ctx context.Context, knowledgeID string) int {
n, err := t.repo.LatestAttempt(ctx, knowledgeID)
if err != nil {
logger.Warnf(ctx, "[SpanTracker] LatestAttempt failed kid=%s: %v", knowledgeID, err)
return 0
}
return n
}
func (t *spanTracker) BeginStage(ctx context.Context, knowledgeID string, attempt int, stage string, input types.JSONMap) *Span {
if knowledgeID == "" || stage == "" {
return nil
}
// Find root span — we need its span_id as parent for stage rows. We
// also reuse this scan to detect an existing row for the same stage
// name in this attempt: re-entry (asynq retry, double-call from
// adjacent code paths) MUST NOT create a second row, otherwise the
// timeline shows two segments for the same stage and LookupStage
// becomes ambiguous.
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
rows, err := t.repo.ListByAttempt(ctx, knowledgeID, attempt)
if err != nil {
logger.Warnf(ctx, "[SpanTracker] BeginStage list failed kid=%s attempt=%d: %v",
knowledgeID, attempt, err)
return nil
}
var (
rootID string
existing *types.KnowledgeProcessingSpan
)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
for i := range rows {
r := rows[i]
if r.Kind == types.SpanKindRoot && rootID == "" {
rootID = r.SpanID
}
if r.Kind == types.SpanKindStage && r.Name == stage {
cp := r
existing = &cp
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
}
}
if rootID == "" {
// Pipeline started before tracker was wired (legacy data,
// or the OpenAttempt repo write failed). Synthesize a
// rootless stage so we still record SOMETHING.
logger.Warnf(ctx, "[SpanTracker] BeginStage: no root for kid=%s attempt=%d, recording rootless",
knowledgeID, attempt)
}
now := time.Now()
// Re-entry path: keep the original span_id so any subspan that
// already references it stays attached. Reset state to running and
// refresh started_at; clear terminal-only fields so the row reads
// cleanly as "running again". Output/error fields go through
// Upsert's DoUpdates list, which only writes the columns we set —
// any nil JSONMap / empty string explicitly clears the column.
if existing != nil {
row := &types.KnowledgeProcessingSpan{
KnowledgeID: existing.KnowledgeID,
Attempt: existing.Attempt,
SpanID: existing.SpanID,
ParentSpanID: existing.ParentSpanID,
Name: existing.Name,
Kind: existing.Kind,
Status: types.SpanStatusRunning,
Input: input,
Output: nil,
StartedAt: &now,
FinishedAt: nil,
DurationMs: 0,
}
if err := t.repo.Upsert(ctx, row); err != nil {
logger.Warnf(ctx, "[SpanTracker] BeginStage re-enter failed kid=%s stage=%s: %v",
knowledgeID, stage, err)
return nil
}
t.recordStart(existing.SpanID, now)
t.touchKnowledgeHeartbeat(ctx, knowledgeID, types.SpanKindStage)
return &Span{
KnowledgeID: existing.KnowledgeID,
Attempt: existing.Attempt,
SpanID: existing.SpanID,
ParentSpanID: existing.ParentSpanID,
Name: existing.Name,
Kind: existing.Kind,
StartedAt: now,
}
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
id := newSpanID()
row := &types.KnowledgeProcessingSpan{
KnowledgeID: knowledgeID,
Attempt: attempt,
SpanID: id,
ParentSpanID: rootID,
Name: stage,
Kind: types.SpanKindStage,
Status: types.SpanStatusRunning,
Input: input,
StartedAt: &now,
}
if err := t.repo.Upsert(ctx, row); err != nil {
logger.Warnf(ctx, "[SpanTracker] BeginStage failed kid=%s stage=%s: %v",
knowledgeID, stage, err)
return nil
}
t.recordStart(id, now)
t.touchKnowledgeHeartbeat(ctx, knowledgeID, types.SpanKindStage)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
return &Span{
KnowledgeID: knowledgeID,
Attempt: attempt,
SpanID: id,
ParentSpanID: rootID,
Name: stage,
Kind: types.SpanKindStage,
StartedAt: now,
}
}
func (t *spanTracker) BeginSubSpan(ctx context.Context, parent *Span, name, kind string, input types.JSONMap) *Span {
if parent == nil || name == "" {
return nil
}
if kind != types.SpanKindGeneration && kind != types.SpanKindSubSpan {
kind = types.SpanKindSubSpan
}
// Asynq retry / server restart can re-run the same handler while the
// previous invocation's span is still status=running (worker died
// without EndSpan). Cancel same-name open rows so the UI shows one
// logical subspan per (attempt, name) instead of duplicate stripes.
if _, err := t.repo.CancelOpenSpansByName(ctx, parent.KnowledgeID, parent.Attempt, name,
"TASK_SUPERSEDED", "superseded by a new run of the same subtask"); err != nil {
logger.Warnf(ctx, "[SpanTracker] supersede %s before BeginSubSpan failed: %v", name, err)
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
now := time.Now()
id := newSpanID()
row := &types.KnowledgeProcessingSpan{
KnowledgeID: parent.KnowledgeID,
Attempt: parent.Attempt,
SpanID: id,
ParentSpanID: parent.SpanID,
Name: name,
Kind: kind,
Status: types.SpanStatusRunning,
Input: input,
StartedAt: &now,
}
if err := t.repo.Upsert(ctx, row); err != nil {
logger.Warnf(ctx, "[SpanTracker] BeginSubSpan failed parent=%s name=%s: %v",
parent.SpanID, name, err)
return nil
}
t.recordStart(id, now)
t.touchKnowledgeHeartbeat(ctx, parent.KnowledgeID, kind)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
return &Span{
KnowledgeID: parent.KnowledgeID,
Attempt: parent.Attempt,
SpanID: id,
ParentSpanID: parent.SpanID,
Name: name,
Kind: kind,
StartedAt: now,
}
}
func (t *spanTracker) EndSpan(ctx context.Context, span *Span, output types.JSONMap) {
if span == nil {
return
}
now := time.Now()
dur := durationSince(t, span, now)
row := &types.KnowledgeProcessingSpan{
KnowledgeID: span.KnowledgeID,
Attempt: span.Attempt,
SpanID: span.SpanID,
ParentSpanID: span.ParentSpanID,
Name: span.Name,
Kind: span.Kind,
Status: types.SpanStatusDone,
Output: output,
StartedAt: &span.StartedAt,
FinishedAt: &now,
DurationMs: dur,
}
if err := t.repo.Upsert(ctx, row); err != nil {
logger.Warnf(ctx, "[SpanTracker] EndSpan failed span=%s: %v", span.SpanID, err)
}
t.touchKnowledgeHeartbeat(ctx, span.KnowledgeID, span.Kind)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
}
func (t *spanTracker) FailSpan(ctx context.Context, span *Span, errorCode, errorMessage string, errorDetail error) {
if span == nil {
return
}
now := time.Now()
dur := durationSince(t, span, now)
detail := ""
if errorDetail != nil {
detail = errorDetail.Error()
if len(detail) > 8192 {
detail = detail[:8192]
}
}
if len(errorMessage) > 1024 {
errorMessage = errorMessage[:1024]
}
row := &types.KnowledgeProcessingSpan{
KnowledgeID: span.KnowledgeID,
Attempt: span.Attempt,
SpanID: span.SpanID,
ParentSpanID: span.ParentSpanID,
Name: span.Name,
Kind: span.Kind,
Status: types.SpanStatusFailed,
ErrorCode: strings.TrimSpace(errorCode),
ErrorMessage: errorMessage,
ErrorDetail: detail,
StartedAt: &span.StartedAt,
FinishedAt: &now,
DurationMs: dur,
}
if err := t.repo.Upsert(ctx, row); err != nil {
logger.Warnf(ctx, "[SpanTracker] FailSpan failed span=%s: %v", span.SpanID, err)
}
// Cascade: anything downstream of this span gets cancelled. The
// reason string is what the UI surfaces under each cancelled
// child's tooltip — keep it short and human.
reason := "upstream " + span.Name + " failed"
if errorCode != "" {
reason = reason + " (" + errorCode + ")"
}
if _, err := t.repo.CancelDescendants(ctx, span.KnowledgeID, span.Attempt, span.SpanID, reason); err != nil {
logger.Warnf(ctx, "[SpanTracker] cancel descendants failed span=%s: %v", span.SpanID, err)
}
// For STAGE failures, also cascade to dependent stages declared
// in StageDependencies (those are siblings, not descendants).
if span.Kind == types.SpanKindStage {
t.cascadeDependentStages(ctx, span, reason)
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
// Any failure in a MAIN pipeline stage means the attempt is
// done — the parse cannot succeed past this point. Close the
// root span as failed so the UI doesn't show "进行中" forever.
// Optional downstream stages (summary/question/wiki/graph) do
// not poison the attempt: they can fail without invalidating
// the parsed document.
if isMainPipelineStage(span.Name) {
t.FinalizeAttempt(ctx, span.KnowledgeID, span.Attempt,
types.SpanStatusFailed, nil, errorCode, errorMessage)
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
}
t.touchKnowledgeHeartbeat(ctx, span.KnowledgeID, span.Kind)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
}
func (t *spanTracker) SkipSpan(ctx context.Context, span *Span, reason string) {
if span == nil {
return
}
now := time.Now()
row := &types.KnowledgeProcessingSpan{
KnowledgeID: span.KnowledgeID,
Attempt: span.Attempt,
SpanID: span.SpanID,
ParentSpanID: span.ParentSpanID,
Name: span.Name,
Kind: span.Kind,
Status: types.SpanStatusSkipped,
ErrorMessage: reason,
StartedAt: &span.StartedAt,
FinishedAt: &now,
}
if err := t.repo.Upsert(ctx, row); err != nil {
logger.Warnf(ctx, "[SpanTracker] SkipSpan failed span=%s: %v", span.SpanID, err)
}
t.touchKnowledgeHeartbeat(ctx, span.KnowledgeID, span.Kind)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
}
func (t *spanTracker) LookupStage(ctx context.Context, knowledgeID string, attempt int, stage string) *Span {
rows, err := t.repo.ListByAttempt(ctx, knowledgeID, attempt)
if err != nil {
logger.Warnf(ctx, "[SpanTracker] LookupStage list failed kid=%s attempt=%d: %v",
knowledgeID, attempt, err)
return nil
}
for i := range rows {
r := rows[i]
if r.Kind != types.SpanKindStage || r.Name != stage {
continue
}
started := time.Time{}
if r.StartedAt != nil {
started = *r.StartedAt
}
return &Span{
KnowledgeID: r.KnowledgeID,
Attempt: r.Attempt,
SpanID: r.SpanID,
ParentSpanID: r.ParentSpanID,
Name: r.Name,
Kind: r.Kind,
StartedAt: started,
}
}
return nil
}
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
func (t *spanTracker) LookupSpanByName(ctx context.Context, knowledgeID string, attempt int, name string) *Span {
if name == "" || knowledgeID == "" || attempt <= 0 {
return nil
}
rows, err := t.repo.ListByAttempt(ctx, knowledgeID, attempt)
if err != nil {
logger.Warnf(ctx, "[SpanTracker] LookupSpanByName list failed kid=%s attempt=%d: %v",
knowledgeID, attempt, err)
return nil
}
for i := range rows {
r := rows[i]
if r.Name != name {
continue
}
started := time.Time{}
if r.StartedAt != nil {
started = *r.StartedAt
}
return &Span{
KnowledgeID: r.KnowledgeID,
Attempt: r.Attempt,
SpanID: r.SpanID,
ParentSpanID: r.ParentSpanID,
Name: r.Name,
Kind: r.Kind,
StartedAt: started,
}
}
return nil
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
// cascadeDependentStages flips downstream STAGE rows to "cancelled" using
// types.StageDependencies. Without this, a Chunking failure leaves
// Embedding / Multimodal as "pending" forever, even though they cannot
// possibly run. After flipping a dependent stage we ALSO cascade-cancel
// any subspan/generation that already attached to it (e.g. an embedding
// batch that started before the chunking failure was observed) — without
// this second walk those subspans would remain in pending/running and
// surface as orphan spinners under a cancelled parent.
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
func (t *spanTracker) cascadeDependentStages(ctx context.Context, failedStage *Span, reason string) {
rows, err := t.repo.ListByAttempt(ctx, failedStage.KnowledgeID, failedStage.Attempt)
if err != nil {
return
}
dependents := stagesDependingOn(failedStage.Name)
if len(dependents) == 0 {
return
}
now := time.Now()
for _, row := range rows {
if row.Kind != types.SpanKindStage {
continue
}
if row.Status != types.SpanStatusPending && row.Status != types.SpanStatusRunning {
continue
}
if !contains(dependents, row.Name) {
continue
}
updated := row // copy
updated.Status = types.SpanStatusCancelled
updated.ErrorCode = "UPSTREAM_FAILED"
updated.ErrorMessage = reason
updated.FinishedAt = &now
if err := t.repo.Upsert(ctx, &updated); err != nil {
logger.Warnf(ctx, "[SpanTracker] cascade dependent stage %s: %v", row.Name, err)
continue
}
// Recurse into the cascaded stage's own subtree so any
// in-flight subspan/generation is cancelled too. The
// repo-level walk is iterative and cheap (small fan-out).
if _, err := t.repo.CancelDescendants(ctx, row.KnowledgeID, row.Attempt, row.SpanID, reason); err != nil {
logger.Warnf(ctx, "[SpanTracker] cascade descendants of dependent %s: %v", row.Name, err)
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
}
}
}
// stagesDependingOn returns the transitive closure of stages that have
// `stage` as an upstream dependency (direct or indirect). Computed by
// reverse-walking StageDependencies; the result is bounded to 5 since
// AllStages has five members, so a naive O(N²) walk is fine.
func stagesDependingOn(stage string) []string {
var out []string
seen := map[string]bool{}
frontier := []string{stage}
for len(frontier) > 0 {
var next []string
for _, candidate := range types.AllStages {
if seen[candidate] {
continue
}
deps := types.StageDependencies[candidate]
for _, d := range deps {
if contains(frontier, d) {
seen[candidate] = true
out = append(out, candidate)
next = append(next, candidate)
break
}
}
}
frontier = next
}
return out
}
func contains(haystack []string, needle string) bool {
for _, h := range haystack {
if h == needle {
return true
}
}
return false
}
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
// isMainPipelineStage reports whether stage is one of the 5 mandatory
// pipeline stages (docreader / chunking / embedding / multimodal /
// postprocess). A failure in any of these terminally invalidates the
// attempt and must close the root as failed. Optional downstream stages
// added later (summary, question, wiki, graph) do NOT match — those
// can fail individually without poisoning the parse result.
func isMainPipelineStage(name string) bool {
for _, s := range types.AllStages {
if s == name {
return true
}
}
return false
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
// durationSince computes elapsed ms preferring the in-process cache;
// falls back to the *Span's StartedAt for cross-process callers.
func durationSince(t *spanTracker, span *Span, now time.Time) int64 {
if start, ok := t.takeStart(span.SpanID); ok {
return now.Sub(start).Milliseconds()
}
if !span.StartedAt.IsZero() {
return now.Sub(span.StartedAt).Milliseconds()
}
return 0
}
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
// FinalizeAttempt closes the root span for (knowledgeID, attempt). The
// pipeline calls this from two places: the success orchestrator
// (PostProcess) when the parse reaches Completed, and FailSpan when a
// MAIN stage fails terminally. Without this, the root row created by
// OpenAttempt would stay in "running" forever even after parse_status
// flips to completed/failed — operators would see a perpetually
// "running" trace despite a terminal knowledge state.
//
// Idempotent: re-closing a root that's already done/failed is a no-op
// so callers from different paths (success vs. cascade-fail vs.
// dead-letter) don't have to coordinate. We deliberately avoid the
// recordStart cache here (cross-process callers won't have it); we
// recompute duration from the persisted row's started_at.
func (t *spanTracker) FinalizeAttempt(ctx context.Context, knowledgeID string, attempt int, status string,
output types.JSONMap, errorCode, errorMessage string,
) {
if knowledgeID == "" || attempt <= 0 {
return
}
if status == "" {
status = types.SpanStatusDone
}
rows, err := t.repo.ListByAttempt(ctx, knowledgeID, attempt)
if err != nil {
logger.Warnf(ctx, "[SpanTracker] FinalizeAttempt list failed kid=%s attempt=%d: %v",
knowledgeID, attempt, err)
return
}
var root *types.KnowledgeProcessingSpan
for i := range rows {
if rows[i].Kind == types.SpanKindRoot {
cp := rows[i]
root = &cp
break
}
}
if root == nil {
// No root means nothing to close — likely an attempt that
// predates the tracker or whose OpenAttempt write failed.
return
}
if root.Status == types.SpanStatusDone || root.Status == types.SpanStatusFailed ||
root.Status == types.SpanStatusCancelled || root.Status == types.SpanStatusSkipped {
return
}
now := time.Now()
var started time.Time
if root.StartedAt != nil {
started = *root.StartedAt
}
dur := int64(0)
if !started.IsZero() {
dur = now.Sub(started).Milliseconds()
}
if len(errorMessage) > 1024 {
errorMessage = errorMessage[:1024]
}
row := &types.KnowledgeProcessingSpan{
KnowledgeID: root.KnowledgeID,
Attempt: root.Attempt,
SpanID: root.SpanID,
ParentSpanID: root.ParentSpanID,
Name: root.Name,
Kind: root.Kind,
Status: status,
Input: root.Input,
Output: output,
Metadata: root.Metadata,
ErrorCode: strings.TrimSpace(errorCode),
ErrorMessage: errorMessage,
StartedAt: root.StartedAt,
FinishedAt: &now,
DurationMs: dur,
}
if err := t.repo.Upsert(ctx, row); err != nil {
logger.Warnf(ctx, "[SpanTracker] FinalizeAttempt upsert failed kid=%s attempt=%d: %v",
knowledgeID, attempt, err)
return
}
t.touchKnowledgeHeartbeat(ctx, knowledgeID, types.SpanKindRoot)
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
// AbortAttempt is the user-cancel counterpart to FinalizeAttempt. It
// flips every still-running / still-pending span for this attempt to
// cancelled — regardless of tree position — and then closes the root.
//
// Why a flat sweep instead of CancelDescendants' BFS: fan-out stages
// (e.g. 多模态识别) call EndSpan on the stage as soon as they finish
// DISPATCHING their async per-image work, so by the time the user
// hits cancel the stage row is already status=done but its image[*]
// children are still status=running. A BFS that stops at terminal
// parents would orphan those leaves. The flat sweep doesn't care
// about the tree shape — anything not-yet-terminal gets flipped.
func (t *spanTracker) AbortAttempt(ctx context.Context, knowledgeID string, attempt int,
errorCode, errorMessage, reason string,
) {
if knowledgeID == "" || attempt <= 0 {
return
}
if reason == "" {
reason = "user cancelled"
}
if errorCode == "" {
errorCode = "USER_CANCELLED"
}
if n, err := t.repo.CancelAllOpenSpans(ctx, knowledgeID, attempt, errorCode, reason); err != nil {
logger.Warnf(ctx, "[SpanTracker] AbortAttempt sweep failed kid=%s attempt=%d: %v",
knowledgeID, attempt, err)
// Fall through to FinalizeAttempt anyway — closing the root
// is more important than perfectly closing every child.
} else if n > 0 {
logger.Infof(ctx,
"[SpanTracker] AbortAttempt swept %d open span(s) for kid=%s attempt=%d",
n, knowledgeID, attempt)
}
t.FinalizeAttempt(ctx, knowledgeID, attempt,
types.SpanStatusCancelled, nil, errorCode, errorMessage)
}
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
// noopSpanTracker collapses every method to a no-op for tests/lite.
type noopSpanTracker struct{}
func (noopSpanTracker) OpenAttempt(_ context.Context, _, _ string) (*Span, int, error) {
return nil, 0, nil
}
func (noopSpanTracker) LatestAttempt(_ context.Context, _ string) int { return 0 }
func (noopSpanTracker) BeginStage(_ context.Context, _ string, _ int, _ string, _ types.JSONMap) *Span {
return nil
}
func (noopSpanTracker) BeginSubSpan(_ context.Context, _ *Span, _, _ string, _ types.JSONMap) *Span {
return nil
}
func (noopSpanTracker) EndSpan(_ context.Context, _ *Span, _ types.JSONMap) {}
func (noopSpanTracker) FailSpan(_ context.Context, _ *Span, _, _ string, _ error) {}
func (noopSpanTracker) SkipSpan(_ context.Context, _ *Span, _ string) {}
func (noopSpanTracker) LookupStage(_ context.Context, _ string, _ int, _ string) *Span { return nil }
fix(knowledge): make finalizing subtask counter reliable + async question fan-out The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span.
2026-05-29 16:09:46 +00:00
func (noopSpanTracker) LookupSpanByName(_ context.Context, _ string, _ int, _ string) *Span {
return nil
}
fix(knowledge): close root span on terminal state, enrich stage metadata The root span created by OpenAttempt was never closed: PostProcess only ended the postprocess stage, so the root row stayed at status=running forever even after parse_status flipped to completed/failed. The timeline rendered "进行中" indefinitely on the root, defeating the whole "is the document done" question the timeline is meant to answer. - SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg): closes the root row idempotently. Re-closing a terminal root no-ops so success / cascade-fail / dead-letter paths can fire without coordination. - PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess) on the success path. Async downstream work (summary/question/wiki/ graph) still records its own spans; their completion extends the trace's wall-clock end-time but doesn't reopen the root. - FailSpan auto-closes the root when a MAIN pipeline stage fails (docreader / chunking / embedding / multimodal / postprocess). Cascade-cancelled siblings stay closed-with-the-cascade as before. - Dead-letter callback (router/task.go) accepts the SpanTracker via DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a document-related task exhausts retries. The probe payload now extracts the Attempt field that Document/Manual/PostProcess payloads already thread through. Stage spans were also being recorded with nil input/output, leaving the new detail panel with timestamps only. Each Begin/End site now emits useful work metrics: - DocReader input: file_name, file_type, is_url, url output: text_length, images_found, is_audio, pages - Chunking input: chunks_planned output: chunks_written, total_text_chars - Embedding input: chunks_to_embed, model_id, dim output: vectors_written, storage_bytes - Multimodal input: image_count, enable_ocr, enable_caption - PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph i18n: add knowledgeStages.root ("Knowledge processing") so the UI can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
func (noopSpanTracker) FinalizeAttempt(_ context.Context, _ string, _ int, _ string, _ types.JSONMap, _, _ string) {
}
feat: add knowledge parse cancellation with finalizing post-process state Lets users stop an in-flight document parse to free up LLM / worker resources without losing the chunks and index already written. The core insight is that the previous parse_status=completed flipped as soon as primary chunks landed, while the most expensive subtasks (graph extract = N LLM calls per chunk, plus summary, question generation) were still running in the background — so "completed" wasn't actually terminal from a resource standpoint. State machine pending -> processing -> finalizing -> completed | +-> cancelled (any of the three in-flight states) +-> failed +-> deleting `finalizing` is the new post-process fan-out window. parse_status only promotes to `completed` once pending_subtasks_count (a new column tracking summary + question + per-chunk graph extract) drains to zero via atomic FinalizeSubtask. Wiki ingest is intentionally excluded from the counter — it's a KB-scoped debounced batch and would otherwise pin parse_status in `finalizing` for the wiki batch window. Backend - New ParseStatusFinalizing + pending_subtasks_count column with migration 000056. - knowledgeRepository.SetFinalizing transitions processing -> finalizing conditionally so a racing cancel cannot be clobbered. - knowledgeRepository.FinalizeSubtask atomically decrements the counter and self-promotes the row to completed when it hits zero. - KnowledgePostProcess restructured to compute expected subtask count up front, flip to finalizing (or completed when no enrichment is enabled), and only then fan out subtasks. Subtask handlers (summary, question, graph extract) defer-decrement on terminal exit using the existing isFinalAsynqAttempt convention. - New POST /api/v1/knowledge/{id}/cancel-parse handler accepting pending / processing / finalizing. Marks the row cancelled, zeroes the counter, best-effort dequeues asynq tasks via a new TaskInspector abstraction (asynq-mode walks pending/scheduled/ retry queues; Lite-mode noop), and scrubs wiki ingest pending op. - SpanTracker.AbortAttempt flat-sweeps every still-running span for the attempt via a new repo.CancelAllOpenSpans helper so the trace viewer's striped bars all flip to cancelled, even leaf generations whose parent stage already EndSpan'd (multimodal fan-out pattern). knowledge_post_process closes its postSpan via SkipSpan on the cancel/deleting entry guard so a worker that opens a span AFTER the cancel sweep doesn't leak it. - Housekeeping and resetPendingTasks sweep finalizing rows identically to processing so a crash/restart can't strand them. - DeleteKnowledge/DeleteKnowledgeList proactively dequeue downstream tasks via the same TaskInspector path. - ChunkExtractService gets a cancel entry guard so the most expensive enrichment (graph extract) bails immediately when the parent knowledge is aborted. Frontend - New cancelKnowledgeParse API client + "Stop parsing" entry in both list view and card view more menus, gated on pending/processing/finalizing. - Polling predicate refactored to a shared isParseInFlight helper that recognises `finalizing` (previously the doc list silently stopped polling once parse_status flipped from processing). - Knowledge processing timeline: isPolling includes finalizing, new isHardTerminal short-circuits LIVE for cancelled/failed/ completed so stranded child spans cannot pin LIVE on. - DocumentListView.computeStatus distinguishes finalizing ("增强中") from completed and shows the previous "生成摘要中" copy when summary_status is still pending under finalizing. Added cancelled badge as well. - i18n: statusFinalizing / statusCancelled / cancelParse* keys across zh-CN, en-US, ko-KR, ru-RU. Docs / SDK - docs/api/knowledge.md: documents the new finalizing state, cancel-parse semantics, and which statuses accept cancel. - client (Go SDK): CancelKnowledgeParse with docstring listing the cancellable statuses.
2026-05-28 10:26:56 +00:00
func (noopSpanTracker) AbortAttempt(_ context.Context, _ string, _ int, _, _, _ string) {}