refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
// Package service: span tracker.
|
|
|
|
|
//
|
|
|
|
|
// SpanTracker is the pipeline-facing facade for recording per-attempt
|
|
|
|
|
// progress trees. It mirrors Langfuse's vocabulary (root / span /
|
|
|
|
|
// generation) so the UI's mental model matches what operators already use
|
|
|
|
|
// for LLM call observability.
|
|
|
|
|
//
|
|
|
|
|
// Lifecycle:
|
|
|
|
|
//
|
|
|
|
|
// attempt := tracker.OpenAttempt(ctx, knowledgeID, langfuseTraceID)
|
|
|
|
|
// // creates the root span; every subsequent Begin* call uses this attempt
|
|
|
|
|
//
|
|
|
|
|
// stage := tracker.BeginStage(ctx, knowledgeID, attempt, types.StageDocReader, input)
|
|
|
|
|
// // ...do work...
|
|
|
|
|
// tracker.EndSpan(ctx, stage, output) // success
|
|
|
|
|
// tracker.FailSpan(ctx, stage, code, msg, err) // error
|
|
|
|
|
// tracker.SkipSpan(ctx, stage, reason) // intentionally not run
|
|
|
|
|
//
|
|
|
|
|
// sub := tracker.BeginSubSpan(ctx, parentSpan, "multimodal.image[0]", types.SpanKindGeneration, input)
|
|
|
|
|
// // ...
|
|
|
|
|
//
|
|
|
|
|
// All operations are best-effort: a DB error is logged and swallowed so a
|
|
|
|
|
// tracker hiccup never breaks the parsing pipeline. Knowledge.parse_status
|
|
|
|
|
// remains the authoritative source of truth for completion.
|
|
|
|
|
package service
|
|
|
|
|
|
|
|
|
|
import (
|
|
|
|
|
"context"
|
|
|
|
|
"strings"
|
|
|
|
|
"sync"
|
|
|
|
|
"time"
|
|
|
|
|
|
|
|
|
|
"github.com/Tencent/WeKnora/internal/application/repository"
|
|
|
|
|
"github.com/Tencent/WeKnora/internal/logger"
|
|
|
|
|
"github.com/Tencent/WeKnora/internal/types"
|
|
|
|
|
"github.com/google/uuid"
|
2026-05-27 07:01:54 +00:00
|
|
|
"gorm.io/gorm"
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
)
|
|
|
|
|
|
|
|
|
|
// Span is the in-memory handle the pipeline holds while a stage / subspan
|
|
|
|
|
// is executing. It carries enough context for End/Fail/Skip to write back
|
|
|
|
|
// without re-querying the DB. Returned (and required) from every Begin*.
|
|
|
|
|
type Span struct {
|
|
|
|
|
KnowledgeID string
|
|
|
|
|
Attempt int
|
|
|
|
|
SpanID string
|
|
|
|
|
ParentSpanID string
|
|
|
|
|
Name string
|
|
|
|
|
Kind string
|
|
|
|
|
StartedAt time.Time
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// SpanTracker is the only public surface — kept as an interface so tests
|
|
|
|
|
// can swap in a no-op without spinning up a database.
|
|
|
|
|
type SpanTracker interface {
|
|
|
|
|
// OpenAttempt creates a new root span for (knowledgeID,
|
|
|
|
|
// nextAttempt) and returns its number plus the root *Span. Call
|
|
|
|
|
// at the start of a parse / reparse, before any other Begin*.
|
|
|
|
|
OpenAttempt(ctx context.Context, knowledgeID, langfuseTraceID string) (root *Span, attempt int, err error)
|
|
|
|
|
|
|
|
|
|
// LatestAttempt returns the highest attempt number recorded for
|
|
|
|
|
// the knowledge, or 0 if it's never been parsed. Used by the API
|
|
|
|
|
// layer to default to "show me the most recent run".
|
|
|
|
|
LatestAttempt(ctx context.Context, knowledgeID string) int
|
|
|
|
|
|
|
|
|
|
// BeginStage starts one of the canonical stages. Looks up the
|
|
|
|
|
// root span for (kid, attempt) — caller passes attempt to make
|
|
|
|
|
// the wiring explicit and let cross-process workers join an
|
|
|
|
|
// existing attempt without new repo lookups.
|
|
|
|
|
BeginStage(ctx context.Context, knowledgeID string, attempt int, stage string, input types.JSONMap) *Span
|
|
|
|
|
|
|
|
|
|
// BeginSubSpan creates a child span under parent. parent may be a
|
|
|
|
|
// stage span (for multimodal.image[i] / embedding.batch[i]) or
|
|
|
|
|
// another subspan. kind is "subspan" or "generation" — generations
|
|
|
|
|
// will be stitched to a Langfuse generation by trace_id.
|
|
|
|
|
BeginSubSpan(ctx context.Context, parent *Span, name, kind string, input types.JSONMap) *Span
|
|
|
|
|
|
|
|
|
|
// EndSpan marks span as done with optional output. Safe with nil.
|
|
|
|
|
EndSpan(ctx context.Context, span *Span, output types.JSONMap)
|
|
|
|
|
|
|
|
|
|
// FailSpan marks span as failed and cascade-cancels its
|
|
|
|
|
// descendants. errorDetail (a Go error) is recorded verbatim in
|
|
|
|
|
// error_detail (truncated to 8 KB) for admin views.
|
|
|
|
|
FailSpan(ctx context.Context, span *Span, errorCode, errorMessage string, errorDetail error)
|
|
|
|
|
|
|
|
|
|
// SkipSpan marks an intentionally not-run span (e.g. multimodal
|
|
|
|
|
// on a text-only document). Distinct from cancelled — skipped is
|
|
|
|
|
// "we chose not to" while cancelled is "an upstream broke".
|
|
|
|
|
SkipSpan(ctx context.Context, span *Span, reason string)
|
|
|
|
|
|
|
|
|
|
// LookupStage returns the stage's *Span for an in-flight attempt
|
|
|
|
|
// — the cross-process bridge that lets an asynq worker (e.g.
|
|
|
|
|
// image_multimodal) attach subspans to the parent stage span
|
|
|
|
|
// created by the upstream pipeline.
|
|
|
|
|
LookupStage(ctx context.Context, knowledgeID string, attempt int, stage string) *Span
|
fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.
- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
closes the root row idempotently. Re-closing a terminal root no-ops
so success / cascade-fail / dead-letter paths can fire without
coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
on the success path. Async downstream work (summary/question/wiki/
graph) still records its own spans; their completion extends the
trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
(docreader / chunking / embedding / multimodal / postprocess).
Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
document-related task exhausts retries. The probe payload now
extracts the Attempt field that Document/Manual/PostProcess
payloads already thread through.
Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph
i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
|
|
|
|
2026-05-29 16:09:46 +00:00
|
|
|
// LookupSpanByName returns the first span of any kind matching name
|
|
|
|
|
// for (knowledgeID, attempt) — the cross-process bridge that lets a
|
|
|
|
|
// fan-out worker (e.g. a question-generation batch) attach its subspan
|
|
|
|
|
// under a grouping span created earlier by the orchestrator. Returns
|
|
|
|
|
// nil when no such span exists (caller should fall back to the stage).
|
|
|
|
|
LookupSpanByName(ctx context.Context, knowledgeID string, attempt int, name string) *Span
|
|
|
|
|
|
fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.
- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
closes the root row idempotently. Re-closing a terminal root no-ops
so success / cascade-fail / dead-letter paths can fire without
coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
on the success path. Async downstream work (summary/question/wiki/
graph) still records its own spans; their completion extends the
trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
(docreader / chunking / embedding / multimodal / postprocess).
Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
document-related task exhausts retries. The probe payload now
extracts the Attempt field that Document/Manual/PostProcess
payloads already thread through.
Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph
i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
|
|
|
// FinalizeAttempt closes the root span for (knowledgeID, attempt)
|
|
|
|
|
// with the given terminal status (done | failed). Idempotent:
|
|
|
|
|
// re-closing an already-terminal root is a no-op so callers from
|
|
|
|
|
// multiple paths (success orchestrator, dead-letter handler,
|
|
|
|
|
// housekeeping) can fire without coordination. status defaults to
|
|
|
|
|
// done; output/error are written verbatim.
|
|
|
|
|
FinalizeAttempt(ctx context.Context, knowledgeID string, attempt int, status string,
|
|
|
|
|
output types.JSONMap, errorCode, errorMessage string)
|
feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
2026-05-28 10:26:56 +00:00
|
|
|
|
|
|
|
|
// AbortAttempt cascade-cancels every still-running descendant of the
|
|
|
|
|
// attempt's root span and then closes the root as cancelled. Used by
|
|
|
|
|
// the user-initiated cancel path so the trace viewer doesn't leave
|
|
|
|
|
// stranded subspans (multimodal images, postprocess subtasks)
|
|
|
|
|
// looking like they're still in flight forever after the user
|
|
|
|
|
// stopped the parse. Idempotent.
|
|
|
|
|
AbortAttempt(ctx context.Context, knowledgeID string, attempt int, errorCode, errorMessage, reason string)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
type spanTracker struct {
|
|
|
|
|
repo repository.KnowledgeSpanRepository
|
2026-05-27 07:01:54 +00:00
|
|
|
// db is held purely for the heartbeat side-channel: every span
|
|
|
|
|
// state transition pokes knowledge.updated_at so the housekeeping
|
|
|
|
|
// sweep can tell "actively running long stage" from "abandoned".
|
|
|
|
|
// nil-safe — when missing (test harness) the heartbeat is skipped.
|
|
|
|
|
db *gorm.DB
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
|
|
|
|
|
// startsMu guards the in-process duration cache. Cross-process
|
|
|
|
|
// workers won't find their parent's start here — that's fine,
|
|
|
|
|
// duration_ms falls back to (FinishedAt - row.StartedAt) computed
|
|
|
|
|
// at write time when the cache misses.
|
|
|
|
|
startsMu sync.Mutex
|
|
|
|
|
starts map[string]time.Time // span_id → started_at
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// NewSpanTracker constructs the GORM-backed tracker. A nil repo collapses
|
2026-05-27 07:01:54 +00:00
|
|
|
// to a no-op so test harnesses don't need to spin up a database. The db
|
|
|
|
|
// is optional too: it's used only for the housekeeping heartbeat (see
|
|
|
|
|
// touchKnowledgeHeartbeat) and a nil db just disables that side-channel.
|
|
|
|
|
func NewSpanTracker(repo repository.KnowledgeSpanRepository, db *gorm.DB) SpanTracker {
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
if repo == nil {
|
|
|
|
|
return noopSpanTracker{}
|
|
|
|
|
}
|
|
|
|
|
return &spanTracker{
|
|
|
|
|
repo: repo,
|
2026-05-27 07:01:54 +00:00
|
|
|
db: db,
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
starts: make(map[string]time.Time),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2026-05-27 07:01:54 +00:00
|
|
|
// touchKnowledgeHeartbeat advances knowledge.updated_at to the current
|
|
|
|
|
// wall-clock so the housekeeping sweep treats this row as actively
|
|
|
|
|
// progressing. Called on every span Begin/End/Fail/Skip — the cost is
|
|
|
|
|
// one indexed UPDATE per transition (≤ a few dozen per knowledge), which
|
|
|
|
|
// is dwarfed by the work the stages themselves do.
|
|
|
|
|
//
|
2026-05-27 07:31:32 +00:00
|
|
|
// touchKnowledgeHeartbeat advances knowledge.updated_at to the current
|
|
|
|
|
// wall-clock so the housekeeping sweep treats this row as actively
|
|
|
|
|
// progressing. Called on root / stage span transitions only — subspan and
|
|
|
|
|
// generation transitions skip this side-channel because:
|
|
|
|
|
//
|
|
|
|
|
// - The spans table itself is updated on every transition, and the
|
|
|
|
|
// housekeeping sweep already reads MAX(spans.updated_at) per
|
|
|
|
|
// knowledge, so subspan progress is observable without poking the
|
|
|
|
|
// parent row.
|
|
|
|
|
// - A multimodal stage with N images would produce 2*N+ extra UPDATEs
|
|
|
|
|
// on the same hot row (Begin+End per image plus retries), which we
|
|
|
|
|
// observed contributing to row-level contention under bursty
|
|
|
|
|
// uploads.
|
|
|
|
|
//
|
2026-05-27 07:01:54 +00:00
|
|
|
// Best-effort. We deliberately do NOT bump status here: the parse_status
|
|
|
|
|
// column remains under the pipeline's control. Only the timestamp gets
|
2026-05-27 07:31:32 +00:00
|
|
|
// nudged, which is exactly what housekeeping reads as the fallback.
|
|
|
|
|
func (t *spanTracker) touchKnowledgeHeartbeat(ctx context.Context, knowledgeID, kind string) {
|
2026-05-27 07:01:54 +00:00
|
|
|
if t.db == nil || knowledgeID == "" {
|
|
|
|
|
return
|
|
|
|
|
}
|
2026-05-27 07:31:32 +00:00
|
|
|
// Subspan / generation transitions are observable through the spans
|
|
|
|
|
// table directly; skip the parent-row UPDATE to avoid write
|
|
|
|
|
// amplification on fan-out workloads.
|
|
|
|
|
if kind != types.SpanKindRoot && kind != types.SpanKindStage {
|
|
|
|
|
return
|
|
|
|
|
}
|
2026-05-27 07:01:54 +00:00
|
|
|
if err := t.db.WithContext(ctx).Model(&types.Knowledge{}).
|
|
|
|
|
Where("id = ?", knowledgeID).
|
|
|
|
|
Update("updated_at", time.Now()).Error; err != nil {
|
|
|
|
|
// Don't log every failure — heartbeat is best-effort and
|
|
|
|
|
// noisy logs would drown out real errors. Single line at
|
|
|
|
|
// warn level is enough for ops to spot a chronic outage.
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] heartbeat update failed kid=%s: %v", knowledgeID, err)
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
func newSpanID() string {
|
|
|
|
|
// Stripping the dashes saves 4 bytes per row — JSON parsers don't
|
|
|
|
|
// care, and operators paste these into queries / Langfuse where a
|
|
|
|
|
// hex-only ID is friendlier.
|
|
|
|
|
return strings.ReplaceAll(uuid.NewString(), "-", "")
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) recordStart(spanID string, at time.Time) {
|
|
|
|
|
t.startsMu.Lock()
|
|
|
|
|
t.starts[spanID] = at
|
|
|
|
|
t.startsMu.Unlock()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) takeStart(spanID string) (time.Time, bool) {
|
|
|
|
|
t.startsMu.Lock()
|
|
|
|
|
defer t.startsMu.Unlock()
|
|
|
|
|
v, ok := t.starts[spanID]
|
|
|
|
|
if ok {
|
|
|
|
|
delete(t.starts, spanID)
|
|
|
|
|
}
|
|
|
|
|
return v, ok
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) OpenAttempt(ctx context.Context, knowledgeID, langfuseTraceID string) (*Span, int, error) {
|
|
|
|
|
attempt, err := t.repo.NextAttempt(ctx, knowledgeID)
|
|
|
|
|
if err != nil {
|
|
|
|
|
return nil, 0, err
|
|
|
|
|
}
|
|
|
|
|
now := time.Now()
|
|
|
|
|
rootID := newSpanID()
|
|
|
|
|
meta := types.JSONMap{}
|
|
|
|
|
if langfuseTraceID != "" {
|
|
|
|
|
// The frontend renders a "open in Langfuse" link from this.
|
|
|
|
|
meta["langfuse_trace_id"] = langfuseTraceID
|
|
|
|
|
}
|
|
|
|
|
row := &types.KnowledgeProcessingSpan{
|
|
|
|
|
KnowledgeID: knowledgeID,
|
|
|
|
|
Attempt: attempt,
|
|
|
|
|
SpanID: rootID,
|
|
|
|
|
Name: "knowledge_processing",
|
|
|
|
|
Kind: types.SpanKindRoot,
|
|
|
|
|
Status: types.SpanStatusRunning,
|
|
|
|
|
Metadata: meta,
|
|
|
|
|
StartedAt: &now,
|
|
|
|
|
}
|
|
|
|
|
if err := t.repo.Upsert(ctx, row); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] OpenAttempt failed kid=%s: %v", knowledgeID, err)
|
|
|
|
|
return nil, attempt, err
|
|
|
|
|
}
|
|
|
|
|
t.recordStart(rootID, now)
|
2026-05-27 07:31:32 +00:00
|
|
|
t.touchKnowledgeHeartbeat(ctx, knowledgeID, types.SpanKindRoot)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
return &Span{
|
|
|
|
|
KnowledgeID: knowledgeID,
|
|
|
|
|
Attempt: attempt,
|
|
|
|
|
SpanID: rootID,
|
|
|
|
|
Name: "knowledge_processing",
|
|
|
|
|
Kind: types.SpanKindRoot,
|
|
|
|
|
StartedAt: now,
|
|
|
|
|
}, attempt, nil
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) LatestAttempt(ctx context.Context, knowledgeID string) int {
|
|
|
|
|
n, err := t.repo.LatestAttempt(ctx, knowledgeID)
|
|
|
|
|
if err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] LatestAttempt failed kid=%s: %v", knowledgeID, err)
|
|
|
|
|
return 0
|
|
|
|
|
}
|
|
|
|
|
return n
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) BeginStage(ctx context.Context, knowledgeID string, attempt int, stage string, input types.JSONMap) *Span {
|
|
|
|
|
if knowledgeID == "" || stage == "" {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
2026-05-27 07:31:32 +00:00
|
|
|
// Find root span — we need its span_id as parent for stage rows. We
|
|
|
|
|
// also reuse this scan to detect an existing row for the same stage
|
|
|
|
|
// name in this attempt: re-entry (asynq retry, double-call from
|
|
|
|
|
// adjacent code paths) MUST NOT create a second row, otherwise the
|
|
|
|
|
// timeline shows two segments for the same stage and LookupStage
|
|
|
|
|
// becomes ambiguous.
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
rows, err := t.repo.ListByAttempt(ctx, knowledgeID, attempt)
|
|
|
|
|
if err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] BeginStage list failed kid=%s attempt=%d: %v",
|
|
|
|
|
knowledgeID, attempt, err)
|
|
|
|
|
return nil
|
|
|
|
|
}
|
2026-05-27 07:31:32 +00:00
|
|
|
var (
|
|
|
|
|
rootID string
|
|
|
|
|
existing *types.KnowledgeProcessingSpan
|
|
|
|
|
)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
for i := range rows {
|
2026-05-27 07:31:32 +00:00
|
|
|
r := rows[i]
|
|
|
|
|
if r.Kind == types.SpanKindRoot && rootID == "" {
|
|
|
|
|
rootID = r.SpanID
|
|
|
|
|
}
|
|
|
|
|
if r.Kind == types.SpanKindStage && r.Name == stage {
|
|
|
|
|
cp := r
|
|
|
|
|
existing = &cp
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
if rootID == "" {
|
|
|
|
|
// Pipeline started before tracker was wired (legacy data,
|
|
|
|
|
// or the OpenAttempt repo write failed). Synthesize a
|
|
|
|
|
// rootless stage so we still record SOMETHING.
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] BeginStage: no root for kid=%s attempt=%d, recording rootless",
|
|
|
|
|
knowledgeID, attempt)
|
|
|
|
|
}
|
|
|
|
|
now := time.Now()
|
2026-05-27 07:31:32 +00:00
|
|
|
// Re-entry path: keep the original span_id so any subspan that
|
|
|
|
|
// already references it stays attached. Reset state to running and
|
|
|
|
|
// refresh started_at; clear terminal-only fields so the row reads
|
|
|
|
|
// cleanly as "running again". Output/error fields go through
|
|
|
|
|
// Upsert's DoUpdates list, which only writes the columns we set —
|
|
|
|
|
// any nil JSONMap / empty string explicitly clears the column.
|
|
|
|
|
if existing != nil {
|
|
|
|
|
row := &types.KnowledgeProcessingSpan{
|
|
|
|
|
KnowledgeID: existing.KnowledgeID,
|
|
|
|
|
Attempt: existing.Attempt,
|
|
|
|
|
SpanID: existing.SpanID,
|
|
|
|
|
ParentSpanID: existing.ParentSpanID,
|
|
|
|
|
Name: existing.Name,
|
|
|
|
|
Kind: existing.Kind,
|
|
|
|
|
Status: types.SpanStatusRunning,
|
|
|
|
|
Input: input,
|
|
|
|
|
Output: nil,
|
|
|
|
|
StartedAt: &now,
|
|
|
|
|
FinishedAt: nil,
|
|
|
|
|
DurationMs: 0,
|
|
|
|
|
}
|
|
|
|
|
if err := t.repo.Upsert(ctx, row); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] BeginStage re-enter failed kid=%s stage=%s: %v",
|
|
|
|
|
knowledgeID, stage, err)
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
t.recordStart(existing.SpanID, now)
|
|
|
|
|
t.touchKnowledgeHeartbeat(ctx, knowledgeID, types.SpanKindStage)
|
|
|
|
|
return &Span{
|
|
|
|
|
KnowledgeID: existing.KnowledgeID,
|
|
|
|
|
Attempt: existing.Attempt,
|
|
|
|
|
SpanID: existing.SpanID,
|
|
|
|
|
ParentSpanID: existing.ParentSpanID,
|
|
|
|
|
Name: existing.Name,
|
|
|
|
|
Kind: existing.Kind,
|
|
|
|
|
StartedAt: now,
|
|
|
|
|
}
|
|
|
|
|
}
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
id := newSpanID()
|
|
|
|
|
row := &types.KnowledgeProcessingSpan{
|
|
|
|
|
KnowledgeID: knowledgeID,
|
|
|
|
|
Attempt: attempt,
|
|
|
|
|
SpanID: id,
|
|
|
|
|
ParentSpanID: rootID,
|
|
|
|
|
Name: stage,
|
|
|
|
|
Kind: types.SpanKindStage,
|
|
|
|
|
Status: types.SpanStatusRunning,
|
|
|
|
|
Input: input,
|
|
|
|
|
StartedAt: &now,
|
|
|
|
|
}
|
|
|
|
|
if err := t.repo.Upsert(ctx, row); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] BeginStage failed kid=%s stage=%s: %v",
|
|
|
|
|
knowledgeID, stage, err)
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
t.recordStart(id, now)
|
2026-05-27 07:31:32 +00:00
|
|
|
t.touchKnowledgeHeartbeat(ctx, knowledgeID, types.SpanKindStage)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
return &Span{
|
|
|
|
|
KnowledgeID: knowledgeID,
|
|
|
|
|
Attempt: attempt,
|
|
|
|
|
SpanID: id,
|
|
|
|
|
ParentSpanID: rootID,
|
|
|
|
|
Name: stage,
|
|
|
|
|
Kind: types.SpanKindStage,
|
|
|
|
|
StartedAt: now,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) BeginSubSpan(ctx context.Context, parent *Span, name, kind string, input types.JSONMap) *Span {
|
|
|
|
|
if parent == nil || name == "" {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
if kind != types.SpanKindGeneration && kind != types.SpanKindSubSpan {
|
|
|
|
|
kind = types.SpanKindSubSpan
|
|
|
|
|
}
|
2026-05-28 12:09:49 +00:00
|
|
|
// Asynq retry / server restart can re-run the same handler while the
|
|
|
|
|
// previous invocation's span is still status=running (worker died
|
|
|
|
|
// without EndSpan). Cancel same-name open rows so the UI shows one
|
|
|
|
|
// logical subspan per (attempt, name) instead of duplicate stripes.
|
|
|
|
|
if _, err := t.repo.CancelOpenSpansByName(ctx, parent.KnowledgeID, parent.Attempt, name,
|
|
|
|
|
"TASK_SUPERSEDED", "superseded by a new run of the same subtask"); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] supersede %s before BeginSubSpan failed: %v", name, err)
|
|
|
|
|
}
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
now := time.Now()
|
|
|
|
|
id := newSpanID()
|
|
|
|
|
row := &types.KnowledgeProcessingSpan{
|
|
|
|
|
KnowledgeID: parent.KnowledgeID,
|
|
|
|
|
Attempt: parent.Attempt,
|
|
|
|
|
SpanID: id,
|
|
|
|
|
ParentSpanID: parent.SpanID,
|
|
|
|
|
Name: name,
|
|
|
|
|
Kind: kind,
|
|
|
|
|
Status: types.SpanStatusRunning,
|
|
|
|
|
Input: input,
|
|
|
|
|
StartedAt: &now,
|
|
|
|
|
}
|
|
|
|
|
if err := t.repo.Upsert(ctx, row); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] BeginSubSpan failed parent=%s name=%s: %v",
|
|
|
|
|
parent.SpanID, name, err)
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
t.recordStart(id, now)
|
2026-05-27 07:31:32 +00:00
|
|
|
t.touchKnowledgeHeartbeat(ctx, parent.KnowledgeID, kind)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
return &Span{
|
|
|
|
|
KnowledgeID: parent.KnowledgeID,
|
|
|
|
|
Attempt: parent.Attempt,
|
|
|
|
|
SpanID: id,
|
|
|
|
|
ParentSpanID: parent.SpanID,
|
|
|
|
|
Name: name,
|
|
|
|
|
Kind: kind,
|
|
|
|
|
StartedAt: now,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) EndSpan(ctx context.Context, span *Span, output types.JSONMap) {
|
|
|
|
|
if span == nil {
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
now := time.Now()
|
|
|
|
|
dur := durationSince(t, span, now)
|
|
|
|
|
row := &types.KnowledgeProcessingSpan{
|
|
|
|
|
KnowledgeID: span.KnowledgeID,
|
|
|
|
|
Attempt: span.Attempt,
|
|
|
|
|
SpanID: span.SpanID,
|
|
|
|
|
ParentSpanID: span.ParentSpanID,
|
|
|
|
|
Name: span.Name,
|
|
|
|
|
Kind: span.Kind,
|
|
|
|
|
Status: types.SpanStatusDone,
|
|
|
|
|
Output: output,
|
|
|
|
|
StartedAt: &span.StartedAt,
|
|
|
|
|
FinishedAt: &now,
|
|
|
|
|
DurationMs: dur,
|
|
|
|
|
}
|
|
|
|
|
if err := t.repo.Upsert(ctx, row); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] EndSpan failed span=%s: %v", span.SpanID, err)
|
|
|
|
|
}
|
2026-05-27 07:31:32 +00:00
|
|
|
t.touchKnowledgeHeartbeat(ctx, span.KnowledgeID, span.Kind)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) FailSpan(ctx context.Context, span *Span, errorCode, errorMessage string, errorDetail error) {
|
|
|
|
|
if span == nil {
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
now := time.Now()
|
|
|
|
|
dur := durationSince(t, span, now)
|
|
|
|
|
detail := ""
|
|
|
|
|
if errorDetail != nil {
|
|
|
|
|
detail = errorDetail.Error()
|
|
|
|
|
if len(detail) > 8192 {
|
|
|
|
|
detail = detail[:8192]
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
if len(errorMessage) > 1024 {
|
|
|
|
|
errorMessage = errorMessage[:1024]
|
|
|
|
|
}
|
|
|
|
|
row := &types.KnowledgeProcessingSpan{
|
|
|
|
|
KnowledgeID: span.KnowledgeID,
|
|
|
|
|
Attempt: span.Attempt,
|
|
|
|
|
SpanID: span.SpanID,
|
|
|
|
|
ParentSpanID: span.ParentSpanID,
|
|
|
|
|
Name: span.Name,
|
|
|
|
|
Kind: span.Kind,
|
|
|
|
|
Status: types.SpanStatusFailed,
|
|
|
|
|
ErrorCode: strings.TrimSpace(errorCode),
|
|
|
|
|
ErrorMessage: errorMessage,
|
|
|
|
|
ErrorDetail: detail,
|
|
|
|
|
StartedAt: &span.StartedAt,
|
|
|
|
|
FinishedAt: &now,
|
|
|
|
|
DurationMs: dur,
|
|
|
|
|
}
|
|
|
|
|
if err := t.repo.Upsert(ctx, row); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] FailSpan failed span=%s: %v", span.SpanID, err)
|
|
|
|
|
}
|
|
|
|
|
// Cascade: anything downstream of this span gets cancelled. The
|
|
|
|
|
// reason string is what the UI surfaces under each cancelled
|
|
|
|
|
// child's tooltip — keep it short and human.
|
|
|
|
|
reason := "upstream " + span.Name + " failed"
|
|
|
|
|
if errorCode != "" {
|
|
|
|
|
reason = reason + " (" + errorCode + ")"
|
|
|
|
|
}
|
|
|
|
|
if _, err := t.repo.CancelDescendants(ctx, span.KnowledgeID, span.Attempt, span.SpanID, reason); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] cancel descendants failed span=%s: %v", span.SpanID, err)
|
|
|
|
|
}
|
|
|
|
|
// For STAGE failures, also cascade to dependent stages declared
|
|
|
|
|
// in StageDependencies (those are siblings, not descendants).
|
|
|
|
|
if span.Kind == types.SpanKindStage {
|
|
|
|
|
t.cascadeDependentStages(ctx, span, reason)
|
fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.
- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
closes the root row idempotently. Re-closing a terminal root no-ops
so success / cascade-fail / dead-letter paths can fire without
coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
on the success path. Async downstream work (summary/question/wiki/
graph) still records its own spans; their completion extends the
trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
(docreader / chunking / embedding / multimodal / postprocess).
Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
document-related task exhausts retries. The probe payload now
extracts the Attempt field that Document/Manual/PostProcess
payloads already thread through.
Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph
i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
|
|
|
// Any failure in a MAIN pipeline stage means the attempt is
|
|
|
|
|
// done — the parse cannot succeed past this point. Close the
|
|
|
|
|
// root span as failed so the UI doesn't show "进行中" forever.
|
|
|
|
|
// Optional downstream stages (summary/question/wiki/graph) do
|
|
|
|
|
// not poison the attempt: they can fail without invalidating
|
|
|
|
|
// the parsed document.
|
|
|
|
|
if isMainPipelineStage(span.Name) {
|
|
|
|
|
t.FinalizeAttempt(ctx, span.KnowledgeID, span.Attempt,
|
|
|
|
|
types.SpanStatusFailed, nil, errorCode, errorMessage)
|
|
|
|
|
}
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
}
|
2026-05-27 07:31:32 +00:00
|
|
|
t.touchKnowledgeHeartbeat(ctx, span.KnowledgeID, span.Kind)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) SkipSpan(ctx context.Context, span *Span, reason string) {
|
|
|
|
|
if span == nil {
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
now := time.Now()
|
|
|
|
|
row := &types.KnowledgeProcessingSpan{
|
|
|
|
|
KnowledgeID: span.KnowledgeID,
|
|
|
|
|
Attempt: span.Attempt,
|
|
|
|
|
SpanID: span.SpanID,
|
|
|
|
|
ParentSpanID: span.ParentSpanID,
|
|
|
|
|
Name: span.Name,
|
|
|
|
|
Kind: span.Kind,
|
|
|
|
|
Status: types.SpanStatusSkipped,
|
|
|
|
|
ErrorMessage: reason,
|
|
|
|
|
StartedAt: &span.StartedAt,
|
|
|
|
|
FinishedAt: &now,
|
|
|
|
|
}
|
|
|
|
|
if err := t.repo.Upsert(ctx, row); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] SkipSpan failed span=%s: %v", span.SpanID, err)
|
|
|
|
|
}
|
2026-05-27 07:31:32 +00:00
|
|
|
t.touchKnowledgeHeartbeat(ctx, span.KnowledgeID, span.Kind)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *spanTracker) LookupStage(ctx context.Context, knowledgeID string, attempt int, stage string) *Span {
|
|
|
|
|
rows, err := t.repo.ListByAttempt(ctx, knowledgeID, attempt)
|
|
|
|
|
if err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] LookupStage list failed kid=%s attempt=%d: %v",
|
|
|
|
|
knowledgeID, attempt, err)
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
for i := range rows {
|
|
|
|
|
r := rows[i]
|
|
|
|
|
if r.Kind != types.SpanKindStage || r.Name != stage {
|
|
|
|
|
continue
|
|
|
|
|
}
|
|
|
|
|
started := time.Time{}
|
|
|
|
|
if r.StartedAt != nil {
|
|
|
|
|
started = *r.StartedAt
|
|
|
|
|
}
|
|
|
|
|
return &Span{
|
|
|
|
|
KnowledgeID: r.KnowledgeID,
|
|
|
|
|
Attempt: r.Attempt,
|
|
|
|
|
SpanID: r.SpanID,
|
|
|
|
|
ParentSpanID: r.ParentSpanID,
|
|
|
|
|
Name: r.Name,
|
|
|
|
|
Kind: r.Kind,
|
|
|
|
|
StartedAt: started,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
|
2026-05-29 16:09:46 +00:00
|
|
|
func (t *spanTracker) LookupSpanByName(ctx context.Context, knowledgeID string, attempt int, name string) *Span {
|
|
|
|
|
if name == "" || knowledgeID == "" || attempt <= 0 {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
rows, err := t.repo.ListByAttempt(ctx, knowledgeID, attempt)
|
|
|
|
|
if err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] LookupSpanByName list failed kid=%s attempt=%d: %v",
|
|
|
|
|
knowledgeID, attempt, err)
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
for i := range rows {
|
|
|
|
|
r := rows[i]
|
|
|
|
|
if r.Name != name {
|
|
|
|
|
continue
|
|
|
|
|
}
|
|
|
|
|
started := time.Time{}
|
|
|
|
|
if r.StartedAt != nil {
|
|
|
|
|
started = *r.StartedAt
|
|
|
|
|
}
|
|
|
|
|
return &Span{
|
|
|
|
|
KnowledgeID: r.KnowledgeID,
|
|
|
|
|
Attempt: r.Attempt,
|
|
|
|
|
SpanID: r.SpanID,
|
|
|
|
|
ParentSpanID: r.ParentSpanID,
|
|
|
|
|
Name: r.Name,
|
|
|
|
|
Kind: r.Kind,
|
|
|
|
|
StartedAt: started,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
// cascadeDependentStages flips downstream STAGE rows to "cancelled" using
|
|
|
|
|
// types.StageDependencies. Without this, a Chunking failure leaves
|
|
|
|
|
// Embedding / Multimodal as "pending" forever, even though they cannot
|
2026-05-27 07:31:32 +00:00
|
|
|
// possibly run. After flipping a dependent stage we ALSO cascade-cancel
|
|
|
|
|
// any subspan/generation that already attached to it (e.g. an embedding
|
|
|
|
|
// batch that started before the chunking failure was observed) — without
|
|
|
|
|
// this second walk those subspans would remain in pending/running and
|
|
|
|
|
// surface as orphan spinners under a cancelled parent.
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
func (t *spanTracker) cascadeDependentStages(ctx context.Context, failedStage *Span, reason string) {
|
|
|
|
|
rows, err := t.repo.ListByAttempt(ctx, failedStage.KnowledgeID, failedStage.Attempt)
|
|
|
|
|
if err != nil {
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
dependents := stagesDependingOn(failedStage.Name)
|
|
|
|
|
if len(dependents) == 0 {
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
now := time.Now()
|
|
|
|
|
for _, row := range rows {
|
|
|
|
|
if row.Kind != types.SpanKindStage {
|
|
|
|
|
continue
|
|
|
|
|
}
|
|
|
|
|
if row.Status != types.SpanStatusPending && row.Status != types.SpanStatusRunning {
|
|
|
|
|
continue
|
|
|
|
|
}
|
|
|
|
|
if !contains(dependents, row.Name) {
|
|
|
|
|
continue
|
|
|
|
|
}
|
|
|
|
|
updated := row // copy
|
|
|
|
|
updated.Status = types.SpanStatusCancelled
|
|
|
|
|
updated.ErrorCode = "UPSTREAM_FAILED"
|
|
|
|
|
updated.ErrorMessage = reason
|
|
|
|
|
updated.FinishedAt = &now
|
|
|
|
|
if err := t.repo.Upsert(ctx, &updated); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] cascade dependent stage %s: %v", row.Name, err)
|
2026-05-27 07:31:32 +00:00
|
|
|
continue
|
|
|
|
|
}
|
|
|
|
|
// Recurse into the cascaded stage's own subtree so any
|
|
|
|
|
// in-flight subspan/generation is cancelled too. The
|
|
|
|
|
// repo-level walk is iterative and cheap (small fan-out).
|
|
|
|
|
if _, err := t.repo.CancelDescendants(ctx, row.KnowledgeID, row.Attempt, row.SpanID, reason); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] cascade descendants of dependent %s: %v", row.Name, err)
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// stagesDependingOn returns the transitive closure of stages that have
|
|
|
|
|
// `stage` as an upstream dependency (direct or indirect). Computed by
|
|
|
|
|
// reverse-walking StageDependencies; the result is bounded to 5 since
|
|
|
|
|
// AllStages has five members, so a naive O(N²) walk is fine.
|
|
|
|
|
func stagesDependingOn(stage string) []string {
|
|
|
|
|
var out []string
|
|
|
|
|
seen := map[string]bool{}
|
|
|
|
|
frontier := []string{stage}
|
|
|
|
|
for len(frontier) > 0 {
|
|
|
|
|
var next []string
|
|
|
|
|
for _, candidate := range types.AllStages {
|
|
|
|
|
if seen[candidate] {
|
|
|
|
|
continue
|
|
|
|
|
}
|
|
|
|
|
deps := types.StageDependencies[candidate]
|
|
|
|
|
for _, d := range deps {
|
|
|
|
|
if contains(frontier, d) {
|
|
|
|
|
seen[candidate] = true
|
|
|
|
|
out = append(out, candidate)
|
|
|
|
|
next = append(next, candidate)
|
|
|
|
|
break
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
frontier = next
|
|
|
|
|
}
|
|
|
|
|
return out
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func contains(haystack []string, needle string) bool {
|
|
|
|
|
for _, h := range haystack {
|
|
|
|
|
if h == needle {
|
|
|
|
|
return true
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return false
|
|
|
|
|
}
|
|
|
|
|
|
fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.
- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
closes the root row idempotently. Re-closing a terminal root no-ops
so success / cascade-fail / dead-letter paths can fire without
coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
on the success path. Async downstream work (summary/question/wiki/
graph) still records its own spans; their completion extends the
trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
(docreader / chunking / embedding / multimodal / postprocess).
Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
document-related task exhausts retries. The probe payload now
extracts the Attempt field that Document/Manual/PostProcess
payloads already thread through.
Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph
i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
|
|
|
// isMainPipelineStage reports whether stage is one of the 5 mandatory
|
|
|
|
|
// pipeline stages (docreader / chunking / embedding / multimodal /
|
|
|
|
|
// postprocess). A failure in any of these terminally invalidates the
|
|
|
|
|
// attempt and must close the root as failed. Optional downstream stages
|
|
|
|
|
// added later (summary, question, wiki, graph) do NOT match — those
|
|
|
|
|
// can fail individually without poisoning the parse result.
|
|
|
|
|
func isMainPipelineStage(name string) bool {
|
|
|
|
|
for _, s := range types.AllStages {
|
|
|
|
|
if s == name {
|
|
|
|
|
return true
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return false
|
|
|
|
|
}
|
|
|
|
|
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
// durationSince computes elapsed ms preferring the in-process cache;
|
|
|
|
|
// falls back to the *Span's StartedAt for cross-process callers.
|
|
|
|
|
func durationSince(t *spanTracker, span *Span, now time.Time) int64 {
|
|
|
|
|
if start, ok := t.takeStart(span.SpanID); ok {
|
|
|
|
|
return now.Sub(start).Milliseconds()
|
|
|
|
|
}
|
|
|
|
|
if !span.StartedAt.IsZero() {
|
|
|
|
|
return now.Sub(span.StartedAt).Milliseconds()
|
|
|
|
|
}
|
|
|
|
|
return 0
|
|
|
|
|
}
|
|
|
|
|
|
fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.
- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
closes the root row idempotently. Re-closing a terminal root no-ops
so success / cascade-fail / dead-letter paths can fire without
coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
on the success path. Async downstream work (summary/question/wiki/
graph) still records its own spans; their completion extends the
trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
(docreader / chunking / embedding / multimodal / postprocess).
Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
document-related task exhausts retries. The probe payload now
extracts the Attempt field that Document/Manual/PostProcess
payloads already thread through.
Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph
i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
|
|
|
// FinalizeAttempt closes the root span for (knowledgeID, attempt). The
|
|
|
|
|
// pipeline calls this from two places: the success orchestrator
|
|
|
|
|
// (PostProcess) when the parse reaches Completed, and FailSpan when a
|
|
|
|
|
// MAIN stage fails terminally. Without this, the root row created by
|
|
|
|
|
// OpenAttempt would stay in "running" forever even after parse_status
|
|
|
|
|
// flips to completed/failed — operators would see a perpetually
|
|
|
|
|
// "running" trace despite a terminal knowledge state.
|
|
|
|
|
//
|
|
|
|
|
// Idempotent: re-closing a root that's already done/failed is a no-op
|
|
|
|
|
// so callers from different paths (success vs. cascade-fail vs.
|
|
|
|
|
// dead-letter) don't have to coordinate. We deliberately avoid the
|
|
|
|
|
// recordStart cache here (cross-process callers won't have it); we
|
|
|
|
|
// recompute duration from the persisted row's started_at.
|
|
|
|
|
func (t *spanTracker) FinalizeAttempt(ctx context.Context, knowledgeID string, attempt int, status string,
|
|
|
|
|
output types.JSONMap, errorCode, errorMessage string,
|
|
|
|
|
) {
|
|
|
|
|
if knowledgeID == "" || attempt <= 0 {
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
if status == "" {
|
|
|
|
|
status = types.SpanStatusDone
|
|
|
|
|
}
|
|
|
|
|
rows, err := t.repo.ListByAttempt(ctx, knowledgeID, attempt)
|
|
|
|
|
if err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] FinalizeAttempt list failed kid=%s attempt=%d: %v",
|
|
|
|
|
knowledgeID, attempt, err)
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
var root *types.KnowledgeProcessingSpan
|
|
|
|
|
for i := range rows {
|
|
|
|
|
if rows[i].Kind == types.SpanKindRoot {
|
|
|
|
|
cp := rows[i]
|
|
|
|
|
root = &cp
|
|
|
|
|
break
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
if root == nil {
|
|
|
|
|
// No root means nothing to close — likely an attempt that
|
|
|
|
|
// predates the tracker or whose OpenAttempt write failed.
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
if root.Status == types.SpanStatusDone || root.Status == types.SpanStatusFailed ||
|
|
|
|
|
root.Status == types.SpanStatusCancelled || root.Status == types.SpanStatusSkipped {
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
now := time.Now()
|
|
|
|
|
var started time.Time
|
|
|
|
|
if root.StartedAt != nil {
|
|
|
|
|
started = *root.StartedAt
|
|
|
|
|
}
|
|
|
|
|
dur := int64(0)
|
|
|
|
|
if !started.IsZero() {
|
|
|
|
|
dur = now.Sub(started).Milliseconds()
|
|
|
|
|
}
|
|
|
|
|
if len(errorMessage) > 1024 {
|
|
|
|
|
errorMessage = errorMessage[:1024]
|
|
|
|
|
}
|
|
|
|
|
row := &types.KnowledgeProcessingSpan{
|
|
|
|
|
KnowledgeID: root.KnowledgeID,
|
|
|
|
|
Attempt: root.Attempt,
|
|
|
|
|
SpanID: root.SpanID,
|
|
|
|
|
ParentSpanID: root.ParentSpanID,
|
|
|
|
|
Name: root.Name,
|
|
|
|
|
Kind: root.Kind,
|
|
|
|
|
Status: status,
|
|
|
|
|
Input: root.Input,
|
|
|
|
|
Output: output,
|
|
|
|
|
Metadata: root.Metadata,
|
|
|
|
|
ErrorCode: strings.TrimSpace(errorCode),
|
|
|
|
|
ErrorMessage: errorMessage,
|
|
|
|
|
StartedAt: root.StartedAt,
|
|
|
|
|
FinishedAt: &now,
|
|
|
|
|
DurationMs: dur,
|
|
|
|
|
}
|
|
|
|
|
if err := t.repo.Upsert(ctx, row); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] FinalizeAttempt upsert failed kid=%s attempt=%d: %v",
|
|
|
|
|
knowledgeID, attempt, err)
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
t.touchKnowledgeHeartbeat(ctx, knowledgeID, types.SpanKindRoot)
|
|
|
|
|
}
|
|
|
|
|
|
feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
2026-05-28 10:26:56 +00:00
|
|
|
// AbortAttempt is the user-cancel counterpart to FinalizeAttempt. It
|
|
|
|
|
// flips every still-running / still-pending span for this attempt to
|
|
|
|
|
// cancelled — regardless of tree position — and then closes the root.
|
|
|
|
|
//
|
|
|
|
|
// Why a flat sweep instead of CancelDescendants' BFS: fan-out stages
|
|
|
|
|
// (e.g. 多模态识别) call EndSpan on the stage as soon as they finish
|
|
|
|
|
// DISPATCHING their async per-image work, so by the time the user
|
|
|
|
|
// hits cancel the stage row is already status=done but its image[*]
|
|
|
|
|
// children are still status=running. A BFS that stops at terminal
|
|
|
|
|
// parents would orphan those leaves. The flat sweep doesn't care
|
|
|
|
|
// about the tree shape — anything not-yet-terminal gets flipped.
|
|
|
|
|
func (t *spanTracker) AbortAttempt(ctx context.Context, knowledgeID string, attempt int,
|
|
|
|
|
errorCode, errorMessage, reason string,
|
|
|
|
|
) {
|
|
|
|
|
if knowledgeID == "" || attempt <= 0 {
|
|
|
|
|
return
|
|
|
|
|
}
|
|
|
|
|
if reason == "" {
|
|
|
|
|
reason = "user cancelled"
|
|
|
|
|
}
|
|
|
|
|
if errorCode == "" {
|
|
|
|
|
errorCode = "USER_CANCELLED"
|
|
|
|
|
}
|
|
|
|
|
if n, err := t.repo.CancelAllOpenSpans(ctx, knowledgeID, attempt, errorCode, reason); err != nil {
|
|
|
|
|
logger.Warnf(ctx, "[SpanTracker] AbortAttempt sweep failed kid=%s attempt=%d: %v",
|
|
|
|
|
knowledgeID, attempt, err)
|
|
|
|
|
// Fall through to FinalizeAttempt anyway — closing the root
|
|
|
|
|
// is more important than perfectly closing every child.
|
|
|
|
|
} else if n > 0 {
|
|
|
|
|
logger.Infof(ctx,
|
|
|
|
|
"[SpanTracker] AbortAttempt swept %d open span(s) for kid=%s attempt=%d",
|
|
|
|
|
n, knowledgeID, attempt)
|
|
|
|
|
}
|
|
|
|
|
t.FinalizeAttempt(ctx, knowledgeID, attempt,
|
|
|
|
|
types.SpanStatusCancelled, nil, errorCode, errorMessage)
|
|
|
|
|
}
|
|
|
|
|
|
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
|
|
|
// noopSpanTracker collapses every method to a no-op for tests/lite.
|
|
|
|
|
type noopSpanTracker struct{}
|
|
|
|
|
|
|
|
|
|
func (noopSpanTracker) OpenAttempt(_ context.Context, _, _ string) (*Span, int, error) {
|
|
|
|
|
return nil, 0, nil
|
|
|
|
|
}
|
|
|
|
|
func (noopSpanTracker) LatestAttempt(_ context.Context, _ string) int { return 0 }
|
|
|
|
|
func (noopSpanTracker) BeginStage(_ context.Context, _ string, _ int, _ string, _ types.JSONMap) *Span {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
func (noopSpanTracker) BeginSubSpan(_ context.Context, _ *Span, _, _ string, _ types.JSONMap) *Span {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
func (noopSpanTracker) EndSpan(_ context.Context, _ *Span, _ types.JSONMap) {}
|
|
|
|
|
func (noopSpanTracker) FailSpan(_ context.Context, _ *Span, _, _ string, _ error) {}
|
|
|
|
|
func (noopSpanTracker) SkipSpan(_ context.Context, _ *Span, _ string) {}
|
|
|
|
|
func (noopSpanTracker) LookupStage(_ context.Context, _ string, _ int, _ string) *Span { return nil }
|
2026-05-29 16:09:46 +00:00
|
|
|
func (noopSpanTracker) LookupSpanByName(_ context.Context, _ string, _ int, _ string) *Span {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.
- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
closes the root row idempotently. Re-closing a terminal root no-ops
so success / cascade-fail / dead-letter paths can fire without
coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
on the success path. Async downstream work (summary/question/wiki/
graph) still records its own spans; their completion extends the
trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
(docreader / chunking / embedding / multimodal / postprocess).
Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
document-related task exhausts retries. The probe payload now
extracts the Attempt field that Document/Manual/PostProcess
payloads already thread through.
Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph
i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
2026-05-27 08:54:09 +00:00
|
|
|
func (noopSpanTracker) FinalizeAttempt(_ context.Context, _ string, _ int, _ string, _ types.JSONMap, _, _ string) {
|
|
|
|
|
}
|
feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
2026-05-28 10:26:56 +00:00
|
|
|
func (noopSpanTracker) AbortAttempt(_ context.Context, _ string, _ int, _, _, _ string) {}
|