WeKnora/migrations/versioned/000055_knowledge_processing...

76 lines
3.9 KiB
MySQL
Raw Normal View History

-- Migration: 000054_knowledge_processing_spans
-- Per-(knowledge, attempt) span tree for the document parsing pipeline,
-- inspired by Langfuse's trace / span / generation hierarchy.
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
--
-- Background: knowledge.parse_status is a 4-state field (pending /
-- processing / completed / failed). When users hit "stuck in processing",
-- they have no way to tell which stage is actually running — DocReader,
-- chunking, embedding, multimodal OCR/VLM, or the final post-process
-- handoff. Operators face the same problem.
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
--
-- This table persists per-stage progress (and finer-grained subspans) so:
-- 1. The frontend can render a five-segment timeline showing where
-- each document is in the pipeline, with collapsible per-image /
-- per-batch subspans underneath.
-- 2. Failures carry a stable error_code (DOCREADER_TIMEOUT,
-- EMBEDDING_RATE_LIMIT, ...) the UI can map to localized
-- remediation text.
-- 3. Reparse history is preserved across attempts — operators can
-- navigate ?attempt=N to post-mortem "why did it fail twice?".
-- 4. Cascade-cancel rules use the parent_span_id tree to flip
-- dependent subspans/stages to "cancelled" when an upstream
-- span fails, so the UI shows a clear blast radius instead of
-- orphan spinners.
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
--
-- Schema mirrors Langfuse's vocabulary:
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
-- * one ROOT span per (knowledge_id, attempt) acting as the trace
-- * STAGE spans (docreader/chunking/embedding/multimodal/postprocess)
-- are children of root
-- * SUBSPANs (multimodal.image[i], embedding.batch[i], postprocess.spawn.X)
-- hang off their stage. The kind="generation" subset corresponds 1:1 to
-- a Langfuse generation; metadata.langfuse_trace_id stitches them.
DO $$ BEGIN RAISE NOTICE '[Migration 000054] Creating table: knowledge_processing_spans'; END $$;
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
CREATE TABLE IF NOT EXISTS knowledge_processing_spans (
id BIGSERIAL PRIMARY KEY,
knowledge_id VARCHAR(64) NOT NULL,
attempt INT NOT NULL DEFAULT 1,
span_id VARCHAR(64) NOT NULL,
parent_span_id VARCHAR(64),
name VARCHAR(64) NOT NULL,
kind VARCHAR(16) NOT NULL, -- root / stage / subspan / generation
status VARCHAR(16) NOT NULL, -- pending/running/done/failed/skipped/cancelled
input JSONB,
output JSONB,
metadata JSONB,
error_code VARCHAR(64),
error_message TEXT,
error_detail TEXT,
started_at TIMESTAMP WITH TIME ZONE,
finished_at TIMESTAMP WITH TIME ZONE,
duration_ms BIGINT,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uq_kpspan_attempt_span UNIQUE (knowledge_id, attempt, span_id)
);
-- Primary read path: fetch every span for a (knowledge, attempt) tuple in
-- one indexed range scan, then build the tree in memory. The unique
-- constraint above already covers point lookups.
CREATE INDEX IF NOT EXISTS idx_kpspan_knowledge_attempt
ON knowledge_processing_spans (knowledge_id, attempt);
-- Operator query: "find spans stuck in running too long". Used by ad-hoc
-- diagnostics and the housekeeping sweep.
refactor(knowledge): replace flat stage table with langfuse-style span tree Addresses review feedback that the PR② design had four shortcomings: 1. The pipeline is a DAG, not a sequence — Embedding and Multimodal are independent of each other, both downstream of Chunking, both upstream of PostProcess. The flat (knowledge_id, stage) table couldn't represent that, so a Chunking failure left dependents stranded as "pending" forever instead of being marked as impossible-to-run. 2. No history across attempts. A reparse erased the previous run's status before the new run started, leaving operators with no way to investigate "why did this fail twice?". 3. Stages had only status + duration. Operators want to know how big the work was — pages parsed, chunks created, tokens embedded, VLM calls made — to distinguish "slow because the file is huge" from "slow because the docreader is wedged". 4. Multimodal fans out N image tasks; Embedding fans out M batches; PostProcess fans out into Summary/Question/Wiki/Graph. Each unit is interesting on its own (Langfuse already captures this for LLM calls). The flat model couldn't express it. The redesign mirrors Langfuse's trace/span/generation hierarchy: * Migration 000053 supersedes 000052: knowledge_processing_spans table with (knowledge_id, attempt, span_id) primary key, plus parent_span_id, kind ∈ {root, stage, subspan, generation}, status ∈ {pending,running,done,failed,skipped,cancelled}, and JSONB input/output/metadata fields. * SpanTracker (replacing StageTracker) exposes OpenAttempt / BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan / LookupStage. Cross-process workers (image_multimodal) get the parent's attempt + span via payload + LookupStage so subspans attach correctly. * StageDependencies declares the DAG; FailSpan now cascades — descendants of the failed span and dependent stages are flipped to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear blast radius instead of orphan spinners. * Reparse now calls OpenAttempt up front so the timeline reflects "new attempt, all pending" instead of letting the previous run's status linger until the worker picks up the task. * Image_multimodal records each image as a generation subspan with its own success/failure on the parent attempt's multimodal stage. The finalize-on-last-attempt counter logic is preserved unchanged. * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns a tree shape with synthesized pending placeholders so the frontend always renders five timeline segments. ?attempt=N enables history navigation.
2026-05-27 06:05:47 +00:00
CREATE INDEX IF NOT EXISTS idx_kpspan_status_started
ON knowledge_processing_spans (status, started_at);
-- Lineage walks: cascade-cancel a stage's downstream needs to find every
-- child by parent_span_id. The cardinality is small (≤ tens per attempt)
-- so we don't need a covering index, just B-tree on parent.
CREATE INDEX IF NOT EXISTS idx_kpspan_parent
ON knowledge_processing_spans (parent_span_id)
WHERE parent_span_id IS NOT NULL;
DO $$ BEGIN RAISE NOTICE '[Migration 000054] knowledge_processing_spans table ready'; END $$;