Commit Graph

2 Commits (a8eb7e28fa4045d2d99966f5ff99b68b9a555a93)

Author SHA1 Message Date
lyingbug 6afb6ca9d5
feat(knowledge): flag documents that stop making progress (#3557)
* feat(knowledge): flag documents that stop making progress

A document stuck in processing looked the same as one still working: a
spinner that polled forever, no hint of where it stopped, and, hours
later, a generic housekeeping failure.

- The batch and spans endpoints return last_activity_at for in-flight
  rows: the later of updated_at and the newest span write. updated_at
  only moves at stage transitions; span writes keep moving while a long
  stage still works.
- After 20 minutes with no progress, the list and card views show
  "may be stuck" with a tooltip. The timeline shows the stage it
  stopped at and a stop-parsing button, plus a "last progress" stat.
  Both pollers back off to 15s while everything in flight is stalled.
- When housekeeping fails a row, the error names the stalled stage and
  the time of the last progress. It closes the row's open spans with
  TASK_STALLED (the running stage fails, the rest are cancelled), so
  the timeline shows where the run stopped instead of a spinner.

* fix(knowledge): locate stalls in finalizing and tell backlog from stuck

- Housekeeping reads the stall site from the latest attempt. It takes
  the running stages, or, when none runs (post-process closes its stage
  once enrichment is fanned out), the innermost running spans, mapped
  back to their stage. Those spans fail with TASK_STALLED and are named
  in the message; every stalled stage is listed in pipeline order. The
  other open spans are cancelled. Every closed span gets a duration, and
  the message leaves out the time when the heartbeat query failed.
- The spans tree prefers a TASK_STALLED failure for last_error over a
  later subtask failure. current_stage falls back to the stage owning a
  running subspan.
- Batch and spans responses flag rows quiet past the stall hint whose
  work is still queued (waiting_in_queue), using the sweep's own probes.
  At most 20 rows are probed per request, cached for a minute. The UI
  shows these as queued, with no stop button in the banner.

* fix(knowledge): answer stall verdicts from one shared queue scan

- Probing a quiet document for a backlog rescanned the whole queue once
  per document. When the document was absent, which is exactly the
  stuck case, the scan ran to the end, and it read a Redis error as
  "nothing queued". QueuedKnowledgeIDs now collects every referenced
  knowledge ID in one pass and returns list errors. HousekeepingService
  keeps that snapshot for a minute, shared across requests and
  single-flight. A failed scan is not cached.
- With the per-request cap of 20 gone, every quiet row on a page gets a
  verdict. The response carries it as stall_state (queued or stalled),
  replacing waiting_in_queue. The UI marks a row as stuck only on a
  "stalled" verdict. A row with no verdict, because the probe failed,
  shows as ordinary processing instead of inviting a stop.
2026-09-22 21:34:08 +08:00
wizardchen 04f56f9cda feat(knowledge): track per-stage parsing progress with /stages API
Adds a five-segment progress model for the document parsing pipeline so
the UI (PR③) can render a timeline showing where each document is
(DocReader → Chunking → Embedding → Multimodal → PostProcess) and
which stage failed with what error code.

- New table `knowledge_processing_stages` (migration 000052) with one
  row per (knowledge_id, stage). UPSERT on Begin/Done/Fail bumps an
  attempt counter so re-parses don't lose history.

- StageTracker service exposes Begin/Done/Fail/Skip; all calls are
  best-effort and never break the pipeline if persistence fails.

- Stable error codes (DOCREADER_TIMEOUT / EMBEDDING_RATE_LIMIT /
  VECTORSTORE_WRITE_FAILED / ...) the UI can map to localized
  remediation hints.

- Tracker call sites added at the four meaningful failure points:
  convert (DocReader), CreateChunks (Chunking), BatchIndex (Embedding),
  enqueueImageMultimodalTasks (Multimodal start),
  KnowledgePostProcess.Handle (Multimodal close + PostProcess).

- New endpoint `GET /api/v1/knowledge/:id/stages` returns the five
  canonical stages — missing rows are synthesized as "pending" so
  the timeline always renders five segments. Includes current_stage
  and last_error block.
2026-05-28 15:14:45 +08:00