Commit Graph

4 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
lyingbug 6afb6ca9d5
feat(knowledge): flag documents that stop making progress (#3557)
* feat(knowledge): flag documents that stop making progress

A document stuck in processing looked the same as one still working: a
spinner that polled forever, no hint of where it stopped, and, hours
later, a generic housekeeping failure.

- The batch and spans endpoints return last_activity_at for in-flight
  rows: the later of updated_at and the newest span write. updated_at
  only moves at stage transitions; span writes keep moving while a long
  stage still works.
- After 20 minutes with no progress, the list and card views show
  "may be stuck" with a tooltip. The timeline shows the stage it
  stopped at and a stop-parsing button, plus a "last progress" stat.
  Both pollers back off to 15s while everything in flight is stalled.
- When housekeeping fails a row, the error names the stalled stage and
  the time of the last progress. It closes the row's open spans with
  TASK_STALLED (the running stage fails, the rest are cancelled), so
  the timeline shows where the run stopped instead of a spinner.

* fix(knowledge): locate stalls in finalizing and tell backlog from stuck

- Housekeeping reads the stall site from the latest attempt. It takes
  the running stages, or, when none runs (post-process closes its stage
  once enrichment is fanned out), the innermost running spans, mapped
  back to their stage. Those spans fail with TASK_STALLED and are named
  in the message; every stalled stage is listed in pipeline order. The
  other open spans are cancelled. Every closed span gets a duration, and
  the message leaves out the time when the heartbeat query failed.
- The spans tree prefers a TASK_STALLED failure for last_error over a
  later subtask failure. current_stage falls back to the stage owning a
  running subspan.
- Batch and spans responses flag rows quiet past the stall hint whose
  work is still queued (waiting_in_queue), using the sweep's own probes.
  At most 20 rows are probed per request, cached for a minute. The UI
  shows these as queued, with no stop button in the banner.

* fix(knowledge): answer stall verdicts from one shared queue scan

- Probing a quiet document for a backlog rescanned the whole queue once
  per document. When the document was absent, which is exactly the
  stuck case, the scan ran to the end, and it read a Redis error as
  "nothing queued". QueuedKnowledgeIDs now collects every referenced
  knowledge ID in one pass and returns list errors. HousekeepingService
  keeps that snapshot for a minute, shared across requests and
  single-flight. A failed scan is not cached.
- With the per-request cap of 20 gone, every quiet row on a page gets a
  verdict. The response carries it as stall_state (queued or stalled),
  replacing waiting_in_queue. The UI marks a row as stuck only on a
  "stalled" verdict. A row with no verdict, because the probe failed,
  shows as ordinary processing instead of inviting a stop.
2026-09-22 21:34:08 +08:00
Fei.Chen c8ebb9ed96
fix(knowledge): recover stranded deleting rows and align the KB count (#3367)
Second slice of #3338 (tracked as #3345).

Count alignment: CountKnowledgeByKnowledgeBaseID now excludes rows
mid-deletion, mirroring the document list's existing filter — a stranded
delete no longer shows as '4 documents, 3 listed' on the KB card. Failed
rows stay counted: they are visible, actionable cards.

Sweep C: housekeeping now recovers rows stuck at 'deleting' whose delete
task no longer exists. The delete dead-letter callback only flips rows
that are still 'deleting' when retries exhaust, so a worker death
mid-delete or a lost queue left the row hidden from the list forever.
Same recovery contract as the parse sweep: flip to failed with a clear
error so the row becomes visible and actionable; the delete executor is
idempotent and plans by row ID, so a late-arriving task still finishes
cleanly.

Liveness gate: a new TaskInspector probe (HasQueuedDeleteTasksForKnowledge)
matches knowledge:list_delete batch payloads — the per-parse matcher
cannot see them. A live queued/active delete defers the row (backlog,
not stranded); a probe error defers that row; a nil inspector defers the
whole sweep. Lite mode's noop returns false by design: inline deletes
never queue, so a stale deleting row there genuinely has no task.

Tests: sweep matrix (stranded recovered / backlogged kept / fresh
untouched / probe-error deferred / nil-inspector deferred), count
alignment, and the payload matcher table.

Refs #3345, #3338
2026-09-18 00:55:44 +08:00
wizardchen c291c6a59f fix(kb): retry KB delete when engine resolution is deferred
ProcessKBDelete must return retryable engine-resolution failures instead
of logging and continuing, which could delete knowledge rows while leaving
embeddings behind. Align the handler with ProcessIndexDelete and cover
cancellation and ErrVectorStoreUnavailable in tests.
2026-07-29 12:25:18 +08:00
zhangzherui 37de0afad0 fix(tasks): dequeue work for deleted knowledge bases 2026-07-23 12:17:14 +08:00