* feat(knowledge): flag documents that stop making progress
A document stuck in processing looked the same as one still working: a
spinner that polled forever, no hint of where it stopped, and, hours
later, a generic housekeeping failure.
- The batch and spans endpoints return last_activity_at for in-flight
rows: the later of updated_at and the newest span write. updated_at
only moves at stage transitions; span writes keep moving while a long
stage still works.
- After 20 minutes with no progress, the list and card views show
"may be stuck" with a tooltip. The timeline shows the stage it
stopped at and a stop-parsing button, plus a "last progress" stat.
Both pollers back off to 15s while everything in flight is stalled.
- When housekeeping fails a row, the error names the stalled stage and
the time of the last progress. It closes the row's open spans with
TASK_STALLED (the running stage fails, the rest are cancelled), so
the timeline shows where the run stopped instead of a spinner.
* fix(knowledge): locate stalls in finalizing and tell backlog from stuck
- Housekeeping reads the stall site from the latest attempt. It takes
the running stages, or, when none runs (post-process closes its stage
once enrichment is fanned out), the innermost running spans, mapped
back to their stage. Those spans fail with TASK_STALLED and are named
in the message; every stalled stage is listed in pipeline order. The
other open spans are cancelled. Every closed span gets a duration, and
the message leaves out the time when the heartbeat query failed.
- The spans tree prefers a TASK_STALLED failure for last_error over a
later subtask failure. current_stage falls back to the stage owning a
running subspan.
- Batch and spans responses flag rows quiet past the stall hint whose
work is still queued (waiting_in_queue), using the sweep's own probes.
At most 20 rows are probed per request, cached for a minute. The UI
shows these as queued, with no stop button in the banner.
* fix(knowledge): answer stall verdicts from one shared queue scan
- Probing a quiet document for a backlog rescanned the whole queue once
per document. When the document was absent, which is exactly the
stuck case, the scan ran to the end, and it read a Redis error as
"nothing queued". QueuedKnowledgeIDs now collects every referenced
knowledge ID in one pass and returns list errors. HousekeepingService
keeps that snapshot for a minute, shared across requests and
single-flight. A failed scan is not cached.
- With the per-request cap of 20 gone, every quiet row on a page gets a
verdict. The response carries it as stall_state (queued or stalled),
replacing waiting_in_queue. The UI marks a row as stuck only on a
"stalled" verdict. A row with no verdict, because the probe failed,
shows as ordinary processing instead of inviting a stop.
Second slice of #3338 (tracked as #3345).
Count alignment: CountKnowledgeByKnowledgeBaseID now excludes rows
mid-deletion, mirroring the document list's existing filter — a stranded
delete no longer shows as '4 documents, 3 listed' on the KB card. Failed
rows stay counted: they are visible, actionable cards.
Sweep C: housekeeping now recovers rows stuck at 'deleting' whose delete
task no longer exists. The delete dead-letter callback only flips rows
that are still 'deleting' when retries exhaust, so a worker death
mid-delete or a lost queue left the row hidden from the list forever.
Same recovery contract as the parse sweep: flip to failed with a clear
error so the row becomes visible and actionable; the delete executor is
idempotent and plans by row ID, so a late-arriving task still finishes
cleanly.
Liveness gate: a new TaskInspector probe (HasQueuedDeleteTasksForKnowledge)
matches knowledge:list_delete batch payloads — the per-parse matcher
cannot see them. A live queued/active delete defers the row (backlog,
not stranded); a probe error defers that row; a nil inspector defers the
whole sweep. Lite mode's noop returns false by design: inline deletes
never queue, so a stale deleting row there genuinely has no task.
Tests: sweep matrix (stranded recovered / backlogged kept / fresh
untouched / probe-error deferred / nil-inspector deferred), count
alignment, and the payload matcher table.
Refs #3345, #3338
ProcessKBDelete must return retryable engine-resolution failures instead
of logging and continuing, which could delete knowledge rows while leaving
embeddings behind. Align the handler with ProcessIndexDelete and cover
cancellation and ErrVectorStoreUnavailable in tests.