Commit Graph

79 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
诉说晚秋 95691daab2
fix(knowledge): retry document processing after transient indexing failures (#3915)
processChunks marked the knowledge failed and returned nil when BatchIndex
failed, so ProcessDocument acked the task and the configured document-task
retries never ran. A short embedding or vector-store outage left the row
permanently failed; unchanged data-source syncs did not reprocess it.

While attempts remain, still clean up the partial chunks and vectors but
keep the row processing and return the error to the queue. The final
attempt (or a caller without a retry loop) records the failure as before.
2026-10-08 01:25:31 +08:00
linus-liu-web a15dfdd941
fix(summary): 坏 JSON 不再冒充文档摘要(截断/畸形 JSON 走失败路径,纯文本兜底保留) (#3769)
* fix(summary): 坏 JSON 不再冒充文档摘要

非流式文档摘要(Chat)解析失败时原先把模型原文当摘要落库:被 MaxTokens
截断的半截 JSON 会写进 knowledges.description、置 SummaryStatus=Completed
(Asynq 不再重试),同一段文本还会被 buildSummaryChunkContent 索引进 RAG。

- parseDocumentSummaryOutput 改为返回 (*documentSummaryResult, error):
  回复形如 JSON(含 markdown 代码围栏)却解析失败、或解析成功但取不到
  summary/gist 时返回 errSummaryOutputNotParsable,绝不回退成原文;
  非 JSON 的纯文本仍走原有的原文兜底,自定义模板 / 老模型不受影响。
- validateSummaryOutput 复用 chatpipeline.IsLengthFinishReason,
  FinishReason 为 length 类取值时返回 errSummaryOutputTruncated。
- 两个 sentinel 经 getSummary 汇入既有失败路径(ProcessSummaryGeneration
  的 handleRetryableSummaryFailure、RegenerateKnowledgeSummary 的
  handleGenerationFailure):保留旧描述、置 pending/failed 交给 Asynq 重试,
  不再以 Completed 落库。

测试:TestParseDocumentSummaryOutput 改为表驱动(截断 JSON、含裸换行 JSON、
无文本 JSON 必须报错且不能把原文当 Summary;纯文本兜底继续通过);
TestValidateSummaryOutput 增加 length / max_tokens 截断用例;
TestGetSummaryRejectsUnusableModelOutput 锁住 getSummary 的错误传播。

* fix(summary): length 截断判定收窄到 JSON 形状回复

非流式文档摘要在校验模型回复时先看 finish_reason:只要 provider 报告
length/max_tokens 就返回 errSummaryOutputTruncated。该判定不认识内容形状,
于是自定义纯文本模板一旦顶到输出预算,就从「截断但可用」变成重试直到
SummaryStatusFailed,与上一提交承诺的「非 JSON 的纯文本仍走原兜底」相反。

- validateSummaryOutput 恢复为只看内容是否为空;新增
  validateStructuredSummaryDocument 供 getSummary 使用:先判形状再判
  length,只有确实在产结构化摘要(能解析成对象/数组,或 { 开头、[ 开头接
  对象/数组/字符串元素)且被预算截断时才返回 errSummaryOutputTruncated。
- 形状判定剥离 UTF-8 BOM、前后空白与一层 markdown 围栏后再尝试
  json.Unmarshal;「[文档摘要] …」「[1] …」这类带方括号引用的纯文本不再被
  误判成坏 JSON,而截断的 {summary: 与截断的 JSON 数组仍然报错。
- parseDocumentSummaryOutput 同步剥离 BOM:BOM 前缀的截断 JSON 之前会被
  当成纯文本落库。
- 表格描述 / 列描述仍走 validateSummaryOutput,截断的描述照旧可索引。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:01 +08:00
lyingbug 9114e4e4f9
fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794)
* fix(citations): verify source locations and recover document previews

* chore: exclude local research notes from pull request
2026-09-27 20:32:26 +08:00
lyingbug ab115c0944
fix(knowledge): stop parses from getting stuck in processing or finalizing (#3696)
Reparse:
- Open the new attempt only after override validation and cleanup pass.
  A rejected reparse used to leave the previous run's subtasks treating
  themselves as superseded, so the row stayed in finalizing for good.
- Cancel the previous run's queued tasks when it is still in flight.
  ProcessDocument, processChunks, post-process, image tasks, wiki ops and
  the dead-letter callback now skip work that belongs to a superseded
  attempt.
- Summary tasks write only their own columns. A full-row save after the
  LLM call could put finalizing over the new run's processing row.

Errors that were acked with the row still in flight:
- A failed post-process enqueue is retried in place and then returned,
  so the task retries instead of leaving the row in processing.
- Transient tenant/knowledge reads and the processing write now retry.
- An image whose orphan check fails on its final attempt is still
  counted. Empty manual content fails the row.

Counters:
- FinalizeSubtask runs its decrement and promote in one transaction.
  Slot releases retry, and the post-process shortfall release keeps
  going past a failed release.

Wiki:
- A synthesis model that cannot be built releases the KB's pending
  documents on the final attempt instead of retrying forever.
- Panics in map/reduce workers become per-document failures.
- Housekeeping protects a row held by a wiki op for at most 48h.

Lite mode:
- SyncTaskExecutor honours options set on asynq.NewTask (MaxRetry,
  Timeout, Deadline, ProcessIn/At), stops on SkipRetry, and runs the
  same final-failure callback as the dead-letter middleware.
- The multimodal fan-in counter is kept in process, so post-process
  waits for every image.

Also:
- Housekeeping recovers summaries left pending on finished rows. Cancel
  and stall recovery close unfinished summaries.
- Housekeeping sweeps no longer overlap.
- Data source URL ingest writes only the metadata column.
- The document list drops cards the batch endpoint no longer returns.
2026-09-25 13:26:38 +08:00
lyingbug 1bf717a54d
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage

Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.

Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.

The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.

* fix(docreader): serialize source blocks outside main so tests import it

test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.

* fix(chat): tighten source locators from review

- Remap rewritten-line-for-line stretches by line so scanned PDF pages
  no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.

* fix(migrations): renumber chunk source locators after main's chunk_images

main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 13:26:16 +08:00
tayadwx 68fe9599cd
feat(gallery): add an image gallery tab with schema-driven filters (#3694)
* feat(image): observe image attributes and let the policy decide OCR

Describe each image with the VLM as before, but ask it to also observe two
attributes about the image in the same round — how much body text it carries
(contain.text) and whether it presents data visually (contain.data_visual) —
then let a small pure function decide whether that image is worth a second,
separate OCR round.

Per-class OCR was the first shape of this idea; a class (chart / photo / logo)
is only an indirect proxy for 'is there text worth reading here', and the class
table has to grow whenever a new kind of image shows up. The decision actually
needs the observed attributes, so it reads them directly, and the attribute set
is a backend registry row rather than a new class.

The observation and the caption share one VLM call; OCR stays a separate call
(prompt isolation, see #3291). An attribute the model does not answer for is
left absent rather than filled with a conservative default, so the trace never
claims an observation that did not happen, and an incomplete observation falls
back to the on_unobserved policy (OCR still runs). The registry also carries the
panel's display text, so a new attribute needs no frontend change.

Off by default: image_attrs_enabled=false keeps today's behaviour (one caption
and one OCR per image). KB config: image_processing_config.image_attrs_enabled /
image_actions; per-upload override via process_config. Registry:
GET /api/v1/image-attrs/schema.

* feat(gallery): add image gallery tab with schema-driven filters

Adds a 4th knowledge-base tab parallel to Documents/Wiki/Graph that
browses every image asset in a KB.

Backend:
- GET /api/v1/knowledge-bases/:id/images lists image assets projected
  from chunks.image_info, with keyword (caption/OCR), multi-value
  attribute (attr_<name>=<value>), is_enabled and sort filters applied
  in memory, plus pagination.
- Resolves the source knowledge item title (SourceName) via a batched
  GetKnowledgeBatch so the viewer can show a human-readable source.

Frontend:
- ImageGallery.vue: thumbnail grid + search/sort + schema-driven filter
  panel (from GET /image-attrs/schema) + single-image viewer with
  prev/next, attributes/caption/OCR/source.
- Wired as the gallery tab in KnowledgeBase.vue; i18n keys in all 5 locales.

* feat(gallery): drive filters/search/sort from a configurable attribute contract

The gallery UI is no longer hardcoded: it renders from a self-describing
contract served by GET /knowledge-bases/:id/gallery-config (live attribute
sources, every attribute's definition with usage flags merged across the
config tiers, and the caller's search activation state).

- Attribute sources form an open registry (types.RegisterGalleryAttrSource);
  a builtin source models caption/ocr_text/timestamps/enabled so the gallery
  has zero special cases.
- Usage resolution: user > KB > system > source, field by field; the system
  tier lives in system_settings (gallery.policy, JSON, no rebuild), the KB
  tier slot is wired in the engine but not stored yet.
- Search activation is the user's own record: mode (all|custom) plus
  per-field on/off in users.preferences.gallery; source declarations never
  set defaults (unrecorded = off in custom mode).
- image_info is read through a gallery-local struct, removing the compile
  dependency on the attribute-pipeline types; the gallery builds and runs
  with or without that feature deployed.
- ListImages validates search_in / sort_by / attr_filters against the
  contract; ineligible fields are dropped, not served.

* fix(frontend): inline the shared graph-drawer stylesheet (@import instead of <style src>)

plugin-vue 6.0.6 keys unscoped src-style descriptors by the imported file
path, so AgentStreamDisplay.vue and WikiBrowser.vue — both importing
components/css/wiki-graph-drawer.less — overwrite each other during the
build and the second one crashes with "reading 'scoped'". Adding the
gallery changed the module order enough to make the collision hit every
time. Inlining the same stylesheet keeps the emitted CSS identical and
removes the shared descriptor key.

* feat(gallery): render attributes from short labels with descriptions in tooltips

A source could only say one thing about a value, so the filter panel had to put
a full sentence inside a checkbox. Attributes and their values now carry two
pieces of text: a short label for the space-constrained controls (checkbox,
search field, sort option) and a description for wherever there is room
(tooltip, detail panel).

The contract's value list becomes an array of {value, label, description}, and
the display helpers resolve each field separately: the gallery stamps the short
name on a control and moves the sentence into a tooltip. Translation overlays
are looked up per field, so a source that ships no gallery-specific text still
picks up the attribute pipeline's own wording.

* fix(gallery): let locale text win over the contract's wording

The filter panel read the backend's own label before asking the locale,
so a translated value never got a chance: the contract ships its own
default-language wording, and it shadowed the overlay for every locale.
Built keys were also off by a segment — imageAttr.<name> has to be
imageAttr.<name>.label, and the gallery's own keys live under
knowledgeEditor.wikiBrowser.gallery.attr — so builtin and system
attributes fell through to English on a Chinese UI.

* feat(gallery): top bar layout, pinned panels and per-value verdicts

The toolbar moves onto its own row above the grid, so the controls no
longer eat into the space the images are there for: search box, the two
scope switches, sort field and direction, and the result count all sit in
one flex row. The settings behind them are no longer inline either — each
opens as a popover under its control and can be pinned to a right-hand
rail, where the two panels take turns as tabs instead of both claiming
width at once.

Filtering gains a third state per attribute value. Where a value could
only be picked or not picked, it can now be "off" (hide those images),
"on" (show them whatever else says) or neutral (leave them alone), which
is what lets one attribute's rules talk about an image another attribute
tried to hide without the pair of them cancelling each other out: a
forced display outranks a forced hide, and anything no rule speaks about
stays visible.

The switch that used to only exist inside the panel now has a toolbar
home as well. "筛选:全显示" genuinely drops the attribute constraint
instead of imposing it against a panel the user cannot see.

Filtering used to empty the list the moment it was switched on. Any
image whose attribute was never observed — every image ingested before
the attribute pipeline existed — read as a mismatch for every filter,
which is not narrowing a list but losing it. Absence now stays silence:
an attribute no image carries places no constraint on it, until the
pipeline catches up and fills the value in.

Co-authored-by: WorkBuddy

* feat(gallery): expose the image attribute pipeline as a gallery source

The gallery's contract resolved to a single source, so the filter panel
could only offer "Enabled" while the attributes the observe-and-describe
pipeline fills in — contain.text, contain.data_visual — never reached the
UI. The registry is a gallery attribute source like any other: every
attribute it declares is offered to the gallery, filterable by default,
with the config tiers able to override the usage flags at runtime.

The registration lives in the attribute pipeline's own commit; this branch
carried an earlier, squapshot of that work which predated the split of an
attribute value into a short label and a description, so the registry's
value shape and this file have been brought in line with it. Both PRs now
hold byte-identical copies of the source hook, which is what lets either
one land first and the other merge cleanly afterwards.

Co-authored-by: WorkBuddy

* fix(gallery): viewer race, pin icon, panel anchoring and scope wording

Opening the viewer rendered the img element with the previous or empty
source while the proxy fetch was still producing a blob, and the error
that empty source fired locked the viewer in its failure state even
after the real image arrived. The resolved source now lands before the
flag is consulted, and a token keeps a slow fetch of one image from
overwriting a faster one that navigated past it.

The pin button drew nothing: TDesign has no "pushpin" glyph, so the
button rendered as an empty shell. It now uses pin / pin-filled, keeps
its place beside the tab bar, and a pinned rail assigns its columns
explicitly — the panel precedes the main section in the DOM, so leaving
the grid to source order handed the wide column to the panel and
squeezed the images into the rail.

The floating panel no longer appears at a hardcoded offset: it anchors
just below the arrow that summoned it, the arrow flips up while its
panel is out, a second press retracts, and any click elsewhere in the
toolbar closes it. When a rail is pinned the arrows turn to the
requested tab instead of spawning a second surface.

The scope words (全部 / 全显示 / 自定义) become plain status text rather
than a second dropdown — they name the current mode, and only the arrow
beside them opens the panel that changes it.

Co-authored-by: WorkBuddy

* fix(web): align the OCR-on-unobserved switch with the other setting rows

The switch sat bare inside its setting-row while every other switch on
the page is wrapped in setting-control, which is what reserves the right
column and centers the control vertically. With the row laid out from
the top, the bare switch hugged the row's upper edge and crowded the
description text — visibly out of place next to its neighbors. Wrapping
it in setting-control gives it the same right-aligned, centered slot as
the rest.

Co-authored-by: WorkBuddy

* fix(web): move the OCR-on-unobserved row out of the vertical schema row

The row lived inside setting-row-vertical, whose descendant rule stretches
every setting-info and setting-control beneath it to 100% width. Two
full-width flex children side by side cannot fit, so the control overflowed
the container and the switch landed past the right edge — visibly further
out than the switch on the row above. The row and its note now sit beside
the vertical block instead of inside it, under their own condition, so the
switch shares the ordinary row layout and its right edge lines up with the
"image attribute observation" switch above.

Co-authored-by: WorkBuddy

* fix(gallery): applying a verdict now activates the filter scope

Setting an off/on verdict while the panel's master switch was off
silently ignored it: buildParams only attaches attr_rules in custom
scope, and setVerdict never flipped the scope. A verdict the user just
expressed now switches the scope to custom; clearing the last meaningful
verdict returns to all, since custom with no rules constrains nothing.

* fix(gallery): search checkboxes stay visible and idle panels read as dimmed

The search panel hid its field checkboxes while the master switch was
on, so the only way to reach them was to flip the switch first. They now
sit next to the switch like the filter panel's verdicts do: dimmed in
all-field mode (where they carry no weight) yet clickable, and touching
one drops the mode to custom. The filter panel's verdict rows dim the
same way while the filter switch is off.

* fix(gallery): bulk verdict row, neutral default mark, panel polish

The clear-filters button never matched what the panel is for, so it is
now a bulk verdict row: one set of -/off/on buttons that applies the
chosen position to every value of every value-typed attribute, with the
default mark turned into a language-neutral dash. The panel head drops
its second divider (the tab bar already draws one) and the tabs follow
the toolbar's order: search left, filter right.

* fix(gallery): drop the all-fields caption and center the order arrow

The caption above the search checkboxes showed and hid with the master
switch, shoving the checkboxes down on every flip; it said nothing the
dimmed state doesn't, so it is gone. The order button's arrow is an
inline svg, which sits on the text baseline and reads as riding high;
it now centers against the label.

* fix(gallery): center the order arrow for real

vertical-align never applied: the icon is a flex item inside the
button's inline-flex text wrapper, and the wrapper's default stretch
pins the explicitly-sized 16px icon to the top of the 22px line box.
align-items: center on the wrapper is what actually centers it.

* fix(gallery): shorten panel tab labels in en and ru

The two-word labels overflowed the 300px panel and forced the tab bar
into scroll arrows; single-word labels fit in every locale that was
affected.

* docs: drop the image pipeline rewrite from this branch

The rewritten 'Go 侧接手后' paragraph belongs to the image attribute observation
work (#3635), which this branch carries as a prerequisite and which already
documents the caption_ocr / observation_driven split. Keeping a competing
rewrite of the same paragraph on the gallery branch makes it collide with
upstream, which extended that paragraph with the custom_instructions and inline
HTML table notes. Restoring the shared wording keeps the gallery branch free of
this file so upstream's edits merge cleanly.

* style: gofmt the gallery attribute types

Struct field tags in the resolved gallery config were one column narrow, so
gofmt rebroke the alignment block. Whitespace only.

* style(gallery): use design tokens in the gallery styles

The style guard ratchet fails on any new violation. ImageGallery.vue shipped
with 47 TDesign token fallbacks, 21 literal font sizes, 4 literal radii and a
0.15s transition; all of them now read the --app-* / --td-* tokens, which is
what the guard asks for and what dark mode depends on.

* lint(gallery): fix the golangci-lint findings in the gallery code

lll: the three over-long swagger @Param lines and one registry row were
shortened. ineffassign: ctx was assigned before the access check and then
reassigned, so the first value was dead; it is now declared once, after the
check. errcheck: the three unchecked c.Error calls in ListImages. revive: the
AttrSpec / AttrValue doc comment block was attached to the wrong type and split
into one block each. staticcheck S1016: the gallery registration converts the
registry value instead of restating its fields.

* lint(gallery): wrap the image query select and check the last c.Error

* style: reformat the wrapped image select in chunk.go

* fix(gallery): address review findings on ordering, KB switch and blob lifetime

- sortImageAssets: swap operands for desc instead of negating, keeping a
  strict weak ordering so ties page stably
- ImageGallery: reload on knowledgeBaseId change, drop stale list
  responses, fetch thumbnails in parallel with one cached blob per URL,
  revoke blobs on unmount, default sort to builtin:created_at
- KnowledgeBase: non-wiki KBs with a stale wiki/graph tab fall back to
  documents again
- handler: move ListImages godoc out of ListKnowledgeFolders' block and
  document attr_filters / attr_rules
- DecideOCR: a payload with no OCR conditions uses the built-in policy
  instead of silently skipping OCR

* docs(parsing): describe the caption_ocr and observation_driven image pipelines

Carried over from #3635, which this PR supersedes.

* fix(gallery): load local:// and other storage handles through the KB file proxy

The gallery's own scheme check only caught resource://, provider:// and
storage://, so local:// (the default storage), minio://, cos:// and s3://
images went straight into <img src> and never rendered. Resolve through
isProviderFileURL, the scheme list chat and wiki already use, and lock the
routing with a test.

* fix(kb-editor): keep API-customised OCR conditions on save and upload

The KB editor always wrote the registry default into image_actions.ocr.on,
so any unrelated edit replaced a knowledge base's custom OCR conditions
with the default table; the upload dialog did the same for the per-document
override. Keep a stored custom list and fall back to the registry default
only when there is none, and show the conditions actually in effect instead
of the default table in both panels.

* perf(gallery): de-duplicate, filter, sort and page images in SQL

Every gallery request used to read every image-bearing chunk of the KB,
OCR text included, decode it in Go and page in memory. A single query now
expands the image_info arrays, keeps one row per image URL, applies the
keyword / attribute / verdict / enabled filters and the sort, and returns
only the page with the filtered total.

The expansion carries narrow scalars only; filters and JSON sort keys are
evaluated on the winning entry of each image, and the page's JSON is read
back for its rows alone. Postgres runs the query with JIT off, since the
fixed 100-rows-per-call estimate of jsonb_array_elements otherwise prices
it ~100x high and compilation dominates.

On 20k images with 2 KB of OCR text each (40k chunks), a page drops from
~750 ms to ~100-200 ms on postgres and ~200 ms on sqlite; keyword search
over OCR text stays a full scan (~400-600 ms) until the text is indexed.
The filter semantics are pinned against both databases.

* perf(gallery): serve the gallery from a trigger-maintained chunk_images table

Reading images out of chunks.image_info meant parsing every image chunk of
the knowledge base on each request, so a page still cost time in
proportion to the KB. chunk_images (versioned/000113, sqlite/000032) keeps
one row per image_info entry with caption, OCR text and attributes as
columns; triggers on chunks keep it in step with every write path, and the
migration backfills existing chunks.

Copies of one image stay separate rows and the query lists the copy on the
most recently updated chunk through an indexed anti-join, so writes decide
nothing and concurrent writers need no locking. On postgres, caption and
OCR text get pg_trgm indexes when the extension is available.

On 20k images with 2 KB of OCR text each (40k chunks):
  postgres  first page 10 ms, page 500 41 ms, keyword 2 ms, filters 23 ms
  sqlite    first page  3 ms, page 500 39 ms, keyword 147 ms, filters 51 ms
down from ~100-600 ms. Writing those 40k image chunks costs 7.0 s instead
of 1.3 s on postgres (1.9 s instead of 1.0 s on sqlite), mostly the trigram
indexes; status and enable-flag updates patch rows in place.

---------

Co-authored-by: tayadwx <tayadwx@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-25 12:33:59 +08:00
wizardchen 0a99a2850b fix(knowledge): address review of the stuck-processing fixes
- Abort check: a read failure now stops the pipeline instead of carrying
  on. Continuing let a later full-row Save overwrite a cancel or delete
  it never saw. processChunks now returns an error, and the handlers
  return it, so asynq retries an unknown state instead of acking it.
- Resolve the vector store before deleting the old chunks, and retry
  when the lookup itself failed. A failed persist or an interrupted
  context no longer acks with the row still in flight.
- Drain and release the wiki slots in one transaction, so a failed
  release rolls the delete back for the retry. Skip any document that
  still has a fresh claim in the lane: its live batch releases it.
- Re-armed and restart-recovered wiki triggers carry the language the
  ops were queued with, rather than the server default.
2026-09-22 20:33:18 +08:00
wizardchen d951cb78d1 fix(knowledge): stop pipeline errors from stranding rows in processing
- Return an error when the processing to finalizing handoff fails to
  persist instead of acking; the ack read as "no longer processing" and
  left a non-wiki row stuck with no retry.
- Only a missing row reads as deleting in the abort check. A transient
  read error made callers wipe a live document's chunks and index and
  return silently.
- Convert handler panics into errors below the dead-letter middleware so
  the final-attempt callback marks the row failed, and recover them in
  the Lite-mode executor, where a panic took down the process.
- Fail the attempt when the KB's vector store cannot be resolved rather
  than dereferencing a nil retrieve engine later.
2026-09-22 20:33:18 +08:00
vicfei 2dd36212d6 fix(knowledge): bind extracted images to their document and unscope shared-KB parent lookups
Org-shared KB viewers got 403 for extracted images on current main even
though the retrieving chunk text carried the resource:// handle and the
share grant was valid (#3342, second report after #3022/#3069).

Root cause is a mismatch between ingestion and authorization: document
parses store extracted images through SaveBytes, which registers the
resource but never claims it, while both file proxies (ResolveKBFile and
the message fallback via resourceAccessibleViaSharedKB) authorize through
IsReferencedByKnowledgeBase — which accepts explicit resource_bindings
rows only. The binding-only hardening landed without adding the missing
bindings, so every extracted image became invisible cross-workspace.

Ingestion: bindStoredImages claims each stored image for its knowledge
(ResourceRelationExtractedImage), mirroring bindContentResources — best
effort, same-workspace resources only. Wired into both the file-parse
path (knowledge_process) and manual knowledge (triggerManualProcessing).

Merge: resolveParentChunks fetched parents, grandparents, and image_info
children with the caller's tenant, so shared-KB hits from the sharing
workspace resolved none of them and lost parent context and image_info
(the second defect in the #3342 audit). Those lookups now use the
existing *Only unscoped variants, matching their shared-KB precedent.

Historical documents parsed before this change still carry no bindings;
a backfill is tracked as the follow-up slice.

Verified end-to-end on a fresh stack: publishing a manual document with
an embedded image now records kind=image plus an extracted_image binding
alongside the content-derived attachment binding.
2026-09-22 17:52:49 +08:00
lyingbug 4a9aa06274
refactor(asr): move the wire format into protocol packages, and cover every vendor whose docs offer ASR (#3503) 2026-09-21 13:25:27 +08:00
lyingbug eaca7364cd
fix(knowledge): stop two parse paths from stranding a row in processing (#3445)
* fix(knowledge): stop two parse paths from stranding a row in processing

Both of these left parse_status on "processing" with nothing left to move
it, so the document sat there until the housekeeping sweep failed it an
hour later with "task stuck in processing, recovered by housekeeping" —
reaching the user as a slow, unexplained parse failure.

processChunks returned silently when the embedding model could not be
resolved (deleted model, expired credentials, unreachable provider). It
reports nothing to its callers, so the attempt ended with the row still
marked processing. Persist the failure with the underlying cause and
record the embedding stage as failed, matching how the sibling
GetASRModel branch already handles a model it cannot resolve.

enqueueImageMultimodalTasks seeded the multimodal fan-in counter with
len(images) before the fan-out, then only warned when a payload failed
to marshal or asynq refused the write. Those slots have no task behind
them to drain them, so the counter never reached zero and post-process
was never triggered. Release the shortfall after the fan-out, and drive
post-process directly when the release itself brings the counter to zero
or when nothing reached the queue at all — the same reconciliation
KnowledgePostProcess already performs for its enrichment subtasks.

Seeding the counter before enqueueing is kept deliberately: an image
task that finishes early would otherwise decrement a key the fan-out is
about to overwrite. The release is what settles the resulting gap.

Also collapse the Redis key into multimodalPendingKey so the fan-out and
fan-in cannot drift, and share one enqueueKnowledgePostProcessTask
between the no-multimodal path and the slot release.

* fix(knowledge): keep the embedding-model guard from clobbering abort state

Review follow-up on the two stranded-row fixes.

The embedding-model guard wrote parse_status=failed unconditionally, which
made two cases worse than the silent return it replaced. Resolving a model
reads the database, so the entry guard's verdict is already stale by the
time the error comes back:

  - A user cancel signals asynq CancelProcessing, which cancels the
    handler context. GetEmbeddingModel then fails with context.Canceled
    and the guard overwrote "cancelled" with "failed: ... context
    canceled" — mislabelling an interrupted run as a broken model. Worker
    preemption and graceful shutdown look the same.
  - ReplaceKnowledgeFile keeps the knowledge ID and swaps file_path.
    UpdateKnowledge is a full-row Save and file_path is not among
    omitFieldsOnUpdate, so the stale row rolled the path back over the
    replacement and marked the replacement's live attempt failed.

So the guard now returns without writing when the context is gone or the
error is a context error, re-checks isKnowledgeAborted before writing, and
persists through updateKnowledgeUnlessSourceReplaced like the completion
path already does.

The multimodal slot release gave up on the first Redis error. Since the
release is the only thing that can drain a slot with no task behind it,
that put the row straight back into the stranded state this fix exists to
remove. It now retries a bounded number of times and logs at error level
when it still cannot settle the counter. DECRBY is not idempotent, so a
retry after a lost reply can over-decrement; that finalizes early rather
than stranding, which is the direction to fail in, and post-process
already tolerates repeat deliveries. The nothing-enqueued branch's Del
also moves onto the detached context, so a cancelled parent no longer
leaves the seeded key to expire on its own 24h TTL.

Two reporting corrections while here: the span now carries the same
message as the document list instead of a fixed generic string, because
the timeline renders error_message and withholds error_detail; and the
comment no longer claims expired credentials or an unreachable provider
surface here — resolving a model only reads the row and constructs a
client, so those fail later on the first BatchIndex round-trip.

Adds 6 tests for the paths above. Two of them initially passed against
the unfixed code because the entry guard caught them before the model
call, so the repository fake now serves the cancelled / replaced row only
after the guard's own reads, which is the window that actually exists.
2026-09-20 13:02:05 +08:00
neoczr 473c8726bb
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404)
* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer

The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.

Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.

* fix(ocr): keep merged-cell HTML tables chunkable

- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
  still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
  pad the block with blank lines so the chunker can split by "\n"
  instead of force-cutting at the absolute max size.

* fix(knowledge): normalize inline HTML tables for all parser engines before chunking

Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.

Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
  before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
  markdown

Converters keep their existing calls (idempotent no-op).

* fix(chunker): bound Markdown link/image protected patterns to one line

The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).

Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.

Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.

* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths

Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.

* style: wrap long lines and avoid shadowing max in table-chunk tests

Pre-push lll/revive failed on the PR diff versus origin/main.

---------

Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 21:10:17 +08:00
lyingbug 5574ebc011
feat(knowledge): derive knowledge base descriptions from document profiles (#3370)
* feat(knowledge): derive knowledge base descriptions from document profiles

Knowledge bases only had a hand-written description, which most users
leave empty, so agents bound to several knowledge bases had nothing to
route on and searched all of them. This adds a generated description
that is derived from per-document data instead of maintained by a model.

Document profile: the summary call now returns JSON with a 2-3 sentence
summary plus a gist, 3-5 topic keywords, a document type and one
typical question (knowledges.profile). Input is capped at 8k chars and
output at 1024 tokens; plain-text replies from custom templates still
work and simply carry no profile. The summary chunk embeds the gist and
short summary.

Knowledge base aggregate: a database-only fold over those rows (counts
per file type, tag, normalized topic and doc type; stratified samples of
titles and questions; a hash of the inputs). Deleting, moving or
re-parsing a document changes nothing but the rows, so the next rebuild
is exact without any model judgement.

Description text: one small model call over the aggregate yields a
gist, merged topics and typical questions (knowledge_bases.
generated_profile). The call is skipped when the aggregate hash is
unchanged; failures keep the previous text and record the error; an
empty knowledge base clears it. The manual description is never
touched, and the agent runtime context renders both so the model can
pass knowledge_base_ids instead of fanning out.

Refreshes are requested from summary task exit, post-process completion,
deletion, cross-KB moves and enabling the option, debounced per
knowledge base in 30 s windows via kb:profile on the summary queue.
POST /knowledge-bases/:id/profile/generate regenerates synchronously.
profile_config (enabled, model_id, custom_instructions) opts a document
knowledge base in; the settings dialog shows the generated card with
regenerate and adopt-as-description actions in five locales. Search
results now log hits per chunk type so the value of indexing summary
chunks can be measured.

Migrations 000101 (PostgreSQL) and 000021 (SQLite) add the columns; the
SQLite schema test expects the new version and columns.

* fix(knowledge): keep generated descriptions out of duplicates and refresh clones

DuplicateKnowledgeBase copies the source row through JSON, which carried
generated_profile onto a knowledge base that has no documents yet, so
an empty duplicate advertised the source's topics. The duplicate now
starts without a generated description and keeps profile_config so it
derives its own once documents arrive.

CopyKnowledgeBase builds the target field by field and skipped
profile_config, and the clone task never asked for a description after
copying the documents, so a clone stayed undescribed until its next
upload. The target now carries a copy of profile_config and both clone
paths request a refresh on completion.

* fix(knowledge): refresh the loaded clone target and carry document profiles

ProcessKBClone requested the description refresh with the reservation
stub it builds before CopyKnowledgeBase runs. For the main create path
that stub has neither a type nor profile_config, so the refresh helper
rejected it and nothing was enqueued. The refresh now uses the loaded
target returned by CopyKnowledgeBase.

cloneKnowledge copied the summary text but not the structured profile,
so even a scheduled refresh would have aggregated titles without topics
until every clone was re-summarized. Cloned documents now carry an
independent copy of the profile.
2026-09-17 22:21:55 +08:00
wizardchen 38be45c9d1 refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00
wizardchen 079e96ae64 fix(knowledge): make in-place file replacement race-safe
Queued ProcessDocument tasks still held the old FilePath, and a crash
between the source UPDATE and reparse could leave completed knowledge
pointing at a new file. Dequeue first, flip pending in the same write,
and skip stale worker Saves when the stored path has moved.
2026-09-16 14:54:47 +08:00
lyingbug 3d81b977f1
refactor(resource-access): unify scoped authorization and transfer recovery (#3097)
* refactor(resource-access): unify scoped authorization and transfer recovery

* fix(resource-access): close reparse and transfer review gaps

* fix: enforce explicit file bindings and recover move cleanup

* fix: align mutation admission and cleanup scope checks
2026-09-08 15:36:33 +08:00
十三分之五六七 786d67609d fix(markdown): sanitize invalid UTF-8 and NUL bytes 2026-08-31 12:24:35 +08:00
lawrence 28dd15ec30 fix(retrieval): 分段中删除的图片不再被检索到
从分段里删掉一张图片后,这张图仍会回到检索结果和模型上下文里。

三个问题串在一起:

1. 删图时 syncEditedChunkImages 会把 image_ocr / image_caption 子块置为
   禁用,但 CollectImageInfoByChunkIDs 查子块时不看 is_enabled,又把它们的
   image_info 挂回了父块。knowledge_search 工具随后把每条 image_info 无条件
   渲染成 markdown 图片写进 LLM 上下文,图片就这样回来了。这里按 is_enabled
   过滤,两级遍历都跳过禁用子块。过滤放在 collector 而不是工具输出层:纯图片
   分块的正文本身就是 OCR 文本、没有 markdown 图片语法,按正文过滤会误杀。
   chat_pipeline 里既有的 fake 仓储 fixture 同步补上 IsEnabled: true:它们不
   走 DB 默认值,零值 false 会被新的过滤当成已禁用子块。

2. UpdateImageInfo 补建 image_ocr / image_caption 子块时没有设置 IsEnabled。
   CreateChunks 用 Select("*") 插入,gorm 的 default:true 不生效,零值 false
   直接落库,这些子块一出生就是禁用的。补上 IsEnabled: true —— 否则第 1 点
   的过滤会把它们永久丢弃。其余 CreateChunks 调用方均已显式设置该字段。

3. 摘要刷新每次都新建一条 summary 分块,旧的留着且仍然启用。原因是它复用了
   ListChunksByKnowledgeID 的结果,而那个查询硬编码 chunk_type = 'text',
   "查找已有 summary 分块"的分支永远匹配不上。残留的旧摘要描述的是编辑前的
   文档,会让用户已经删掉的内容继续被检索到。新增按 chunk_type 过滤的仓储
   方法,刷新时原地更新已有 summary 分块。
2026-08-20 12:01:28 +08:00
wizardchen e123f8247e feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档
通过 cgo 链接 anydoc Rust 静态库,把 docx/pptx/xlsx 等转为带原位图片的 Markdown;扫描版 PDF 无文字层时回落到 DocReader OCR。默认构建不链接该库。

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-16 11:37:27 +08:00
sandaogouchen 262b172c28
fix(chunking): preserve parent-child text child embeddings (#2680)
* fix(chunking): preserve parent-child text child embeddings

* refactor(chunking): simplify text chunk index selection
2026-08-13 11:20:03 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
ttommybot 0827cd6cf0 fix: retry document summaries before fallback 2026-08-05 19:29:32 +08:00
xiaowang 520be78766 fix(chunking): normalize line endings before splitting / 在分块前统一换行符 2026-08-05 19:24:45 +08:00
wizardchen 1985a59b69 fix(chunking): sync preview swagger and normalize splitter config
Centralize ingestion base splitter defaults in chunker.NormalizeSplitterConfig
so the preview endpoint applies the same normalization as upload. Regenerate
Swagger for parent-child preview fields and add regression tests for single-
level preview, default parent/child sizes, and context headers.
2026-08-05 16:17:48 +08:00
wizardchen 39f4220c13 fix(wiki): ensure prompt language is always resolved
Wiki generation sometimes rendered "Write in ." because async workers
read empty locale payloads without falling back to context or defaults.
Unify language resolution helpers and apply them at enqueue and consume
time so every wiki and related async prompt gets a non-empty language.
2026-08-05 16:14:50 +08:00
xiaowang d2888e9565 fix(chunking): align preview with parent-child splitting / 修复预览与父子分块不一致 2026-08-05 15:09:14 +08:00
xiaowang 51b88bbae3 fix(summary): preserve rows across table chunks / 修复表格分块摘要丢行 2026-08-05 13:12:59 +08:00
lyingbug fa67dfd44b feat(knowledge): store folder path for folder uploads and expose folder tree API
Folder uploads used to encode their relative directory inside file_name,
which made the document list render the whole path as the title and left
no way to browse or query a single folder. The directory now lives in a
dedicated folder_path column (with a backfill for existing rows), the
document list accepts folder_path / folder_recursive filters, and a new
GET /knowledge-bases/:id/knowledge/folders endpoint returns the folder
tree with per-folder document counts.
2026-08-04 20:45:29 +08:00
lyingbug 82ea91f18e fix(knowledge): clear stale error_message on the remaining reprocess paths
Follow-up to the error_message clearing added for SetFinalizing,
FinalizeSubtask and ProcessDocument. Two routes still persisted a
previous attempt's failure message:

- ProcessManualUpdate flips a row back to 'processing' without clearing
  error_message. A manual knowledge item that failed earlier (for
  example when the knowledge base lookup failed, or after an Asynq
  retry) re-enters processing still carrying that message, so the UI
  shows a failure on a document it also shows as running.

- finalizeIndexedKnowledgeState is the only route to 'completed' that
  bypasses FinalizeSubtask. When a document has no text chunks and no
  pending multimodal work it is completed in place, so the promotion
  that would otherwise clear the column never runs and a successfully
  indexed row keeps reporting the earlier failure.

The 'processing' transition is extracted into markKnowledgeProcessing so
ProcessDocument and ProcessManualUpdate share one definition of what
starting a new attempt means, and a future worker cannot reintroduce the
same omission. The helper also replaces the "processing" string literal
with types.ParseStatusProcessing.

Tests: markKnowledgeProcessing gets direct coverage, and the
finalizeIndexedKnowledgeState table now seeds a stale error_message and
asserts it survives the processing branches (FinalizeSubtask clears it
later) and is cleared on the completed branch.
2026-08-03 20:14:09 +08:00
shangfeng-liu 44206e4ad4 fix(knowledge): clear stale error_message on reprocessing and finalize
A knowledge item that failed once keeps error_message set; when it is
reprocessed or driven through the finalizing pipeline, the stale message
was never cleared, so the UI kept showing an outdated failure on items
that were processing or had completed successfully.

- SetFinalizing and FinalizeSubtask now clear error_message in the same
  atomic UPDATE that transitions parse_status
- ProcessDocument clears knowledge.ErrorMessage when flipping the row
  to processing for a new attempt

Add a SQLite-backed regression test covering both transitions.
2026-08-03 19:38:22 +08:00
Myang_print 5780affdfc fix(knowledge): surface batch reparse failures
Signed-off-by: Myang_print <1182470296@qq.com>
2026-07-31 19:06:16 +08:00
wizardchen 8dd997f0f4 refactor(knowledge): make the supported import extension set the single gate
The xlsx URL-import fix left the accepted extension set reachable through two
indirections and widened ApplyKnowledgeProcessOverrides to gate file types for
callers that are not file imports (web URL fetch, connector sync, reparse).

Collapse the set into one map that both isValidFileType and the extension check
read, move the shared validation next to the other file-type predicates, and
return ApplyKnowledgeProcessOverrides to its override-only scope so
resolveFileImportProcessConfig is the only import gate.

Also drop the unreachable branch in createKnowledgeFromFileURL: getFileType
never yields an empty string, so the resolvedFileType fallback and its else arm
could not run. Normalize the resolved type before it reaches the record and the
task payload, and add an invariant test asserting direct upload and URL import
accept the same extensions so #2447 cannot drift back.
2026-07-31 13:46:56 +08:00
wizardchen 43afa4c1f8 fix(knowledge): align URL file import validation with direct upload
Extract shared file-import checks (type, VLM/ASR, video) into
resolveKnowledgeFileImportConfig and reuse them for file URL import.
Also enqueue DataTableSummaryTask for spreadsheet URL imports and reparse.
2026-07-31 13:46:56 +08:00
wizardchen 48756d13d6 fix(knowledge): allow xlsx and other file types in URL import (#2447)
Align URL-based file import validation with direct file upload by
delegating to isValidFileType instead of a narrower whitelist.
2026-07-31 13:46:56 +08:00
chenjunhong 595d19a806 fix(docreader): restore optional XLSX header context 2026-07-28 22:00:36 +08:00
zhangzherui eb447365cd fix(knowledge): clear stale errors on reparse 2026-07-28 20:44:34 +08:00
wizardchen 497e8312e3 fix(knowledge): harden summary refresh after review feedback
Discard stale async refreshes without clobbering newer summaries, queue manual refreshes instead of blocking HTTP, sort edited chunks by index for summaries, escape document metadata in chat headers, and distinguish lookup failures from true staleness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-28 20:23:48 +08:00
wizardchen 48b661134b fix(chunks): retain questions across content edits 2026-07-28 20:23:48 +08:00
wizardchen 6a439a4c3b fix(chunks): keep edited retrieval content consistent 2026-07-28 20:23:48 +08:00
wizardchen 850a073c83 fix(summary): restore tenant context for refresh tasks 2026-07-28 20:23:48 +08:00
wizardchen 1b9858ecb8 fix(knowledge): enqueue stale summary refreshes 2026-07-28 20:23:48 +08:00
wizardchen 1320c50bde refactor(knowledge): keep metadata document scoped 2026-07-28 20:23:48 +08:00
wizardchen 95775ae909 fix(knowledge): bound generated question source IDs 2026-07-28 20:23:48 +08:00
wizardchen ac3eccddf5 feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
wizardchen 6fb858105a feat(llm): add prompt-cache observability and wiki cache-friendly prompts
Normalize provider-specific cache counters into TokenUsage, propagate call
purpose and prefix fingerprints through context for logging and Langfuse,
and restructure wiki page-modify prompts with warmup/singleflight to improve
prefix cache hit rates during reduce batches.
2026-07-23 19:17:30 +08:00
wizardchen 5af8eae81d feat(knowledge): enhance activity logging for FAQ imports
- Added new actions for FAQ import lifecycle: `faq.import_started`, `faq.import_completed`, and `faq.import_failed`.
- Updated activity details to include success, failure, and skipped counts for FAQ imports.
- Enhanced UI to display additional activity information, including filtering options and clearer summaries.
- Implemented tests to validate the new FAQ import outcomes and activity details.

This commit improves the tracking and visibility of FAQ import activities within the knowledge base, ensuring better audit trails and user feedback.
2026-07-23 10:50:32 +08:00
wizardchen a3a0308b62 feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
darcy f7ec759cc6 fix(knowledge): propagate chunking strategy to parent-child splitter configs
buildParentChildConfigs only copied ChunkSize/ChunkOverlap/Separators into
the parent and child SplitterConfigs, dropping base.Strategy. Since an empty
Strategy resolves to the legacy tier (resolveChainWithProfile treats "" as
legacy for backwards compatibility), parent-child chunking always ran the
legacy recursive splitter regardless of the configured strategy: no heading
alignment and no ContextHeader breadcrumbs in child embeddings, silently.

The flat path (buildSplitterConfigFromChunking) already propagates Strategy;
parent-child predates the strategy system (b3334990, 2026-03) and was not
updated when the 3-tier strategy landed (ae6fde23, 2026-05). SplitParentChild's
docstring ("Child splitting honours childCfg.Strategy") and mergeBreadcrumbs
both assume the strategy reaches the parent-child path.

Copy base.Strategy into both configs and add a regression test.
2026-07-14 11:56:29 +08:00
wizardchen 07ba01ec93 feat(worker): replace aggregate asynq concurrency with per-pool governance
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
2026-07-13 15:51:06 +08:00
wizardchen 5eefa70e6f Enhance knowledge base configuration by normalizing and validating custom instruction fields. Implemented length checks for prompt instructions and added tests for validation logic. Updated GraphSettings to retain custom instructions when extraction is disabled. 2026-07-13 12:16:19 +08:00