Reject invalid agent_source_tenant_id instead of silently falling back, fix a resolveAgent nil panic when a shared selector fails, and add regression tests for wiki write-tool filtering and agent resolution.
Replace the broad channel-runtime principal check with owner-scoped rules so
API callers can still read their own api_tenant_key and api_external_user
sessions without opening IM sessions matched via legacy empty user_id scope.
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.
中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
ProcessKBDelete must return retryable engine-resolution failures instead
of logging and continuing, which could delete knowledge rows while leaving
embeddings behind. Align the handler with ProcessIndexDelete and cover
cancellation and ErrVectorStoreUnavailable in tests.
The engine registry is per-process, so a store's engine is missing
whenever this process did not create it. That happens three ways:
- Startup skips a store whose engine fails to construct and never
retries it, so that instance cannot serve the store again -- not
even after a restart, if the backend is slow to come up.
- A store registered on one instance is absent on every other until
they restart, because registration is not broadcast.
- Registration can fail on the instance handling the create, leaving
the store missing even there.
Each one surfaces as "vector store is currently unavailable" and is
cleared today only by restarting the process.
Add GetOrLoadByStoreID, which falls back to building the engine from the
store row when the lookup misses, and route the two bound-store lookups
through it. The method goes on RetrieveEngineRegistry rather than
StoreRegistry because that is the interface the factory functions
receive; the two declare GetByStoreID separately and neither embeds the
other.
Concurrent callers collapse onto one build. That build runs on a context
detached from the caller that started it: callers share the result, so
letting the first one's cancellation abort it would fail everyone else
waiting. A panic inside the build is contained, because singleflight
re-raises it on a goroutine the HTTP recovery middleware cannot reach
and the build calls third-party client constructors.
Publishing is guarded by a per-store generation sampled before the
build, so an engine that finishes after the entry was registered or
removed by someone else does not overwrite it and leave that engine
orphaned with its connections open. A failed build starts a cooldown:
collapsing only helps concurrent callers, so without it a backend that
stays down would cost a full build timeout on every sequential request.
Separate "the store is gone" from "its engine cannot be built right
now". Async delete handlers answer ErrVectorStoreNotFound with
asynq.SkipRetry, so reporting a database blip or a backend outage that
way would permanently discard knowledge-base and index delete tasks and
orphan the vector data they exist to remove. ErrVectorStoreUnavailable
covers the retryable cases; the repository already distinguishes them,
returning (nil, nil) for a missing row and an error for a failure.
For the same reason a caller's own cancellation is never folded into a
store sentinel. The ownership check runs first and queries the database
with that context, so on a shutdown it is usually where cancellation is
noticed.
A search resolves one engine per distinct store, sequentially, and this
server sets no HTTP read or write timeout, so the resolution loop gets a
budget above a single build. An exhausted budget still leaves the
engines warming, since the build is detached, and is reported as an
unavailable store because retrying is likely to succeed.
Build failures are logged where they happen, with the panic stack. The
cause cannot travel back to the caller -- factory errors embed endpoints
and credentials -- so without this the one thing an operator needs when
self-healing fails to heal would be discarded with the error.
Note a deliberate contract change: an ownership lookup failure used to
be reported as not-found, and a test asserted that on purpose. A
database failure says nothing about whether the store exists, and
treating it as permanent is what discards the task, so that assertion is
inverted rather than preserved.
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
Discard stale async refreshes without clobbering newer summaries, queue manual refreshes instead of blocking HTTP, sort edited chunks by index for summaries, escape document metadata in chat headers, and distinguish lookup failures from true staleness.
Co-authored-by: Cursor <cursoragent@cursor.com>
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.
Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
Markdown tables — searchable text, instead of being lost or (post-export)
rendered as unsearchable images. Bitable date/datetime cells render in the
table's timezone (configurable via Settings["timezone"], default GMT+8) rather
than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
pipeline, gated by the KB's multimodal setting: the image's external_id always
stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
records) break on an empty page so a malformed has_more response cannot spin
until the task deadline.
Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
"<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
means a single failed attachment can't wipe its siblings, and dupIsSameNode
guards the sweep so a file_hash-only dedup collision against a different node
cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
(never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
metadata external_id prefix scan.
Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
Cross-KB moves updated knowledges/chunks but left wiki_pages source_refs
behind in the source wiki KB, so the folder tree and wiki graph still showed
moved documents. Run the same wiki cleanup as delete on the source KB and
enqueue wiki ingest on the target for reuse_vectors moves.
Sort mergedChunks after groupAndMergeOverlapping concatenates results
from different merge units. The unit order depends on Go map iteration
in the grouping phase, which is non-deterministic.
Uses the same tie-breakers as filter_top_k.go to restore consistent
ordering at the merge source.
groupAndMergeOverlapping groups results through Go maps whose
iteration order is undefined. The concatenated output has no global
sort, so different runs can return chunks in different sequences
even for identical inputs.
This test exercises cross-unit tie-breaking (Score, KnowledgeID,
ChunkType) with chunks that land in separate merge units. The test
currently fails because the post-merge sort is missing.
Single SearchTarget must intersect document whitelists with tag scope
instead of treating tags as an independent authorization path. SQL scope
filters now AND document and tag constraints; Agent tools use an explicit
scopeEnforced flag so empty targets fail closed. Also add missing built-in
tool handle policies with drift tests, cache handle regexes at registration,
encode tool error text, and avoid wikiKBIDs slice aliasing.
Unify request-local model-handle handling behind a single
modelcontext.Registry (source citations, durable resource handles, wiki
issue handles) with per-tool argument policies and unresolved-handle
rejection before tool execution.
Also fixes issues found in review:
- decode wiki_read_issue's iN issue_id argument (was never resolved)
- preserve stored wiki page aliases when wiki_write_page omits aliases
- validate http/https scheme before registering url fields as web sources
- gate tool-role MultiContent text behind the tool result policy
- dedup KB IDs in NewWikiScopesFromKBIDs to avoid false ambiguity
- remove dead codec facades (decodeJSON, EncodeMessagesWithPolicy,
UnresolvedToolHandles)
Weak LLMs routinely mangle high-entropy wiki slugs (especially UUID-based
summary/<id> slugs), producing dead cross-links and cross-item dedup
hallucinations. This tightens integrity end to end:
- Ingest dedup now groups each new item with only its own trigram
candidates and validates merges against that per-item scope, rejecting
cross-item mispairings; candidate top-K narrowed 20 -> 5.
- Editor path aliases out-link slugs behind low-entropy ref-N tokens and
translates them back after generation, removing the chance to mangle.
- Agent write tool normalizes/validates slugs, blocks manual creation of
system-owned summary pages, and auto-repairs dead [[slug]] links
(rewrite-only, never strips) via bigram/display-text resolution.
- llmresource registry aliases summary/<uuid> slugs the same way as
storage handles so they round-trip safely through chat.
Adds unit tests for slug validation, aliasing round-trip, dead-link
repair, and dedup rejection rules.
Doris Stream Load previously used a plain http.Client, leaving FE→BE
redirects and stored FE URLs without outbound SSRF checks. Use the shared
SSRFSafeHTTPClient, validate URLs at send time, and only forward Basic
credentials to whitelisted BE hosts after FE 307 redirects.
Treat limit as the authoritative upper bound instead of clamping it down
to the agent's Starters.Count, so ?limit=N no longer always returns 6.
Fixes#2238
Normalize provider-specific cache counters into TokenUsage, propagate call
purpose and prefix fingerprints through context for logging and Langfuse,
and restructure wiki page-modify prompts with warmup/singleflight to improve
prefix cache hit rates during reduce batches.
Prevent concurrent refresh-token rotation across instances, distinguish
expired-but-refreshable tokens from active authorization, and bind OAuth
popup polling to a per-attempt ID so stale tokens cannot satisfy new consent.
Run the synchronous best-effort queue scrub in DeleteKnowledgeBase with
context.WithoutCancel and kbTaskCleanupTimeout so client disconnects
cannot truncate cancellation mid-scan, matching ProcessKBDelete cleanup
semantics.
- Updated the listFAQEntries function to support multiple tag IDs for filtering.
- Modified the frontend components to accommodate the new tag_ids parameter.
- Enhanced the KbTagManageDrawer and FAQEntryManager components to reflect changes in tag handling.
- Improved API documentation to clarify the new tag filtering capabilities.
This commit enhances the flexibility of FAQ entry retrieval, allowing users to filter by multiple tags for more precise results.
- Updated the exportFAQEntries function to support exporting FAQ entries in both CSV and JSON formats.
- Modified the frontend to include options for selecting the export format.
- Enhanced localization files to reflect the new export options in English, Russian, and Chinese.
- Implemented new tests to validate the export functionality for both formats.
This commit improves the flexibility of FAQ exports, allowing users to choose their preferred format for better integration with other tools.
- Added new fields for tracking added, merged, and partially failed entries during FAQ imports.
- Updated the import result statistics to include detailed counts for added and merged entries.
- Enhanced the UI to display progress hints and improved import result summaries.
- Implemented a new method for saving chunks in a single transaction to improve performance.
This commit improves the visibility and accuracy of FAQ import results, providing better user feedback and tracking capabilities.
- Added new actions for FAQ import lifecycle: `faq.import_started`, `faq.import_completed`, and `faq.import_failed`.
- Updated activity details to include success, failure, and skipped counts for FAQ imports.
- Enhanced UI to display additional activity information, including filtering options and clearer summaries.
- Implemented tests to validate the new FAQ import outcomes and activity details.
This commit improves the tracking and visibility of FAQ import activities within the knowledge base, ensuring better audit trails and user feedback.