Commit Graph

12 Commits (e6ecb9adcc8ddbb500957f77490ca5cb4c2fa8d4)

Author SHA1 Message Date
lyingbug 04adc13272
fix(retrieval): fix recall correctness, score scales, cost and agent tools (#3687)
* fix(retrieval): keep index writes and shared-KB lookups correct

Index writes
- Image OCR/caption chunks were indexed without IsEnabled, so every engine
  but Postgres stored them disabled and retrieval never returned them.
  The SQLite engine re-enables the affected rows at startup.
- FAQ edits, similar-question additions and import merges re-index under
  the old SourceID and relied on the engine overwriting it. Postgres and
  SQLite kept the stale row (ON CONFLICT DO NOTHING); ES v8 and Qdrant
  mint a new document ID and kept both. FAQ re-indexing now deletes the
  old rows first, and Postgres/SQLite BatchSave upserts by
  (source_id, source_type).
- Postgres: drop the GORM default tag on is_enabled, which replaced false
  with true on insert and indexed disabled chunks as enabled.
- SQLite: BatchSave no longer attaches vectors and FTS rows to the wrong
  rows when part of a batch already exists, and vecTables is guarded
  against concurrent map writes.
- CopyIndices: Postgres filters to the copied knowledge, pages by id and
  keeps is_enabled; SQLite copies every row of a chunk (generated
  questions included) with stable source IDs and enabled state.

Shared knowledge bases
- Chat merge: FAQ answer population and neighbor expansion look chunks up
  without the caller-tenant filter, like parent resolution (#3342).
- Neighbor expansion never cuts the retrieval hit; the length budget goes
  to the text adjacent to it.
- FAQ negative questions are applied whenever an FAQ KB is in scope and
  to chunks of shared FAQ KBs.
- Search results get image_info for shared-KB chunks.
- search_knowledge reads FAQ metadata from the search result instead of
  one tenant-scoped query per hit, so shared FAQs render their answers.

* fix(retrieval): put every retrieval score on one scale

Scores from different retrieval paths were ranked together while living on
different scales, so ordering depended on which path produced a hit.

Fusion
- RRF ranked a chunk by its position in the concatenation of every
  vector (or keyword) list, appended in goroutine completion order. The
  second list's best hit ranked behind every hit of the first, and the
  order changed between identical requests. Each list is now ranked on
  its own by score and a chunk takes its best rank.
- RRF is divided by its maximum, so hybrid scores land in [0, 1] like
  vector-only (cosine) and keyword-only (rescaled BM25) results. Raw RRF
  topped out near 0.016 and lost to any single-retriever hit it was ranked
  against across searches (document targets, model groups, FAQ calls);
  rerank's base-score term and MMR's relevance term had nothing to use.
- Keyword-only BM25 is rescaled per list; engines' BM25 is not comparable.
- FAQ iterative retrieval fuses each round like the first pass instead of
  sorting raw BM25 and cosine together, starts at twice the depth already
  searched, triggers when any single vector list is full (the old summed
  count never matched with more than one list) and replaces results per
  round.

Rerank and chat pipeline
- The FAQ boost is no longer capped at 1; capping made strong FAQs tie at
  1.0 and fall to the tie-breaker. The composite score before boosts is
  kept as composite_score.
- The FAQ direct-answer threshold is compared against that pre-boost
  score, and the FAQ that cleared it is the one marked match="exact"
  rather than the first FAQ.
- Graph hits no longer carry a fabricated retrieval score of 1.0, which
  ranked them above every scored hit when rerank did not run; the
  composite score uses the model score in its place.

Engines
- OpenSearch and Weaviate return (1 + cos) / 2; their drivers now report
  cosine similarity and map thresholds onto the engine scale, so a
  threshold means the same on every engine.
- Milvus L2 returned squared distance, ranked farthest-first downstream,
  and took the threshold as a radius; scores and radius are converted.
- The multi-store normalizer no longer maps Milvus through (s + 1) / 2,
  which put it on a different scale from every other engine and from its
  own single-store results.

Docs updated for fusion, normalization, FAQ boost and iterative recall.

* perf(retrieval): bound rerank, agent and pipeline cost; stop failing on slow or missing models

Rerank
- Rerankers report the longest document a request can carry
  (max_document_chars / max_request_chars minus the query) and the rerank
  stage fits its own passages to it, trimming appended captions, OCR and
  generated questions first. One chunk with a large screenshot's OCR used
  to exceed the vendor limit and fail the whole batch, so every candidate
  lost its score and the threshold stopped applying.
- Rerank HTTP requests default to a 60 s timeout; chat and agent contexts
  carry no deadline, so a stalled self-hosted rerank server held the turn
  open.
- OCR and caption chunks no longer repeat their own text from image_info in
  the passage.
- The chat pipeline and the agent tool rerank at most 200 candidates, best
  retrieval score first.
- A rerank model that cannot be loaded degrades to retrieval order instead
  of failing every chat turn, as the search APIs already do; without rerank
  results the merge stage keeps only RerankTopK hits instead of enriching
  every one.

Retrieval APIs
- knowledge-search match_count and rerank.top_k are capped at 200
  (unbounded values multiplied into tens of thousands of billed rerank
  passages); normalizedMatchCount clamps to the pool size so the
  over-retrieval arithmetic cannot overflow.
- hybrid-search with rerank recalls at least rerank.top_k deep.

Agent
- search_knowledge fits its rendered output to four fifths of the tool
  output budget, dropping the lowest-ranked rows and telling the model how
  many it omitted; the model's view is rebuilt from Data, which registry
  truncation never reached.
- Agent searches skip neighbor/parent/related enrichment rows, which were
  never retrieved yet tripled the rerank input.
- ParallelToolCalls is enabled. The model was always told it may call tools
  in parallel while the engine ran them one by one; only read-only tools
  overlap, anything else stays a barrier.

Chat pipeline
- Query expansion searches keywords only (its variants are local rewrites
  whose embeddings repeat the main search) and is skipped when keyword
  matching is off.
- Entity extraction, a second LLM call, is skipped for intents that need no
  KB retrieval and uses the rewritten query.
- Partial-overlap dedup tokenizes each text once instead of per pair, and
  only compares token overlap between texts within 3x of each other's size.

Storage
- Migration 000112 adds btree indexes on embeddings(knowledge_id) and
  (chunk_id); document deletes, chunk re-index and enable/tag updates
  scanned the whole table.
- Shared-KB knowledge rows are fetched in one query instead of one per ID.
- Drop GORM Debug() from the ParadeDB keyword query and the knowledge batch
  lookup, which logged SQL and user queries on every search.

Docs updated for the API limits, rerank timeout and passage fitting, agent
parallel tools and output budget, and pipeline changes.

* fix(retrieval): make agent tools and pipeline stages report what they did

Agent tools
- query_knowledge_graph now queries the graph store: entity names from the
  query are matched in each KB's graph, relations are returned (and shown to
  the model), and the entities' source chunks lead the results, with text
  search as a supplement. It used to run text search only, so a graph-only
  KB always answered "no relevant graph information".
- search_knowledge reports failures instead of "no matching chunks": when
  every search fails the tool fails with the reason; partial failures are
  listed. A failed query embedding falls back to keyword search for hybrid
  KBs, as the chat pipeline does.
- Keyword mode uses rerank to order hits, not to reject them (identifiers
  score low), and the empty-result hint no longer sends keyword mode back to
  keyword mode.
- Dedup keeps siblings under one parent, no longer merges every image chunk
  of a document through chunk_index 0, and keeps the better-scored copy.
- Search rows omit a <match> excerpt that only repeats the content; snippet
  terms are segmented for Chinese queries and skip stopwords, and FAQ
  matched questions are chosen by match strength.
- read_document query mode returns next_offset when truncated and resumes
  from offset.
- wiki_search over-fetches under document/tag scopes before filtering and
  says when every match was out of scope.
- Prompts point to read_document(id=cN, context=k) for surrounding text.
- The agent editor hides the retrieval-strategy section in agent mode,
  where search_knowledge uses the global retrieval settings.

Chat pipeline
- Intent labels are normalized; an unknown label retrieves instead of
  silently skipping the knowledge base, and web_search without web search
  enabled searches the knowledge bases.
- Image turns get a 2048-token rewrite budget, and a reply cut off by the
  cap is salvaged field by field instead of being discarded.
- History references are matched by how much of the query they cover;
  Jaccard against chunk-sized text never reached its threshold.
- Entity search resolves chunks of org-shared KBs within the turn's KB
  scope, skips chunks whose document is gone (it used to panic) and caps
  graph chunks at 30.
- Empty history is not fetched twice; fetched pages are cut on a rune
  boundary.

Retrieval
- A multi-KB search whose primary KB has no vectors embeds the query with a
  vector KB's model instead of failing on an empty model ID.
- Rerank passages include the chunk's heading breadcrumb, which is part of
  what was embedded.
- SQLite FTS searches the content column only and turns a lone Han
  character into a prefix query; TencentVectorDB ef follows TopK.

Docs updated for the agent tools and pipeline stages.

* fix(retrieval): address review findings on the retrieval fixes

- Restore NeedsRetrieval for web_search without web search enabled: the
  intent prompt "web_search" is how the user learns web search is off, and
  running KB retrieval instead bypassed it.
- query_knowledge_graph keeps the case of entity terms; Neo4j matches node
  names with case-sensitive CONTAINS, so lowercased terms never matched
  English entities.
- An empty search_knowledge or query_knowledge_graph result states partial
  failures in its text, which is all the model sees for an empty result.
- SQLite BatchSave replaces a source in one transaction, so a failed insert
  keeps the old rows instead of leaving the source unindexed.
- Reuse stringSliceValue and an exported searchutil.ContainsChinese instead
  of new copies; drop an unused named return and the FAQ obsolete-source
  delete that the full per-chunk delete now covers.

* fix(retrieval): address second review of the retrieval fixes

- query_knowledge_graph: under a document/tag scope, keep only relations
  whose entities come from an in-scope evidence chunk (the graph namespace
  is the whole KB); drop chunks whose document no longer exists; report a
  graph or text search failure even when the other source returned hits.
- search_knowledge: record the hybrid -> keyword fallback taken when the
  query embedding fails in mode_fallbacks, so an empty result is not read
  as "no semantic match".
- query understanding salvage: take a cut-off value only for the image
  description; a truncated rewrite_query or intent is dropped instead of
  replacing the user's query.
- Graph hits (no retrieval score) are kept past the no-rerank RerankTopK
  cut and the rerank candidate cap instead of always falling below them.
- FAQ boost is capped at 1 again so scores stay on the [0, 1] scale; FAQs
  tied at the cap are ordered by their pre-boost score.
- Negative-question filtering loads only candidates of FAQ knowledge bases.
- read_document query continuation starts one chunk before offset so the
  first match keeps its preceding context at a page boundary.
- graphSearchTerms and searchQueryTokens share one queryTerms splitter.
2026-09-25 10:48:18 +08:00
lyingbug 9fc08c2079
feat(retrieval): unify rerank across search APIs and expose diagnostics (#3675)
* feat(retrieval): unify rerank across search APIs and expose diagnostics

The two retrieval APIs each missed half of the job: hybrid-search could
tune recall but never reranked, knowledge-search always reranked with the
tenant config and exposed no recall parameters. Rerank itself lived in
four places with drifting thresholds, passages and composite scores.

- internal/reranking: one rerank stage (passage = title + cleaned body +
  captions/OCR + generated questions, threshold with degradation and
  top-1 fallback, composite score, MMR) used by the chat pipeline, the
  agent search_knowledge tool and both APIs.
- hybrid-search: optional `rerank` object; absent keeps the old behavior.
- knowledge-search: vector/keyword thresholds, match_count, recall-path
  switches and a `rerank` object that can override or disable rerank.
- Rerank model resolution: request > tenant RetrievalConfig > first
  rerank model; an unknown requested model is a 400. Load or call
  failures degrade to retrieval order instead of failing the search.
- Responses carry meta.rerank (outcome, model, thresholds, top score) so
  an empty result explains itself (#2391).
- Go SDK and website-docs updated, including a "which API" section.

Refs #2976

* docs(retrieval): lead with which search API to use and when

knowledge-search is the default for callers; hybrid-search is the
low-level recall interface. Add a scenario table so the choice does not
hinge on reading the whole capability comparison.
2026-09-24 20:41:49 +08:00
lyingbug be46eabaf2
docs: sync website-docs, homepage and changelog for v0.8.2 (#3632)
* docs: sync website-docs with changes since v0.8.0

Audit every change between v0.8.0 and the v0.8.2 release against the
documentation site, adding what users and operators need and correcting
statements that no longer match the code.

- Add local browser (BrowserSkill), built-in MCP Server endpoints, the
  toolbox, conversation steer/fork/rewind and reasoning effort, the
  artifacts library, sandbox terminal and desktop, the Lite host sandbox,
  the rebuilt model catalog (including a protocol-compat field reference),
  Confluence and DingTalk Docs sources, Bocha/Serply search, per-channel IM
  locale, whitelist-only outbound, admin user creation, new configuration
  variables and migrations 000091-000110.
- Fix wrong API examples and parameters (model type values, knowledge
  search params, FAQ match_count cap, Notion credentials), tenant deletion
  semantics, default timeouts, parser engine claims, removed tables and
  commands, stale navigation paths, and route locations after the router
  split into routes_*.go.
- Add screenshots for the new pages and refresh outdated ones.

* fix(frontend): link the protocol-compat field to its documentation

The model editor told users to read catalog/compat.go on the backend, a
path that no longer exists after the model catalog refactor. The hint now
says when the override is needed and links to the field reference in
website-docs/03-features/06-models.md; the config examples and the API
type comment point to the same place.

* feat(website): showcase v0.8.2 on the homepage

- Add a v0.8.2 section for the local browser (BrowserSkill), the built-in
  MCP Server and conversation control, and turn the v0.8.0 release block
  into a standing skills & sandbox topic with the sandbox desktop and
  terminal.
- Replace the three-up galleries with a feature list beside one large
  screenshot: every item's text stays visible, clicking an item or one of
  its views switches the image with a cross-fade, and the list plays itself
  while on screen until the visitor picks something.
- Add Confluence, DingTalk, BrowserSkill and more model vendors to the
  integrations, bump the header badge to v0.8.2 and link the changelog.
- check-site reports missing docs screenshots as pending instead of
  failing, since the Screenshot component renders a placeholder for them.

* docs(changelog): add v0.8.2 release notes

* docs(website): add per-version release notes pages

Add a "版本发布" docs section with one page per release, starting with
v0.8.2 and a backfilled v0.8.0. Each page walks through the highlights
with the existing product screenshots, the other improvements and fixes,
and an upgrade checklist with the actions each breaking change needs.

The sidebar orders these pages by version, newest first; the docs header
gains a 版本 entry, and the homepage's v0.8.2 section now links to the
release notes page instead of the raw CHANGELOG.

* feat(website): let the site image listen on a configurable port

The Docker image hard-coded `listen 80`, so running it with host
networking or on a platform that assigns the container port required
editing the Nginx config and rebuilding. The image now renders its
config from the official nginx template directory at startup and reads
the listen port from `PORT` (default 80); deploy/nginx.conf stays a plain
config for static deployments.

test-docker starts a second container with PORT=8080 and checks the
rendered config and that the site answers on that port; the README
documents the variable.

* fix(website): namespace the nginx listen port variable

* fix(website): support bash-wrapped container startup

* fix(website): align nginx startup with frontend entrypoint

* fix(website): pin nginx runtime for older host compatibility

* fix(website): bake the configured nginx port into the image
2026-09-24 11:26:34 +08:00
lyingbug 63ef76dd80
docs: 迁移有效文档至 website-docs 并清理旧文档 (#3580)
* docs: consolidate maintained documentation in website-docs

* docs: remove migrated legacy documentation and update remaining references
2026-09-22 23:25:40 +08:00
lyingbug 9e0c28fbc9
feat(artifacts): let users delete generated files (#3447)
* feat(artifacts): let users delete generated files

技能生成的文件此前只能列出、预览和下载,没有任何删除入口,后端
handler/service/repository 三层都没有实现。DeleteSession 里那句注释
("a dedicated GC job or explicit hard-delete API is the right place to
reclaim storage") 留的就是这个位置。

Tombstone, not a row delete. message_artifacts 新增 deleted_at,删除只
打标记不删行。position 是下载接口寻址用的下标(msg.Artifacts[index]),
物理删掉中间一行会让它后面的文件整体前移,已有链接下到错的文件;而
ArtifactCollector 的去重集就是从这张表读的,行没了之后沙箱里 mtime 未变
的同名文件下一轮就会被重新采集回来,删了又出现。墓碑同时挡住这两条。
loadArtifactsByMessage 因此刻意保留墓碑,展示侧用 MessageArtifacts.Live
过滤;NewMessageArtifactRecords 带上 DeletedAt,任何一次消息重写都能把
墓碑原样写回去。

Bytes. 字节是真回收的,走 catalog.Release 引用计数:一份文件可能同时被
知识库条目和后续回答持有,只有所有持有者都放手才 DeleteFile。删多个版本
时同一个对象可能来自多条消息,每条消息的绑定都要释放,字节只删一次。
回收发生在打标记之后,且只针对本次调用真正认领到的行——并发的第二次删除
认领不到任何行,也就不会去动别人正在回收的对象。释放失败一律保留字节:
留个孤儿 blob 以后还能靠墓碑上的 url 回收,删错了则是别人的文档里的图片
凭空消失。

Scope. 会话内面板每个版本单独一行,删的就是点的那一版;产物库一行代表
一个文件(带版本数),默认连同该文件在这个会话里的所有版本一起删,否则
删完这一行还在、只是显示上一版。权限比下载严一档:下载认共享智能体的
只读访问,删除只认会话归属人。

Inline references. 正文里的 resource:// 引用渲染成置灰加删除线的卡片,
不可点击,和「模型引用了不存在的文件」的 --pending 态区分开。图片类产物
也降级成卡片:走 renderImage 只会拉取失败,留下一张永远加载不出来的图。
墓碑必须留在传给渲染器的产物列表里——句柄形式与知识库图片同形,过滤掉
之后句柄对不上本条回答,会被当成外部受保护图片去拉一个已经不存在的
blob,结果是裂图。组件里因此拆出 liveArtifacts 只给计数和文件列表用。

Also: 产物库是 offset 分页的无限列表,删一行会让后续页整体前移漏掉一条,
改成按已显示行数取下一页并按 key 去重;access.ResolveMessageArtifact 与
MessageReferencesFile 不再让墓碑授权访问;SessionArtifacts 过滤墓碑,
避免新回答把引用绑到已删文件上;版本澄清文案不再把已删版本算进版本数。

docs/swagger.* 未重新生成:仓库里那份已经落后若干功能,重生成会带进约
1000 行无关漂移,另开 PR 同步。

* fix(artifacts): close the delete holes found in review

Regeneration. loadKnownSet 把墓碑的 ContentHash 也放进了去重集,于是
knownSameContent 会把「删除后技能重新生成、字节恰好相同」当成 fork
checkout 那种 restore:不上传、不挂回本轮回答,文件再也回不来。墓碑现在
只登记 (path, mtime),不进 hash 也不进 byPath —— 前者才是「沙箱文件没动
过就别复活」需要的键,后者是「同内容恢复」的短路,两件事不该共用。
RecordRestoredArtifactMtime 也不再改墓碑:它的 mtime 就是那把锁。

Message file proxy. /sessions/:id/messages/:message_id/files 走
MessageReferencesFile,而正文检查在最前面且无条件为真。删除不会重写回答,
resource:// 句柄还留在正文里,于是这条代理会继续把已删文件交出去——包括
交给共享智能体的只读访客,也包括「字节因为知识库还持有而没被回收」的情形。
现在先看 artifacts:命中本条消息的墓碑直接 false,命中 live 直接 true,
其余(知识库图、附件)才回落到正文。同一文件挂在多个位置时只要还有一个
live 就仍然可取。

Cross-session bytes. CreateForked 复制产物行时连同存储 URL 一起复制,但
不会给副本重新 Bind,所以 catalog 的引用计数看不到分叉。在父会话里删除会
把分叉还在用的字节删掉。无 catalog 部署、以及存的是裸 provider 路径的行,
同样不在计数覆盖内。回收前新增一道 CountLiveArtifactsByURL:只要还有任何
未删除的行指向同一对象就保留字节,查不出来也保留。配套加了 url 的部分索引。

Atomicity. SoftDeleteSessionArtifacts 原来逐条 UPDATE 不带事务,删多版本
时中途失败会留下「一部分已隐藏、字节没回收、接口 500、用户重试 404」。
现在整批放进一个事务。writeMessageArtifacts 是整表替换,拿着删除前快照的
并发写会让墓碑复活而字节已在回收,现在按 position 把已存在的 deleted_at
合并回去——删除是唯一不允许被过期写覆盖的属性。

all_versions 两个路由改用同一个解析器(strconv.ParseBool),此前一个是
== "true"、另一个是 != "false",显式传 1/TRUE 在两边语义不同;缺省值仍
按路由区分。

测试:收集器三条(未动不复活 / 重新生成要回来 / live 的同内容恢复仍短路)、
access 三条(正文不能复活已删授权、多副本只要有 live 就可取、非产物引用
仍走正文)、service 七条(归属人校验、他人会话拒绝、并发认领为空、live
行仍在则保留字节、guard 失败保留字节、多版本按对象归并 owner、无
source_path 不扫全会话)、repository 三条(墓碑抗过期写、事务回滚、跨会话
URL 计数)、handler 一条(两个路由的 all_versions 解析矩阵)。其中
「重新生成要回来」在改回旧实现时确认会失败。

client/ 没有 artifacts 相关 API(只有一个 SSE 事件类型常量),无需同步。
2026-09-20 13:01:46 +08:00
lyingbug b82fd6c0f9
fix(security): close the low-severity shared-space findings (#3416)
* fix(security): close the low-severity shared-space findings

Verified and fixed from the shared-space permission review:

- Embed channels kept knowledge_ids, tag_ids, mentioned_items, skill_names
  and summary_model_id from anonymous visitors. Mentions are only scoped
  for shared agents, so a visitor could query any KB (and pick any model)
  of the channel workspace by ID. The payload now drops them; the widget
  never sends them.
- PUT /knowledgebase/:kb_id/wiki/issues/:issue_id/status updated any issue
  by ID. The update is now scoped to the path's KB (404 otherwise).
- A direct org invite let the inviter attach any user of the target
  workspace (or any user by ID) as its representative, whose username and
  email the roster then showed to all members. Direct adds attach no
  representative, and the roster returns emails only for the caller's own
  workspace.
- COS and KS3 dropped the bucket (and COS the region) segment when turning
  a path into an object key, while the tenant check scans every segment:
  cos://42/<region>/prefix/7/... passed as tenant 42 and read tenant 7's
  object from a shared bucket. Both must now match the service's config.
- Shared agent listings returned the full agent to receivers. Prompts and
  the creator's user ID are withheld; capabilities and scope stay visible.
- Temporary document uploads let the caller pick the parser engine (and
  its credentials) of a shared agent's workspace; only the agent's rules
  choose one there now.
- Suggested questions used request / message-snapshot knowledge_ids
  without a permission check. Each document is now resolved to its KB and
  checked like KB IDs, including a shared agent's scope.
- IM channels accepted any knowledge_base_id (and any agent ID on create).
  Both must belong to the channel's workspace, the KB also to a restricted
  API key's allow-list.
- KB-restricted API keys could list org-shared KBs outside their allow-list
  in the organization views; those listings are now filtered.
- With enable_rbac=false, RequireKBAccess let cross-workspace denials
  through. That switch is for roles inside a workspace; the guard now
  always denies another workspace's KB, since handlers behind it load by
  ID without a tenant filter.
- The org member limit was counted before and enforced apart from the
  insert; it is now checked in the insert's transaction with the
  organization row locked (SQLite serializes writers instead).
- The UpdateOrganization route comment and API doc said owner-only; the
  service and UI allow org admins. Comment and doc corrected.

Documented, not changed: issued /r/ and presigned links stay valid until
they expire after a share is revoked.

* fix(security): address review of the low-severity shared-space fixes

- Embed payloads also drop question_origin. It selects a KB like an
  @mention for agents that retrieve only when mentioned, and the widget
  never sends it; the selection stays inside the agent's scope, but an
  anonymous visitor should not pick among its KBs. suggestion_attribution is
  kept (the widget sends it and it is validated).
- The receiver view of a shared agent also withholds the follow-up
  suggestion instruction, a prompt like the others. Starter questions are
  shown to receivers anyway, and the sharer's name is displayed by design.
- Follow-up suggestion generation moves into the agent's workspace with
  types.WithExecutionTenant instead of a bare tenant rewrite, so the
  caller is pinned before the document permission check runs. The auth
  middleware and the shared run already pinned it; this removes the
  dependency on that.
- KS3 CopyFile copies only from the service's own bucket (as COS and S3
  do); another bucket's path is a cross-backend copy.

Tests cover the dropped question_origin, the follow-up instruction and the
/shared-agents listing, a follow-up context with the tenant switched, a
repeated wiki issue status update, and the KS3 copy source.
2026-09-18 22:58:46 +08:00
lyingbug db8e7611be
fix(security): close cross-workspace access paths in shared spaces (#3408)
* fix(security): close cross-workspace access paths in shared spaces

A review of how shared-space (organization) members reach each other's
resources found six paths that bypass the org-share permission model:

- presigned-preview signed any resource handle or storage path, turning a
  known resource:// handle into an anonymous /r/ URL that outlives share
  revocation. It now requires the path to belong to the caller's tenant,
  like /files.
- POST /initialization/initialize let a shared-KB editor rewrite the owner
  workspace's models (base URL, API key) in place. Only the KB's own
  workspace may call it, and rewriting a stored model needs the same
  authority as PUT /models/:id.
- KB JSON carried the legacy inline credentials (storage_config
  secret_id/secret_key, which initialization fills with the MinIO env
  keys, and vlm_config api_key) to every share receiver. They are withheld
  from JSON; their DB columns are unchanged.
- A chat without agent_source_tenant_id resolved shares before the
  caller's own agent, so a member sharing a same-ID built-in agent took
  over other workspaces' default agent. Own agents now win, built-in
  agents can no longer be shared (existing such shares no longer resolve),
  and embed channels drop a visitor-supplied source workspace.
- Any org-admin tenant (even a tenant Viewer inside it) could raise another
  workspace's KB share to editor/admin, and the route guard was bypassable
  via an unrelated :id. Share management is now: the sharer from the source
  tenant (Contributor+), source-tenant Admin+, or an org-admin tenant's
  Admin+ that may only remove or lower. share_id must belong to :id.
- Wiki write tools were registered for every wiki KB in the search targets,
  which only need read access, so a viewer share (e.g. via the built-in
  wiki fixer) could edit or delete another workspace's wiki pages. They now
  act only on KBs the caller may edit.

* fix(security): close the KB settings path and owner details for share receivers

Follow-up to the shared-space review:

- PUT /initialization/config shares the route guard that admits share
  editors, so an editor could still rebind the owner's KB models, chunking
  and storage, and storage backends resolved against the caller's workspace
  let it bind the owner's KB to its own backend. Another workspace now needs
  an admin share (the only share level the frontend offers settings to) and
  may not change the storage binding at all.
- GET /initialization/config showed model base URLs and bucket locations to
  any admin of the caller's own workspace, including share receivers. They
  now only see whether credentials are configured.
- Wiki write tools on the caller's own KBs had no role floor, so a tenant
  Viewer (and the IM, embed and MCP endpoint principals that run as Viewer)
  could write through an agent what the HTTP routes deny. Writes now need
  Contributor+, or the ingest capability for a scoped API key, following
  the RBAC rollout switch.
- resolveAgent logs the share lookup error instead of a nil own-agent error
  when a shared selector fails.

* fix(security): close the medium shared-space access findings

Follow-up to the high-risk fixes, from the same review:

- Wiki fixer moved into a shared KB's workspace ran with the caller's own
  config there: an unset MCP mode registered all of the owner's MCP
  services, and an "all" KB scope reached the owner's other KBs. The scoped
  run now uses the built-in config pinned to that one KB, without MCP,
  skills, sandbox or web search, and falls back to the KB's own model.
- An unset MCP mode means "all" at runtime but is shown as none in the share
  scope; shared runs now treat it as none.
- Quick-answer mode used the request's web_search_enabled alone, so a
  receiver could turn on search the agent disables and spend the owner's
  provider quota. It now requires the agent setting too, as agent mode does.
- summary_model_id picked any model of the workspace a run executes in. It
  is ignored for shared agents and the relocated wiki fixer.
- A workspace that left or was removed from an organization kept sharing its
  KBs and agents. Its shares are revoked on departure, and share queries and
  permission checks require the source workspace's membership and a live
  organization, which also covers rows left behind earlier.
- Sharing an agent exposes its KB scope to all members, so a Contributor
  could share colleagues' KBs (or "all") through an agent. The sharer must
  now be able to share each KB directly (creator or Admin+; "all" needs
  Admin+), also when a shared agent's scope is widened later.
- A scoped API key with manage_channels could mint an MCP endpoint token
  wider than itself. Endpoints it creates, updates or rotates must stay
  within its KB allow-list and capabilities.
- A shared agent's run indexed the receiver's Q&A into the owner's chat
  history KB; indexing now uses the session owner's configuration.
- Sharing an agent with skills keeps injecting the owner's skill
  environment values (product decision); the share panel now warns that
  members can read them. Preview of originals through read-only shares is
  kept as designed and documented.
2026-09-18 22:00:02 +08:00
lyingbug 6db8fdd601
feat(artifacts): artifacts library page backed by a message_artifacts table (#3384)
* feat(artifacts): add an artifacts library with its own message_artifacts table

Skill-generated files were only reachable from inside the conversation that
produced them. This adds a sidebar entry and a page that lists them across
every conversation the user can see.

Storage: artifacts move out of the messages.artifacts JSONB column into a new
message_artifacts table (PostgreSQL 000103, SQLite 000023), backfilled on
upgrade. The table is now the only source of truth; the old column is kept and
the down migration writes the table back into it. The message repository
loads artifacts for every message it returns and writes them on create,
update (when non-nil) and fork, so callers still use Message.Artifacts.
mod_time is stored as RFC 3339 text because the collector matches files on
nanosecond mtimes, which PostgreSQL timestamps would truncate.

API: GET /api/v1/artifacts lists the latest version of each file (grouped by
session + source path, with version_count), scoped like the session list,
with keyword / file type filters and paging. Downloads keep using the
per-session endpoint.

Frontend: "产物/Artifacts" sidebar entry (hidden without sandbox support), a
library page with type tabs, search, date groups, infinite loading, preview
drawer, download and jump-to-conversation; strings in all five locales. The
sidebar's top-menu paths are now one shared list.

* fix(artifacts): keep the restore hash guard and count versions by URL

RecordRestoredArtifactMtime again only stamps rows whose content hash matches
the restored content (and ignores empty hashes), and never rewrites the hash,
matching MessageArtifacts.WithRestoredMtime. Without the guard a restore of
v1's content also moved v2's mtime and relabelled v2 with v1's hash.

The library's version_count now counts distinct URLs per (session, source
path). A later answer that references an earlier file stores another row with
the same URL, which COUNT(*) reported as an extra version.

* fix(artifacts): scope the library to web chats and clear the legacy column

The library used the session list's default scope, which counts sessions
without an owner as legacy tenant-level rows. IM sessions are created without
an owner, so every member saw the file names and titles of every IM chat
(downloads then 404 for non-admins). The library now applies the same "web"
bucket predicate as the home sidebar, extracted from QueryPaged so the two
cannot drift: no IM mapping (including cleared ones), no embed marker, no
API-key owner.

Migrations 000103 / SQLite 000023 now clear messages.artifacts after the
backfill. Message loads still SELECT *, so the stale copy was read on every
history load. Only rows that carried artifacts are rewritten; the column stays
for instances on the old binary during a rolling upgrade, and the down
migration still rebuilds it from the table.

* fix(migrations): keep the 000103 backfill alive on malformed artifact JSON

A single file_size that is not an integer ("12.5", "") or a created_at that
does not parse ("garbage", month 13) raised a cast error and failed the whole
migration, blocking startup. Casts now go through pg_temp helpers that return
NULL on bad input, falling back to 0 and the message's created_at; valid rows
are unchanged. SQLite casts never raise; its migration test now covers the
same malformed values.

* docs(artifacts): describe the web-only library scope and the cleared legacy column
2026-09-18 12:10:19 +08:00
lyingbug ef9cbf456d
docs: update website guides and improve writing style (#3095)
* docs: sync website guides with current product capabilities

* docs: improve website guide structure and writing style
2026-09-08 10:54:41 +08:00
wizardchen f780f81fc0 docs(website-docs): add comprehensive product documentation site
Import and update WeKnora documentation with VitePress: getting started,
architecture, features, API reference, clients, and development guides.
Includes landing page with inline SVG illustrations, theme components,
doc check scripts, and product screenshots.
2026-08-06 19:54:40 +08:00
lyingbug 8d9309e8fc Revert "docs: add comprehensive WeKnora documentation"
This reverts commit 4aeef20b46.
2026-08-06 19:52:25 +08:00
wizardchen 4aeef20b46 docs: add comprehensive WeKnora documentation
This commit introduces a complete set of documentation for WeKnora, organized into six main sections: Getting Started, Architecture, Features, API Reference, Clients, and Development Guide. Key highlights include:

- Detailed installation instructions and quick start guides.
- In-depth architectural overview and backend design documentation.
- Comprehensive feature descriptions, including multi-tenant support, document parsing, and agent capabilities.
- API reference covering approximately 360 endpoints with usage examples.
- Client documentation for various interfaces including web, CLI, and SDK.

This documentation aims to facilitate user onboarding and provide a thorough understanding of the WeKnora system.
2026-08-06 18:29:35 +08:00