Commit Graph

26 Commits (22cbcdfa5789b33dbb8d47532c21b24da2fdf5aa)

Author SHA1 Message Date
vicfei 2dd36212d6 fix(knowledge): bind extracted images to their document and unscope shared-KB parent lookups
Org-shared KB viewers got 403 for extracted images on current main even
though the retrieving chunk text carried the resource:// handle and the
share grant was valid (#3342, second report after #3022/#3069).

Root cause is a mismatch between ingestion and authorization: document
parses store extracted images through SaveBytes, which registers the
resource but never claims it, while both file proxies (ResolveKBFile and
the message fallback via resourceAccessibleViaSharedKB) authorize through
IsReferencedByKnowledgeBase — which accepts explicit resource_bindings
rows only. The binding-only hardening landed without adding the missing
bindings, so every extracted image became invisible cross-workspace.

Ingestion: bindStoredImages claims each stored image for its knowledge
(ResourceRelationExtractedImage), mirroring bindContentResources — best
effort, same-workspace resources only. Wired into both the file-parse
path (knowledge_process) and manual knowledge (triggerManualProcessing).

Merge: resolveParentChunks fetched parents, grandparents, and image_info
children with the caller's tenant, so shared-KB hits from the sharing
workspace resolved none of them and lost parent context and image_info
(the second defect in the #3342 audit). Those lookups now use the
existing *Only unscoped variants, matching their shared-KB precedent.

Historical documents parsed before this change still carry no bindings;
a backfill is tracked as the follow-up slice.

Verified end-to-end on a fresh stack: publishing a manual document with
an embedded image now records kind=image plus an extracted_image binding
alongside the content-derived attachment binding.
2026-09-22 17:52:49 +08:00
neoczr 473c8726bb
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404)
* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer

The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.

Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.

* fix(ocr): keep merged-cell HTML tables chunkable

- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
  still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
  pad the block with blank lines so the chunker can split by "\n"
  instead of force-cutting at the absolute max size.

* fix(knowledge): normalize inline HTML tables for all parser engines before chunking

Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.

Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
  before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
  markdown

Converters keep their existing calls (idempotent no-op).

* fix(chunker): bound Markdown link/image protected patterns to one line

The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).

Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.

Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.

* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths

Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.

* style: wrap long lines and avoid shadowing max in table-chunk tests

Pre-push lll/revive failed on the PR diff versus origin/main.

---------

Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 21:10:17 +08:00
jacentsao 01f0700b35
feat(datasource): add Confluence sync (#3287)
* feat(datasource): add Confluence sync

* fix(confluence): resume full sync and list Server pages without CQL

CQL search silently truncates large spaces and can delete pages that
dropped out of the index. Full-sync checkpoints now keep a deletion
baseline so Asynq retries continue instead of restarting.

* fix(confluence): satisfy package lint for pre-push against main

* fix(confluence): parse Server page lists and keep version expand

The default Server/DC edition unmarshaled /content/page as a nested
envelope, so sync imported zero pages. Pagination next links also drop
expand=version,space, which made later pages versionless and hid edits.

* fix(confluence): classify errors and fail closed on empty listings

Raw Confluence error bodies were copied into sync-log UI, and pagination
next links could leave the configured context path via `..`. Empty
listings against a non-empty baseline now abort instead of deleting.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-16 14:38:16 +08:00
wizardchen 1f82fbc9b5 fix(datasource): preserve distinct GitLab files during deduplication 2026-09-14 19:32:41 +08:00
lyingbug 3d81b977f1
refactor(resource-access): unify scoped authorization and transfer recovery (#3097)
* refactor(resource-access): unify scoped authorization and transfer recovery

* fix(resource-access): close reparse and transfer review gaps

* fix: enforce explicit file bindings and recover move cleanup

* fix: align mutation admission and cleanup scope checks
2026-09-08 15:36:33 +08:00
lyingbug 270b64f6ec
feat(agent): preview skill artifacts and verify installs without running them (#2823)
* feat(agent): preview skill artifacts and verify installs without running them

Generated files open in the chat drawer with overlay/Esc returning to the
list, and install verification no longer executes guessed skill entry
points. A failed install can be retried from the saved archive.

* ui(chat): use a folder icon and corner count for skill artifacts

The eye and document glyphs read as preview or copy, and TDesign's
loading spinner sat beside the icon. A folder with a top-right count
matches the generated-files drawer without covering the toolbar action.

* feat(artifact): implement inline artifact references with previews

Add support for inline artifact references in Markdown, allowing generated files to be displayed as clickable cards. Implemented a new rendering mechanism for artifacts, enabling previews directly from the chat interface. Updated localization files for new inline preview hints and missing file messages. Introduced tests for artifact reference normalization and rendering logic.

* fix(client): document AgentResponseType constants for revive

Touching the const block made golangci-lint require comments on every
exported value. Match the rest of the SDK and describe artifacts_pending.

* fix(agent): wrap artifact-reference prompt lines for lll

Pre-push golangci-lint rejected the branch on three lines over 120
characters in the sandbox: reference guidance.

* fix(agent): scope HTML scripts and tighten install verification

Keep scripted HTML preview on skill artifacts only, recreate sandbox
workspace dirs after they are deleted, and treat nested skill files as
dependencies rather than first-party imports.

* refactor(artifact): enhance artifact reference handling and metadata

Updated artifact metadata to include a stable resource handle for better identification. Adjusted artifact reference resolution to support both handle and name forms, ensuring compatibility with existing references. Enhanced tests to validate the new handling logic and ensure proper rendering of artifacts in chat messages. Improved documentation for clarity on artifact reference formats and their usage in the system.
2026-08-27 10:30:32 +08:00
xiaowang 520be78766 fix(chunking): normalize line endings before splitting / 在分块前统一换行符 2026-08-05 19:24:45 +08:00
wizardchen 39f4220c13 fix(wiki): ensure prompt language is always resolved
Wiki generation sometimes rendered "Write in ." because async workers
read empty locale payloads without falling back to context or defaults.
Unify language resolution helpers and apply them at enqueue and consume
time so every wiki and related async prompt gets a non-empty language.
2026-08-05 16:14:50 +08:00
lyingbug fa67dfd44b feat(knowledge): store folder path for folder uploads and expose folder tree API
Folder uploads used to encode their relative directory inside file_name,
which made the document list render the whole path as the title and left
no way to browse or query a single folder. The directory now lives in a
dedicated folder_path column (with a backfill for existing rows), the
document list accepts folder_path / folder_recursive filters, and a new
GET /knowledge-bases/:id/knowledge/folders endpoint returns the folder
tree with per-folder document counts.
2026-08-04 20:45:29 +08:00
wizardchen 8dd997f0f4 refactor(knowledge): make the supported import extension set the single gate
The xlsx URL-import fix left the accepted extension set reachable through two
indirections and widened ApplyKnowledgeProcessOverrides to gate file types for
callers that are not file imports (web URL fetch, connector sync, reparse).

Collapse the set into one map that both isValidFileType and the extension check
read, move the shared validation next to the other file-type predicates, and
return ApplyKnowledgeProcessOverrides to its override-only scope so
resolveFileImportProcessConfig is the only import gate.

Also drop the unreachable branch in createKnowledgeFromFileURL: getFileType
never yields an empty string, so the resolvedFileType fallback and its else arm
could not run. Normalize the resolved type before it reaches the record and the
task payload, and add an invariant test asserting direct upload and URL import
accept the same extensions so #2447 cannot drift back.
2026-07-31 13:46:56 +08:00
wizardchen 43afa4c1f8 fix(knowledge): align URL file import validation with direct upload
Extract shared file-import checks (type, VLM/ASR, video) into
resolveKnowledgeFileImportConfig and reuse them for file URL import.
Also enqueue DataTableSummaryTask for spreadsheet URL imports and reparse.
2026-07-31 13:46:56 +08:00
wizardchen 48756d13d6 fix(knowledge): allow xlsx and other file types in URL import (#2447)
Align URL-based file import validation with direct file upload by
delegating to isValidFileType instead of a narrower whitelist.
2026-07-31 13:46:56 +08:00
wizardchen bf68db4653 fix(storage): allow image uploads to use file service fallback 2026-07-29 11:36:24 +08:00
koishi514-Z 18b95ebb3a fix(knowledge): distinguish duplicate files by type 2026-07-29 10:36:17 +08:00
wizardchen a3a0308b62 feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
wizardchen f8b3e5f148 feat(storage): add multi-instance storage backends per workspace
Introduce tenant-scoped storage backend registry so workspaces can register
multiple storage instances and bind each knowledge base to a specific backend,
with migration from legacy tenant-level storage config and new settings UI.
2026-07-15 14:54:39 +08:00
wizardchen d496d0fc73 feat(task-queue): split upstream workers into isolated core/enrichment/maintenance pools
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.
2026-07-10 23:51:29 +08:00
wizardchen 2f29054110 fix(knowledge): harden multi-tag data integrity and UX follow-ups
Clear tag relations on move/delete, validate tags on create, batch-load tags,
restore KB-scoped reference counts, dedupe SetKnowledgeTags, fix filter
double-fetch and ResizeObserver leaks, and regenerate Swagger for tag_ids.
2026-06-26 10:52:52 +08:00
Anky c007a897cf
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge

- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)

* feat(knowledge): tag edit dialog with dynamic overflow and inline creation

- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder

* fix: exclude soft-deleted knowledge from tag reference counts

CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.

* fix: refresh tag counts after async knowledge deletion completes

Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.

---------

Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-26 00:18:48 +08:00
wizardchen ed9e780c8c feat(client): enhance knowledge file upload with process configuration support
- Updated the `CreateKnowledgeFromFile` function to accept an optional `processConfig` parameter for per-upload parse configuration overrides.
- Introduced the `KnowledgeProcessOverrides` type to define various parsing settings, including parser engine rules and chunking configurations.
- Modified the API client and related components to handle the new process configuration, improving flexibility in knowledge file uploads.
- Updated documentation and API specifications to reflect the new `process_config` parameter in the upload process.
- Added a new `UploadConfirmHost` component to facilitate user confirmation for uploads, integrating with the updated upload logic.
- Enhanced user experience by providing detailed configuration options during the upload process.
2026-06-09 20:38:24 +08:00
wolfkill 2a54bb2d45 fix: avoid stale knowledge records on upload failure 2026-05-29 16:49:07 +08:00
HelloWeit edfabf3135 bugfix 2026-05-28 20:22:55 +08:00
wizardchen 31471f5efa refactor(knowledgeService): update checkStorageEngineConfigured method for improved clarity and functionality
- Refactored checkStorageEngineConfigured to be a method of knowledgeService, enhancing encapsulation and readability.
- Updated logic to allow fallback to global file service when no storage provider is configured at the KB or tenant level, improving error handling.
- Added detailed comments to clarify the method's behavior and internal logic, ensuring better understanding for future maintenance.
2026-05-25 19:16:08 +08:00
wizardchen d4c30126a9 refactor(knowledge): split 9.8k-line service file by responsibility
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.

Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.

  knowledge.go              497  struct + constructor + errs + CRUD reads + Search
  knowledge_create.go      1144  CreateFrom{File,URL,Passage,Manual} + helpers
  knowledge_delete.go       686  Delete + wiki cleanup + ProcessKnowledgeListDelete
  knowledge_process.go     2407  ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
  knowledge_clone_move.go  1049  Clone* + Move* + progress persistence
  knowledge_faq.go         1932  FAQ CRUD + Search + Export
  knowledge_faq_import.go  1939  FAQ import/validate/index + runningFAQImportInfo
  knowledge_util.go         352  file-type/url/hash + VLM+Storage config + resolveFileService

Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
2026-05-06 17:40:10 +08:00
lyingbug 0d8540c93c Revert "refactor(knowledge): split 9.8k-line service file by responsibility"
This reverts commit f67d0b53ce.
2026-05-06 17:09:24 +08:00
wizardchen f67d0b53ce refactor(knowledge): split 9.8k-line service file by responsibility
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.

Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.

  knowledge.go              497  struct + constructor + errs + CRUD reads + Search
  knowledge_create.go      1144  CreateFrom{File,URL,Passage,Manual} + helpers
  knowledge_delete.go       686  Delete + wiki cleanup + ProcessKnowledgeListDelete
  knowledge_process.go     2407  ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
  knowledge_clone_move.go  1049  Clone* + Move* + progress persistence
  knowledge_faq.go         1932  FAQ CRUD + Search + Export
  knowledge_faq_import.go  1939  FAQ import/validate/index + runningFAQImportInfo
  knowledge_util.go         352  file-type/url/hash + VLM+Storage config + resolveFileService

Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
2026-05-06 17:06:05 +08:00