* ci(build): add manual docker build workflow
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects
- folder mirror: wiki/drive connectors emit path-qualified FileName so
knowledge lands in KB folders mirroring the Feishu tree; shortcut
subtrees deduped; sanitized directory names; legacy datasources
converge via one-shot resync marker (incremental sync upgrades to
full once, then clears)
- parse mode: replace FEISHU_DOCX_PARSE_MODE env with per-datasource
settings.parse_mode (blocks default, export escape hatch); frontend
selector + 5 locales; zero-migration read-time fallback
- block-level GFM: extend docx block coverage (callout/grid/iframe/
board/mindnote/agenda/synced blocks), rich text elements (links,
inline code, KaTeX equations, mentions), real ordered-list numbering,
code language mapping, merged-cell fill, truncation note
- embedded objects: images (gated off multimodal toggle) and boards
(download_as_image, 403-degraded) stored as knowledge with
weknora-img://<n> markers resolved in the document-processing
pipeline via knowledge FilePath; document attachments whitelisted
(pdf/md/xls(x)/csv/doc(x)/ppt(x)) as standalone knowledge in the
parent folder with bidirectional parent/child metadata and filename
list in the body; videos skipped with placeholder + source link
- golden baseline explicitly re-recorded; docs (.env.example,
docker-compose, docs/*, website-docs) synced
* fix(datasource): address Feishu deep-adaptation smoke-test findings
- folder mapping: drop the space/root-folder name prefix — the selected
sync root maps to the knowledge-base root, subdirectories stay relative
- docx blocks: render view(33) blocks silently as containers (children
only), removing the stray '[飞书块: 视图]' placeholder before file lists
- sheet inline tables: fetch merge ranges via sheets/v3 sheets/query and
fill the top-left value across the merged region (closed-interval
interpretation, flagged for real-environment verification)
- knowledge list: hide embedded image/board rows (metadata
embedded_image=true) from the document list and folder-tree counts;
rows remain for VLM indexing and marker resolution; search untouched
- frontend: hydrate protected images (resource:// etc.) in the document
preview so embedded Feishu images/boards actually render
* fix(datasource): resolve embedded image markers at ingest time
The weknora-img://<n> markers were only resolved during document
processing, so chunks carried real resource:// URLs while the stored
markdown file and source_content kept raw markers — file preview
showed blank images. Resolve them in ingestItem before
CreateKnowledgeFromFile (children are emitted first, so image rows
with FilePath already exist); markers with missing children are kept
for the ProcessDocument fallback to degrade. Stored file,
source_content and chunks now agree. Existing datasources converge on
the next full re-sync (update=delete+recreate).
* feat(datasource): resolve Feishu mention users to real names at ingest
docx mention_user elements only carry an open_id, so they rendered as
a generic '@成员'. Resolve the display name via
GET /contact/v3/users/:id (user_id_type=open_id) while generating the
markdown during FetchDocxWithBlocks — before storage, per user
decision; the frontend performs no lookups. A per-sync closure caches
positive and negative lookups; permission errors and empty names
silently degrade to '@成员' (debug log only, ingest never blocked).
Requires one of the contact read scopes (e.g.
contact:user.base:readonly) on the Feishu app.
* refactor(datasource): polish round — defect fixes and structural cleanup
- attachment placeholder patching now targets the unique
'- weknora-att://<token>' marker line rendered per attachment instead of
matching by display name, so an identical earlier bullet is never
replaced by mistake (regression-tested)
- markdown link targets with spaces or parens are percent-escaped
(iframe / mention_doc / inline links), keeping them parseable
- drive folder-tree walking consolidated into core.WalkDriveTree,
deleting the 174-line copy in drive/paths.go
- parse mode threading made explicit: DocxFetchInput.ParseMode replaces
the private client field (connector entry resolves via
ParseFeishuConfig and passes it through ops)
- maxFeishuDownloadBytes restored to a const; the oversize-degrade test
now declares an over-cap Content-Length and downloadRawBytes rejects
early on the header
* refactor(datasource): align Feishu image handling with the platform image pipeline
Replace the image-as-separate-knowledge-row channel with the platform's
orthodox flow: the connector downloads image/board bytes at fetch time and
embeds them in the parent markdown as base64 data URIs; the document
processing image resolver (image_resolver.ResolveAndStore, the same stage
used for parsed PDFs/Word) stores them on the tenant file service, swaps in
persistent provider:// URLs and produces OCR/caption child chunks when a VLM
is configured.
Removed with the marker pipeline:
- weknora-img:// ingest resolution in ingestItem
- the ProcessDocument Step-1.9 marker hook
- the document-list/folder-tree filter for embedded_image rows
- the feishu-specific runtime parse_mode backfill; legacy data sources are
now stamped by migration 000096 (sqlite 000017) which writes the one-shot
resync_required marker — the generic full-upgrade + clear consumption stays
Attachment (PDF/Word/md) sub-items are unchanged.
* style: gofmt test files
* fix(datasource): resolve Drive sub-selection kind from the root listing
A single-file Drive selection (rootFolderToken:fileToken) was probed with
folder APIs before falling back: folder meta returned 91202 and the folder
list 1061002 on every sync, logging API errors for a working configuration.
listDriveFilesForResource now walks the root subtree first and reads the
selected node's kind off the listing: a file filters straight out of the
already-fetched results (zero error calls), a sub-folder still gets its
dedicated walk, and only a token missing from the listing (partial listing /
permission edge) keeps the legacy probe + fallback.
Also pins sub-folder selection semantics with a dedicated test.
* fix(datasource): address review findings across the Feishu deep-adaptation branch
Four parallel reviewer slices audited the branch; this applies the accepted
findings.
Content fidelity (blocks rendering):
- todo blocks parse style.done — completed items render "- [x]" instead of
always unchecked
- code fences size past the longest backtick run in the content, so a line
of ``` inside code can no longer close the fence early
- ordered-list "auto" numbering only resets on a sibling of the last item;
child blocks rendered flat between items no longer renumber the run
Image pipeline:
- inline base64 images are capped at the document-processing resolver's
10 MB budget; larger images degrade to a placeholder instead of leaving a
permanent raw-base64 blob in stored markdown
- image markers carry a per-document nonce so user-typed look-alike text
can never be patched in place of a real marker
- attachment marker patching anchors on the token's line end, closing the
token-prefix collision
- SupportedImageExt recovers image/svg+xml (board exports), which the
platform resolver already stores natively
Sync correctness:
- clearResyncMarker re-reads the row before writing: a long upgraded sync
no longer risks reverting concurrent user edits with a stale full-row
update
- wiki EditTime keys on max(obj_edit_time, node_edit_time): pure
moves/renames re-file the document on the next incremental sync
- file-typed wiki/drive base names are sanitised; a "/" in a title can no
longer fake an extra folder level
Docs/tests/comments:
- datasources docs rewritten to the v3 image contract (no image rows,
no image_map, no parse_mode backfill claim)
- DataSourceEditorDialog drops settings.parse_mode when backtracking to a
non-Feishu connector type
- stale comments corrected (types.go parse-mode transport, repository
folder-counts, wiki List partial-error branch, test headers); regression
tests added for todo done, fence sizing, ordered numbering, svg sniff
* fix(datasource): repair markdown conversion defects found in live smoke test
Live re-sync of the all-block-type Feishu test document surfaced three
converter defects:
- Code blocks: the closing fence reused the opening fence string, so a
tagged block ended with ```java — CommonMark requires a bare closing
fence, and every renderer left the block unterminated. The closing
fence is now backticks only.
- Feishu heading7-9 emitted 7-9 hash marks; GFM stops at H6, so those
lines rendered as literal '#######' text. They now degrade to bold.
- The docx API percent-encodes link targets (http%3A%2F%2F...); escapeURL
now reverses that encoding before markdown-escaping, so stored links
point at real URLs instead of percent-encoded strings.
Verified against the live document via a throwaway API probe: the
'i|g|k' table cells questioned during review are the document's own
content (the sheets copy says i|j|k; the two docx tables genuinely
contain 'g'), not a converter bug.
* fix(frontend): render GFM task lists in document preview
Task list items (- [x] / - [ ]) rendered as plain bullets: marked emits
<input type="checkbox"> for them, but the preview DOMPurify config's
ALLOWED_TAGS allowlist has no 'input', so the element was stripped whole.
The preview renderer now overrides checkbox to emit a styled
task-checkbox span (class/attr both allowlisted) and the preview styles
draw the box with a check mark, hiding the list bullet via :has().
Same gap exists in the chat markdown surface; left untouched there.
* fix(datasource): crop blank canvas borders from whiteboard exports
The board download_as_image API renders the WHOLE board canvas, not the
content bounding box — editors routinely leave the canvas far larger than
the drawing, so board/mindnote/diagram exports surfaced with large blank
areas below the content. The API has no crop parameter, so the connector
now trims near-uniform blank borders after download (bottom-right pixel
seeds the background color, 14/255 per-channel tolerance absorbs JPEG
noise, 8px margin kept). PNG and JPEG are re-encoded; anything
undecodable, fully blank, or already tight passes through unchanged, and
oversized images (>64MP) skip the scan to bound memory.
* fix(datasource): require corner majority before trimming board exports
The blank-border trim seeded the background color from the bottom-right
pixel alone. A large solid shape reaching that corner would seed the
background with its own fill color, and fully-filled rows could then be
trimmed as blank — cutting real content. The background color is now the
majority of the four corner pixels (>=3 must agree); on a split vote
content reaches the corners and the image passes through untrimmed.
Content visible against the background is untouched either way: a row or
column survives if even one pixel differs from the background by more
than the 14/255 JPEG-noise tolerance.
* fix(datasource): widen board trim margin and lower JPEG quality to 80
The 8px margin read as a tight crop — a 32px white frame now visually
bounds the content region. JPEG re-encode quality drops 90 -> 80 to keep
the re-encoded size closer to the original export.
* feat(datasource): timeline widget Markdown degradation + table column alignment
Two content-fidelity gaps from the live smoke test:
- 文档小组件 (add_ons, block_type 40) carry their full data inline in
record; there is no server-side render to download. The timeline
component (component_type_id blk_6358a421bca0001c...) now degrades to
a Markdown list (- **time · title**:text), honoring contentShow and
omitting empty segments; unknown component types and malformed records
keep the previous placeholder note.
- docx text blocks carry style.align (1 left / 2 center / 3 right,
absent unless the author set it). Native-table rendering now collects
per-column alignment from the first explicit cell and emits GFM colon
separators (:---: / ---:). Embedded sheets/bitable keep the plain
separator — their value APIs expose no alignment.
* feat(datasource): show document titles for 云文档 mention links
The docx renderer emitted mention_doc references as [url](url) because
the payload carries no title. Feishu doc URLs now emit a
[weknora-docname://token](url) placeholder and FetchDocxWithBlocks
resolves the titles in one drive metadata batch call
(POST /drive/v1/metas/batch_query, docx/doc/sheet/bitable/wiki/...),
patching the link text to [文档名](url). Refs whose metadata cannot be
fetched — no permission (970003), deleted/mismatched (970005), API
absent — degrade to the previous URL-as-text form with a warn log.
This is display-only: referenced docs are not pulled as sub-documents
(deferred design lives in the task PRD).
* docs(datasource): document mention rendering enhancement and drive rename limitation
- Add optional mention scopes (contact:user.base:readonly,
drive:drive.metadata:readonly, board:whiteboard:node:read) with
degrade behavior to the Feishu permission tables
- Describe mention_user real-name lookup, mention_doc title backfill
and timeline widget placeholders in the blocks rendering flow
- Document the Drive known limitation: folder renames / file moves do
not bump modified_time, so incremental sync keeps the old KB folder
structure until a full sync; Wiki sources are unaffected
- Clarify the directory-mapping wording in website-docs to distinguish
Wiki (rename-aware) from Drive (rename-blind) sources
* docs(datasource): add blocks vs export tradeoff summary to website-docs
* docs(datasource): document docx block-type coverage for blocks mode
- Add a full block-type support table (block_type 1-52 grouped) to the
datasource dev doc: text/heading/list/task/quote structures, code
fences with language aliases, containers (callout/quote/grid/view),
native tables with merged-cell splitting (top-left value fill) and
column alignment, embedded sheet/bitable converted to in-document
Markdown tables, image/board inline pipeline, attachment whitelist,
iframe links, timeline add-on degradation and the placeholder set
- Summarize the same coverage in the drive user guide, the wiki
quick-start doc and website-docs; fix the stale claim that board
blocks degrade to placeholders (they ride the image pipeline now)
* style: resolve diff-scoped lint findings across feishu packages
- Wrap over-length signatures and calls (lll, max 120)
- Add doc comments to exported Block* payload types (revive)
- Rename unused closure parameters to _ (revive)
- Check w.Write / resp.Body.Close error returns (errcheck)
- Apply De Morgan rewrite in docname token validation (staticcheck)
- gofumpt-format touched files
golangci-lint run --new-from-rev=main ./... now reports 0 issues.
* ci: drop fork-only docker-build workflow
The manual docker-build workflow lives on the fork's main and is not
part of the upstream contribution; it entered the branch through the
fork-main merge.
* fix(migrations): renumber feishu resync migrations to avoid version collision with upstream
Upstream added 000017_dingtalk_stream_only (sqlite) and 000096_dingtalk_stream_only
(versioned) before our merge, colliding with our 000017/000096 feishu resync
migrations. golang-migrate rejects duplicate versions. Renumber to the next free
slots (sqlite 000027, versioned 000108) and update comments + test constant.
* fix(migrations): renumber feishu resync migrations ahead of upstream 000027/000108
Upstream main now also claims sqlite 000027 and versioned 000108
(session_sandbox_config_tenant), colliding with our just-renumbered
resync migrations. Move to 000028 / 000109.
* fix(feishu): address review — deprecated env fallback and upgrade notes
- Keep FEISHU_DOCX_PARSE_MODE as a one-release deprecated fallback when a
data source has no per-source parse_mode (deprecation warning logged);
annotate it in .env.example / docker-compose.yml instead of removing.
- Point stale migration-number comments at the final numbers
(versioned 000110 / sqlite 000030) in datasource_service.go and tests.
- website-docs: add Feishu upgrade notes (env deprecation, default mode
flip to blocks, one-shot full resync with embedding/VLM recompute);
refresh 24-feishu-drive.md parse-mode section to the per-source
setting and new blocks semantics.
* docs(feishu): document necessary and optional connector permissions
Restore the permission matrix lost with the docs/ removal: necessary
scopes (wiki/drive/export/docx) with missing-permission symptoms and
optional enhancement scopes (embedded sheets/bitable, board export,
mention user/doc rendering) with their degrade behavior, in
10-datasource.md's Feishu section; make 24-feishu-drive.md list the
three necessary drive scopes and link to the full table.
* Revert "docs(feishu): document necessary and optional connector permissions"
This reverts commit 3be19cb1b656b3596f2cb6afb2cda8c2a1d12be1.
* docs(feishu): permission matrix by connector, parse mode and necessity
Necessary-scope tables split by connector (feishu/lark wiki vs
feishu_drive/lark_drive) with blocks/export applicability columns and
missing-permission symptoms; optional enhancement scopes with their
degrade behavior; scope names cross-checked against the Feishu open
platform API docs (files/folder list, medias download, export tasks,
metas batch_query). Full matrix lives in 10-datasource.md; the drive
guide mirrors its own table and links back.
* test(feishu): adapt mention_user_test to upstream retryTestServer signature
* feat(feishu): mirror the sync root (wiki space / drive folder) as a top-level KB folder
So a knowledge base linking several feishu sources can tell them apart at a
glance: syncing folder 测试1 yields 测试1/... and syncing wiki space
飞书知识库同步测试 yields 飞书知识库同步测试/.... Root-name resolution
failure falls back to the previous root-less behaviour without failing the
sync; existing sources converge through the one-shot resync marker.
* refactor(feishu): cache wiki space folder prefix per run, refresh stale comments
Review follow-ups for d012e593f: cache the sanitized space prefix per spaceID
for the whole sync run (K selected spaces now cost one ListWikiSpaces paging
pass instead of K), fix the misleading warn wording, and realign the
WalkDriveTree / driveFolderName doc comments with root-mirror semantics.
* docs(feishu): drop stale and duplicated comments from the deep-adaptation
- board.go: trimBoardImage no longer claims a bottom-right seed (the code
votes the background over the four corners)
- shared.go: rewrite the FetchDocxWithBlocks note (images/boards inline via
base64 now; removes a stale-marker splice artefact)
- markdown.go: drop an internal task-number leftover
- datasource_service.go: collapse the ProcessSync/processSyncStreaming notes
duplicated verbatim from resyncRequired's doc comment, keep the call-site
why (marker cleared only on success)
- datasource_resync_test.go: drop the obsolete 'backfill' wording
* feat(i18n): 添加解析模式及其提示信息到繁體中文語言包
* fix(frontend): use radius token for task checkbox; fix simplified chars in zh-TW parse mode
CI guard fixes: the PR-added task-checkbox style pushed the radius-literal
count to 161 (> baseline 160); the new zh-TW keys missed the locale parity
audit (user's dc6fdcff9) — also normalise the simplified 文档 leftovers.
* fix(feishu): preserve sync scope, nested content, and legacy settings
Cover shortcut selection, ancestor path changes, nested containers and table boards with regression tests. Avoid redundant Drive traversal, repair resync migrations for legacy settings, and preserve inherited parse modes. Bound board image dimensions before decoding.
* docs(feishu): clarify shortcut scope and incremental path tracking
* Fix Feishu sync reconciliation and embedded media regressions
* Bound Feishu table rendering by actual cell data
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>