* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer
The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.
Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.
* fix(ocr): keep merged-cell HTML tables chunkable
- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
pad the block with blank lines so the chunker can split by "\n"
instead of force-cutting at the absolute max size.
* fix(knowledge): normalize inline HTML tables for all parser engines before chunking
Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.
Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
markdown
Converters keep their existing calls (idempotent no-op).
* fix(chunker): bound Markdown link/image protected patterns to one line
The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).
Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.
Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.
* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths
Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.
* style: wrap long lines and avoid shadowing max in table-chunk tests
Pre-push lll/revive failed on the PR diff versus origin/main.
---------
Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Centralize ingestion base splitter defaults in chunker.NormalizeSplitterConfig
so the preview endpoint applies the same normalization as upload. Regenerate
Swagger for parent-child preview fields and add regression tests for single-
level preview, default parent/child sizes, and context headers.
Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.
Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
not accept a context.Context, so when previewTimeout fires the
worker keeps running. Bounding input size keeps worst-case CPU
per request well under a second on commodity hardware. Sized so
10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
follow-up: real cancellation needs the splitter to take a ctx.
Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
reuse a single rune-count per chunk for stats + size + token
estimation. Eliminates the previous triple []rune allocation per
chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
instead of []chunker.Chunk; sumSq computed in float64 to avoid
the int*int overflow at l > ~46k.
Correctness / UX
- Preview panel sends strategy / token_limit / languages
unconditionally, mirroring the buildSubmitData convention so the
preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
gin's cryptic 'Field validation failed on the required tag'.
Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
made the original name a misnomer; the test still asserts the
right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
cases (empty / single / varying / no-variance underflow) plus
PreviewChunking httptest scenarios (auto path, legacy strategy,
empty-text rejection, oversize rejection, chunk truncation with
full-set stats).
Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
obsolete since the heading and heuristic splitters shipped.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves the issues surfaced by the post-Phase-3 self-review.
High severity
- Strategy / TokenLimit / Languages can now be reset. The KB-config
handler uses pointer DTOs so the absence of a field in the payload
means "no change" while an explicit empty string / 0 / [] clears
the value. The frontend now sends the fields unconditionally so
the user can revert to defaults via the clear icon — previously
these fields were write-once.
- DocProfile is no longer computed twice in the preview endpoint.
Diagnostics carries the profile that drove tier selection (when
auto-strategy ran); the handler only re-profiles for explicit
strategies that bypass the profiler.
- Chunk-size stats (avg / min / max / stddev) are now computed over
the FULL chunk set before the response is truncated to
previewMaxChunks. Avoids systematically misleading metrics for
large documents.
- SplitWithDiagnostics defaults SelectedTier to TierLegacy so an
empty-text request never leaves the diag with a blank tier string.
Medium severity
- Split() reverts to its own loop instead of delegating to
SplitWithDiagnostics. Diagnostics struct allocation is reserved
for callers that actually want the trace (preview endpoint).
Matters in SplitParentChild where Split runs once per parent.
- Preview endpoint runs the splitter on a goroutine wrapped in a
5s context.WithTimeout. CJK input at the 256k-rune ceiling
could otherwise tie up the worker for several seconds.
Low severity
- KBChunkingSettings emits arrays via spread copies so parent-side
mutation cannot drift the form's reactive state.
- KBChunkingDebug font-family includes a monospace fallback so the
chunk content stays monospaced even when the TDesign CSS variable
is missing.
Documentation
- DocProfile JSON shape is now annotated as part of the public
preview API; field renames would be breaking.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds POST /api/v1/chunker/preview — a read-only endpoint that runs the
adaptive chunker on supplied text and returns the chunks plus
diagnostic info (which tier won, which were rejected and why, the
DocProfile that the selector saw). Used by the upcoming KB editor
debug panel so users can experiment with chunking parameters without
re-uploading documents.
Backend pieces:
- chunker.SplitWithDiagnostics(text, cfg) returns chunks plus a
Diagnostics struct (SelectedTier, TierChain, Rejected). Split() now
delegates here.
- chunker.DocProfile gets JSON tags so it serializes cleanly.
- handler.PreviewChunking caps input at 256k chars and chunks at 500
per response, computes a size-distribution summary, and never
touches DB or embedding APIs.
Auth: lives under the v1 group so the standard JWT middleware applies.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k