WeKnora/internal/infrastructure/docparser/html_table_normalizer.go

160 lines
5.5 KiB
Go
Raw Normal View History

package docparser
import (
"regexp"
"strings"
"github.com/JohannesKaufmann/html-to-markdown/v2/converter"
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/base"
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/commonmark"
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/table"
)
var (
// htmlTableBlockPattern matches a single (non-nested) <table>...</table>
// block, the form OCR/layout engines such as PaddleOCR-VL emit tables in.
htmlTableBlockPattern = regexp.MustCompile(`(?is)<table\b[^>]*>.*?</table>`)
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
// fencedCodeBlockPattern matches CommonMark fenced code blocks so HTML
// <table> examples inside them are left untouched.
fencedCodeBlockPattern = regexp.MustCompile("(?s)(?:```|~~~)[^\\n]*\\n.*?(?:```|~~~)")
// htmlLayoutAttrPattern matches presentational HTML attributes that carry
// no semantic value (text-align styles, CSS classes, sizing). Structural
// attributes like rowspan/colspan are intentionally excluded.
htmlLayoutAttrPattern = regexp.MustCompile(
`(?is)\s+(?:style|class|align|valign|width|height|bgcolor)\s*=\s*(?:"[^"]*"|'[^']*'|[^\s>]+)`,
)
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
// htmlSpanAttrPattern detects rowspan/colspan values greater than 1, which
// Markdown tables cannot represent; such tables keep their HTML form
// (attributes stripped) instead. Span values of 1 (and the invalid 0) do
// not merge anything and must stay convertible.
htmlSpanAttrPattern = regexp.MustCompile(`(?i)\b(?:row|col)span\s*=\s*["']?(?:[2-9]|\d{2,})`)
// htmlTableRowPattern matches the opening <tr> tag of each table row, used
// to put every row on its own line so the chunker can split degraded HTML
// tables at "\n" boundaries.
htmlTableRowPattern = regexp.MustCompile(`(?i)<tr\b`)
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
// markdownTableSeparatorPattern matches the |---| delimiter row that a
// valid GFM table must contain. The repeating column group is optional so
// single-column tables (|---|) are accepted; requiring two or more columns
// discarded successful conversions of MinerU figure/TOC tables.
markdownTableSeparatorPattern = regexp.MustCompile(`(?m)^\s*\|?\s*:?-+:?\s*(?:\|\s*:?-+:?\s*)*\|?\s*$`)
)
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
// NormalizeHTMLTables rewrites inline HTML <table> blocks embedded in OCR
// markdown output. PaddleOCR-VL emits tables as HTML with per-cell text-align
// styles, which (1) waste tokens on layout markup and (2) are not recognized
// by the chunker's table-protection logic, so large tables get split mid-row.
//
// Each table block is converted to a GFM Markdown table when possible. Tables
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
// that use rowspan/colspan (which Markdown cannot express), or that the
// converter cannot turn into a valid GFM table (cell <br>, nested lists, …),
// fall back to presentational-attribute stripping plus one row per line so
// the chunker can still split on "\n". Tables inside fenced code blocks are
// left unchanged.
func NormalizeHTMLTables(md string) string {
if !strings.Contains(strings.ToLower(md), "<table") {
return md
}
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
locs := htmlTableBlockPattern.FindAllStringIndex(md, -1)
if len(locs) == 0 {
return md
}
conv := converter.NewConverter(
converter.WithPlugins(
base.NewBasePlugin(),
commonmark.NewCommonmarkPlugin(),
table.NewTablePlugin(),
),
)
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
fences := fencedCodeBlockPattern.FindAllStringIndex(md, -1)
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
var b strings.Builder
last := 0
for _, loc := range locs {
b.WriteString(md[last:loc[0]])
block := md[loc[0]:loc[1]]
if indexInsideSpans(loc[0], fences) {
b.WriteString(block)
last = loc[1]
continue
}
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
// Isolate the rewritten table with blank lines without stacking pads
// on repeated NormalizeHTMLTables calls.
out := strings.TrimRight(b.String(), "\n")
b.Reset()
b.WriteString(out)
b.WriteString("\n\n")
b.WriteString(normalizeOneHTMLTable(conv, block))
last = loc[1]
for last < len(md) && md[last] == '\n' {
last++
}
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
b.WriteString("\n\n")
}
b.WriteString(md[last:])
return b.String()
}
func normalizeOneHTMLTable(conv *converter.Converter, block string) string {
fallback := func() string {
return splitHTMLTableRows(stripHTMLLayoutAttrs(block))
}
if htmlSpanAttrPattern.MatchString(block) {
return fallback()
}
converted, err := conv.ConvertString(block)
if err != nil {
return fallback()
}
converted = unescapeMarkdownImageSyntax(strings.TrimSpace(converted))
if converted == "" || !markdownTableSeparatorPattern.MatchString(converted) {
return fallback()
}
return converted
}
func indexInsideSpans(pos int, spans [][]int) bool {
for _, s := range spans {
if pos >= s[0] && pos < s[1] {
return true
}
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
}
return false
}
// stripHTMLLayoutAttrs removes presentational attributes from an HTML fragment
// while preserving structural attributes (rowspan/colspan) and text content.
func stripHTMLLayoutAttrs(html string) string {
return htmlLayoutAttrPattern.ReplaceAllString(html, "")
}
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) * fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML does not match markdown text with an embedded table, so ocrHTMLToMarkdown (which also lacked the table plugin) never ran. The chunker then split the raw markup mid-row, tripping the 7500-char hard limit and invalidating the heading layer for the whole document. Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead: export it and call it unconditionally from sanitizeOCRText, since it is a safe no-op when no <table> is present. * fix(ocr): keep merged-cell HTML tables chunkable - Narrow span detection to values >1 so rowspan=1/colspan=1 tables still convert to GFM instead of falling back to raw HTML. - On the merged-cell fallback, split each <tr> onto its own line and pad the block with blank lines so the chunker can split by "\n" instead of force-cutting at the absolute max size. * fix(knowledge): normalize inline HTML tables for all parser engines before chunking Previously NormalizeHTMLTables was only wired into the paddleocr-vl converters, so tables emitted by other engines (notably MinerU) reached the chunker as raw single-line <table> blocks and were force-cut at the 7500 char hard limit. Add the normalization at the two central pre-chunk boundaries instead: - knowledge_process.go: after all converter/ASR/image markdown rewrites, before the chunker step (covers every parser engine) - knowledge_create.go: before building the splitter config for manual markdown Converters keep their existing calls (idempotent no-op). * fix(chunker): bound Markdown link/image protected patterns to one line The unbounded [^\]]* / [^)]+ character classes in the Markdown link and image protectedPatterns also matched newlines, so a stray '[' left behind by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with a distant '](' and swallow whole paragraphs as one protected atomic span. buildUnitsWithProtection then treated the region as a single unit and mergeUnits could not split it, defeating chunking (observed 1 chunk of 1753 runes where 16 chunks were expected). Exclude newlines and bound the link text (<=200) / destination (<=500), matching CommonMark's rule that neither may span a blank line. Regression: protected_spans_test.go — same input now yields multiple chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune chunk; well-formed single-line links/images stay protected. * fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths Conversion fallbacks for <br>, nested lists, and one-column tables still left single-line HTML, so MinerU output could hit the same oversized-chunk bug. Split those rows, accept one-column GFM, skip fenced examples, and normalize before image resolution so cell images are not escaped. * style: wrap long lines and avoid shadowing max in table-chunk tests Pre-push lll/revive failed on the PR diff versus origin/main. --------- Co-authored-by: neoczr <neoczr@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 13:10:17 +00:00
// splitHTMLTableRows puts each table row on its own line. The chunker splits
// on "\n", so a single-line HTML table would otherwise be unsplittable and get
// force-cut at the absolute max size. Whitespace between tags is insignificant
// in HTML, and newlines are only inserted before <tr> (never inside a tag).
// A <tr> that already starts on its own line is left alone.
func splitHTMLTableRows(block string) string {
locs := htmlTableRowPattern.FindAllStringIndex(block, -1)
if len(locs) == 0 {
return strings.TrimSpace(block)
}
var b strings.Builder
last := 0
for _, loc := range locs {
b.WriteString(block[last:loc[0]])
if loc[0] == 0 || block[loc[0]-1] != '\n' {
b.WriteByte('\n')
}
b.WriteString(block[loc[0]:loc[1]])
last = loc[1]
}
b.WriteString(block[last:])
return strings.TrimSpace(b.String())
}