feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
"""Source locators: map ranges of parsed markdown back to the original file.
|
|
|
|
|
|
|
|
|
|
|
|
A source block is ``{"start": int, "end": int, "locator": {...}}`` where
|
|
|
|
|
|
``start``/``end`` are code point offsets into the document content (end
|
|
|
|
|
|
exclusive) and the locator says where that text sits in the original file.
|
|
|
|
|
|
For PDFs the locator is ``{"type": "pdf", "page": n}`` plus an optional
|
|
|
|
|
|
``"bbox": [x0, y0, x1, y1]`` given as fractions of the displayed page with
|
|
|
|
|
|
the origin at its top-left corner, so viewers can draw it at any zoom.
|
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
|
|
|
|
|
|
|
import re
|
2026-09-27 12:32:26 +00:00
|
|
|
|
from collections import Counter
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
import statistics
|
|
|
|
|
|
import unicodedata
|
|
|
|
|
|
from typing import Iterable, List, Optional, Sequence, Tuple
|
|
|
|
|
|
|
|
|
|
|
|
# How far past the cursor a line is searched for, in normalized characters.
|
|
|
|
|
|
_SEARCH_WINDOW = 6000
|
|
|
|
|
|
# Lines shorter than this (normalized) are too ambiguous to place.
|
|
|
|
|
|
_MIN_KEY_CHARS = 2
|
|
|
|
|
|
# Lines shorter than this are placed only when their text is unique on the
|
|
|
|
|
|
# page: labels inside figures ("LLM", "EBAE") otherwise steal body offsets.
|
|
|
|
|
|
# Wide (CJK) characters count as four, since each carries about a word.
|
|
|
|
|
|
_SHORT_KEY_WEIGHT = 16
|
|
|
|
|
|
|
|
|
|
|
|
_LIST_START_RE = re.compile(
|
|
|
|
|
|
r"^\s*(?:"
|
|
|
|
|
|
r"\d{1,3}(?:\.\d{1,3}){0,4}[\.、.))]?\s" # 1 1. 8.14.11 2)
|
|
|
|
|
|
r"|[((]\d{1,3}[))]" # (1) (1)
|
|
|
|
|
|
r"|[一二三四五六七八九十百]+[、..]" # 一、
|
|
|
|
|
|
r"|[•·●○■□◆◇▪\-–*]\s" # bullets
|
|
|
|
|
|
r"|第[一二三四五六七八九十百零\d]+[章节条款部分篇]" # 第三章
|
|
|
|
|
|
r")"
|
|
|
|
|
|
)
|
|
|
|
|
|
_SENTENCE_END = tuple("。..!?!?::;;」』”))")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _fold(ch: str) -> str:
|
|
|
|
|
|
"""Letters and digits only, lower-cased, full-width folded."""
|
|
|
|
|
|
if not ch.isalnum():
|
|
|
|
|
|
return ""
|
|
|
|
|
|
return unicodedata.normalize("NFKC", ch).lower()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def normalize_with_positions(text: str) -> Tuple[str, List[int]]:
|
|
|
|
|
|
"""Project text onto letters and digits.
|
|
|
|
|
|
|
|
|
|
|
|
Returns the projection and, for each of its characters, the offset in
|
|
|
|
|
|
``text`` of the character it came from.
|
|
|
|
|
|
"""
|
|
|
|
|
|
out: List[str] = []
|
|
|
|
|
|
pos: List[int] = []
|
|
|
|
|
|
for i, ch in enumerate(text):
|
2026-09-27 12:32:26 +00:00
|
|
|
|
value = unicodedata.normalize("NFKC", ch)
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
folded = _fold(ch)
|
2026-09-27 12:32:26 +00:00
|
|
|
|
if value in ".,:/+−-%‰<>=≤≥≠":
|
|
|
|
|
|
before, after = i - 1, i + 1
|
|
|
|
|
|
while before >= 0 and text[before] in " \t":
|
|
|
|
|
|
before -= 1
|
|
|
|
|
|
while after < len(text) and text[after] in " \t":
|
|
|
|
|
|
after += 1
|
|
|
|
|
|
prev = before >= 0 and text[before].isdigit()
|
|
|
|
|
|
next_digit = after < len(text) and text[after].isdigit()
|
|
|
|
|
|
semantic = ((value in ".,/:" and prev and next_digit)
|
|
|
|
|
|
or (value in "+-−" and next_digit)
|
|
|
|
|
|
or (value in "%‰" and prev)
|
|
|
|
|
|
or value in "<>=≤≥≠")
|
|
|
|
|
|
if semantic:
|
|
|
|
|
|
folded = value.replace("−", "-")
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
for f in folded:
|
|
|
|
|
|
out.append(f)
|
|
|
|
|
|
pos.append(i)
|
|
|
|
|
|
return "".join(out), pos
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def normalize(text: str) -> str:
|
2026-09-27 12:32:26 +00:00
|
|
|
|
return normalize_with_positions(text)[0]
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _key_weight(key: str) -> int:
|
|
|
|
|
|
return sum(4 if unicodedata.east_asian_width(ch) in ("W", "F") else 1 for ch in key)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _column_cursor(boxes, idx: int, ends: List[Optional[int]]) -> Optional[int]:
|
|
|
|
|
|
"""End of the nearest placed line above ``boxes[idx]`` in the same column."""
|
|
|
|
|
|
x0, y0, x1, _ = boxes[idx]
|
|
|
|
|
|
for j in range(idx - 1, max(-1, idx - 80), -1):
|
|
|
|
|
|
if ends[j] is None:
|
|
|
|
|
|
continue
|
|
|
|
|
|
px0, py0, px1, _ = boxes[j]
|
|
|
|
|
|
if px0 < x1 and x0 < px1 and py0 > y0:
|
|
|
|
|
|
return ends[j]
|
|
|
|
|
|
return None
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-09-27 12:32:26 +00:00
|
|
|
|
def _numeric_safe_matches(text: str, key: str) -> List[int]:
|
|
|
|
|
|
out = []
|
|
|
|
|
|
if not key:
|
|
|
|
|
|
return out
|
|
|
|
|
|
start = 0
|
|
|
|
|
|
while (at := text.find(key, start)) >= 0:
|
|
|
|
|
|
before = text[at - 1] if at else ""
|
|
|
|
|
|
end = at + len(key)
|
|
|
|
|
|
after = text[end] if end < len(text) else ""
|
|
|
|
|
|
left = key[0].isdigit() and bool(before) and (before.isdigit() or before in ".,:/+−-")
|
|
|
|
|
|
right = key[-1].isdigit() and bool(after) and (after.isdigit() or after in ".,:/%‰")
|
|
|
|
|
|
if not left and not right:
|
|
|
|
|
|
out.append(at)
|
|
|
|
|
|
start = at + 1
|
|
|
|
|
|
return out
|
|
|
|
|
|
|
|
|
|
|
|
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
def locate_lines(
|
|
|
|
|
|
page_text: str,
|
|
|
|
|
|
line_texts: Sequence[str],
|
|
|
|
|
|
boxes: Optional[Sequence[Tuple[float, float, float, float]]] = None,
|
|
|
|
|
|
) -> List[Optional[int]]:
|
|
|
|
|
|
"""Find where each visual line starts in the page text.
|
|
|
|
|
|
|
|
|
|
|
|
Lines are searched in order after the previous hit so repeated phrases
|
|
|
|
|
|
resolve to the right occurrence. With ``boxes`` (PDF points, bottom-left
|
|
|
|
|
|
origin) the search first continues from the line above in the same
|
|
|
|
|
|
column, since glyph lines of side-by-side columns arrive interleaved. A
|
|
|
|
|
|
line not found after either cursor is searched from the start of the page,
|
|
|
|
|
|
which tolerates text extractors that order columns differently from the
|
|
|
|
|
|
glyph grouping. Returns one offset (or None) per line.
|
|
|
|
|
|
"""
|
|
|
|
|
|
norm, pos = normalize_with_positions(page_text)
|
|
|
|
|
|
hits: List[Optional[int]] = []
|
|
|
|
|
|
ends: List[Optional[int]] = []
|
|
|
|
|
|
cursor = 0
|
2026-09-27 12:32:26 +00:00
|
|
|
|
counts = Counter(normalize(line) for line in line_texts)
|
|
|
|
|
|
occurrences = {key: _numeric_safe_matches(norm, key) for key in counts}
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
for idx, line in enumerate(line_texts):
|
2026-09-27 12:32:26 +00:00
|
|
|
|
key = normalize(line)
|
|
|
|
|
|
if len(key) < _MIN_KEY_CHARS or len(occurrences[key]) != counts[key] or (
|
|
|
|
|
|
_key_weight(key) < _SHORT_KEY_WEIGHT and len(occurrences[key]) > 1
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
):
|
|
|
|
|
|
hits.append(None)
|
|
|
|
|
|
ends.append(None)
|
|
|
|
|
|
continue
|
|
|
|
|
|
cursors = [cursor]
|
|
|
|
|
|
if boxes is not None:
|
|
|
|
|
|
column = _column_cursor(boxes, idx, ends)
|
|
|
|
|
|
if column is not None:
|
|
|
|
|
|
cursors.insert(0, column)
|
|
|
|
|
|
at = -1
|
|
|
|
|
|
for start_at in cursors:
|
2026-09-27 12:32:26 +00:00
|
|
|
|
at = next((hit for hit in occurrences[key] if start_at <= hit <= start_at + _SEARCH_WINDOW), -1)
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
if at >= 0:
|
|
|
|
|
|
break
|
|
|
|
|
|
if at < 0:
|
2026-09-27 12:32:26 +00:00
|
|
|
|
at = occurrences[key][0] if len(occurrences[key]) == 1 else -1
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
if at < 0:
|
|
|
|
|
|
hits.append(None)
|
|
|
|
|
|
ends.append(None)
|
|
|
|
|
|
continue
|
|
|
|
|
|
else:
|
|
|
|
|
|
cursor = at + len(key)
|
|
|
|
|
|
ends.append(at + len(key))
|
|
|
|
|
|
start = pos[at]
|
|
|
|
|
|
# Pull in leading punctuation or markup on the same text line
|
|
|
|
|
|
# (brackets, "## ", list bullets) that normalization skipped.
|
|
|
|
|
|
while start > 0 and page_text[start - 1] != "\n" and not page_text[start - 1].isalnum():
|
|
|
|
|
|
start -= 1
|
|
|
|
|
|
hits.append(start)
|
|
|
|
|
|
return hits
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class PageGeometry:
|
|
|
|
|
|
"""Converts PDF user-space points to fractions of the displayed page."""
|
|
|
|
|
|
|
|
|
|
|
|
def __init__(self, cropbox: Tuple[float, float, float, float], rotation: int = 0):
|
|
|
|
|
|
left, bottom, right, top = cropbox
|
|
|
|
|
|
self.left, self.bottom = min(left, right), min(bottom, top)
|
|
|
|
|
|
self.width = abs(right - left) or 1.0
|
|
|
|
|
|
self.height = abs(top - bottom) or 1.0
|
|
|
|
|
|
self.rotation = int(rotation or 0) % 360
|
|
|
|
|
|
|
|
|
|
|
|
def bbox(self, box: Tuple[float, float, float, float]) -> List[float]:
|
|
|
|
|
|
x0, y0, x1, y1 = box
|
|
|
|
|
|
# Unrotated, top-left origin.
|
|
|
|
|
|
ux0 = (x0 - self.left) / self.width
|
|
|
|
|
|
ux1 = (x1 - self.left) / self.width
|
|
|
|
|
|
uy0 = 1.0 - (y1 - self.bottom) / self.height
|
|
|
|
|
|
uy1 = 1.0 - (y0 - self.bottom) / self.height
|
|
|
|
|
|
corners = [self._rotate(ux0, uy0), self._rotate(ux1, uy1)]
|
|
|
|
|
|
xs = [c[0] for c in corners]
|
|
|
|
|
|
ys = [c[1] for c in corners]
|
|
|
|
|
|
return [
|
|
|
|
|
|
round(_clamp(min(xs)), 4),
|
|
|
|
|
|
round(_clamp(min(ys)), 4),
|
|
|
|
|
|
round(_clamp(max(xs)), 4),
|
|
|
|
|
|
round(_clamp(max(ys)), 4),
|
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
def _rotate(self, x: float, y: float) -> Tuple[float, float]:
|
|
|
|
|
|
# Viewers display /Rotate clockwise.
|
|
|
|
|
|
if self.rotation == 90:
|
|
|
|
|
|
return 1.0 - y, x
|
|
|
|
|
|
if self.rotation == 180:
|
|
|
|
|
|
return 1.0 - x, 1.0 - y
|
|
|
|
|
|
if self.rotation == 270:
|
|
|
|
|
|
return y, 1.0 - x
|
|
|
|
|
|
return x, y
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _clamp(v: float) -> float:
|
|
|
|
|
|
return 0.0 if v < 0 else 1.0 if v > 1 else v
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _union(boxes: Iterable[Tuple[float, float, float, float]]):
|
|
|
|
|
|
boxes = list(boxes)
|
|
|
|
|
|
return (
|
|
|
|
|
|
min(b[0] for b in boxes),
|
|
|
|
|
|
min(b[1] for b in boxes),
|
|
|
|
|
|
max(b[2] for b in boxes),
|
|
|
|
|
|
max(b[3] for b in boxes),
|
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _line_gap(prev: dict, cur: dict) -> Optional[float]:
|
|
|
|
|
|
"""Vertical gap from ``prev`` down to ``cur`` when they share a column."""
|
|
|
|
|
|
px0, py0, px1, _ = prev["bbox"]
|
|
|
|
|
|
cx0, _, cx1, cy1 = cur["bbox"]
|
|
|
|
|
|
if cx1 < px0 or cx0 > px1 or cy1 > py0:
|
|
|
|
|
|
return None
|
|
|
|
|
|
return py0 - cy1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _starts_paragraph(
|
|
|
|
|
|
prev: dict, cur: dict, line_h: float, usual_gap: float, col_right: float
|
|
|
|
|
|
) -> bool:
|
|
|
|
|
|
"""Whether ``cur`` opens a new visual paragraph after ``prev``.
|
|
|
|
|
|
|
|
|
|
|
|
Boxes are PDF points with a bottom-left origin, so reading downwards means
|
|
|
|
|
|
decreasing y. ``usual_gap`` is the page's typical gap between consecutive
|
|
|
|
|
|
lines of a paragraph, which absorbs double-spaced layouts.
|
|
|
|
|
|
"""
|
|
|
|
|
|
px0, py0, px1, _ = prev["bbox"]
|
|
|
|
|
|
cx0, _, cx1, cy1 = cur["bbox"]
|
|
|
|
|
|
if cx1 < px0 or cx0 > px1:
|
|
|
|
|
|
return True # moved to another column
|
|
|
|
|
|
if cy1 > py0 + line_h * 0.5:
|
|
|
|
|
|
return True # went back up the page: a new column or region
|
|
|
|
|
|
if py0 - cy1 > usual_gap * 1.5 + line_h * 0.3:
|
|
|
|
|
|
return True # vertical gap wider than the usual line spacing
|
|
|
|
|
|
if _LIST_START_RE.match(cur["text"]):
|
|
|
|
|
|
return True
|
|
|
|
|
|
ends_sentence = prev["text"].rstrip().endswith(_SENTENCE_END)
|
|
|
|
|
|
short = col_right - px1 > line_h * 2
|
|
|
|
|
|
return ends_sentence and short
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def page_blocks(
|
|
|
|
|
|
page_text: str,
|
|
|
|
|
|
offset: int,
|
|
|
|
|
|
page_number: int,
|
|
|
|
|
|
lines: Sequence[dict],
|
|
|
|
|
|
geometry: Optional[PageGeometry],
|
|
|
|
|
|
) -> List[dict]:
|
|
|
|
|
|
"""Source blocks for one text page placed at ``offset`` in the document.
|
|
|
|
|
|
|
|
|
|
|
|
``lines`` are visual lines ``{"text", "bbox"}`` (bbox in PDF points). Lines
|
|
|
|
|
|
are placed in ``page_text``, grouped into paragraphs by geometry, and each
|
2026-09-27 12:32:26 +00:00
|
|
|
|
paragraph becomes a block spanning only its matched lines, boxed by their
|
|
|
|
|
|
union. Unmatched gaps keep page-only provenance and never inherit the
|
|
|
|
|
|
previous paragraph's geometry.
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
"""
|
|
|
|
|
|
end = offset + len(page_text)
|
2026-09-27 12:32:26 +00:00
|
|
|
|
page_only = {"type": "pdf", "page": page_number, "mapping": "exact"}
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
if not page_text:
|
|
|
|
|
|
return []
|
|
|
|
|
|
placed = []
|
|
|
|
|
|
if lines and geometry is not None:
|
|
|
|
|
|
starts = locate_lines(
|
|
|
|
|
|
page_text, [ln["text"] for ln in lines], [ln["bbox"] for ln in lines]
|
|
|
|
|
|
)
|
|
|
|
|
|
placed = sorted(
|
|
|
|
|
|
(
|
|
|
|
|
|
(at, ln)
|
|
|
|
|
|
for at, ln in zip(starts, lines)
|
|
|
|
|
|
if at is not None
|
|
|
|
|
|
),
|
|
|
|
|
|
key=lambda item: item[0],
|
|
|
|
|
|
)
|
|
|
|
|
|
if not placed:
|
|
|
|
|
|
return [{"start": offset, "end": end, "locator": dict(page_only)}]
|
|
|
|
|
|
|
|
|
|
|
|
heights = [ln["bbox"][3] - ln["bbox"][1] for _, ln in placed if ln["bbox"][3] > ln["bbox"][1]]
|
|
|
|
|
|
line_h = statistics.median(heights) if heights else 10.0
|
|
|
|
|
|
gaps = [
|
|
|
|
|
|
g
|
|
|
|
|
|
for g in (_line_gap(a[1], b[1]) for a, b in zip(placed, placed[1:]))
|
|
|
|
|
|
if g is not None and g >= 0
|
|
|
|
|
|
]
|
2026-09-27 12:32:26 +00:00
|
|
|
|
usual_gap = min(statistics.median(gaps), line_h * 1.5) if gaps else line_h * 0.5
|
|
|
|
|
|
|
|
|
|
|
|
norm, positions = normalize_with_positions(page_text)
|
|
|
|
|
|
def line_end(at, line):
|
|
|
|
|
|
key = normalize(line["text"])
|
|
|
|
|
|
norm_start = next((i for i, p in enumerate(positions) if p >= at), len(norm))
|
|
|
|
|
|
hit = norm.find(key, norm_start)
|
|
|
|
|
|
if hit < 0:
|
|
|
|
|
|
return at
|
|
|
|
|
|
end_at = positions[hit + len(key) - 1] + 1
|
|
|
|
|
|
while end_at < len(page_text) and not page_text[end_at].isalnum():
|
|
|
|
|
|
if page_text[end_at] == "\n":
|
|
|
|
|
|
return end_at + 1
|
|
|
|
|
|
end_at += 1
|
|
|
|
|
|
return end_at
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
|
|
|
|
|
|
paragraphs: List[Tuple[int, List[dict]]] = []
|
|
|
|
|
|
for at, ln in placed:
|
2026-09-27 12:32:26 +00:00
|
|
|
|
ln = {**ln, "_source_end": line_end(at, ln)}
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
if paragraphs:
|
|
|
|
|
|
prev = paragraphs[-1][1][-1]
|
|
|
|
|
|
col_right = max(p["bbox"][2] for p in paragraphs[-1][1])
|
|
|
|
|
|
col_right = max(col_right, ln["bbox"][2])
|
2026-09-27 12:32:26 +00:00
|
|
|
|
previous_end = prev["_source_end"]
|
|
|
|
|
|
gap_is_empty = not normalize(page_text[previous_end:at])
|
|
|
|
|
|
if gap_is_empty and not _starts_paragraph(prev, ln, line_h, usual_gap, col_right):
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
paragraphs[-1][1].append(ln)
|
|
|
|
|
|
continue
|
|
|
|
|
|
paragraphs.append((at, [ln]))
|
|
|
|
|
|
|
|
|
|
|
|
blocks: List[dict] = []
|
2026-09-27 12:32:26 +00:00
|
|
|
|
cursor = 0
|
|
|
|
|
|
for at, members in paragraphs:
|
|
|
|
|
|
stop = members[-1]["_source_end"]
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
if stop <= at:
|
|
|
|
|
|
continue
|
2026-09-27 12:32:26 +00:00
|
|
|
|
if at > cursor and page_text[cursor:at].strip():
|
|
|
|
|
|
blocks.append({"start": offset + cursor, "end": offset + at, "locator": dict(page_only)})
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
locator = dict(page_only)
|
|
|
|
|
|
locator["bbox"] = geometry.bbox(_union(m["bbox"] for m in members))
|
2026-09-27 12:32:26 +00:00
|
|
|
|
locator["source_id"] = f"pdf:{page_number}:{at}:{stop}"
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
blocks.append({"start": offset + at, "end": offset + stop, "locator": locator})
|
2026-09-27 12:32:26 +00:00
|
|
|
|
cursor = stop
|
|
|
|
|
|
if cursor < len(page_text) and page_text[cursor:].strip():
|
|
|
|
|
|
blocks.append({"start": offset + cursor, "end": end, "locator": dict(page_only)})
|
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage
Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.
Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.
The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.
* fix(docreader): serialize source blocks outside main so tests import it
test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.
* fix(chat): tighten source locators from review
- Remap rewritten-line-for-line stretches by line so scanned PDF pages
no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.
* fix(migrations): renumber chunk source locators after main's chunk_images
main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 05:26:16 +00:00
|
|
|
|
return blocks
|