WeKnora/internal
linus-liu-web 4eb23a8560
fix(qdrant): CopyIndices 用服务端返回的 next_page_offset 翻页,不再按页尾 id 回填 (#3882)
* fix(qdrant): 复制索引改用服务端返回的游标,重复游标报错并给分页加跳数上限

CopyIndices 用"上一页最后一个点的 id"当游标(repository.go:987)。Qdrant 的
scroll offset 是**包含**语义 —— proto 里写的是 "Start with this ID",实测下一页
正是从该 id 本身开始返回,所以每翻一页都会把上一页的最后一个点再读一次、再写一份
到目标库。服务端本来就把下一页该用的游标放在 ScrollResponse.next_page_offset
("Use this offset for the next query")里,而客户端只在 ScrollAndOffset() 中返回它,
本函数调用的 Scroll() 恰好把它丢掉了;终止条件也只能退化成"这一页比 64 短"。

在真实 Qdrant 1.19.1 上实测:150 个源 chunk 复制出 152 个目标点,chunk-63 与
chunk-126 各被复制两次(每个页边界一个),totalCopied 同步虚高。源库只要超过一页
(>64 个 chunk)就会命中,不需要服务端有任何异常行为。

本次修改:

- 改用 q.client.ScrollAndOffset,游标直接取服务端返回的 next_page_offset;终止
  条件改为"服务端没有下一页"(next_page_offset == nil),不再用短页推断读完。
- 记录用过的游标,重复即返回 "copy indices made no progress at cursor ...",
  检测点在整页处理之前,重复的那一页不会再多写一份。
- 分页跳数上限 maxCopyPaginationHops(10000 页);触顶打印 Warn 并返回错误,
  而不是把这一轮复制当成成功。
- 修复后同一份数据在真实 Qdrant 上测得 150 个目标点、0 重复。

回归用例用假 gRPC 服务端(拦截 unary RPC,对齐 repository_test.go 的
newInterceptedQdrantClient)复刻真实服务端的游标语义:150 个源 chunk 必须各复制
一次(修复前 152 个点、2 个重复);游标恒定不动的源端必须报 no progress
(修复前 5 秒写入 3127040 个点仍未返回);一直推进但永不结束的源端在第 10000 页
被拦下。

* fix(qdrant): drop the CopyIndices page cap

10000 pages of 64 points capped a copy at 640k points and failed the
whole clone beyond that. The scroll walks every point of the source
knowledge base (a single-knowledge clone included), so a cap derived from
the chunk map would cut off small clones out of large bases too.
next_page_offset only moves forward, so the repeated-cursor check already
stops a walk that cannot finish, and the task context bounds the rest.

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:04:43 +08:00
..
agent fix(data-analysis): skip style-only worksheets before DuckDB loading (#3660) 2026-10-08 13:04:29 +08:00
application fix(qdrant): CopyIndices 用服务端返回的 next_page_offset 翻页,不再按页尾 id 回填 (#3882) 2026-10-08 13:04:43 +08:00
assets feat(assets): add ASR test audio file and embed it in the application 2026-04-02 21:27:27 +08:00
browserskill fix(browser): refine connection and preview UI with extension version 2026-09-23 10:57:59 +08:00
common fix(markdown): sanitize invalid UTF-8 and NUL bytes 2026-08-31 12:24:35 +08:00
config feat(auth): 新增仅限邀请注册档位 (#3900) 2026-09-30 13:32:55 +08:00
container fix: use consistent qdrant env settings (#3960) 2026-10-07 23:59:56 +08:00
database revert: take the plugin system back out of main 2026-09-26 14:58:18 +08:00
datasource fix(notion): 分页循环补重复游标防护与跳数上限 (#3890) 2026-10-08 01:29:20 +08:00
embedpolicy fix(embed): enforce host origin allowlists on framed pages 2026-09-14 18:03:57 +08:00
errors feat(knowledge): flag documents that stop making progress (#3557) 2026-09-22 21:34:08 +08:00
event fix(agent): end the turn when the completion cap truncates an answer (#3449) 2026-09-20 13:17:50 +08:00
filetransport refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
handler fix(docs): make `make docs` reproduce the committed Swagger again (#3751) 2026-10-08 01:26:08 +08:00
im fix(im/wecom): preserve image in mixed messages with text captions (#3901) 2026-10-08 01:25:25 +08:00
infrastructure fix(docparser): anydoc 内嵌 EMF/WMF 图片的扩展名与 Rust 侧对齐 (#3938) 2026-10-08 01:21:57 +08:00
ipclass fix: align client sessions and integration request handling (#3305) 2026-09-16 16:20:57 +08:00
localsandbox feat(lite): install and run skills on the host (#3661) 2026-09-24 21:30:34 +08:00
logger fix(logger): keep LLM debug records inside their directory 2026-09-25 18:22:14 +08:00
mcp fix(mcp): pass embedded resources in tool results to the agent (#3766) 2026-10-08 01:25:04 +08:00
mcpserver fix(mcp): return accessible image URLs from existing tools (#3650) 2026-09-24 11:56:25 +08:00
middleware fix(middleware): stop buffering whole request bodies in the access log (#3927) 2026-10-08 01:29:05 +08:00
modelcontext fix(agent): 图谱查询的三处截断必须可见,模型不再把子图当全集 (#3885) 2026-10-07 23:59:50 +08:00
models fix(models): honor OpenRouter native thinking for legacy defaults (#3807) 2026-10-08 01:22:04 +08:00
ratelimit feat(embed): add backend embed channels with auth and rate limiting (#1657) 2026-06-12 23:10:39 +08:00
reranking fix(retrieval): fix recall correctness, score scales, cost and agent tools (#3687) 2026-09-25 10:48:18 +08:00
router fix(frontend): 修复复制模型时模型名称被改写的问题 (#3830) 2026-09-29 19:32:08 +08:00
runtime fix(server): run ResourceCleaner on signal shutdown instead of racing Fatalf exit (#3804) 2026-09-28 23:37:36 +08:00
sandbox fix(sandbox): reclassify exec races with the idle sweep as NotFound so rebinding heals (#3980) 2026-10-08 01:28:53 +08:00
searchutil feat(wiki): 增加跨知识库 Wiki 搜索 POST /wiki-search (#3535) 2026-09-25 22:22:58 +08:00
sourceloc fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794) 2026-09-27 20:32:26 +08:00
storageallowlist fix(storage): align default provider fallback with STORAGE_ALLOW_LIST 2026-07-15 11:05:00 +08:00
storageurl fix(mcp): return accessible image URLs from existing tools (#3650) 2026-09-24 11:56:25 +08:00
stream feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
textconv fix: remove GPL converter and bundle third-party license sources 2026-09-08 15:19:10 +08:00
tracing/langfuse fix(milvus): preserve keyword retrieval scores 2026-09-17 11:59:40 +08:00
types fix(wiki): 重建链接逐页写失败不再静默,半成品不再回 200 (#3884) 2026-10-08 01:25:53 +08:00
utils feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393) 2026-09-23 12:02:06 +08:00