Commit Graph

9 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
linus-liu-web a2aeb97a01
fix(gitlab): 分页循环补重复页防护与跳数上限 (#3841)
projects() 与 tree() 只依据服务端返回的 X-Next-Page 判断是否继续,
该值重复或不前进时循环不会结束:同一页被反复请求、结果切片持续
增长,只能等上层 context 到期。

两条循环改用 trackPage:页码已请求过即返回错误,不静默当作读完;
页数达到 maxPaginationHops 时记警告并返回错误。正常翻页路径不变。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:37 +08:00
linus-liu-web 1dbe9d0c1a
fix(gitlab): 仓库文件下载补体积上限,超限报错而不是无界读入 (#3887)
* fix(gitlab): 仓库文件下载没有体积上限,超限改为显式报错

getRaw 直接用 io.ReadAll 缓冲整个 blob,get/getPage 的 JSON 读取同样无界。
GitLab 的 tree/compare 接口不返回 blob 大小,下载处是唯一能拦的地方,所以
同步一个音频或大 PDF 就会把一个完整 blob 读进进程内存。

对齐兄弟 connector(钉钉 16MiB JSON 上限、飞书 512MiB 下载上限):

- maxJSONResponseBytes(16 MiB)用于 get/getPage;
- maxRawFileBytes(512 MiB,与飞书 maxFeishuDownloadBytes 对齐)用于 getRaw;
- base64 回退走 getCapped,其请求体是 base64 后的 blob(约 4/3)加信封。

超限返回错误而不是截断:截断的正文会被当成完整文档入库。非 2xx 状态先于
读取判定,避免超大错误页把真实状态码盖掉。

* fix(gitlab): 将 compare 响应上限与元数据分开

compare 正文包含完整提交和 patch,沿用 16MiB 元数据上限会中断增量同步且游标无法前进。超限错误改为带上未编码的仓库路径。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-30 14:33:15 +08:00
yeliheng-010 72e105f46a fix(gitlab): 按提交快照比较增量同步变更 2026-09-29 00:50:05 +08:00
yeliheng-010 608027179f fix(gitlab): 补齐项目资源列表分页 2026-09-24 07:52:32 +08:00
hahahahahayesyeseys 11edd16819 fix(datasource): fail closed on GitLab compare errors 2026-09-24 07:49:07 +08:00
wizardchen 1f82fbc9b5 fix(datasource): preserve distinct GitLab files during deduplication 2026-09-14 19:32:41 +08:00
小妍儿 ✨ 061df8efdc
feat(datasource): persist source created time; Feishu uses content edit time for source_updated_at (#3083)
* feat(datasource): persist source document creation time and use content edit time for Feishu

Follow-up to 160c806b (Refs #3016), which started persisting the source
last-modified time as metadata.source_updated_at. Two gaps remained:

1. Creation time was never carried. FetchedItem gains CreatedAt and
   ingestItem persists it as metadata.source_created_at (RFC3339, UTC)
   with the same absent-means-unknown rule as source_updated_at.

2. The Feishu wiki connector filled FetchedItem.UpdatedAt from
   NodeEditTime, which moves when a node is renamed or dragged in the
   tree without any content change. The connector's own cursor already
   prefers ObjEditTime for exactly this reason; use the same rule for the
   persisted value (contentEditTime / contentCreateTime, node_* as
   fallback), so source_updated_at reflects document edits.

Feishu Drive populates CreatedAt from created_time. GitLab stops
fabricating UpdatedAt = time.Now(): a fetch timestamp is not a document
timestamp, and an absent value is the documented "unknown" (the commit
time is not fetched today; leaving it unset is honest until it is).

Tests: ingestItem covers both keys present/absent; the wiki fetch test
asserts obj_* times win and node_* are the fallback.

* style(gitlab): break up the FetchedItem literal (lll)

---------

Co-authored-by: xiaoyaner0201 <>
2026-09-07 19:39:01 +08:00
lyingbug 0cc3c33530
fix(datasource/gitlab): 修复项目路径编码与连接器单例缓存问题 (#2673)
* fix(datasource/gitlab): 修复项目路径编码与连接器单例缓存问题

避免 project_id 二次 URL 编码,并确保每个数据源使用独立 client;
保存时校验 projects 配置,补齐增量同步过滤与相关单测。

* style(datasource/gitlab): gofmt connector sources
2026-08-12 16:22:02 +08:00
Numblgw d3524d8d04
feat(datasource): 支持 GitLab 数据源文档同步 (#2656)
* 支持gitlab数据源

* 添加单元测试 && 修复gitlab数据源配置多个的UI展示问题

* 补充韩语、俄语文案

* gitlab数据源同步:新增文件类型过滤 && 改为streaming实现

* fix: remove duplicate RSS locale keys

---------

Co-authored-by: liguowei <liguowei@xinye.com>
2026-08-12 16:09:55 +08:00