WeKnora/internal/database/migration_sqlite_versioned_...

371 lines
13 KiB
Go
Raw Normal View History

package database
import (
"database/sql"
feat(session): fork conversations from a historical user message (#3304) * feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 06:20:18 +00:00
"fmt"
"os"
"path/filepath"
feat(session): fork conversations from a historical user message (#3304) * feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 06:20:18 +00:00
"strings"
"testing"
"github.com/stretchr/testify/require"
)
// versionedSQLiteTables is the set of tables that SQLite migrations must
// create to stay in sync with the versioned (PostgreSQL) migrations:
// 000041 task queue, 000053 system settings, 000055 processing spans,
feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) * feat(artifacts): add an artifacts library with its own message_artifacts table Skill-generated files were only reachable from inside the conversation that produced them. This adds a sidebar entry and a page that lists them across every conversation the user can see. Storage: artifacts move out of the messages.artifacts JSONB column into a new message_artifacts table (PostgreSQL 000103, SQLite 000023), backfilled on upgrade. The table is now the only source of truth; the old column is kept and the down migration writes the table back into it. The message repository loads artifacts for every message it returns and writes them on create, update (when non-nil) and fork, so callers still use Message.Artifacts. mod_time is stored as RFC 3339 text because the collector matches files on nanosecond mtimes, which PostgreSQL timestamps would truncate. API: GET /api/v1/artifacts lists the latest version of each file (grouped by session + source path, with version_count), scoped like the session list, with keyword / file type filters and paging. Downloads keep using the per-session endpoint. Frontend: "产物/Artifacts" sidebar entry (hidden without sandbox support), a library page with type tabs, search, date groups, infinite loading, preview drawer, download and jump-to-conversation; strings in all five locales. The sidebar's top-menu paths are now one shared list. * fix(artifacts): keep the restore hash guard and count versions by URL RecordRestoredArtifactMtime again only stamps rows whose content hash matches the restored content (and ignores empty hashes), and never rewrites the hash, matching MessageArtifacts.WithRestoredMtime. Without the guard a restore of v1's content also moved v2's mtime and relabelled v2 with v1's hash. The library's version_count now counts distinct URLs per (session, source path). A later answer that references an earlier file stores another row with the same URL, which COUNT(*) reported as an extra version. * fix(artifacts): scope the library to web chats and clear the legacy column The library used the session list's default scope, which counts sessions without an owner as legacy tenant-level rows. IM sessions are created without an owner, so every member saw the file names and titles of every IM chat (downloads then 404 for non-admins). The library now applies the same "web" bucket predicate as the home sidebar, extracted from QueryPaged so the two cannot drift: no IM mapping (including cleared ones), no embed marker, no API-key owner. Migrations 000103 / SQLite 000023 now clear messages.artifacts after the backfill. Message loads still SELECT *, so the stale copy was read on every history load. Only rows that carried artifacts are rewritten; the column stays for instances on the old binary during a rolling upgrade, and the down migration still rebuilds it from the table. * fix(migrations): keep the 000103 backfill alive on malformed artifact JSON A single file_size that is not an integer ("12.5", "") or a created_at that does not parse ("garbage", month 13) raised a cast error and failed the whole migration, blocking startup. Casts now go through pg_temp helpers that return NULL on bad input, falling back to 0 and the message's created_at; valid rows are unchanged. SQLite casts never raise; its migration test now covers the same malformed values. * docs(artifacts): describe the web-only library scope and the cleared legacy column
2026-09-18 04:10:19 +00:00
// 000063 knowledge multi-tags, 000093 browser authorization, 000103 message
// artifacts.
var versionedSQLiteTables = []string{
"memory_extraction_sessions",
"task_pending_ops",
"task_dead_letters",
"system_settings",
"knowledge_processing_spans",
"knowledge_tag_relations",
"browser_devices",
"browser_pairings",
"browser_task_interruptions",
feat(session): fork conversations from a historical user message (#3304) * feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 06:20:18 +00:00
"fork_snapshot_leases",
feat(mcp): add built-in MCP server with per-workspace endpoints (#3371) * feat(mcp): add built-in MCP server with per-workspace endpoints WeKnora previously exposed itself to MCP clients only through the Python package under mcp-server/, which binds one process to one API key and therefore to one workspace, and mirrors the REST API tool-for-tool. This adds a native MCP server surface to the Go backend: - New mcp_endpoints table (PostgreSQL 000100, SQLite 000020). A workspace can publish any number of endpoints, each with its own bearer token (SHA-256 stored, rotatable, shown once), knowledge-base allowlist, tool allowlist, default agent and per-minute rate limit. - Management API under /api/v1/mcp-endpoints (Viewer reads, Admin writes, API keys need manage_channels) plus a tool catalog endpoint. - internal/mcpserver hosts one mcp-go Streamable HTTP server in stateless mode at /mcp/:endpoint_id. tools/list is filtered per endpoint and tools/call re-checks the allowlist, applies rate limiting and touches last_used_at. MCPEndpointAuth resolves the token and injects the workspace, an mcp_endpoint principal and an API-key style scope derived from the endpoint, so existing knowledge-base and capability checks apply unchanged. - Redesigned tool set (12 tools in four groups): retrieval and reading reuse the agent's native tools; ask runs the default or a named agent synchronously and returns a cited answer with a session_id for follow-ups; wiki tools cover search, read and index; write tools are off by default. Knowledge bases accept ids or names. - Frontend: new "MCP Server" integrations tab (tab id mcpserver, to avoid clashing with the existing MCP-client settings section) with endpoint cards, a create/edit drawer, a one-time token dialog and ready-to-paste snippets for Cursor/VS Code/Claude Desktop, Claude Code and mcp-remote. Locales updated in all five languages. - Deployment capability integrations.mcpserver wired on both sides. - Docs: website-docs MCP page now leads with the built-in server; the Python mcp-server README carries a deprecation notice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(mcp): wire the MCP server into DI/router and tighten authorization The previous commit lost eight staged files to the repo pre-commit hook, which resolves ROOT to the main checkout and re-adds the main-checkout copies of tracked files into the worktree index. This commit restores them (container providers, RouterParams and route registration, context key, principal, clone table entry, deployment capability key, SQLite parity test) and addresses the review findings: - Knowledge-base scope is authorized, not trusted: endpoint allowlists and every tool call resolve each knowledge base through access.ResolveKB for the calling principal (ownership or organization share). Foreign IDs are rejected on create/update and dropped at call time. - Write tools mint explicit editor grants via access.KBAccess.WithGrant before calling the knowledge service, so add/update/delete go through the same RequireKBWrite path as the REST API. add_document also checks the grant itself. - ask no longer accepts agent_id. It runs only the endpoint's configured agent; tenant-owned agents and user-facing builtins qualify, internal builtins (wiki fixer, skill installer) are refused both on save and at call time. Web search is forced off for MCP calls. - Tool annotations match behaviour: ask is not read-only, add_document is a mutation. - CORS allows MCP-Protocol-Version / Mcp-Session-Id / Last-Event-ID and exposes Mcp-Session-Id for browser-based MCP clients. - Rate limiter runs its local cleanup goroutine; SQLite migration gains the deleted_at index for parity with PostgreSQL; non-ASCII endpoint names get unique mcpServers keys derived from the endpoint id. - Swagger annotations on the management handler. - Tests: KB isolation (service and server), write grants, endpoint-agent restriction, tool annotations, and a router test that mounts the public /mcp/:endpoint_id surface and checks the deployment capability. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(mcp): renumber migrations after rebase onto main main took 000100 / 000020 for chunks_index_diet, so the mcp_endpoints migrations move to 000101 (PostgreSQL) and 000021 (SQLite); the SQLite parity test and docs follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(mcp): run document tools under the knowledge base owner tenant Shared knowledge bases resolved through access.ResolveKB were only half usable: documents of an organization-shared knowledge base are stored under the owner tenant, so the document-level tools that queried by the calling tenant found nothing, and add_document would have created rows under the caller's tenant inside someone else's knowledge base. - knowledgeInScope loads documents tenant-agnostically and then requires the document to belong to an allowed knowledge base and to that base's tenant. - scopedKBContext replaces grantContext: it resolves the grant, switches the execution tenant to the knowledge base owner (KBAccess.Context) and swaps TenantInfo to the owner so storage/model settings resolve there. list_documents, read_document and every write tool run under it. - Locale text and docs no longer claim clients can pass agent_id. - Go SDK gains MCP endpoint methods (client/mcp_endpoint.go) with docs, and the Swagger artifacts are regenerated. - Tests: shared knowledge base read/write under the owner tenant, spoofed document tenant rejected, viewer share cannot write. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(mcp): renumber migrations after rebase onto main main now carries 000101 / 000021 (knowledge_profiles), so the mcp_endpoints migrations move to 000102 (PostgreSQL) and 000022 (SQLite); the SQLite parity test expects version 22 and the docs follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(mcp): guard last_used touch without a repository and wrap SDK line The transport tests build the server without a repository; the background last_used_at write then dereferenced a nil interface and, on the slower CI runner, took the test binary down. touchLastUsed now returns early when no repository is wired, and the tests inject a recording repository and assert the touch instead. Also wraps a 123-column SDK signature for the client module's lll check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-17 14:40:05 +00:00
"mcp_endpoints",
feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) * feat(artifacts): add an artifacts library with its own message_artifacts table Skill-generated files were only reachable from inside the conversation that produced them. This adds a sidebar entry and a page that lists them across every conversation the user can see. Storage: artifacts move out of the messages.artifacts JSONB column into a new message_artifacts table (PostgreSQL 000103, SQLite 000023), backfilled on upgrade. The table is now the only source of truth; the old column is kept and the down migration writes the table back into it. The message repository loads artifacts for every message it returns and writes them on create, update (when non-nil) and fork, so callers still use Message.Artifacts. mod_time is stored as RFC 3339 text because the collector matches files on nanosecond mtimes, which PostgreSQL timestamps would truncate. API: GET /api/v1/artifacts lists the latest version of each file (grouped by session + source path, with version_count), scoped like the session list, with keyword / file type filters and paging. Downloads keep using the per-session endpoint. Frontend: "产物/Artifacts" sidebar entry (hidden without sandbox support), a library page with type tabs, search, date groups, infinite loading, preview drawer, download and jump-to-conversation; strings in all five locales. The sidebar's top-menu paths are now one shared list. * fix(artifacts): keep the restore hash guard and count versions by URL RecordRestoredArtifactMtime again only stamps rows whose content hash matches the restored content (and ignores empty hashes), and never rewrites the hash, matching MessageArtifacts.WithRestoredMtime. Without the guard a restore of v1's content also moved v2's mtime and relabelled v2 with v1's hash. The library's version_count now counts distinct URLs per (session, source path). A later answer that references an earlier file stores another row with the same URL, which COUNT(*) reported as an extra version. * fix(artifacts): scope the library to web chats and clear the legacy column The library used the session list's default scope, which counts sessions without an owner as legacy tenant-level rows. IM sessions are created without an owner, so every member saw the file names and titles of every IM chat (downloads then 404 for non-admins). The library now applies the same "web" bucket predicate as the home sidebar, extracted from QueryPaged so the two cannot drift: no IM mapping (including cleared ones), no embed marker, no API-key owner. Migrations 000103 / SQLite 000023 now clear messages.artifacts after the backfill. Message loads still SELECT *, so the stale copy was read on every history load. Only rows that carried artifacts are rewritten; the column stays for instances on the old binary during a rolling upgrade, and the down migration still rebuilds it from the table. * fix(migrations): keep the 000103 backfill alive on malformed artifact JSON A single file_size that is not an integer ("12.5", "") or a created_at that does not parse ("garbage", month 13) raised a cast error and failed the whole migration, blocking startup. Casts now go through pg_temp helpers that return NULL on bad input, falling back to 0 and the message's created_at; valid rows are unchanged. SQLite casts never raise; its migration test now covers the same malformed values. * docs(artifacts): describe the web-only library scope and the cleared legacy column
2026-09-18 04:10:19 +00:00
"message_artifacts",
}
// versionedSQLiteColumns maps each existing table to the columns that the
// versioned migrations add and the SQLite baseline was missing.
var versionedSQLiteColumns = map[string][]string{
feat(agent): persist compaction checkpoints across turns (#3407) * feat(agent): persist compaction checkpoints across turns Agent compaction only lived inside the turn that ran it: the summary was discarded at turn end and LoadAgentHistory rebuilt raw history from the messages table, so every later turn past the threshold paid for another summarization of nearly the same history. A compaction whose history summary ends exactly on a stored turn is now written onto that turn's assistant message (messages.context_checkpoint). LoadAgentHistory starts from the newest checkpoint: the summary replaces that turn and everything before it, and later turns are replayed verbatim. The compactor already treats a leading summary as the previous summary, so checkpoints fold forward incrementally. History messages carry their turn's assistant message ID (chat.Message TurnID, engine-internal) so compaction can tell whether its cut lands on a stored-turn boundary. Nothing is persisted when the cut ends inside a stored turn, covers only the live turn, or the history summary degraded to a raw archive; a split turn's prefix summary is never persisted because that turn is replayed whole next time. Writes are column-scoped and best-effort, and a failed lookup falls back to history without a checkpoint. Storing the checkpoint on the covered turn keeps the boundary implicit, so session forks copy it with the turn and deleting the turn drops it. The HistoryTurns cap still applies, now to the turns after the checkpoint. * test(skill): wait for the background install in the name-conflict test InstallSkill queues the install on a goroutine, and the fixture row already starts at installing, so reading the status right after the call raced the mock install and failed whenever it landed on ready first. Wait for the run to finish on the reused row instead, as the other install tests do. * feat(agent): size agent history by tokens instead of turn count Agent history was capped at history_turns (default 5) because every turn used to rebuild raw history and compaction could not persist. With compaction checkpoints persisted, the cap is no longer what keeps a session inside the window, and it threw away context that would have fit. LoadAgentHistory now loads the newest checkpoint summary plus the newest turns after it that fit a token budget: the run's compaction threshold (agent.HistoryTokenBudget, the window less the reply reserve). History that large already triggers the first round's compaction, which persists a new checkpoint, so compaction rather than a turn count bounds the session. Loading more would only hand that compaction a summarizer input larger than the model can read. Rows are paged backwards by (created_at, id) and the read stops at the checkpoint or once the budget is full, so a long session is never read whole. Kept turns stay contiguous, the newest turn is kept even when it alone exceeds the budget (compaction splits it), and a turn cut by the read is left out rather than replayed without its original question. Agent mode no longer reads history_turns: AgentConfig.HistoryTurns is removed, the editor hides the input in smart-reasoning mode and explains the automatic behavior, and the smart-reasoning builtins drop the setting. Quick-answer (KnowledgeQA) keeps its turn cap since it has no compaction. * Revert "feat(agent): size agent history by tokens instead of turn count" This reverts commit d05fb6e5f1f161813a88e9127609f7a059f38f12. Loading history up to the compaction threshold lets the loader drop the oldest turns before compaction can trigger: whenever a single turn's replay is larger than the system prompt plus the new question, the trimmed history stays under the threshold, nothing is summarized, and the session degrades into a large sliding window that never gets a checkpoint. A turn-by-turn simulation of LoadAgentHistory with the real compactor showed zero compactions over 60 turns in those cases. Keep the persisted checkpoints with the HistoryTurns cap for now; token budgeted history returns in a separate change with a budget that leaves the overflow to compaction.
2026-09-18 12:38:56 +00:00
"memory_subjects": {"extraction_state"}, // 000094
"memory_items": {"replaces_id"}, // 000094
"tenants": {"api_principal_config"}, // 000064
"users": {"is_system_admin"}, // 000053
"knowledges": {"pending_subtasks_count", "profile"}, // 000056, 000101
"knowledge_bases": {"profile_config", "generated_profile"}, // 000101
"messages": {"attachments", "usage", "sandbox_checkpoint", "context_checkpoint"}, // 000034/085/097/105
feat(agent): persist compaction checkpoints across turns (#3407) * feat(agent): persist compaction checkpoints across turns Agent compaction only lived inside the turn that ran it: the summary was discarded at turn end and LoadAgentHistory rebuilt raw history from the messages table, so every later turn past the threshold paid for another summarization of nearly the same history. A compaction whose history summary ends exactly on a stored turn is now written onto that turn's assistant message (messages.context_checkpoint). LoadAgentHistory starts from the newest checkpoint: the summary replaces that turn and everything before it, and later turns are replayed verbatim. The compactor already treats a leading summary as the previous summary, so checkpoints fold forward incrementally. History messages carry their turn's assistant message ID (chat.Message TurnID, engine-internal) so compaction can tell whether its cut lands on a stored-turn boundary. Nothing is persisted when the cut ends inside a stored turn, covers only the live turn, or the history summary degraded to a raw archive; a split turn's prefix summary is never persisted because that turn is replayed whole next time. Writes are column-scoped and best-effort, and a failed lookup falls back to history without a checkpoint. Storing the checkpoint on the covered turn keeps the boundary implicit, so session forks copy it with the turn and deleting the turn drops it. The HistoryTurns cap still applies, now to the turns after the checkpoint. * test(skill): wait for the background install in the name-conflict test InstallSkill queues the install on a goroutine, and the fixture row already starts at installing, so reading the status right after the call raced the mock install and failed whenever it landed on ready first. Wait for the run to finish on the reused row instead, as the other install tests do. * feat(agent): size agent history by tokens instead of turn count Agent history was capped at history_turns (default 5) because every turn used to rebuild raw history and compaction could not persist. With compaction checkpoints persisted, the cap is no longer what keeps a session inside the window, and it threw away context that would have fit. LoadAgentHistory now loads the newest checkpoint summary plus the newest turns after it that fit a token budget: the run's compaction threshold (agent.HistoryTokenBudget, the window less the reply reserve). History that large already triggers the first round's compaction, which persists a new checkpoint, so compaction rather than a turn count bounds the session. Loading more would only hand that compaction a summarizer input larger than the model can read. Rows are paged backwards by (created_at, id) and the read stops at the checkpoint or once the budget is full, so a long session is never read whole. Kept turns stay contiguous, the newest turn is kept even when it alone exceeds the budget (compaction splits it), and a turn cut by the read is left out rather than replayed without its original question. Agent mode no longer reads history_turns: AgentConfig.HistoryTurns is removed, the editor hides the input in smart-reasoning mode and explains the automatic behavior, and the smart-reasoning builtins drop the setting. Quick-answer (KnowledgeQA) keeps its turn cap since it has no compaction. * Revert "feat(agent): size agent history by tokens instead of turn count" This reverts commit d05fb6e5f1f161813a88e9127609f7a059f38f12. Loading history up to the compaction threshold lets the loader drop the oldest turns before compaction can trigger: whenever a single turn's replay is larger than the system prompt plus the new question, the trimmed history stays under the threshold, nothing is summarized, and the session degrades into a large sliding window that never gets a checkpoint. A turn-by-turn simulation of LoadAgentHistory with the real compactor showed zero compactions over 60 turns in those cases. Keep the persisted checkpoints with the HistoryTurns cap for now; token budgeted history returns in a separate change with a budget that leaves the overflow to compaction.
2026-09-18 12:38:56 +00:00
"sessions": {"parent_session_id", "forked_from_message_id", "fork_bootstrap"}, // 000097
"tenant_invitations": {"token", "accepted_count"}, // 000054
"embed_channels": {"allow_memory"}, // 000060
"mcp_oauth_tokens": {"principal_type", "principal_id"}, // 000064
"mcp_tool_approvals": {"enabled"}, // 000091
}
feat(agent): size agent history by the window and calibrate token estimates (#3413) * feat(agent): size agent history by the window and calibrate token estimates Agent history was capped at history_turns (default 5) because compaction could not survive a turn. With checkpoints persisted, load history by tokens instead, and let compaction bound the session. Budget. The loader drops the oldest turns that do not fit, and what it drops is lost: neither replayed nor summarized. A budget equal to the compaction threshold (the earlier attempt, since reverted) let the loader trim first, so whenever one turn was larger than the system prompt plus the new question the request never crossed the threshold; nothing was summarized and the session became a sliding window with no checkpoint. agent.HistoryTokenBudget is now the whole window: overflow reaches the first round's compaction, which summarizes it into a new checkpoint. A turn-by-turn regression test drives the real loader, budget and compactor and fails as soon as a stored turn after the checkpoint is missing from the loaded history; restoring the old budget fails it. Summarizer input. History can now reach the window, so the part being summarized can outgrow one request. An overflowing request degrades to a raw archive that is never persisted, so every later turn would fail the same way. Prepare keeps the newest messages that fit, the prompt states how many were left out, and file paths still come from all of them. Calibration. cl100k counts Chinese at about a token a character where Qwen and DeepSeek spend about 0.6. The estimator now carries a scale (clamped to 0.5-1.5, text only), learned from consecutive requests in a turn: they share tools, system prompt and earlier messages, so the provider's growth between them prices exactly the appended messages, without mixing in how a provider renders tool schemas. The turn's scale is stored on its usage (context_token_scale); the next turn's loader prices history with it and hands it to the engine. Loading, the first compaction check and compaction itself share one estimator, so they agree; calibrating only the trigger would let the loader drop turns again, which the regression test covers. Also: history is priced as the engine sends it (redacted KB/Wiki results cost one line); a checkpoint older than the rows read is matched by the assistant rows' (created_at, id) order; a read ending on a turn's question keeps that turn; a compaction discarded for freeing under 5% still persists its checkpoint; a checkpoint write matching no row is an error; a partial index backs the newest-checkpoint lookup. Agent mode no longer reads history_turns: the field leaves AgentConfig, the editor hides it in smart-reasoning mode, and quick-answer keeps its turn cap. * fix(agent): address review of window-sized history loading Summarization. The summarizer call had a fixed 60s total timeout. With history loaded up to the window, a request could carry ~175k tokens and be cut off while prefilling or writing normally; the degraded result left no checkpoint, so the next turn reloaded the same history and failed the same way. Summarization now streams and is cancelled only after no output for the engine's stall timeout (120s by default, LLMCallTimeout overrides), with the overall ceiling left to the transport, as for the engine's own rounds and as in Codex and Pi. A stopped turn is not retried. When the summarizer still fails, the raw archive is persisted as a checkpoint marked degraded so the loop is broken, and the next compaction refines it as the previous summary. The archive is held to the summary budget, newest messages first, so a degraded compaction always frees room. Loading. The loader priced turns as sent but returned and held them as stored, so wiki pages stored in full and sent as one line could pile up in memory. It now returns the as-sent history and releases each turn's stored rows once replayed, keeping about a page of rows at a time. Index. The partial checkpoint index is replaced by a composite (session_id, created_at DESC, id DESC) index serving the backwards page, the newest-checkpoint lookup and the recent-messages reads. It is built with CREATE INDEX CONCURRENTLY so writes to messages are not blocked; the migration holds that single statement, since a multi-statement query runs as an implicit transaction. A PostgreSQL integration test, skipped without WEKNORA_MIGRATION_TEST_POSTGRES_DSN, applies the chain, runs 000106 down and up through golang-migrate, and checks both queries use the index without a sort; the SQLite migration test checks the same plans. Calibration. It no longer waits for a total token count, and a usage with only a scale is persisted. A turn that measures nothing carries forward the scale it started from, so the newest turn always has the latest one.
2026-09-18 14:51:31 +00:00
const expectedSQLiteMigrationVersion = 25
func TestSQLiteMigrationsCreateVersionedSchema(t *testing.T) {
repoRoot := sqliteRepoRoot(t)
chdirAndRestore(t, repoRoot)
dbPath := filepath.Join(t.TempDir(), "fresh.db")
require.NoError(t, RunMigrationsWithOptions("sqlite3://unused", MigrationOptions{SQLiteDBPath: dbPath}))
db := openSQLiteDB(t, dbPath)
version, dirty := sqliteMigrationState(t, db)
require.Equal(t, expectedSQLiteMigrationVersion, version)
require.False(t, dirty)
for _, table := range versionedSQLiteTables {
require.Truef(t, sqliteTableExists(t, db, table), "SQLite migrations must create table %s", table)
}
for table, columns := range versionedSQLiteColumns {
for _, column := range columns {
require.Truef(
t,
sqliteColumnExists(t, db, table, column),
"SQLite migrations must add column %s.%s",
table,
column,
)
}
}
feat(agent): size agent history by the window and calibrate token estimates (#3413) * feat(agent): size agent history by the window and calibrate token estimates Agent history was capped at history_turns (default 5) because compaction could not survive a turn. With checkpoints persisted, load history by tokens instead, and let compaction bound the session. Budget. The loader drops the oldest turns that do not fit, and what it drops is lost: neither replayed nor summarized. A budget equal to the compaction threshold (the earlier attempt, since reverted) let the loader trim first, so whenever one turn was larger than the system prompt plus the new question the request never crossed the threshold; nothing was summarized and the session became a sliding window with no checkpoint. agent.HistoryTokenBudget is now the whole window: overflow reaches the first round's compaction, which summarizes it into a new checkpoint. A turn-by-turn regression test drives the real loader, budget and compactor and fails as soon as a stored turn after the checkpoint is missing from the loaded history; restoring the old budget fails it. Summarizer input. History can now reach the window, so the part being summarized can outgrow one request. An overflowing request degrades to a raw archive that is never persisted, so every later turn would fail the same way. Prepare keeps the newest messages that fit, the prompt states how many were left out, and file paths still come from all of them. Calibration. cl100k counts Chinese at about a token a character where Qwen and DeepSeek spend about 0.6. The estimator now carries a scale (clamped to 0.5-1.5, text only), learned from consecutive requests in a turn: they share tools, system prompt and earlier messages, so the provider's growth between them prices exactly the appended messages, without mixing in how a provider renders tool schemas. The turn's scale is stored on its usage (context_token_scale); the next turn's loader prices history with it and hands it to the engine. Loading, the first compaction check and compaction itself share one estimator, so they agree; calibrating only the trigger would let the loader drop turns again, which the regression test covers. Also: history is priced as the engine sends it (redacted KB/Wiki results cost one line); a checkpoint older than the rows read is matched by the assistant rows' (created_at, id) order; a read ending on a turn's question keeps that turn; a compaction discarded for freeing under 5% still persists its checkpoint; a checkpoint write matching no row is an error; a partial index backs the newest-checkpoint lookup. Agent mode no longer reads history_turns: the field leaves AgentConfig, the editor hides it in smart-reasoning mode, and quick-answer keeps its turn cap. * fix(agent): address review of window-sized history loading Summarization. The summarizer call had a fixed 60s total timeout. With history loaded up to the window, a request could carry ~175k tokens and be cut off while prefilling or writing normally; the degraded result left no checkpoint, so the next turn reloaded the same history and failed the same way. Summarization now streams and is cancelled only after no output for the engine's stall timeout (120s by default, LLMCallTimeout overrides), with the overall ceiling left to the transport, as for the engine's own rounds and as in Codex and Pi. A stopped turn is not retried. When the summarizer still fails, the raw archive is persisted as a checkpoint marked degraded so the loop is broken, and the next compaction refines it as the previous summary. The archive is held to the summary budget, newest messages first, so a degraded compaction always frees room. Loading. The loader priced turns as sent but returned and held them as stored, so wiki pages stored in full and sent as one line could pile up in memory. It now returns the as-sent history and releases each turn's stored rows once replayed, keeping about a page of rows at a time. Index. The partial checkpoint index is replaced by a composite (session_id, created_at DESC, id DESC) index serving the backwards page, the newest-checkpoint lookup and the recent-messages reads. It is built with CREATE INDEX CONCURRENTLY so writes to messages are not blocked; the migration holds that single statement, since a multi-statement query runs as an implicit transaction. A PostgreSQL integration test, skipped without WEKNORA_MIGRATION_TEST_POSTGRES_DSN, applies the chain, runs 000106 down and up through golang-migrate, and checks both queries use the index without a sort; the SQLite migration test checks the same plans. Calibration. It no longer waits for a total token count, and a usage with only a scale is persisted. A turn that measures nothing carries forward the scale it started from, so the newest turn always has the latest one.
2026-09-18 14:51:31 +00:00
require.True(t, sqliteIndexExists(t, db, "idx_messages_session_created_id"),
"SQLite migrations must add the session/created_at index") // 000106
assertSQLiteAgentHistoryQueriesUseTheIndex(t, db)
assertSQLiteShareLinkInvitationsWork(t, db)
assertSQLiteMCPOAuthPrincipalUpsertWorks(t, db)
require.False(t, sqliteColumnExists(t, db, "knowledges", "tag_id"),
"SQLite migrations must drop legacy knowledges.tag_id after multi-tag migration")
}
func TestSQLiteMigrationsUpgradeV4PreservesData(t *testing.T) {
repoRoot := sqliteRepoRoot(t)
// Build a legacy v4 migration root (000000_init .. 000004_memory) so we
// can prove the new migrations upgrade an existing Lite database without
// replaying the baseline.
legacyRoot := copySQLiteMigrationsV4(t, repoRoot)
chdirAndRestore(t, legacyRoot)
dbPath := filepath.Join(t.TempDir(), "upgrade.db")
require.NoError(t, RunMigrationsWithOptions("sqlite3://unused", MigrationOptions{SQLiteDBPath: dbPath}))
db := openSQLiteDB(t, dbPath)
versionBefore, dirtyBefore := sqliteMigrationState(t, db)
require.Equal(t, 4, versionBefore)
require.False(t, dirtyBefore)
_, err := db.Exec("INSERT INTO tenants (name, business) VALUES (?, ?)", "upgrade-sentinel", "migration-test")
require.NoError(t, err)
_, err = db.Exec(
"INSERT INTO knowledges (id, tenant_id, knowledge_base_id, type, title, source, tag_id) "+
"VALUES (?, 1, ?, 'document', 'tagged-doc', 'manual', ?)",
"legacy-knowledge-1", "legacy-kb-1", "legacy-tag-1",
)
require.NoError(t, err)
// Run the full migration set from the repo root.
chdirAndRestore(t, repoRoot)
require.NoError(t, RunMigrationsWithOptions("sqlite3://unused", MigrationOptions{SQLiteDBPath: dbPath}))
db = openSQLiteDB(t, dbPath)
versionAfter, dirtyAfter := sqliteMigrationState(t, db)
require.Equal(t, expectedSQLiteMigrationVersion, versionAfter)
require.False(t, dirtyAfter)
for _, table := range versionedSQLiteTables {
require.Truef(t, sqliteTableExists(t, db, table), "upgraded SQLite DB must have table %s", table)
}
for table, columns := range versionedSQLiteColumns {
for _, column := range columns {
require.Truef(
t,
sqliteColumnExists(t, db, table, column),
"upgraded SQLite DB must have column %s.%s",
table,
column,
)
}
}
var sentinelName string
require.NoError(t, db.QueryRow("SELECT name FROM tenants WHERE business = ?", "migration-test").Scan(&sentinelName))
require.Equal(t, "upgrade-sentinel", sentinelName)
var relationCount int
require.NoError(t, db.QueryRow(
"SELECT COUNT(*) FROM knowledge_tag_relations WHERE knowledge_id = ? AND tag_id = ?",
"legacy-knowledge-1", "legacy-tag-1",
).Scan(&relationCount))
require.Equal(t, 1, relationCount)
require.False(t, sqliteColumnExists(t, db, "knowledges", "tag_id"))
}
feat(session): fork conversations from a historical user message (#3304) * feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 06:20:18 +00:00
func TestSQLiteMigrationsUpgradeV16AddsSessionForkColumns(t *testing.T) {
repoRoot := sqliteRepoRoot(t)
legacyRoot := copySQLiteMigrationsThrough(t, repoRoot, 16)
chdirAndRestore(t, legacyRoot)
dbPath := filepath.Join(t.TempDir(), "upgrade-v16.db")
require.NoError(t, RunMigrationsWithOptions("sqlite3://unused", MigrationOptions{SQLiteDBPath: dbPath}))
db := openSQLiteDB(t, dbPath)
versionBefore, dirtyBefore := sqliteMigrationState(t, db)
require.Equal(t, 16, versionBefore)
require.False(t, dirtyBefore)
require.False(t, sqliteColumnExists(t, db, "sessions", "parent_session_id"))
require.False(t, sqliteColumnExists(t, db, "sessions", "forked_from_message_id"))
require.False(t, sqliteColumnExists(t, db, "sessions", "fork_bootstrap"))
require.False(t, sqliteColumnExists(t, db, "messages", "sandbox_checkpoint"))
chdirAndRestore(t, repoRoot)
require.NoError(t, RunMigrationsWithOptions("sqlite3://unused", MigrationOptions{SQLiteDBPath: dbPath}))
db = openSQLiteDB(t, dbPath)
versionAfter, dirtyAfter := sqliteMigrationState(t, db)
require.Equal(t, expectedSQLiteMigrationVersion, versionAfter)
require.False(t, dirtyAfter)
for _, column := range versionedSQLiteColumns["sessions"] {
require.Truef(
t,
sqliteColumnExists(t, db, "sessions", column),
"upgraded SQLite DB must have column sessions.%s",
column,
)
}
require.True(t, sqliteColumnExists(t, db, "messages", "sandbox_checkpoint"))
}
func sqliteRepoRoot(t *testing.T) string {
t.Helper()
repoRoot, err := filepath.Abs(filepath.Join("..", ".."))
require.NoError(t, err)
return repoRoot
}
func chdirAndRestore(t *testing.T, dir string) {
t.Helper()
previousDir, err := os.Getwd()
require.NoError(t, err)
require.NoError(t, os.Chdir(dir))
t.Cleanup(func() { _ = os.Chdir(previousDir) })
}
func openSQLiteDB(t *testing.T, dbPath string) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite3", dbPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
return db
}
func sqliteMigrationState(t *testing.T, db *sql.DB) (version int, dirty bool) {
t.Helper()
require.NoError(t, db.QueryRow("SELECT version, dirty FROM schema_migrations").Scan(&version, &dirty))
return version, dirty
}
func sqliteTableExists(t *testing.T, db *sql.DB, table string) bool {
t.Helper()
var n int
require.NoError(t, db.QueryRow(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?",
table,
).Scan(&n))
return n == 1
}
feat(agent): size agent history by the window and calibrate token estimates (#3413) * feat(agent): size agent history by the window and calibrate token estimates Agent history was capped at history_turns (default 5) because compaction could not survive a turn. With checkpoints persisted, load history by tokens instead, and let compaction bound the session. Budget. The loader drops the oldest turns that do not fit, and what it drops is lost: neither replayed nor summarized. A budget equal to the compaction threshold (the earlier attempt, since reverted) let the loader trim first, so whenever one turn was larger than the system prompt plus the new question the request never crossed the threshold; nothing was summarized and the session became a sliding window with no checkpoint. agent.HistoryTokenBudget is now the whole window: overflow reaches the first round's compaction, which summarizes it into a new checkpoint. A turn-by-turn regression test drives the real loader, budget and compactor and fails as soon as a stored turn after the checkpoint is missing from the loaded history; restoring the old budget fails it. Summarizer input. History can now reach the window, so the part being summarized can outgrow one request. An overflowing request degrades to a raw archive that is never persisted, so every later turn would fail the same way. Prepare keeps the newest messages that fit, the prompt states how many were left out, and file paths still come from all of them. Calibration. cl100k counts Chinese at about a token a character where Qwen and DeepSeek spend about 0.6. The estimator now carries a scale (clamped to 0.5-1.5, text only), learned from consecutive requests in a turn: they share tools, system prompt and earlier messages, so the provider's growth between them prices exactly the appended messages, without mixing in how a provider renders tool schemas. The turn's scale is stored on its usage (context_token_scale); the next turn's loader prices history with it and hands it to the engine. Loading, the first compaction check and compaction itself share one estimator, so they agree; calibrating only the trigger would let the loader drop turns again, which the regression test covers. Also: history is priced as the engine sends it (redacted KB/Wiki results cost one line); a checkpoint older than the rows read is matched by the assistant rows' (created_at, id) order; a read ending on a turn's question keeps that turn; a compaction discarded for freeing under 5% still persists its checkpoint; a checkpoint write matching no row is an error; a partial index backs the newest-checkpoint lookup. Agent mode no longer reads history_turns: the field leaves AgentConfig, the editor hides it in smart-reasoning mode, and quick-answer keeps its turn cap. * fix(agent): address review of window-sized history loading Summarization. The summarizer call had a fixed 60s total timeout. With history loaded up to the window, a request could carry ~175k tokens and be cut off while prefilling or writing normally; the degraded result left no checkpoint, so the next turn reloaded the same history and failed the same way. Summarization now streams and is cancelled only after no output for the engine's stall timeout (120s by default, LLMCallTimeout overrides), with the overall ceiling left to the transport, as for the engine's own rounds and as in Codex and Pi. A stopped turn is not retried. When the summarizer still fails, the raw archive is persisted as a checkpoint marked degraded so the loop is broken, and the next compaction refines it as the previous summary. The archive is held to the summary budget, newest messages first, so a degraded compaction always frees room. Loading. The loader priced turns as sent but returned and held them as stored, so wiki pages stored in full and sent as one line could pile up in memory. It now returns the as-sent history and releases each turn's stored rows once replayed, keeping about a page of rows at a time. Index. The partial checkpoint index is replaced by a composite (session_id, created_at DESC, id DESC) index serving the backwards page, the newest-checkpoint lookup and the recent-messages reads. It is built with CREATE INDEX CONCURRENTLY so writes to messages are not blocked; the migration holds that single statement, since a multi-statement query runs as an implicit transaction. A PostgreSQL integration test, skipped without WEKNORA_MIGRATION_TEST_POSTGRES_DSN, applies the chain, runs 000106 down and up through golang-migrate, and checks both queries use the index without a sort; the SQLite migration test checks the same plans. Calibration. It no longer waits for a total token count, and a usage with only a scale is persisted. A turn that measures nothing carries forward the scale it started from, so the newest turn always has the latest one.
2026-09-18 14:51:31 +00:00
// assertSQLiteAgentHistoryQueriesUseTheIndex checks the two per-turn agent
// history queries walk idx_messages_session_created_id in order instead of
// sorting every message of the session.
func assertSQLiteAgentHistoryQueriesUseTheIndex(t *testing.T, db *sql.DB) {
t.Helper()
for name, query := range map[string]string{
"backwards page": `SELECT * FROM messages WHERE session_id = 's'
AND (created_at < '2026-01-01' OR (created_at = '2026-01-01' AND id < 'x'))
AND deleted_at IS NULL ORDER BY created_at DESC, id DESC LIMIT 200`,
"newest checkpoint": `SELECT id FROM messages WHERE session_id = 's' AND role = 'assistant'
AND context_checkpoint IS NOT NULL AND deleted_at IS NULL
ORDER BY created_at DESC, id DESC LIMIT 1`,
} {
rows, err := db.Query("EXPLAIN QUERY PLAN " + query)
require.NoError(t, err, name)
var plan strings.Builder
for rows.Next() {
var id, parent, unused int
var detail string
require.NoError(t, rows.Scan(&id, &parent, &unused, &detail), name)
plan.WriteString(detail + "\n")
}
require.NoError(t, rows.Err(), name)
require.NoError(t, rows.Close(), name)
require.Contains(t, plan.String(), "idx_messages_session_created_id", "%s plan:\n%s", name, plan.String())
require.NotContains(t, plan.String(), "TEMP B-TREE", "%s must not sort:\n%s", name, plan.String())
}
}
func sqliteIndexExists(t *testing.T, db *sql.DB, index string) bool {
t.Helper()
var n int
require.NoError(t, db.QueryRow(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'index' AND name = ?",
index,
).Scan(&n))
return n == 1
}
func sqliteColumnExists(t *testing.T, db *sql.DB, table, column string) bool {
t.Helper()
var n int
require.NoError(t, db.QueryRow(
"SELECT COUNT(*) FROM pragma_table_info(?) WHERE name = ?",
table,
column,
).Scan(&n))
return n == 1
}
func assertSQLiteShareLinkInvitationsWork(t *testing.T, db *sql.DB) {
t.Helper()
_, err := db.Exec("INSERT INTO tenants (name, business) VALUES (?, ?)", "share-link-tenant", "share-link-test")
require.NoError(t, err)
expiresAt := "2099-01-01 00:00:00"
shareLinkInsert := "INSERT INTO tenant_invitations " +
"(tenant_id, invitee_user_id, token, role, status, expires_at) " +
"VALUES (1, '', ?, 'member', 'pending', ?)"
_, err = db.Exec(shareLinkInsert, "token-a", expiresAt)
require.NoError(t, err)
_, err = db.Exec(shareLinkInsert, "token-b", expiresAt)
require.NoError(t, err)
var count int
require.NoError(t, db.QueryRow(
"SELECT COUNT(*) FROM tenant_invitations WHERE tenant_id = 1 AND invitee_user_id = '' AND status = 'pending'",
).Scan(&count))
require.Equal(t, 2, count)
}
func assertSQLiteMCPOAuthPrincipalUpsertWorks(t *testing.T, db *sql.DB) {
t.Helper()
_, err := db.Exec(
"INSERT INTO mcp_services (id, tenant_id, name, transport_type) VALUES (?, 1, 'svc', 'http')",
"svc-migration-1",
)
require.NoError(t, err)
tokenInsertPrefix := "INSERT INTO mcp_oauth_tokens " +
"(id, tenant_id, user_id, service_id, principal_type, principal_id, access_token) "
_, err = db.Exec(
tokenInsertPrefix +
"VALUES ('tok-1', 1, 'u1', 'svc-migration-1', 'web_user', 'u1', 'token-1')",
)
require.NoError(t, err)
_, err = db.Exec(
tokenInsertPrefix +
"VALUES ('tok-2', 1, 'u1', 'svc-migration-1', 'web_user', 'u1', 'token-2') " +
"ON CONFLICT(tenant_id, principal_type, principal_id, service_id) " +
"DO UPDATE SET access_token = excluded.access_token",
)
require.NoError(t, err)
var accessToken string
require.NoError(t, db.QueryRow(
"SELECT access_token FROM mcp_oauth_tokens "+
"WHERE tenant_id = 1 AND principal_type = 'web_user' "+
"AND principal_id = 'u1' AND service_id = 'svc-migration-1'",
).Scan(&accessToken))
require.Equal(t, "token-2", accessToken)
var rowCount int
require.NoError(t, db.QueryRow(
"SELECT COUNT(*) FROM mcp_oauth_tokens WHERE tenant_id = 1 AND service_id = 'svc-migration-1'",
).Scan(&rowCount))
require.Equal(t, 1, rowCount)
}
func copySQLiteMigrationsV4(t *testing.T, repoRoot string) string {
feat(session): fork conversations from a historical user message (#3304) * feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 06:20:18 +00:00
t.Helper()
return copySQLiteMigrationsThrough(t, repoRoot, 4)
}
func copySQLiteMigrationsThrough(t *testing.T, repoRoot string, maxVersion int) string {
t.Helper()
dest := t.TempDir()
srcDir := filepath.Join(repoRoot, "migrations", "sqlite")
destDir := filepath.Join(dest, "migrations", "sqlite")
require.NoError(t, os.MkdirAll(destDir, 0o755))
feat(session): fork conversations from a historical user message (#3304) * feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 06:20:18 +00:00
entries, err := os.ReadDir(srcDir)
require.NoError(t, err)
copied := 0
for _, entry := range entries {
name := entry.Name()
if entry.IsDir() || !strings.HasSuffix(name, ".up.sql") {
continue
}
var version int
_, scanErr := fmt.Sscanf(name, "%d_", &version)
require.NoError(t, scanErr, "sqlite migration filename %s", name)
if version > maxVersion {
continue
}
data, readErr := os.ReadFile(filepath.Join(srcDir, name))
require.NoError(t, readErr)
require.NoError(t, os.WriteFile(filepath.Join(destDir, name), data, 0o600))
feat(session): fork conversations from a historical user message (#3304) * feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 06:20:18 +00:00
copied++
}
feat(session): fork conversations from a historical user message (#3304) * feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 06:20:18 +00:00
require.Greater(t, copied, 0)
return dest
}