Commit Graph

6 Commits (014f95f1394a7bfd1c7bbbdcdc19057a80dcaca5)

Author SHA1 Message Date
lyingbug 68d99d7d21
perf(chunk): speed up chunk ingestion and batch updates (#3369)
* perf(chunk): speed up chunk ingestion and batch updates

Chunk writes were dominated by avoidable work in the repository layer:

- UpdateChunks built one CASE WHEN id = ? chain per column, so a batch of
  N rows evaluated 5N branches per row (O(N²)). On PostgreSQL it now joins
  against a VALUES list keyed by the primary key (O(N), batched by 1000);
  other dialects run one small UPDATE per row inside a single transaction.
- CreateChunks copied Content into SourceContent for every new row, doubling
  insert volume. SourceContent is only needed once a chunk is edited and the
  edit path already backfills it lazily, so it is left empty on create.
  Batch size raised from 100 to 500 and the insert (including the SQLite
  seq_id pre-assignment) now runs inside one transaction.
- New UpdateChunkFieldsByIDs applies uniform field values with a single
  UPDATE ... WHERE id IN per batch. FAQ import uses it to flip status after
  indexing instead of shipping every row's content back, and resolves tag
  names per batch instead of per entry.
- Drop three chunks indexes that only slow down writes: chunk_type (low
  cardinality), content_hash (never used as a filter) and
  (tenant_id, knowledge_id) (covered by idx_chunks_knowledge_enabled).
  Migrations 000100 (PostgreSQL) and 000020 (SQLite) with down scripts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* perf(chunk): rewrite chunk batch writes and FAQ import status updates

Go changes belonging to the chunk write-path optimization. The previous
commit on this branch only carried the migrations because the repo's
pre-commit hook re-staged files from the main checkout when run inside a
git worktree, silently dropping the staged Go changes.

- UpdateChunks: O(N²) CASE chains replaced by a VALUES join on PostgreSQL
  (batched by 1000) and per-row updates in one transaction elsewhere.
- CreateChunks: stop copying Content into SourceContent, batch 500 rows,
  run the insert and the SQLite seq_id pre-assignment in one transaction.
- New UpdateChunkFieldsByIDs for uniform, tenant-scoped field updates.
- FAQ import flips status with UpdateChunkFieldsByIDs and resolves tag
  names once per batch instead of once per entry.
- Drop the gorm index tag on ContentHash to match migration 000100;
  SQLite schema test expects migration version 20.
- SQLite tests for the new and rewritten repository methods.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-17 21:38:15 +08:00
langcaiye 5db13a131e
feat(faq): filter entries by enabled status (#3157)
* docs: design FAQ enabled-status filter

* feat(faq): filter entries by enabled status

* test(faq): cover disabled list filtering

* docs(faq): publish enabled filter contract
2026-09-10 14:42:59 +08:00
shangfeng-liu b6b1148bbb fix(chunk): sanitize context headers before persistence 2026-08-14 00:33:09 +08:00
wizardchen 7d2a6311cb fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
wizardchen eedd7d4641 feat(agent): enhance suggested questions API with tag scopes
Update the SuggestedQuestionsRequest to include TagScopes for better filtering of knowledge items based on their parent knowledge bases. Modify related API documentation and frontend interfaces to accommodate the new structure. Ensure backend services handle the new tag scope logic correctly, including tests for the updated functionality.
2026-07-17 10:19:43 +08:00
swim2sun d2c6814f22 fix: SQLite/Lite mode compatibility for FAQ import and chunk operations
Multiple fixes for WeKnora Lite mode (SQLite, no Redis):

1. Redis nil pointer panic on FAQ import (knowledge.go)
   - Add in-memory fallback (sync.Map) for FAQ import progress tracking
   - Add nil guards for running task locks and progress queries

2. seq_id not auto-incrementing in SQLite (chunk.go, tag.go)
   - GORM autoIncrement on non-PK columns doesn't work in SQLite
   - Add AssignChunkSeqIDs() to pre-assign seq_ids before batch insert
   - Add BeforeCreate hook for KnowledgeTag
   - Use Unscoped() to include soft-deleted records in MAX(seq_id) query

3. NOW() function not available in SQLite (chunk.go)
   - Replace raw SQL NOW() with time.Now() or datetime('now')
   - Use dialector check for raw SQL that must differ between PG and SQLite

4. FAQ KB should reject file uploads (knowledge.go)
   - Add kb.Type == "faq" check in CreateKnowledgeFromFile

5. Linux/macOS build compatibility (Makefile)
   - Conditionally apply macOS-only linker flag based on uname

6. Add SQLite integration tests (chunk_sqlite_test.go)
   - TestCreateChunks_SQLite_SeqIDAutoAssigned
   - TestCreateChunks_SQLite_SeqIDContinuesFromExisting
   - TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
   - TestCreateChunks_SQLite_SeqIDAfterSoftDelete
   - TestKnowledgeTag_SQLite_SeqIDAutoAssigned
   - TestUpdateChunk_SQLite_NoNOWError

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 20:37:10 +08:00