* perf(chunk): speed up chunk ingestion and batch updates
Chunk writes were dominated by avoidable work in the repository layer:
- UpdateChunks built one CASE WHEN id = ? chain per column, so a batch of
N rows evaluated 5N branches per row (O(N²)). On PostgreSQL it now joins
against a VALUES list keyed by the primary key (O(N), batched by 1000);
other dialects run one small UPDATE per row inside a single transaction.
- CreateChunks copied Content into SourceContent for every new row, doubling
insert volume. SourceContent is only needed once a chunk is edited and the
edit path already backfills it lazily, so it is left empty on create.
Batch size raised from 100 to 500 and the insert (including the SQLite
seq_id pre-assignment) now runs inside one transaction.
- New UpdateChunkFieldsByIDs applies uniform field values with a single
UPDATE ... WHERE id IN per batch. FAQ import uses it to flip status after
indexing instead of shipping every row's content back, and resolves tag
names per batch instead of per entry.
- Drop three chunks indexes that only slow down writes: chunk_type (low
cardinality), content_hash (never used as a filter) and
(tenant_id, knowledge_id) (covered by idx_chunks_knowledge_enabled).
Migrations 000100 (PostgreSQL) and 000020 (SQLite) with down scripts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perf(chunk): rewrite chunk batch writes and FAQ import status updates
Go changes belonging to the chunk write-path optimization. The previous
commit on this branch only carried the migrations because the repo's
pre-commit hook re-staged files from the main checkout when run inside a
git worktree, silently dropping the staged Go changes.
- UpdateChunks: O(N²) CASE chains replaced by a VALUES join on PostgreSQL
(batched by 1000) and per-row updates in one transaction elsewhere.
- CreateChunks: stop copying Content into SourceContent, batch 500 rows,
run the insert and the SQLite seq_id pre-assignment in one transaction.
- New UpdateChunkFieldsByIDs for uniform, tenant-scoped field updates.
- FAQ import flips status with UpdateChunkFieldsByIDs and resolves tag
names once per batch instead of once per entry.
- Drop the gorm index tag on ContentHash to match migration 000100;
SQLite schema test expects migration version 20.
- SQLite tests for the new and rewritten repository methods.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Update the SuggestedQuestionsRequest to include TagScopes for better filtering of knowledge items based on their parent knowledge bases. Modify related API documentation and frontend interfaces to accommodate the new structure. Ensure backend services handle the new tag scope logic correctly, including tests for the updated functionality.
Multiple fixes for WeKnora Lite mode (SQLite, no Redis):
1. Redis nil pointer panic on FAQ import (knowledge.go)
- Add in-memory fallback (sync.Map) for FAQ import progress tracking
- Add nil guards for running task locks and progress queries
2. seq_id not auto-incrementing in SQLite (chunk.go, tag.go)
- GORM autoIncrement on non-PK columns doesn't work in SQLite
- Add AssignChunkSeqIDs() to pre-assign seq_ids before batch insert
- Add BeforeCreate hook for KnowledgeTag
- Use Unscoped() to include soft-deleted records in MAX(seq_id) query
3. NOW() function not available in SQLite (chunk.go)
- Replace raw SQL NOW() with time.Now() or datetime('now')
- Use dialector check for raw SQL that must differ between PG and SQLite
4. FAQ KB should reject file uploads (knowledge.go)
- Add kb.Type == "faq" check in CreateKnowledgeFromFile
5. Linux/macOS build compatibility (Makefile)
- Conditionally apply macOS-only linker flag based on uname
6. Add SQLite integration tests (chunk_sqlite_test.go)
- TestCreateChunks_SQLite_SeqIDAutoAssigned
- TestCreateChunks_SQLite_SeqIDContinuesFromExisting
- TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
- TestCreateChunks_SQLite_SeqIDAfterSoftDelete
- TestKnowledgeTag_SQLite_SeqIDAutoAssigned
- TestUpdateChunk_SQLite_NoNOWError
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>