2025-08-05 07:08:07 +00:00
package handler
import (
2026-01-30 08:28:01 +00:00
"context"
2025-08-05 07:08:07 +00:00
"encoding/json"
2026-09-08 07:36:33 +00:00
goerrors "errors"
2025-08-05 07:08:07 +00:00
"fmt"
"net/http"
2025-08-08 09:19:06 +00:00
"strconv"
2025-12-31 10:18:22 +00:00
"strings"
2026-03-09 05:14:42 +00:00
"time"
2025-08-05 07:08:07 +00:00
2026-09-08 07:36:33 +00:00
"github.com/Tencent/WeKnora/internal/application/access"
2026-03-09 05:14:42 +00:00
"github.com/Tencent/WeKnora/internal/application/repository"
2026-07-02 06:49:27 +00:00
"github.com/Tencent/WeKnora/internal/config"
2025-08-05 07:08:07 +00:00
"github.com/Tencent/WeKnora/internal/errors"
2026-09-08 07:36:33 +00:00
"github.com/Tencent/WeKnora/internal/filetransport"
2025-08-05 07:08:07 +00:00
"github.com/Tencent/WeKnora/internal/logger"
2026-07-02 06:49:27 +00:00
"github.com/Tencent/WeKnora/internal/middleware"
feat(observability): extend Langfuse tracing across asynq pipeline
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.
This change threads one trace end-to-end:
- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
worker can attach to an upstream trace without emitting a duplicate
trace-create; StartGeneration now auto-picks parentObservationId from
ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
payloads so trace_id / parent_obs_id / user_id / session_id serialise
into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
resumes the upstream trace or opens a standalone asynq.<type> trace
for scheduled jobs, and wraps the handler in a SPAN with task metadata
(id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
ResumeTrace no-trace-create guarantee; asynq_test.go covers
InjectTracing round-trip, middleware resume path, and standalone
trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
and documents the cross-process propagation model.
No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
2026-04-24 04:23:30 +00:00
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
2025-08-05 07:08:07 +00:00
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
2026-03-09 05:14:42 +00:00
"github.com/Tencent/WeKnora/internal/utils"
2025-11-25 14:19:41 +00:00
secutils "github.com/Tencent/WeKnora/internal/utils"
2025-08-05 07:08:07 +00:00
"github.com/gin-gonic/gin"
2026-03-09 05:14:42 +00:00
"github.com/hibiken/asynq"
2025-08-05 07:08:07 +00:00
)
// KnowledgeHandler processes HTTP requests related to knowledge resources
type KnowledgeHandler struct {
2026-07-02 06:49:27 +00:00
cfg * config . Config
2026-02-10 04:15:21 +00:00
kgService interfaces . KnowledgeService
kbService interfaces . KnowledgeBaseService
kbShareService interfaces . KBShareService
2026-02-04 12:12:29 +00:00
agentShareService interfaces . AgentShareService
2026-03-09 05:14:42 +00:00
asynqClient interfaces . TaskEnqueuer
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
spanRepo repository . KnowledgeSpanRepository
2025-08-05 07:08:07 +00:00
}
// NewKnowledgeHandler creates a new knowledge handler instance
func NewKnowledgeHandler (
2026-07-02 06:49:27 +00:00
cfg * config . Config ,
2025-08-05 07:08:07 +00:00
kgService interfaces . KnowledgeService ,
kbService interfaces . KnowledgeBaseService ,
2026-01-30 08:28:01 +00:00
kbShareService interfaces . KBShareService ,
2026-02-04 12:12:29 +00:00
agentShareService interfaces . AgentShareService ,
2026-03-09 05:14:42 +00:00
asynqClient interfaces . TaskEnqueuer ,
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
spanRepo repository . KnowledgeSpanRepository ,
2025-08-05 07:08:07 +00:00
) * KnowledgeHandler {
2026-01-30 08:28:01 +00:00
return & KnowledgeHandler {
2026-07-02 06:49:27 +00:00
cfg : cfg ,
2026-02-10 04:15:21 +00:00
kgService : kgService ,
kbService : kbService ,
kbShareService : kbShareService ,
2026-02-04 12:12:29 +00:00
agentShareService : agentShareService ,
2026-03-09 05:14:42 +00:00
asynqClient : asynqClient ,
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
spanRepo : spanRepo ,
2026-01-30 08:28:01 +00:00
}
2025-08-05 07:08:07 +00:00
}
2026-07-02 06:49:27 +00:00
// requireKBOwnershipOrAdmin enforces the same "KB creator OR Admin+" matrix
// used by OwnedKBOrAdmin for routes whose KB id comes from the request body.
func ( h * KnowledgeHandler ) requireKBOwnershipOrAdmin ( c * gin . Context , kbID string ) error {
evalErr := middleware . EvaluateOwnershipOrRole (
c . Request . Context ( ) ,
h . cfg ,
types . TenantRoleAdmin ,
2026-09-08 07:36:33 +00:00
func ( ) ( string , error ) { return resolveKBCreatorByKBID ( c , h . kbService , kbID ) } ,
2026-07-02 06:49:27 +00:00
)
if evalErr == nil {
return nil
}
if goerrors . Is ( evalErr , middleware . ErrResourceNotFound ) {
return errors . NewNotFoundError ( "knowledge base not found" )
}
if goerrors . Is ( evalErr , middleware . ErrOwnershipForbidden ) {
return errors . NewForbiddenError ( "No permission to operate on this knowledge base" )
}
logger . ErrorWithFields ( c . Request . Context ( ) , evalErr , map [ string ] interface { } {
"kb_id" : secutils . SanitizeForLog ( kbID ) ,
} )
return errors . NewInternalServerError ( "cannot verify knowledge base ownership" )
}
2025-08-05 07:08:07 +00:00
// validateKnowledgeBaseAccess validates access permissions to a knowledge base
2026-02-28 10:11:37 +00:00
// using the ":id" URL path parameter. It delegates to validateKnowledgeBaseAccessWithKBID.
2026-01-30 08:28:01 +00:00
func ( h * KnowledgeHandler ) validateKnowledgeBaseAccess ( c * gin . Context ) ( * types . KnowledgeBase , string , uint64 , types . OrgMemberRole , error ) {
2025-11-26 02:50:53 +00:00
kbID := secutils . SanitizeForLog ( c . Param ( "id" ) )
2026-02-28 10:11:37 +00:00
return h . validateKnowledgeBaseAccessWithKBID ( c , kbID )
2026-01-30 08:28:01 +00:00
}
// validateKnowledgeBaseAccessWithKBID validates access to the given knowledge base ID (e.g. from query or body).
2026-07-05 02:55:00 +00:00
// Enforces per-API-key KB scope before tenant/share/agent resolution.
2026-01-30 08:28:01 +00:00
// Returns the knowledge base, kbID, effective tenant ID, permission, and error.
2026-09-08 07:36:33 +00:00
func ( h * KnowledgeHandler ) validateKnowledgeBaseAccessWithKBID (
c * gin . Context ,
kbID string ,
) ( * types . KnowledgeBase , string , uint64 , types . OrgMemberRole , error ) {
2026-01-30 08:28:01 +00:00
kbID = secutils . SanitizeForLog ( kbID )
2026-09-08 07:36:33 +00:00
grant , err := resolveHandlerKBAccess ( c , kbID , h . kbService , h . kbShareService , h . agentShareService )
if err != nil {
2026-07-05 02:55:00 +00:00
return nil , kbID , 0 , "" , err
}
2026-09-08 07:36:33 +00:00
return grant . KnowledgeBase , kbID , grant . EffectiveTenantID , grant . Permission , nil
}
func ( h * KnowledgeHandler ) validateKnowledgeBaseWriteAccessWithKBID (
c * gin . Context ,
kbID string ,
) ( * types . KnowledgeBase , string , uint64 , types . OrgMemberRole , error ) {
grant , err := resolveHandlerKBAccessFor (
c ,
kbID ,
h . kbService ,
h . kbShareService ,
h . agentShareService ,
types . OrgRoleEditor ,
)
2026-01-30 08:28:01 +00:00
if err != nil {
2026-09-08 07:36:33 +00:00
return nil , kbID , 0 , "" , err
2025-08-05 07:08:07 +00:00
}
2026-09-08 07:36:33 +00:00
if err := access . RequireKBWrite ( grant . Context ( c . Request . Context ( ) ) , grant . KnowledgeBase ) ; err != nil {
return nil , kbID , 0 , "" , kbAccessHTTPError ( err )
2026-02-08 06:32:18 +00:00
}
2026-09-08 07:36:33 +00:00
return grant . KnowledgeBase , kbID , grant . EffectiveTenantID , grant . Permission , nil
2026-01-30 08:28:01 +00:00
}
2026-09-08 07:36:33 +00:00
// resolveKnowledgeAndValidateKBAccess resolves a document and checks its parent
// KB using the same policy as KB routes. The persisted document supplies the KB
// and owner reference, so unguarded callers do not need an additional KB lookup.
func ( h * KnowledgeHandler ) resolveKnowledgeAndValidateKBAccess (
c * gin . Context ,
knowledgeID string ,
requiredPermission types . OrgMemberRole ,
) ( * types . Knowledge , context . Context , error ) {
2026-01-30 08:28:01 +00:00
ctx := c . Request . Context ( )
2026-09-08 07:36:33 +00:00
request := middleware . KBAccessRequest ( c )
if request . Caller . TenantID == 0 {
2026-01-30 08:28:01 +00:00
return nil , ctx , errors . NewUnauthorizedError ( "Unauthorized" )
}
knowledge , err := h . kgService . GetKnowledgeByIDOnly ( ctx , knowledgeID )
2026-09-08 07:36:33 +00:00
if err != nil || knowledge == nil {
2026-01-30 08:28:01 +00:00
return nil , ctx , errors . NewNotFoundError ( "Knowledge not found" )
}
2026-07-05 02:55:00 +00:00
if err := requireTenantAPIKeyKnowledgeBase ( ctx , knowledge . KnowledgeBaseID ) ; err != nil {
return nil , ctx , err
}
2026-09-08 07:36:33 +00:00
if grant , ok := resolvedKBAccess ( c , knowledge . KnowledgeBaseID , requiredPermission ) ; ok {
if grant . EffectiveTenantID != knowledge . TenantID {
return nil , ctx , errors . NewForbiddenError ( "Permission denied to access this knowledge" )
2026-02-04 12:12:29 +00:00
}
2026-09-08 07:36:33 +00:00
return knowledge , grant . Context ( ctx ) , nil
2026-01-30 08:28:01 +00:00
}
2026-09-08 07:36:33 +00:00
kb := & types . KnowledgeBase { ID : knowledge . KnowledgeBaseID , TenantID : knowledge . TenantID }
grant , err := access . ResolveKB ( ctx , request , kb , requiredPermission , h . kbShareService , h . agentShareService )
if goerrors . Is ( err , access . ErrForbidden ) {
return nil , ctx , errors . NewForbiddenError ( "Permission denied to access this knowledge" )
}
if err != nil {
return nil , ctx , kbAccessHTTPError ( err )
2026-01-30 08:28:01 +00:00
}
2026-09-08 07:36:33 +00:00
return knowledge , grant . Context ( ctx ) , nil
2025-08-05 07:08:07 +00:00
}
// handleDuplicateKnowledgeError handles cases where duplicate knowledge is detected
// Returns true if the error was a duplicate error and was handled, false otherwise
func ( h * KnowledgeHandler ) handleDuplicateKnowledgeError ( c * gin . Context ,
err error , knowledge * types . Knowledge , duplicateType string ,
) bool {
if dupErr , ok := err . ( * types . DuplicateKnowledgeError ) ; ok {
ctx := c . Request . Context ( )
2025-11-25 15:33:34 +00:00
logger . Warnf ( ctx , "Detected duplicate %s: %s" , duplicateType , secutils . SanitizeForLog ( dupErr . Error ( ) ) )
2025-08-05 07:08:07 +00:00
c . JSON ( http . StatusConflict , gin . H {
"success" : false ,
"message" : dupErr . Error ( ) ,
"data" : knowledge , // knowledge contains the existing document
"code" : fmt . Sprintf ( "duplicate_%s" , duplicateType ) ,
} )
return true
}
return false
}
2026-04-29 03:47:59 +00:00
// enqueueKnowledgeListDelete enqueues an async batch-delete task for the
// given knowledge IDs and returns the asynq task ID.
func ( h * KnowledgeHandler ) enqueueKnowledgeListDelete (
2026-09-08 07:36:33 +00:00
ctx context . Context , tenantID uint64 , kbID string , ids [ ] string ,
2026-04-29 03:47:59 +00:00
) ( string , error ) {
payload := types . KnowledgeListDeletePayload {
2026-09-08 07:36:33 +00:00
KnowledgeBaseID : kbID ,
TenantID : tenantID ,
KnowledgeIDs : ids ,
Initiator : types . TaskInitiatorFromContext ( ctx ) ,
2026-04-29 03:47:59 +00:00
}
langfuse . InjectTracing ( ctx , & payload )
payloadBytes , err := json . Marshal ( payload )
if err != nil {
return "" , fmt . Errorf ( "marshal payload: %w" , err )
}
task := asynq . NewTask ( types . TypeKnowledgeListDelete , payloadBytes ,
2026-07-10 15:19:41 +00:00
asynq . Queue ( types . QueueMaintenance ) , asynq . MaxRetry ( 3 ) , asynq . Timeout ( 2 * time . Hour ) )
2026-04-29 03:47:59 +00:00
info , err := h . asynqClient . Enqueue ( task )
if err != nil {
return "" , fmt . Errorf ( "enqueue task: %w" , err )
}
return info . ID , nil
}
2026-06-25 15:51:38 +00:00
// enqueueKnowledgeListReparse enqueues an async batch-reparse task for the
// given knowledge IDs and returns the asynq task ID.
func ( h * KnowledgeHandler ) enqueueKnowledgeListReparse (
2026-09-08 07:36:33 +00:00
ctx context . Context , tenantID uint64 , kbID string , ids [ ] string , processConfig * types . KnowledgeProcessOverrides ,
2026-06-25 15:51:38 +00:00
) ( string , error ) {
payload := types . KnowledgeListReparsePayload {
2026-09-08 07:36:33 +00:00
KnowledgeBaseID : kbID ,
TenantID : tenantID ,
KnowledgeIDs : ids ,
ProcessConfig : processConfig ,
Initiator : types . TaskInitiatorFromContext ( ctx ) ,
2026-06-25 15:51:38 +00:00
}
langfuse . InjectTracing ( ctx , & payload )
payloadBytes , err := json . Marshal ( payload )
if err != nil {
return "" , fmt . Errorf ( "marshal payload: %w" , err )
}
task := asynq . NewTask ( types . TypeKnowledgeListReparse , payloadBytes ,
2026-07-10 15:19:41 +00:00
asynq . Queue ( types . QueueMaintenance ) , asynq . MaxRetry ( 3 ) , asynq . Timeout ( time . Hour ) )
2026-06-25 15:51:38 +00:00
info , err := h . asynqClient . Enqueue ( task )
if err != nil {
return "" , fmt . Errorf ( "enqueue task: %w" , err )
}
return info . ID , nil
}
2025-12-16 08:40:13 +00:00
// CreateKnowledgeFromFile godoc
// @Summary 从文件创建知识
// @Description 上传文件并创建知识条目
// @Tags 知识管理
// @Accept multipart/form-data
// @Produce json
// @Param id path string true "知识库ID"
// @Param file formData file true "上传的文件"
// @Param fileName formData string false "自定义文件名"
// @Param metadata formData string false "元数据JSON"
// @Param enable_multimodel formData bool false "启用多模态处理"
2026-06-25 18:18:10 +00:00
// @Param tag_ids formData string false "分类ID列表, 逗号分隔"
2026-06-09 09:49:39 +00:00
// @Param process_config formData string false "处理配置JSON( KnowledgeProcessOverrides) "
2025-12-16 08:40:13 +00:00
// @Success 200 {object} map[string]interface{} "创建的知识"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 409 {object} map[string]interface{} "文件重复"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge-bases/{id}/knowledge/file [post]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) CreateKnowledgeFromFile ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start creating knowledge from file" )
2026-01-30 08:28:01 +00:00
// Validate access to the knowledge base (only owner or admin/editor can create)
_ , kbID , effectiveTenantID , permission , err := h . validateKnowledgeBaseAccess ( c )
2025-08-05 07:08:07 +00:00
if err != nil {
c . Error ( err )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-01-30 08:28:01 +00:00
// Check write permission
if permission != types . OrgRoleAdmin && permission != types . OrgRoleEditor {
c . Error ( errors . NewForbiddenError ( "No permission to create knowledge" ) )
return
}
2025-08-05 07:08:07 +00:00
2026-09-02 03:49:19 +00:00
// Validate file size — read MAX_FILE_SIZE_MB env (50MB default).
// Deliberately not a runtime system_setting; see filesize.go for the
// rationale (nginx / docreader / browser bundle all cache this at
// container startup, so a UI knob would silently mismatch).
maxSizeMB := utils . GetMaxFileSizeMB ( )
maxSize := maxSizeMB * 1024 * 1024
// Capped before the multipart parse, not after: FormFile buffers the whole
// body first, so the size check below only ever sees an upload we already
2026-09-02 04:07:06 +00:00
// accepted. nginx location /api/ still enforces MAX_FILE_SIZE; this is the
// same cap for requests that reach the app without that proxy.
2026-09-02 03:49:19 +00:00
limitUploadBody ( c , maxSize )
2025-08-05 07:08:07 +00:00
// Get the uploaded file
file , err := c . FormFile ( "file" )
if err != nil {
2026-09-02 03:49:19 +00:00
if isRequestBodyTooLarge ( err ) {
logger . Error ( ctx , "File size too large" )
c . Error ( errors . NewBadRequestError ( fmt . Sprintf ( "文件大小不能超过%dMB" , maxSizeMB ) ) )
return
}
2025-08-05 07:08:07 +00:00
logger . Error ( ctx , "File upload failed" , err )
c . Error ( errors . NewBadRequestError ( "File upload failed" ) . WithDetails ( err . Error ( ) ) )
return
}
2025-12-30 03:08:05 +00:00
if file . Size > maxSize {
logger . Error ( ctx , "File size too large" )
2026-05-24 13:38:05 +00:00
c . Error ( errors . NewBadRequestError ( fmt . Sprintf ( "文件大小不能超过%dMB" , maxSizeMB ) ) )
2025-12-30 03:08:05 +00:00
return
}
2025-11-23 09:27:30 +00:00
// Get custom filename if provided (for folder uploads with path)
customFileName := c . PostForm ( "fileName" )
2025-11-26 02:50:53 +00:00
customFileName = secutils . SanitizeForLog ( customFileName )
2025-11-23 09:27:30 +00:00
displayFileName := file . Filename
2025-11-26 02:50:53 +00:00
displayFileName = secutils . SanitizeForLog ( displayFileName )
2025-11-23 09:27:30 +00:00
if customFileName != "" {
displayFileName = customFileName
2025-11-26 02:50:53 +00:00
logger . Infof ( ctx , "Using custom filename: %s (original: %s)" , customFileName , displayFileName )
2025-11-23 09:27:30 +00:00
}
2025-11-26 02:50:53 +00:00
logger . Infof ( ctx , "File upload successful, filename: %s, size: %.2f KB" , displayFileName , float64 ( file . Size ) / 1024 )
logger . Infof ( ctx , "Creating knowledge, knowledge base ID: %s, filename: %s" , kbID , displayFileName )
2025-08-05 07:08:07 +00:00
// Parse metadata if provided
var metadata map [ string ] string
metadataStr := c . PostForm ( "metadata" )
if metadataStr != "" {
if err := json . Unmarshal ( [ ] byte ( metadataStr ) , & metadata ) ; err != nil {
logger . Error ( ctx , "Failed to parse metadata" , err )
c . Error ( errors . NewBadRequestError ( "Invalid metadata format" ) . WithDetails ( err . Error ( ) ) )
return
}
2025-11-25 15:33:34 +00:00
logger . Infof ( ctx , "Received file metadata: %s" , secutils . SanitizeForLog ( fmt . Sprintf ( "%v" , metadata ) ) )
2025-08-05 07:08:07 +00:00
}
2025-08-08 09:19:06 +00:00
enableMultimodelForm := c . PostForm ( "enable_multimodel" )
var enableMultimodel * bool
if enableMultimodelForm != "" {
parseBool , err := strconv . ParseBool ( enableMultimodelForm )
if err != nil {
logger . Error ( ctx , "Failed to parse enable_multimodel" , err )
c . Error ( errors . NewBadRequestError ( "Invalid enable_multimodel format" ) . WithDetails ( err . Error ( ) ) )
return
}
enableMultimodel = & parseBool
}
2026-06-09 09:49:39 +00:00
var processOverrides * types . KnowledgeProcessOverrides
if raw := c . PostForm ( "process_config" ) ; raw != "" {
processOverrides = & types . KnowledgeProcessOverrides { }
if err := json . Unmarshal ( [ ] byte ( raw ) , processOverrides ) ; err != nil {
logger . Error ( ctx , "Failed to parse process_config" , err )
c . Error ( errors . NewBadRequestError ( "Invalid process_config format" ) . WithDetails ( err . Error ( ) ) )
return
}
}
if enableMultimodel != nil && ( processOverrides == nil || processOverrides . EnableMultimodel == nil ) {
if processOverrides == nil {
processOverrides = & types . KnowledgeProcessOverrides { EnableMultimodel : enableMultimodel }
} else {
processOverrides . EnableMultimodel = enableMultimodel
}
}
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-25 16:18:48 +00:00
// 获取分类ID列表( 如果提供) , 逗号分隔, 用于知识多标签分类管理
tagIDs := parseCommaSeparatedTagIDs ( c . PostForm ( "tag_ids" ) )
2026-01-14 10:51:12 +00:00
2026-03-24 11:50:13 +00:00
channel := c . PostForm ( "channel" )
2025-08-05 07:08:07 +00:00
// Create knowledge entry from the file
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-25 16:18:48 +00:00
knowledge , err := h . kgService . CreateKnowledgeFromFile ( ctx , kbID , file , metadata , enableMultimodel , customFileName , tagIDs , channel , processOverrides )
2025-08-05 07:08:07 +00:00
// Check for duplicate knowledge error
if err != nil {
if h . handleDuplicateKnowledgeError ( c , err , knowledge , "file" ) {
return
}
2025-08-10 09:04:39 +00:00
if appErr , ok := errors . IsAppError ( err ) ; ok {
c . Error ( appErr )
return
}
2025-08-05 07:08:07 +00:00
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
2025-11-28 08:17:52 +00:00
logger . Infof (
ctx ,
"Knowledge created successfully, ID: %s, title: %s" ,
secutils . SanitizeForLog ( knowledge . ID ) ,
secutils . SanitizeForLog ( knowledge . Title ) ,
)
2025-08-05 07:08:07 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : knowledge ,
} )
}
2025-12-16 08:40:13 +00:00
// CreateKnowledgeFromURL godoc
// @Summary 从URL创建知识
2026-02-26 07:26:03 +00:00
// @Description 从指定URL抓取内容并创建知识条目。当提供 file_name/file_type 或 URL 路径含已知文件扩展名时,自动切换为文件下载模式
2025-12-16 08:40:13 +00:00
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识库ID"
2026-06-25 18:18:10 +00:00
// @Param request body object{url=string,file_name=string,file_type=string,enable_multimodel=bool,title=string,tag_ids=[]string} true "URL请求"
2025-12-16 08:40:13 +00:00
// @Success 201 {object} map[string]interface{} "创建的知识"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 409 {object} map[string]interface{} "URL重复"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge-bases/{id}/knowledge/url [post]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) CreateKnowledgeFromURL ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start creating knowledge from URL" )
2026-01-30 08:28:01 +00:00
// Validate access to the knowledge base (only owner or admin/editor can create)
_ , kbID , effectiveTenantID , permission , err := h . validateKnowledgeBaseAccess ( c )
2025-08-05 07:08:07 +00:00
if err != nil {
c . Error ( err )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-01-30 08:28:01 +00:00
// Check write permission
if permission != types . OrgRoleAdmin && permission != types . OrgRoleEditor {
c . Error ( errors . NewForbiddenError ( "No permission to create knowledge" ) )
return
}
2025-08-05 07:08:07 +00:00
// Parse URL from request body
var req struct {
2026-06-09 09:49:39 +00:00
URL string ` json:"url" binding:"required" `
FileName string ` json:"file_name" `
FileType string ` json:"file_type" `
EnableMultimodel * bool ` json:"enable_multimodel" `
Title string ` json:"title" `
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-25 16:18:48 +00:00
TagIDs [ ] string ` json:"tag_ids" `
2026-06-09 09:49:39 +00:00
Channel string ` json:"channel" `
ProcessConfig * types . KnowledgeProcessOverrides ` json:"process_config" `
2025-08-05 07:08:07 +00:00
}
if err := c . ShouldBindJSON ( & req ) ; err != nil {
logger . Error ( ctx , "Failed to parse URL request" , err )
c . Error ( errors . NewBadRequestError ( err . Error ( ) ) )
return
}
2026-02-26 07:26:03 +00:00
logger . Infof ( ctx , "Received URL request: %s, file_name: %s, file_type: %s" ,
secutils . SanitizeForLog ( req . URL ) ,
secutils . SanitizeForLog ( req . FileName ) ,
secutils . SanitizeForLog ( req . FileType ) ,
)
2026-03-17 07:33:19 +00:00
// SSRF validation for user-supplied URL
if err := secutils . ValidateURLForSSRF ( req . URL ) ; err != nil {
logger . Warnf ( ctx , "SSRF validation failed for knowledge URL: %v" , err )
fix(initialization): surface upstream and SSRF errors verbatim in test-connection responses
Test-connection's "连接失败" was nearly useless for debugging because
two layers conspired to swallow the actual upstream error:
Backend swallow
---------------
checkChatModelConnection and CheckASRModel pattern-matched on the
upstream error string (401 / 404 / timeout / ...) and replaced it with
a fixed Chinese hint. The original SDK message (request URL, response
body, provider's error code description) was discarded.
Refactor: extract classifyConnectionError(errMsg) — returns just the
short Chinese hint. Callers combine "<hint>:<raw err>" so the hint
indicates "where to start looking" and the raw error shows "what
actually happened". The status-400 special-case (endpoint reachable,
parameter mismatch only) still returns success.
Frontend swallow
----------------
ModelEditorDialog.testConnection ignored result.message on success
paths (fine), but also overwrote it with a generic i18n string on
failure paths and 4xx rejections — even though axios's interceptor
already lifts the backend's error.message to error.message. Failure
branch now uses error?.message || result.message as the toast content;
i18n fallback only when neither is present.
SSRF allowlist hint
-------------------
SSRF validation failures previously returned
"Base URL 未通过安全校验: <reason>" with no mention of how to recover
(the allowlist is server-side, not configurable in the UI). Operators
hit it and were stuck.
New utils.FormatSSRFError(label, rawURL, err) wraps the error with
guidance to add the host to SSRF_WHITELIST_EXTRA (the operator-facing
append-only escape hatch — SSRF_WHITELIST itself is the project's
baseline). The hint echoes the failed host so it can be copy-pasted
straight into the env var:
Base URL 未通过安全校验:DNS resolution failed for hostname
test.local.com ... 如该地址确实可信,请联系运维在服务端环境变量
SSRF_WHITELIST_EXTRA 中加入该主机(支持精确域名 / *.example.com
通配 / IP / CIDR),示例:SSRF_WHITELIST_EXTRA=test.local.com,
*.example.com,10.0.0.0/8
All SSRF call sites switch to the helper:
initialization (CheckRemoteModel / TestEmbedding / CheckRerank /
CheckASR / TestMultimodal / batch validation), model CRUD,
mcp_service CRUD, knowledge URL ingest, system ReconnectDocReader.
2026-05-17 07:10:42 +00:00
c . Error ( errors . NewBadRequestError ( secutils . FormatSSRFError ( "URL" , req . URL , err ) ) )
2026-03-17 07:33:19 +00:00
return
}
2026-02-26 07:26:03 +00:00
logger . Infof ( ctx ,
2025-11-28 08:17:52 +00:00
"Creating knowledge from URL, knowledge base ID: %s, URL: %s" ,
secutils . SanitizeForLog ( kbID ) ,
secutils . SanitizeForLog ( req . URL ) ,
)
2025-08-05 07:08:07 +00:00
// Create knowledge entry from the URL
2026-06-09 09:49:39 +00:00
knowledge , err := h . kgService . CreateKnowledgeFromURL (
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-25 16:18:48 +00:00
ctx , kbID , req . URL , req . FileName , req . FileType , req . EnableMultimodel , req . Title , req . TagIDs , req . Channel , req . ProcessConfig ,
2026-06-09 09:49:39 +00:00
)
2025-08-05 07:08:07 +00:00
// Check for duplicate knowledge error
if err != nil {
if h . handleDuplicateKnowledgeError ( c , err , knowledge , "url" ) {
return
}
2026-02-26 06:28:14 +00:00
if appErr , ok := errors . IsAppError ( err ) ; ok {
c . Error ( appErr )
return
}
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
2026-02-26 07:26:03 +00:00
logger . Infof (
ctx ,
"Knowledge created successfully from URL, ID: %s, title: %s" ,
2026-02-26 06:28:14 +00:00
secutils . SanitizeForLog ( knowledge . ID ) ,
secutils . SanitizeForLog ( knowledge . Title ) ,
)
c . JSON ( http . StatusCreated , gin . H {
"success" : true ,
"data" : knowledge ,
} )
}
2025-12-16 08:40:13 +00:00
// CreateManualKnowledge godoc
// @Summary 手工创建知识
// @Description 手工录入Markdown格式的知识内容
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识库ID"
// @Param request body types.ManualKnowledgePayload true "手工知识内容"
// @Success 200 {object} map[string]interface{} "创建的知识"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge-bases/{id}/knowledge/manual [post]
2025-11-12 13:27:56 +00:00
func ( h * KnowledgeHandler ) CreateManualKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start creating manual knowledge" )
2026-01-30 08:28:01 +00:00
// Validate access to the knowledge base (only owner or admin/editor can create)
_ , kbID , effectiveTenantID , permission , err := h . validateKnowledgeBaseAccess ( c )
2025-11-12 13:27:56 +00:00
if err != nil {
c . Error ( err )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-01-30 08:28:01 +00:00
// Check write permission
if permission != types . OrgRoleAdmin && permission != types . OrgRoleEditor {
c . Error ( errors . NewForbiddenError ( "No permission to create knowledge" ) )
return
}
2025-11-12 13:27:56 +00:00
var req types . ManualKnowledgePayload
if err := c . ShouldBindJSON ( & req ) ; err != nil {
logger . Error ( ctx , "Failed to parse manual knowledge request" , err )
c . Error ( errors . NewBadRequestError ( err . Error ( ) ) )
return
}
2026-03-24 11:50:13 +00:00
knowledge , err := h . kgService . CreateKnowledgeFromManual ( ctx , kbID , & req , req . Channel )
2025-11-12 13:27:56 +00:00
if err != nil {
if appErr , ok := errors . IsAppError ( err ) ; ok {
c . Error ( appErr )
return
}
logger . ErrorWithFields ( ctx , err , map [ string ] interface { } {
"kb_id" : kbID ,
} )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
2025-11-26 11:52:07 +00:00
logger . Infof ( ctx , "Manual knowledge created successfully, knowledge ID: %s" ,
secutils . SanitizeForLog ( knowledge . ID ) )
2025-11-12 13:27:56 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : knowledge ,
} )
}
2025-12-16 08:40:13 +00:00
// GetKnowledge godoc
// @Summary 获取知识详情
// @Description 根据ID获取知识条目详情
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识ID"
// @Success 200 {object} map[string]interface{} "知识详情"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 404 {object} errors.AppError "知识不存在"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge/{id} [get]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) GetKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start retrieving knowledge" )
2025-11-25 16:19:26 +00:00
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
2025-08-05 07:08:07 +00:00
if id == "" {
logger . Error ( ctx , "Knowledge ID is empty" )
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
2026-01-30 08:28:01 +00:00
// Resolve knowledge and validate KB access (at least viewer)
2026-06-26 04:23:27 +00:00
knowledge , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleViewer )
2025-08-05 07:08:07 +00:00
if err != nil {
2026-01-30 08:28:01 +00:00
c . Error ( err )
2025-08-05 07:08:07 +00:00
return
}
2026-06-26 04:23:27 +00:00
// Re-fetch with tenant-scoped service so tags and other joined fields are populated.
if knowledge , err = h . kgService . GetKnowledgeByID ( effCtx , id ) ; err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewNotFoundError ( "Knowledge not found" ) )
return
}
2026-01-30 08:28:01 +00:00
logger . Infof ( ctx , "Knowledge retrieved successfully, ID: %s, title: %s" ,
secutils . SanitizeForLog ( knowledge . ID ) , secutils . SanitizeForLog ( knowledge . Title ) )
2025-08-05 07:08:07 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : knowledge ,
} )
}
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
// GetKnowledgeSpans godoc
// @Summary 获取知识文档解析的 Span 树(含历史尝试)
// @Description 返回该知识在解析流水线的 trace tree( root → stage → subspan) : 每段状态、耗时、input/output、错误码、langfuse_trace_id。支持 ?attempt=N 查看历史尝试;不传则返回最新尝试。前端用于渲染时间线 + 多模态/embedding 子节点 + 一键跳转 Langfuse。
2026-05-27 05:31:06 +00:00
// @Tags 知识管理
// @Accept json
// @Produce json
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
// @Param id path string true "知识ID"
// @Param attempt query int false "指定尝试号;省略=最新"
// @Success 200 {object} map[string]interface{}
// @Router /api/v1/knowledge/{id}/spans [get]
2026-05-27 05:31:06 +00:00
//
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
// Always returns the canonical 5-stage timeline; missing stage rows are
2026-05-27 05:31:06 +00:00
// synthesized as "pending" so the frontend timeline always renders five
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
// segments. Subspans (multimodal.image[i], generation.*) ride along under
// each stage as children when present.
func ( h * KnowledgeHandler ) GetKnowledgeSpans ( c * gin . Context ) {
2026-05-27 05:31:06 +00:00
ctx := c . Request . Context ( )
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
if id == "" {
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
knowledge , _ , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleViewer )
if err != nil {
c . Error ( err )
return
}
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
// Pick attempt: explicit ?attempt=N wins; otherwise pull the
// latest attempt from the spans table. Lite-mode / fresh installs
// with zero rows fall through to attempt=0, in which case we
// return a placeholder tree (5 pending stages, no root, no
// children) so the UI still renders.
requestedAttempt := 0
if v := strings . TrimSpace ( c . Query ( "attempt" ) ) ; v != "" {
if n , perr := strconv . Atoi ( v ) ; perr == nil && n > 0 {
requestedAttempt = n
2026-05-27 05:31:06 +00:00
}
}
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
rows := [ ] types . KnowledgeProcessingSpan { }
currentAttempt := 0
2026-07-14 08:40:49 +00:00
latestAttempt := 0
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
if h . spanRepo != nil {
2026-07-14 08:40:49 +00:00
latest , lerr := h . spanRepo . LatestAttempt ( ctx , knowledge . ID )
if lerr != nil {
logger . Warnf ( ctx , "spans LatestAttempt failed for %s: %v" , knowledge . ID , lerr )
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
} else {
2026-07-14 08:40:49 +00:00
latestAttempt = latest
}
if requestedAttempt > 0 {
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
currentAttempt = requestedAttempt
2026-07-14 08:40:49 +00:00
} else {
currentAttempt = latestAttempt
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
}
if currentAttempt > 0 {
rows , err = h . spanRepo . ListByAttempt ( ctx , knowledge . ID , currentAttempt )
if err != nil {
logger . Warnf ( ctx , "spans ListByAttempt failed kid=%s attempt=%d: %v" ,
knowledge . ID , currentAttempt , err )
rows = nil
2026-05-27 05:31:06 +00:00
}
}
}
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
// Build tree: index by SpanID, then attach to parents. Stages
// missing from the DB are synthesized as "pending" placeholders
// under a synthetic (or real, if present) root so the timeline
2026-05-27 09:35:58 +00:00
// always renders five segments. parse_status threads through so
// pre-tracker historical knowledge (no rows but parse_status is
// already terminal) renders as done/failed instead of pending —
// otherwise legacy completed documents would forever look like
// they're still waiting in the queue.
tree , currentStageName , lastErr := buildSpanTree ( knowledge . ID , currentAttempt , rows , knowledge . ParseStatus )
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
2026-05-27 05:31:06 +00:00
resp := gin . H {
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
"knowledge_id" : knowledge . ID ,
2026-07-14 08:40:49 +00:00
"attempt" : currentAttempt ,
"latest_attempt" : latestAttempt ,
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
"parse_status" : knowledge . ParseStatus ,
"current_attempt" : currentAttempt ,
"current_stage" : currentStageName ,
"trace" : tree ,
2026-05-27 05:31:06 +00:00
}
2026-07-14 08:48:30 +00:00
if lastError := knowledgeSpansLastError (
currentAttempt ,
latestAttempt ,
knowledge . ParseStatus ,
knowledge . ErrorMessage ,
knowledge . UpdatedAt ,
lastErr ,
) ; lastError != nil {
resp [ "last_error" ] = lastError
2026-05-27 05:31:06 +00:00
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : resp ,
} )
}
2026-07-14 08:48:30 +00:00
// knowledgeSpansLastError builds the last_error payload for GetKnowledgeSpans.
// Span failures win when present; otherwise a failed knowledge row without a
// matching span error (recovery / dead-letter paths) surfaces ErrorMessage.
func knowledgeSpansLastError (
currentAttempt , latestAttempt int ,
parseStatus , knowledgeErrorMessage string ,
knowledgeUpdatedAt time . Time ,
spanFailure * types . KnowledgeProcessingSpan ,
) gin . H {
if spanFailure != nil {
return gin . H {
"stage" : spanFailure . Name ,
"code" : spanFailure . ErrorCode ,
"message" : spanFailure . ErrorMessage ,
"name" : spanFailure . Name ,
"error_code" : spanFailure . ErrorCode ,
"error_message" : spanFailure . ErrorMessage ,
"finished_at" : spanFailure . FinishedAt ,
}
}
if currentAttempt != latestAttempt || parseStatus != types . ParseStatusFailed || knowledgeErrorMessage == "" {
return nil
}
2026-07-14 09:20:22 +00:00
errorCode := "UNKNOWN"
if strings . EqualFold ( strings . TrimSpace ( knowledgeErrorMessage ) ,
"Task interrupted due to application restart" ) {
errorCode = "SERVER_RESTART"
}
2026-07-14 08:48:30 +00:00
return gin . H {
"stage" : "knowledge_processing" ,
2026-07-14 09:20:22 +00:00
"code" : errorCode ,
2026-07-14 08:48:30 +00:00
"message" : knowledgeErrorMessage ,
"name" : "knowledge_processing" ,
2026-07-14 09:20:22 +00:00
"error_code" : errorCode ,
2026-07-14 08:48:30 +00:00
"error_message" : knowledgeErrorMessage ,
"finished_at" : knowledgeUpdatedAt ,
}
}
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
// buildSpanTree assembles a flat list of span rows into a parent-child
// tree rooted at the (knowledge, attempt)'s root span. Missing canonical
// stages are filled in with pending placeholders so the UI always renders
// the five timeline segments. Returns the root, the current_stage name
// (the running stage if any), and the most recent failed span if one
// exists.
2026-05-27 09:35:58 +00:00
//
// parseStatus is the knowledge.parse_status string. When the spans table
// has zero rows for this attempt (legacy data parsed before tracking, or
// a fresh knowledge before the pipeline starts), the placeholder status
// is inferred from parseStatus: completed → done, failed → failed,
// otherwise pending. Without this, every historical knowledge would
// render as "all 5 stages pending" forever despite having actually
// completed parsing.
func buildSpanTree ( knowledgeID string , attempt int , rows [ ] types . KnowledgeProcessingSpan , parseStatus string ) (
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
root * types . SpanTreeNode , currentStage string , lastFailure * types . KnowledgeProcessingSpan ,
) {
now := time . Now ( )
// Build node lookup, identify root.
nodes := make ( map [ string ] * types . SpanTreeNode , len ( rows ) )
var rootRow * types . KnowledgeProcessingSpan
stageRowByName := map [ string ] * types . KnowledgeProcessingSpan { }
for i := range rows {
r := rows [ i ]
nodes [ r . SpanID ] = & types . SpanTreeNode { KnowledgeProcessingSpan : r }
if r . Kind == types . SpanKindRoot && rootRow == nil {
cp := r
rootRow = & cp
}
if r . Kind == types . SpanKindStage {
cp := r
stageRowByName [ r . Name ] = & cp
}
if r . Status == types . SpanStatusRunning && r . Kind == types . SpanKindStage && currentStage == "" {
currentStage = r . Name
}
if r . Status == types . SpanStatusFailed {
cp := r
lastFailure = & cp
}
}
2026-05-27 09:35:58 +00:00
// Pick the synthesized stage status from parse_status. Without this,
// historical knowledge that completed before span tracking was wired
// would render as "5 pending stages" forever — the rows simply
// weren't recorded, but parse_status correctly reads "completed".
// The synthesized stages don't carry duration/timing data; they
// just communicate the inferred terminal state.
syntheticStatus := types . SpanStatusPending
switch parseStatus {
case types . ParseStatusCompleted :
syntheticStatus = types . SpanStatusDone
case types . ParseStatusFailed :
syntheticStatus = types . SpanStatusFailed
}
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
// Synthesize root if no rows came back so the API contract stays
// stable (frontend always expects a `trace` object).
if rootRow == nil {
root = & types . SpanTreeNode { KnowledgeProcessingSpan : types . KnowledgeProcessingSpan {
KnowledgeID : knowledgeID ,
Attempt : attempt ,
SpanID : "" ,
Name : "knowledge_processing" ,
Kind : types . SpanKindRoot ,
2026-05-27 09:35:58 +00:00
Status : syntheticStatus ,
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
CreatedAt : now ,
UpdatedAt : now ,
} }
} else {
root = nodes [ rootRow . SpanID ]
}
2026-05-27 17:24:49 +00:00
// Link real children to their parents. Walk `rows` (not the
// `nodes` map!) so the append order matches the repo's stable
// `ORDER BY id ASC`. Iterating the map directly would give
// callers a different child ordering on every request — Go map
// iteration is intentionally randomised — and the UI would
// flicker subspans into a different order on each refresh.
for i := range rows {
r := & rows [ i ]
n := nodes [ r . SpanID ]
if n == nil || n == root {
continue
}
if r . ParentSpanID == "" {
// Real top-level row with no parent and not the root
// itself — attach to root so it doesn't dangle.
root . Children = append ( root . Children , n )
continue
}
parent , ok := nodes [ r . ParentSpanID ]
if ! ok {
// Unknown parent (orphan); attach to root.
root . Children = append ( root . Children , n )
continue
}
parent . Children = append ( parent . Children , n )
}
2026-05-27 09:35:58 +00:00
// Synthesize missing stage rows as children of root so the timeline
// always shows 5 segments. Status mirrors the synthesized root —
// pending while the pipeline is still running, done/failed for
// historical knowledge whose terminal state we know but whose
2026-05-27 17:24:49 +00:00
// per-stage timing was never recorded. Appended in AllStages order
// so the canonical stage layout is deterministic regardless of
// which rows are missing.
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
for _ , name := range types . AllStages {
if _ , ok := stageRowByName [ name ] ; ok {
continue
}
placeholder := types . KnowledgeProcessingSpan {
KnowledgeID : knowledgeID ,
Attempt : attempt ,
Name : name ,
Kind : types . SpanKindStage ,
2026-05-27 09:35:58 +00:00
Status : syntheticStatus ,
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
CreatedAt : now ,
UpdatedAt : now ,
}
2026-05-27 17:24:49 +00:00
root . Children = append ( root . Children , & types . SpanTreeNode { KnowledgeProcessingSpan : placeholder } )
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
2026-05-27 06:05:47 +00:00
}
return root , currentStage , lastFailure
}
2025-12-16 08:40:13 +00:00
// ListKnowledge godoc
// @Summary 获取知识列表
// @Description 获取知识库下的知识列表,支持分页和筛选
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识库ID"
// @Param page query int false "页码"
// @Param page_size query int false "每页数量"
2026-06-25 18:18:10 +00:00
// @Param tag_ids query string false "标签ID筛选, 逗号分隔( OR语义) "
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
2026-05-12 10:16:34 +00:00
// @Param keyword query string false "关键词搜索"
// @Param file_type query string false "文件类型筛选"
// @Param parse_status query string false "解析状态筛选 (pending/processing/completed/failed)"
// @Param source query string false "来源/渠道筛选 (web/api/feishu/notion/yuque/wechat/...,或 manual/url 按 type 过滤)"
// @Param start_time query string false "更新时间起点, RFC3339 格式"
// @Param end_time query string false "更新时间终点, RFC3339 格式"
2026-08-04 08:45:51 +00:00
// @Param folder_path query string false "文件夹路径筛选,空字符串表示知识库根目录;不传该参数则不按文件夹过滤"
// @Param folder_recursive query bool false "为 true 时同时返回子文件夹内的文档"
2025-12-16 08:40:13 +00:00
// @Success 200 {object} map[string]interface{} "知识列表"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge-bases/{id}/knowledge [get]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) ListKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start retrieving knowledge list" )
2026-01-30 08:28:01 +00:00
// Validate access to the knowledge base (read access - any permission level)
_ , kbID , effectiveTenantID , _ , err := h . validateKnowledgeBaseAccess ( c )
if err != nil {
c . Error ( err )
2025-08-05 07:08:07 +00:00
return
}
2026-01-30 08:28:01 +00:00
// Update context with effective tenant ID for shared KB access
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-01-30 08:28:01 +00:00
2025-08-05 07:08:07 +00:00
// Parse pagination parameters from query string
var pagination types . Pagination
if err := c . ShouldBindQuery ( & pagination ) ; err != nil {
logger . Error ( ctx , "Failed to parse pagination parameters" , err )
c . Error ( errors . NewBadRequestError ( err . Error ( ) ) )
return
}
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
2026-05-12 10:16:34 +00:00
filter := types . KnowledgeListFilter {
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-25 16:18:48 +00:00
TagIDs : parseCommaSeparatedTagIDs ( c . Query ( "tag_ids" ) ) ,
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
2026-05-12 10:16:34 +00:00
Keyword : c . Query ( "keyword" ) ,
FileType : c . Query ( "file_type" ) ,
ParseStatus : c . Query ( "parse_status" ) ,
Source : c . Query ( "source" ) ,
}
if raw := c . Query ( "start_time" ) ; raw != "" {
t , err := parseFilterTime ( raw )
if err != nil {
c . Error ( errors . NewBadRequestError ( "invalid start_time: " + err . Error ( ) ) )
return
}
filter . UpdatedFrom = t
}
if raw := c . Query ( "end_time" ) ; raw != "" {
t , err := parseFilterTime ( raw )
if err != nil {
c . Error ( errors . NewBadRequestError ( "invalid end_time: " + err . Error ( ) ) )
return
}
filter . UpdatedTo = t
}
2026-08-04 08:45:51 +00:00
// The folder dimension is opt-in by parameter *presence*: an empty
// folder_path is meaningful (the knowledge base root), so it cannot be
// distinguished from "no folder filter" by value alone.
if raw , ok := c . GetQuery ( "folder_path" ) ; ok {
filter . FolderPath = types . NormalizeKnowledgeFolderPath ( raw )
filter . FolderScope = types . FolderScopeExact
if recursive , err := strconv . ParseBool ( c . DefaultQuery ( "folder_recursive" , "false" ) ) ; err == nil && recursive {
filter . FolderScope = types . FolderScopeSubtree
}
}
2025-11-17 10:15:02 +00:00
2025-11-28 08:17:52 +00:00
logger . Infof (
ctx ,
2026-08-04 08:45:51 +00:00
"Retrieving knowledge list under knowledge base, kb_id=%s tag_ids=%s keyword=%s file_type=%s parse_status=%s source=%s start_time=%s end_time=%s folder_path=%s folder_scope=%s page=%d page_size=%d effectiveTenantID=%d" ,
2025-11-28 08:17:52 +00:00
secutils . SanitizeForLog ( kbID ) ,
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-25 16:18:48 +00:00
secutils . SanitizeForLog ( strings . Join ( filter . TagIDs , "," ) ) ,
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
2026-05-12 10:16:34 +00:00
secutils . SanitizeForLog ( filter . Keyword ) ,
secutils . SanitizeForLog ( filter . FileType ) ,
secutils . SanitizeForLog ( filter . ParseStatus ) ,
secutils . SanitizeForLog ( filter . Source ) ,
secutils . SanitizeForLog ( c . Query ( "start_time" ) ) ,
secutils . SanitizeForLog ( c . Query ( "end_time" ) ) ,
2026-08-04 08:45:51 +00:00
secutils . SanitizeForLog ( filter . FolderPath ) ,
string ( filter . FolderScope ) ,
2025-11-28 08:17:52 +00:00
pagination . Page ,
pagination . PageSize ,
2026-01-30 08:28:01 +00:00
effectiveTenantID ,
2025-11-28 08:17:52 +00:00
)
2025-08-05 07:08:07 +00:00
// Retrieve paginated knowledge entries
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
2026-05-12 10:16:34 +00:00
result , err := h . kgService . ListPagedKnowledgeByKnowledgeBaseID ( ctx , kbID , & pagination , filter )
2025-08-05 07:08:07 +00:00
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
2025-11-28 08:17:52 +00:00
logger . Infof (
ctx ,
"Knowledge list retrieved successfully, knowledge base ID: %s, total: %d" ,
secutils . SanitizeForLog ( kbID ) ,
result . Total ,
)
2025-08-05 07:08:07 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : result . Data ,
"total" : result . Total ,
"page" : result . Page ,
"page_size" : result . PageSize ,
} )
}
2026-08-04 08:45:51 +00:00
// ListKnowledgeFolders godoc
// @Summary 获取知识库文件夹目录树
// @Description 返回知识库内由文件夹上传形成的目录树,包含每个文件夹的直接文档数与含子目录的总数
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识库ID"
// @Success 200 {object} map[string]interface{} "目录树"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge-bases/{id}/knowledge/folders [get]
func ( h * KnowledgeHandler ) ListKnowledgeFolders ( c * gin . Context ) {
ctx := c . Request . Context ( )
// Read access mirrors ListKnowledge so the sidebar tree is available to
// every viewer of a shared knowledge base.
_ , kbID , effectiveTenantID , _ , err := h . validateKnowledgeBaseAccess ( c )
if err != nil {
c . Error ( err )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-08-04 08:45:51 +00:00
tree , err := h . kgService . ListKnowledgeFolderTree ( ctx , kbID )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
logger . Infof ( ctx , "Knowledge folder tree retrieved, kb_id=%s folders=%d" ,
secutils . SanitizeForLog ( kbID ) , len ( tree . Folders ) )
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : tree ,
} )
}
2026-08-04 10:15:23 +00:00
// MoveKnowledgeToFolderRequest is the body schema for POST /knowledge/folder.
type MoveKnowledgeToFolderRequest struct {
KBID string ` json:"kb_id" binding:"required" `
IDs [ ] string ` json:"knowledge_ids" binding:"required" `
// FolderPath is the destination folder; the empty string is the knowledge
// base top level. It is deliberately not `binding:"required"` so documents
// can be moved back out of every folder.
FolderPath string ` json:"folder_path" `
}
// MoveKnowledgeToFolder godoc
// @Summary 移动知识到文件夹
// @Description 批量修改知识条目所属文件夹。文件夹由路径推导而来,因此目标路径不存在时会自动创建;空路径表示知识库顶层。仅调整归类,不会重新解析文档
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param request body MoveKnowledgeToFolderRequest true "移动请求"
// @Success 200 {object} map[string]interface{} "移动成功"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 403 {object} errors.AppError "权限不足"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge/folder [post]
func ( h * KnowledgeHandler ) MoveKnowledgeToFolder ( c * gin . Context ) {
ctx := c . Request . Context ( )
var req MoveKnowledgeToFolderRequest
if err := c . ShouldBindJSON ( & req ) ; err != nil {
c . Error ( errors . NewBadRequestError ( "Invalid request parameters: " + err . Error ( ) ) )
return
}
ids := dedupeKnowledgeIDs ( req . IDs )
if len ( ids ) == 0 {
c . Error ( errors . NewBadRequestError ( "knowledge_ids cannot be empty" ) )
return
}
const maxBatch = 200
if len ( ids ) > maxBatch {
c . Error ( errors . NewBadRequestError ( fmt . Sprintf ( "too many ids (max %d per batch)" , maxBatch ) ) )
return
}
kbID , effectiveTenantID , err := h . requireKnowledgeWriteAccess ( c , req . KBID )
if err != nil {
c . Error ( err )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-08-04 10:15:23 +00:00
// Guard against cross-KB moves: the service layer scopes by tenant, so the
// handler must confirm every entry belongs to the requested knowledge base.
if err := h . requireKnowledgeInKB ( ctx , effectiveTenantID , kbID , ids ) ; err != nil {
c . Error ( err )
return
}
affected , err := h . kgService . MoveKnowledgeToFolder ( ctx , kbID , ids , req . FolderPath )
if err != nil {
if appErr , ok := errors . IsAppError ( err ) ; ok {
c . Error ( appErr )
return
}
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : gin . H {
"moved_count" : affected ,
"folder_path" : types . NormalizeKnowledgeFolderPath ( req . FolderPath ) ,
} ,
} )
}
// RenameKnowledgeFolderRequest is the body schema for
// PUT /knowledge-bases/:id/knowledge/folders.
type RenameKnowledgeFolderRequest struct {
From string ` json:"from" binding:"required" `
To string ` json:"to" binding:"required" `
}
// RenameKnowledgeFolder godoc
// @Summary 重命名或移动文件夹
// @Description 把一个文件夹及其所有子目录改到新路径。目标路径已存在时两个文件夹合并;不能移动到自身子目录下
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识库ID"
// @Param request body RenameKnowledgeFolderRequest true "重命名请求"
// @Success 200 {object} map[string]interface{} "重命名成功"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 403 {object} errors.AppError "权限不足"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge-bases/{id}/knowledge/folders [put]
func ( h * KnowledgeHandler ) RenameKnowledgeFolder ( c * gin . Context ) {
ctx := c . Request . Context ( )
var req RenameKnowledgeFolderRequest
if err := c . ShouldBindJSON ( & req ) ; err != nil {
c . Error ( errors . NewBadRequestError ( "Invalid request parameters: " + err . Error ( ) ) )
return
}
2026-09-08 07:36:33 +00:00
_ , kbID , effectiveTenantID , permission , err := h . validateKnowledgeBaseWriteAccessWithKBID ( c , c . Param ( "id" ) )
2026-08-04 10:15:23 +00:00
if err != nil {
c . Error ( err )
return
}
if permission != types . OrgRoleAdmin && permission != types . OrgRoleEditor {
c . Error ( errors . NewForbiddenError ( "No permission to modify knowledge" ) )
return
}
2026-08-04 12:30:43 +00:00
if err := h . requireKBOwnershipOrAdmin ( c , kbID ) ; err != nil {
c . Error ( err )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-08-04 10:15:23 +00:00
affected , err := h . kgService . RenameKnowledgeFolder ( ctx , kbID , req . From , req . To )
if err != nil {
if appErr , ok := errors . IsAppError ( err ) ; ok {
c . Error ( appErr )
return
}
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : gin . H {
"moved_count" : affected ,
"folder_path" : types . NormalizeKnowledgeFolderPath ( req . To ) ,
} ,
} )
}
// dedupeKnowledgeIDs trims, drops empty and de-duplicates a batch of IDs while
// preserving the caller's order.
func dedupeKnowledgeIDs ( raw [ ] string ) [ ] string {
seen := make ( map [ string ] struct { } , len ( raw ) )
ids := make ( [ ] string , 0 , len ( raw ) )
for _ , item := range raw {
id := strings . TrimSpace ( item )
if id == "" {
continue
}
if _ , ok := seen [ id ] ; ok {
continue
}
seen [ id ] = struct { } { }
ids = append ( ids , id )
}
return ids
}
// requireKnowledgeWriteAccess resolves a knowledge base from a request body and
// enforces the editor-or-admin plus ownership gate shared by the batch routes.
func ( h * KnowledgeHandler ) requireKnowledgeWriteAccess (
c * gin . Context ,
requestedKBID string ,
) ( string , uint64 , error ) {
2026-09-08 07:36:33 +00:00
_ , kbID , effectiveTenantID , permission , err := h . validateKnowledgeBaseWriteAccessWithKBID ( c , requestedKBID )
2026-08-04 10:15:23 +00:00
if err != nil {
return "" , 0 , err
}
if permission != types . OrgRoleAdmin && permission != types . OrgRoleEditor {
return "" , 0 , errors . NewForbiddenError ( "No permission to modify knowledge" )
}
if err := h . requireKBOwnershipOrAdmin ( c , kbID ) ; err != nil {
return "" , 0 , err
}
return kbID , effectiveTenantID , nil
}
// requireKnowledgeInKB verifies every ID exists and belongs to the given
// knowledge base, so a batch operation cannot reach across knowledge bases.
func ( h * KnowledgeHandler ) requireKnowledgeInKB (
ctx context . Context ,
tenantID uint64 ,
kbID string ,
ids [ ] string ,
) error {
knowledgeList , err := h . kgService . GetKnowledgeBatch ( ctx , tenantID , ids )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
return errors . NewInternalServerError ( err . Error ( ) )
}
if len ( knowledgeList ) != len ( ids ) {
return errors . NewBadRequestError ( "One or more knowledge entries not found" )
}
for _ , k := range knowledgeList {
if k . KnowledgeBaseID != kbID {
return errors . NewBadRequestError (
fmt . Sprintf ( "Knowledge %s does not belong to knowledge base %s" ,
secutils . SanitizeForLog ( k . ID ) , secutils . SanitizeForLog ( kbID ) ) )
}
}
return nil
}
2025-12-16 08:40:13 +00:00
// DeleteKnowledge godoc
// @Summary 删除知识
2026-05-25 08:54:26 +00:00
// @Description 根据ID异步删除知识条目。请求会被入队到与批量删除相同的异步管道( asynq) ;
// @Description 接口返回 200 仅表示任务已提交(响应 data.task_id 为任务 ID) , 实际删除由后台 worker 完成。
2025-12-16 08:40:13 +00:00
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识ID"
2026-05-25 08:54:26 +00:00
// @Success 200 {object} map[string]interface{} "任务已提交,返回 task_id"
2025-12-16 08:40:13 +00:00
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge/{id} [delete]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) DeleteKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start deleting knowledge" )
2025-11-25 16:19:26 +00:00
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
2025-08-05 07:08:07 +00:00
if id == "" {
logger . Error ( ctx , "Knowledge ID is empty" )
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
2026-09-08 07:36:33 +00:00
knowledge , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleEditor )
2026-01-30 08:28:01 +00:00
if err != nil {
c . Error ( err )
return
}
2026-05-25 08:54:26 +00:00
2026-09-08 07:36:33 +00:00
if err := access . RejectMovingKnowledge ( knowledge ) ; err != nil {
_ = c . Error ( err )
return
}
2026-05-25 08:54:26 +00:00
// Reuse the batch async pipeline so single-item delete shares the same
// hardening (asynq retries, business-aware queue routing, marking-as-deleting
// inside the worker) as BatchDeleteKnowledge / ClearKnowledgeBaseContents.
effectiveTenantID , _ := effCtx . Value ( types . TenantIDContextKey ) . ( uint64 )
if effectiveTenantID == 0 {
logger . Error ( ctx , "Effective tenant ID missing after access validation" )
2026-07-14 08:13:49 +00:00
c . Error ( errors . NewInternalServerError ( "workspace context unavailable" ) )
2026-05-25 08:54:26 +00:00
return
}
logger . Infof ( ctx , "Enqueuing knowledge delete, ID: %s" , secutils . SanitizeForLog ( id ) )
2026-09-08 07:36:33 +00:00
taskID , err := h . enqueueKnowledgeListDelete ( effCtx , effectiveTenantID , knowledge . KnowledgeBaseID , [ ] string { id } )
2025-08-05 07:08:07 +00:00
if err != nil {
2026-05-25 08:54:26 +00:00
logger . Errorf ( ctx , "Failed to enqueue knowledge delete task: %v" , err )
c . Error ( errors . NewInternalServerError ( "Failed to enqueue delete task" ) )
2025-08-05 07:08:07 +00:00
return
}
2026-05-25 08:54:26 +00:00
logger . Infof ( ctx , "Knowledge delete task enqueued: %s, knowledge_id: %s" , taskID , secutils . SanitizeForLog ( id ) )
2025-08-05 07:08:07 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
2026-05-25 08:54:26 +00:00
"message" : "Delete task submitted" ,
"data" : gin . H {
"task_id" : taskID ,
} ,
2025-08-05 07:08:07 +00:00
} )
}
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
// BatchDeleteKnowledgeRequest is the body schema for POST /knowledge/batch-delete.
type BatchDeleteKnowledgeRequest struct {
KBID string ` json:"kb_id" binding:"required" `
IDs [ ] string ` json:"ids" binding:"required" `
}
// BatchDeleteKnowledge godoc
// @Summary 批量删除知识
// @Description 按 ID 列表批量删除单个知识库下的多个知识条目
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param request body BatchDeleteKnowledgeRequest true "批量删除请求"
// @Success 200 {object} map[string]interface{} "删除成功"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 403 {object} errors.AppError "权限不足"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge/batch-delete [post]
func ( h * KnowledgeHandler ) BatchDeleteKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
var req BatchDeleteKnowledgeRequest
if err := c . ShouldBindJSON ( & req ) ; err != nil {
c . Error ( errors . NewBadRequestError ( "Invalid request parameters: " + err . Error ( ) ) )
return
}
2026-08-04 10:15:23 +00:00
ids := dedupeKnowledgeIDs ( req . IDs )
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
if len ( ids ) == 0 {
c . Error ( errors . NewBadRequestError ( "ids cannot be empty" ) )
return
}
const maxBatch = 200
if len ( ids ) > maxBatch {
c . Error ( errors . NewBadRequestError ( fmt . Sprintf ( "too many ids (max %d per batch)" , maxBatch ) ) )
return
}
// Validate KB access (editor or admin) using the kb_id from body.
2026-09-08 07:36:33 +00:00
_ , kbID , effectiveTenantID , permission , err := h . validateKnowledgeBaseWriteAccessWithKBID ( c , req . KBID )
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
if err != nil {
c . Error ( err )
return
}
if permission != types . OrgRoleAdmin && permission != types . OrgRoleEditor {
c . Error ( errors . NewForbiddenError ( "No permission to delete knowledge" ) )
return
}
2026-07-02 06:49:27 +00:00
if err := h . requireKBOwnershipOrAdmin ( c , kbID ) ; err != nil {
c . Error ( err )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
// Single batch fetch to validate that every id exists and belongs to the
2026-09-08 07:36:33 +00:00
// requested KB before admitting a task with that exact KB scope.
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
knowledgeList , err := h . kgService . GetKnowledgeBatch ( ctx , effectiveTenantID , ids )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
if len ( knowledgeList ) != len ( ids ) {
c . Error ( errors . NewBadRequestError ( "One or more knowledge entries not found" ) )
return
}
for _ , k := range knowledgeList {
2026-09-08 07:36:33 +00:00
if err := access . RejectMovingKnowledge ( k ) ; err != nil {
_ = c . Error ( err )
return
}
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
if k . KnowledgeBaseID != kbID {
c . Error ( errors . NewBadRequestError (
fmt . Sprintf ( "Knowledge %s does not belong to knowledge base %s" ,
secutils . SanitizeForLog ( k . ID ) , secutils . SanitizeForLog ( kbID ) ) ) )
return
}
}
2026-09-08 07:36:33 +00:00
taskID , err := h . enqueueKnowledgeListDelete ( ctx , effectiveTenantID , kbID , ids )
2026-04-29 03:47:59 +00:00
if err != nil {
logger . Errorf ( ctx , "Failed to enqueue batch knowledge delete task: %v" , err )
c . Error ( errors . NewInternalServerError ( "Failed to enqueue batch delete task" ) )
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
return
}
2026-04-29 03:47:59 +00:00
logger . Infof ( ctx , "Batch knowledge delete task enqueued: %s, kb_id: %s, count: %d" ,
taskID , secutils . SanitizeForLog ( kbID ) , len ( ids ) )
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
2026-04-29 03:47:59 +00:00
"message" : "Batch delete task submitted" ,
"data" : gin . H {
"task_id" : taskID ,
"deleted_count" : len ( ids ) ,
} ,
feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.
Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
permission, then delegates to the existing DeleteKnowledgeList service
which already cascades vector/file/graph cleanup. Caps batch size at
200 and uses a single GetKnowledgeBatch call for membership validation.
Frontend
- DocumentListView: table layout with sticky header, checkbox column
(with indeterminate select-all), file-type icons, status badges,
per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
with the new list view and avoid further drift from the existing
inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.
Closes #1045
Closes #957
2026-04-28 19:24:35 +00:00
} )
}
2026-03-24 08:29:55 +00:00
// ClearKnowledgeBaseContents godoc
// @Summary 清空知识库内容
// @Description 删除知识库下的所有知识条目(异步任务)。知识库本身保留,仅清空其中的内容
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识库ID"
// @Success 200 {object} map[string]interface{} "清空任务已提交"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 403 {object} errors.AppError "权限不足"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge-bases/{id}/knowledge [delete]
func ( h * KnowledgeHandler ) ClearKnowledgeBaseContents ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start clearing knowledge base contents" )
2026-09-08 07:36:33 +00:00
kb , kbID , effectiveTenantID , permission , err := h . validateKnowledgeBaseWriteAccessWithKBID ( c , c . Param ( "id" ) )
2026-03-24 08:29:55 +00:00
if err != nil {
c . Error ( err )
return
}
// Only owner (admin with matching tenant) can clear knowledge base contents
tenantID := c . GetUint64 ( types . TenantIDContextKey . String ( ) )
if kb . TenantID != tenantID || permission != types . OrgRoleAdmin {
c . Error ( errors . NewForbiddenError ( "Only knowledge base owner can clear contents" ) )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-03-24 08:29:55 +00:00
knowledgeList , err := h . kgService . ListKnowledgeByKnowledgeBaseID ( ctx , kbID )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( "Failed to list knowledge entries" ) . WithDetails ( err . Error ( ) ) )
return
}
if len ( knowledgeList ) == 0 {
logger . Infof ( ctx , "Knowledge base %s is already empty" , secutils . SanitizeForLog ( kbID ) )
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"message" : "Knowledge base is already empty" ,
"data" : gin . H { "deleted_count" : 0 } ,
} )
return
}
knowledgeIDs := make ( [ ] string , 0 , len ( knowledgeList ) )
for _ , knowledge := range knowledgeList {
2026-09-08 07:36:33 +00:00
if err := access . RejectMovingKnowledge ( knowledge ) ; err != nil {
_ = c . Error ( err )
return
}
2026-03-24 08:29:55 +00:00
knowledgeIDs = append ( knowledgeIDs , knowledge . ID )
}
2026-09-08 07:36:33 +00:00
taskID , err := h . enqueueKnowledgeListDelete ( ctx , effectiveTenantID , kbID , knowledgeIDs )
2026-03-24 08:29:55 +00:00
if err != nil {
logger . Errorf ( ctx , "Failed to enqueue knowledge list delete task: %v" , err )
c . Error ( errors . NewInternalServerError ( "Failed to enqueue cleanup task" ) )
return
}
logger . Infof ( ctx , "Knowledge base contents clear task enqueued: %s, kb_id: %s, count: %d" ,
2026-04-29 03:47:59 +00:00
taskID , secutils . SanitizeForLog ( kbID ) , len ( knowledgeIDs ) )
2026-03-24 08:29:55 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"message" : "Knowledge base contents clear task submitted" ,
"data" : gin . H { "deleted_count" : len ( knowledgeIDs ) } ,
} )
}
2025-12-16 08:40:13 +00:00
// DownloadKnowledgeFile godoc
// @Summary 下载知识文件
// @Description 下载知识条目关联的原始文件
// @Tags 知识管理
// @Accept json
// @Produce application/octet-stream
// @Param id path string true "知识ID"
// @Success 200 {file} file "文件内容"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge/{id}/download [get]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) DownloadKnowledgeFile ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start downloading knowledge file" )
2025-11-25 16:19:26 +00:00
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
2025-08-05 07:08:07 +00:00
if id == "" {
logger . Error ( ctx , "Knowledge ID is empty" )
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
2026-07-23 13:11:37 +00:00
// Keep a handler-level Editor check in addition to the route guard. The
// original file is more sensitive than parsed-content reads and must not
// be downloadable through a read-only organization share.
_ , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleEditor )
2026-01-30 08:28:01 +00:00
if err != nil {
c . Error ( err )
return
}
2025-11-25 15:33:34 +00:00
logger . Infof ( ctx , "Retrieving knowledge file, ID: %s" , secutils . SanitizeForLog ( id ) )
2025-08-05 07:08:07 +00:00
2026-01-30 08:28:01 +00:00
file , filename , err := h . kgService . GetKnowledgeFile ( effCtx , id )
2025-08-05 07:08:07 +00:00
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( "Failed to retrieve file" ) . WithDetails ( err . Error ( ) ) )
return
}
c . Header ( "Content-Description" , "File Transfer" )
c . Header ( "Content-Transfer-Encoding" , "binary" )
c . Header ( "Expires" , "0" )
2026-09-08 07:36:33 +00:00
if err := filetransport . Serve ( c . Writer , c . Request , file , filetransport . Options {
Filename : filename , Download : true , ContentType : "application/octet-stream" , CacheControl : "private, no-store" ,
} ) ; err != nil {
logger . Errorf ( ctx , "Failed to send file: %v" , err )
}
2025-08-05 07:08:07 +00:00
}
2026-02-28 05:22:07 +00:00
// PreviewKnowledgeFile godoc
// @Summary 预览知识文件
// @Description 返回知识条目关联的原始文件, Content-Type 根据文件类型设置,用于浏览器内嵌预览
// @Tags 知识管理
// @Accept json
// @Produce application/pdf,image/jpeg,image/png,text/plain
// @Param id path string true "知识ID"
// @Success 200 {file} file "文件内容"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge/{id}/preview [get]
func ( h * KnowledgeHandler ) PreviewKnowledgeFile ( c * gin . Context ) {
ctx := c . Request . Context ( )
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
if id == "" {
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
_ , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleViewer )
if err != nil {
c . Error ( err )
return
}
file , filename , err := h . kgService . GetKnowledgeFile ( effCtx , id )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( "Failed to retrieve file" ) . WithDetails ( err . Error ( ) ) )
return
}
2026-09-08 07:36:33 +00:00
if err := filetransport . Serve ( c . Writer , c . Request , file , filetransport . Options {
Filename : filename , CacheControl : "private, no-store" ,
} ) ; err != nil {
logger . Errorf ( ctx , "Failed to stream preview: %v" , err )
2026-07-02 10:17:51 +00:00
}
2026-02-28 05:22:07 +00:00
}
2025-08-05 07:08:07 +00:00
// GetKnowledgeBatchRequest defines parameters for batch knowledge retrieval
type GetKnowledgeBatchRequest struct {
2026-07-30 04:10:31 +00:00
IDs [ ] string ` form:"ids" binding:"required" ` // List of knowledge IDs
KBID string ` form:"kb_id" ` // Optional: scope to this KB (validates access and uses effective tenant for shared KB)
AgentID string ` form:"agent_id" ` // Optional: when using a shared agent, use agent's tenant for retrieval (validates shared agent access)
AgentSourceTenantID uint64 ` form:"agent_source_tenant_id" ` // Optional source selector, verified against the share relation
2025-08-05 07:08:07 +00:00
}
2025-12-16 08:40:13 +00:00
// GetKnowledgeBatch godoc
// @Summary 批量获取知识
2026-07-14 08:13:49 +00:00
// @Description 根据ID列表批量获取知识条目。可选 kb_id: 指定时按该知识库校验权限并用于共享知识库的空间解析; 可选 agent_id: 使用共享智能体时传此参数, 后端按智能体所属空间查询( 用于刷新后恢复共享知识库下的文件)
2025-12-16 08:40:13 +00:00
// @Tags 知识管理
// @Accept json
// @Produce json
2026-02-04 12:12:29 +00:00
// @Param ids query []string true "知识ID列表"
// @Param kb_id query string false "可选, 知识库ID( 用于共享知识库时指定范围) "
2026-07-14 08:13:49 +00:00
// @Param agent_id query string false "可选, 共享智能体ID( 用于按智能体空间批量拉取文件详情) "
2026-02-04 12:12:29 +00:00
// @Success 200 {object} map[string]interface{} "知识列表"
// @Failure 400 {object} errors.AppError "请求参数错误"
2025-12-16 08:40:13 +00:00
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge/batch [get]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) GetKnowledgeBatch ( c * gin . Context ) {
ctx := c . Request . Context ( )
2026-09-08 07:36:33 +00:00
effectiveTenantID := middleware . KBAccessRequest ( c ) . Caller . TenantID
if effectiveTenantID == 0 {
2025-08-05 07:08:07 +00:00
c . Error ( errors . NewUnauthorizedError ( "Unauthorized" ) )
return
}
var req GetKnowledgeBatchRequest
if err := c . ShouldBindQuery ( & req ) ; err != nil {
logger . Error ( ctx , "Failed to parse request parameters" , err )
c . Error ( errors . NewBadRequestError ( "Invalid request parameters" ) . WithDetails ( err . Error ( ) ) )
return
}
2026-07-30 04:26:25 +00:00
if _ , parseErr := types . ParseAgentSourceTenantID ( c . Query ( types . AgentSourceTenantIDParam ) ) ; parseErr != nil {
c . Error ( errors . NewBadRequestError ( parseErr . Error ( ) ) )
return
}
2025-08-05 07:08:07 +00:00
2026-09-08 07:36:33 +00:00
// A nil scope means no agent selector. Every resolved scope, including
// mode=all, binds results to the authorized agent's source tenant.
var agentScope * types . SharedAgentKBScope
if agentID := secutils . SanitizeForLog ( req . AgentID ) ; agentID != "" {
agent , err := resolveSharedAgentForRequest ( c , agentID , h . agentShareService )
if err != nil {
_ = c . Error ( err )
2026-02-04 12:12:29 +00:00
return
}
2026-09-08 07:36:33 +00:00
ctx = access . WithSharedAgent ( ctx , agent )
scope := types . NewSharedAgentKBScope ( agent )
agentScope = & scope
2026-02-04 12:12:29 +00:00
effectiveTenantID = agent . TenantID
2026-09-08 07:36:33 +00:00
if scope . IsEmpty ( ) {
2026-04-01 13:45:35 +00:00
c . JSON ( http . StatusOK , gin . H { "success" : true , "data" : [ ] * types . Knowledge { } } )
return
}
2026-02-04 12:12:29 +00:00
}
2026-01-30 08:28:01 +00:00
var knowledges [ ] * types . Knowledge
var err error
2026-04-01 13:45:35 +00:00
// scopeKBID tracks the single KB the results must belong to (set by explicit kb_id).
var scopeKBID string
2026-01-30 08:28:01 +00:00
// Optional kb_id: validate KB access and use effective tenant for shared KB
if kbID := secutils . SanitizeForLog ( req . KBID ) ; kbID != "" {
2026-09-08 07:36:33 +00:00
_ , _ , effID , _ , accessErr := h . validateKnowledgeBaseAccessWithKBID ( c , kbID )
if accessErr != nil {
_ = c . Error ( accessErr )
2026-01-30 08:28:01 +00:00
return
}
2026-09-08 07:36:33 +00:00
if agentScope != nil && ! agentScope . Allows ( kbID , effID ) {
2026-04-01 13:45:35 +00:00
c . Error ( errors . NewForbiddenError ( "Knowledge base not accessible through this agent" ) )
return
}
scopeKBID = kbID
2026-01-30 08:28:01 +00:00
effectiveTenantID = effID
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-01-30 08:28:01 +00:00
logger . Infof ( ctx , "Batch retrieving knowledge with kb_id, effective tenant ID: %d, IDs count: %d" ,
effectiveTenantID , len ( req . IDs ) )
2026-09-08 07:36:33 +00:00
knowledges , err = h . kgService . GetKnowledgeBatch ( ctx , effectiveTenantID , req . IDs )
} else if agentScope != nil {
// Keep the authorized agent's read scope and the original caller.
ctx = types . WithExecutionTenant ( ctx , effectiveTenantID )
2026-01-30 08:28:01 +00:00
knowledges , err = h . kgService . GetKnowledgeBatch ( ctx , effectiveTenantID , req . IDs )
} else {
knowledges , err = h . kgService . GetKnowledgeBatchWithSharedAccess ( ctx , effectiveTenantID , req . IDs )
}
2026-09-08 07:36:33 +00:00
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
_ = c . Error ( errors . NewInternalServerError ( "Failed to retrieve knowledge list" ) . WithDetails ( err . Error ( ) ) )
return
}
2025-08-05 07:08:07 +00:00
2026-07-05 02:55:00 +00:00
// Build the effective allowed-KB set from explicit kb_id, shared agent
// scope, and per-API-key KB restrictions.
2026-04-01 13:45:35 +00:00
var allowedKBSet map [ string ] bool
if scopeKBID != "" {
allowedKBSet = map [ string ] bool { scopeKBID : true }
}
2026-09-08 07:36:33 +00:00
if agentScope != nil {
knowledges = filterKnowledgeByAgentScope ( knowledges , * agentScope )
}
2026-07-05 02:55:00 +00:00
if apiKeySet := tenantAPIKeyAllowedKBSet ( ctx ) ; apiKeySet != nil {
allowedKBSet = intersectKBAllowSet ( allowedKBSet , apiKeySet )
}
if allowedKBSet != nil {
knowledges = filterKnowledgesByKBAllowSet ( knowledges , allowedKBSet )
2026-04-01 13:45:35 +00:00
}
2026-01-30 08:28:01 +00:00
logger . Infof ( ctx , "Batch knowledge retrieval successful, requested count: %d, returned count: %d" ,
len ( req . IDs ) , len ( knowledges ) )
2025-08-05 07:08:07 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : knowledges ,
} )
}
2026-08-20 06:54:53 +00:00
// UpdateKnowledgeRequest defines the partial-update body for PUT /knowledge/:id.
// Omitted fields are left unchanged; an explicit empty description clears the summary.
type UpdateKnowledgeRequest struct {
Title * string ` json:"title" `
Description * string ` json:"description" `
CustomMetadata json . RawMessage ` json:"custom_metadata" `
}
2025-12-16 08:40:13 +00:00
// UpdateKnowledge godoc
// @Summary 更新知识
2026-08-20 06:54:53 +00:00
// @Description 部分更新知识条目(标题/描述/自定义元数据);未传字段保持不变,显式传空 description 可清空摘要
2025-12-16 08:40:13 +00:00
// @Tags 知识管理
// @Accept json
// @Produce json
2026-08-20 06:54:53 +00:00
// @Param id path string true "知识ID"
// @Param request body UpdateKnowledgeRequest true "更新字段(均可选)"
2025-12-16 08:40:13 +00:00
// @Success 200 {object} map[string]interface{} "更新成功"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge/{id} [put]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) UpdateKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
2026-01-30 08:28:01 +00:00
2025-11-25 16:19:26 +00:00
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
2025-08-05 07:08:07 +00:00
if id == "" {
logger . Error ( ctx , "Knowledge ID is empty" )
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
2026-01-30 08:28:01 +00:00
_ , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleEditor )
if err != nil {
c . Error ( err )
return
}
2026-08-20 06:54:53 +00:00
var req UpdateKnowledgeRequest
if err := c . ShouldBindJSON ( & req ) ; err != nil {
2025-08-05 07:08:07 +00:00
logger . Error ( ctx , "Failed to parse request parameters" , err )
c . Error ( errors . NewBadRequestError ( err . Error ( ) ) )
return
}
2026-08-20 06:54:53 +00:00
knowledge := types . Knowledge { ID : id , CustomMetadata : types . JSON ( req . CustomMetadata ) }
if req . Title != nil {
knowledge . Title = * req . Title
}
if req . Description != nil {
knowledge . Description = * req . Description
knowledge . DescriptionSpecified = true
}
2025-08-05 07:08:07 +00:00
2026-01-30 08:28:01 +00:00
if err := h . kgService . UpdateKnowledge ( effCtx , & knowledge ) ; err != nil {
2025-08-05 07:08:07 +00:00
logger . ErrorWithFields ( ctx , err , nil )
2026-09-08 07:36:33 +00:00
var appErr * errors . AppError
if goerrors . As ( err , & appErr ) {
_ = c . Error ( appErr )
} else {
_ = c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
}
2025-08-05 07:08:07 +00:00
return
}
2026-07-28 08:04:50 +00:00
updated , getErr := h . kgService . GetKnowledgeByID ( effCtx , id )
if getErr != nil {
logger . Warnf ( ctx , "Knowledge updated but failed to reload status for %s: %v" , id , getErr )
}
2025-08-05 07:08:07 +00:00
2025-11-25 16:19:26 +00:00
logger . Infof ( ctx , "Knowledge updated successfully, knowledge ID: %s" , id )
2025-08-05 07:08:07 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
2026-07-28 08:04:50 +00:00
"message" : "Knowledge updated successfully" ,
"data" : updated ,
2025-08-05 07:08:07 +00:00
} )
}
2026-07-28 03:39:20 +00:00
// RegenerateKnowledgeSummary refreshes a stale summary after chunk or metadata edits.
func ( h * KnowledgeHandler ) RegenerateKnowledgeSummary ( c * gin . Context ) {
ctx := c . Request . Context ( )
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
if id == "" {
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
_ , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleEditor )
if err != nil {
c . Error ( err )
return
}
2026-07-28 12:04:02 +00:00
knowledge , err := h . kgService . GetKnowledgeByID ( effCtx , id )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
2026-09-08 07:36:33 +00:00
var appErr * errors . AppError
if goerrors . As ( err , & appErr ) {
_ = c . Error ( appErr )
} else {
_ = c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
}
2026-07-28 12:04:02 +00:00
return
}
if knowledge . SummaryStatus == "" || knowledge . SummaryStatus == types . SummaryStatusNone {
knowledge , err = h . kgService . RegenerateKnowledgeSummary ( effCtx , id )
} else {
err = h . kgService . RequestKnowledgeSummaryRefresh ( effCtx , id )
if err == nil {
knowledge , err = h . kgService . GetKnowledgeByID ( effCtx , id )
}
}
2026-07-28 03:39:20 +00:00
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
2026-09-08 07:36:33 +00:00
var appErr * errors . AppError
if goerrors . As ( err , & appErr ) {
_ = c . Error ( appErr )
} else {
_ = c . Error ( errors . NewBadRequestError ( err . Error ( ) ) )
}
2026-07-28 03:39:20 +00:00
return
}
c . JSON ( http . StatusOK , gin . H { "success" : true , "data" : knowledge } )
}
2025-12-16 08:40:13 +00:00
// UpdateManualKnowledge godoc
// @Summary 更新手工知识
// @Description 更新手工录入的Markdown知识内容
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识ID"
// @Param request body types.ManualKnowledgePayload true "手工知识内容"
// @Success 200 {object} map[string]interface{} "更新后的知识"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge/manual/{id} [put]
2025-11-12 13:27:56 +00:00
func ( h * KnowledgeHandler ) UpdateManualKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start updating manual knowledge" )
2025-11-25 16:19:26 +00:00
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
if id == "" {
2025-11-12 13:27:56 +00:00
logger . Error ( ctx , "Knowledge ID is empty" )
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
2026-01-30 08:28:01 +00:00
_ , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleEditor )
if err != nil {
c . Error ( err )
return
}
2025-11-12 13:27:56 +00:00
var req types . ManualKnowledgePayload
if err := c . ShouldBindJSON ( & req ) ; err != nil {
logger . Error ( ctx , "Failed to parse manual knowledge update request" , err )
c . Error ( errors . NewBadRequestError ( err . Error ( ) ) )
return
}
2026-01-30 08:28:01 +00:00
knowledge , err := h . kgService . UpdateManualKnowledge ( effCtx , id , & req )
2025-11-12 13:27:56 +00:00
if err != nil {
if appErr , ok := errors . IsAppError ( err ) ; ok {
c . Error ( appErr )
return
}
logger . ErrorWithFields ( ctx , err , map [ string ] interface { } {
2025-11-25 16:19:26 +00:00
"knowledge_id" : id ,
2025-11-12 13:27:56 +00:00
} )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
2025-11-25 16:19:26 +00:00
logger . Infof ( ctx , "Manual knowledge updated successfully, knowledge ID: %s" , id )
2025-11-12 13:27:56 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : knowledge ,
2026-02-10 04:15:21 +00:00
} )
}
// ReparseKnowledge godoc
// @Summary 重新解析知识
// @Description 删除知识中现有的文档内容并重新解析,使用异步任务方式处理
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识ID"
2026-06-09 12:34:16 +00:00
// @Param body body object false "可选的处理配置覆盖:{\"process_config\": KnowledgeProcessOverrides}"
2026-02-10 04:15:21 +00:00
// @Success 200 {object} map[string]interface{} "重新解析任务已提交"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 403 {object} errors.AppError "权限不足"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge/{id}/reparse [post]
func ( h * KnowledgeHandler ) ReparseKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start re-parsing knowledge" )
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
if id == "" {
logger . Error ( ctx , "Knowledge ID is empty" )
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
// Validate KB access with editor permission (reparse requires write access)
_ , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleEditor )
if err != nil {
c . Error ( err )
return
}
2026-06-09 12:34:16 +00:00
// Optional per-reparse parse config override. Empty body keeps the
// overrides stored at upload time.
var processOverrides * types . KnowledgeProcessOverrides
if c . Request . ContentLength != 0 {
var req struct {
ProcessConfig * types . KnowledgeProcessOverrides ` json:"process_config" `
}
if err := c . ShouldBindJSON ( & req ) ; err != nil {
logger . Error ( ctx , "Failed to parse reparse request body" , err )
c . Error ( errors . NewBadRequestError ( "Invalid reparse request body" ) . WithDetails ( err . Error ( ) ) )
return
}
processOverrides = req . ProcessConfig
}
2026-02-10 04:15:21 +00:00
// Call service to reparse knowledge
2026-06-09 12:34:16 +00:00
knowledge , err := h . kgService . ReparseKnowledge ( effCtx , id , processOverrides )
2026-02-10 04:15:21 +00:00
if err != nil {
if appErr , ok := errors . IsAppError ( err ) ; ok {
c . Error ( appErr )
return
}
logger . ErrorWithFields ( ctx , err , map [ string ] interface { } {
"knowledge_id" : id ,
} )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
logger . Infof ( ctx , "Knowledge reparse task submitted successfully, knowledge ID: %s" , id )
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"message" : "Knowledge reparse task submitted" ,
"data" : knowledge ,
2025-11-12 13:27:56 +00:00
} )
}
feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
2026-05-28 10:26:56 +00:00
// CancelKnowledgeParse godoc
// @Summary 取消知识解析
// @Description 取消进行中的知识解析任务。当前已写入的 chunk / 索引保留,可通过 reparse 接口重新触发解析。已完成 / 已失败 / 删除中的知识不支持取消。
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识ID"
// @Success 200 {object} map[string]interface{} "取消已提交"
// @Failure 400 {object} errors.AppError "状态不支持取消"
// @Failure 403 {object} errors.AppError "权限不足"
// @Failure 404 {object} errors.AppError "知识不存在"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge/{id}/cancel-parse [post]
func ( h * KnowledgeHandler ) CancelKnowledgeParse ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start cancelling knowledge parse" )
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
if id == "" {
logger . Error ( ctx , "Knowledge ID is empty" )
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
// Editor permission — same gate as ReparseKnowledge / DeleteKnowledge.
_ , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleEditor )
if err != nil {
c . Error ( err )
return
}
knowledge , err := h . kgService . CancelKnowledgeParse ( effCtx , id )
if err != nil {
if appErr , ok := errors . IsAppError ( err ) ; ok {
c . Error ( appErr )
return
}
logger . ErrorWithFields ( ctx , err , map [ string ] interface { } {
"knowledge_id" : id ,
} )
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
logger . Infof ( ctx , "Knowledge parse cancelled successfully, knowledge ID: %s" , id )
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"message" : "Knowledge parse cancelled" ,
"data" : knowledge ,
} )
}
2025-11-18 03:48:37 +00:00
type knowledgeTagBatchRequest struct {
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-25 16:18:48 +00:00
Updates map [ string ] [ ] string ` json:"updates" binding:"required,min=1" `
KBID string ` json:"kb_id" ` // Optional: scope to this KB (validates editor access and uses effective tenant for shared KB)
2025-11-18 03:48:37 +00:00
}
2025-12-16 08:40:13 +00:00
// UpdateKnowledgeTagBatch godoc
// @Summary 批量更新知识标签
2026-07-14 08:13:49 +00:00
// @Description 批量更新知识条目的标签。可选 kb_id: 指定时按该知识库校验编辑权限并用于共享知识库的空间解析
2025-12-16 08:40:13 +00:00
// @Tags 知识管理
// @Accept json
// @Produce json
2026-01-30 08:28:01 +00:00
// @Param request body object true "标签更新请求( updates 必填, kb_id 可选)"
2025-12-16 08:40:13 +00:00
// @Success 200 {object} map[string]interface{} "更新成功"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge/tags [put]
2025-11-18 03:48:37 +00:00
func ( h * KnowledgeHandler ) UpdateKnowledgeTagBatch ( c * gin . Context ) {
ctx := c . Request . Context ( )
2026-01-30 08:28:01 +00:00
// Ensure tenant ID is in context (service reads it; may be missing if request context was not set by auth)
tenantID := c . GetUint64 ( types . TenantIDContextKey . String ( ) )
if tenantID == 0 {
c . Error ( errors . NewUnauthorizedError ( "Unauthorized" ) )
return
}
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( ctx , tenantID )
2026-01-30 08:28:01 +00:00
2025-11-18 03:48:37 +00:00
var req knowledgeTagBatchRequest
if err := c . ShouldBindJSON ( & req ) ; err != nil {
logger . Error ( ctx , "Failed to parse knowledge tag batch request" , err )
c . Error ( errors . NewBadRequestError ( "请求参数不合法" ) . WithDetails ( err . Error ( ) ) )
return
}
2026-04-01 13:45:35 +00:00
// Resolve effective tenant and the authorized KB scope.
var authorizedKBID string
2026-01-30 08:28:01 +00:00
if kbID := secutils . SanitizeForLog ( req . KBID ) ; kbID != "" {
2026-09-08 07:36:33 +00:00
_ , _ , effID , permission , err := h . validateKnowledgeBaseWriteAccessWithKBID ( c , kbID )
2026-01-30 08:28:01 +00:00
if err != nil {
c . Error ( err )
return
}
if permission != types . OrgRoleAdmin && permission != types . OrgRoleEditor {
c . Error ( errors . NewForbiddenError ( "No permission to update knowledge tags" ) )
return
}
2026-04-01 13:45:35 +00:00
authorizedKBID = kbID
2026-09-08 07:36:33 +00:00
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effID )
2026-01-30 08:28:01 +00:00
} else if len ( req . Updates ) > 0 {
// No kb_id: infer from first knowledge ID so shared-KB updates work without client sending kb_id
var firstKnowledgeID string
for id := range req . Updates {
firstKnowledgeID = id
break
}
if firstKnowledgeID != "" {
2026-04-01 13:45:35 +00:00
knowledge , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , firstKnowledgeID , types . OrgRoleEditor )
2026-01-30 08:28:01 +00:00
if err != nil {
c . Error ( err )
return
}
2026-04-01 13:45:35 +00:00
authorizedKBID = knowledge . KnowledgeBaseID
2026-01-30 08:28:01 +00:00
ctx = effCtx
}
}
2026-09-08 07:36:33 +00:00
if err := h . requireKBOwnershipOrAdmin ( c , authorizedKBID ) ; err != nil {
_ = c . Error ( err )
return
}
2026-04-01 13:45:35 +00:00
if err := h . kgService . UpdateKnowledgeTagBatch ( ctx , authorizedKBID , req . Updates ) ; err != nil {
2025-11-18 03:48:37 +00:00
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( err )
return
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
} )
}
2025-12-16 08:40:13 +00:00
// UpdateImageInfo godoc
// @Summary 更新图像信息
// @Description 更新知识分块的图像信息
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param id path string true "知识ID"
// @Param chunk_id path string true "分块ID"
// @Param request body object{image_info=string} true "图像信息"
// @Success 200 {object} map[string]interface{} "更新成功"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-16 08:40:13 +00:00
// @Router /knowledge/image/{id}/{chunk_id} [put]
2025-08-05 07:08:07 +00:00
func ( h * KnowledgeHandler ) UpdateImageInfo ( c * gin . Context ) {
ctx := c . Request . Context ( )
logger . Info ( ctx , "Start updating image info" )
2025-11-25 16:19:26 +00:00
id := secutils . SanitizeForLog ( c . Param ( "id" ) )
2025-08-05 07:08:07 +00:00
if id == "" {
logger . Error ( ctx , "Knowledge ID is empty" )
c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
2025-11-25 16:19:26 +00:00
chunkID := secutils . SanitizeForLog ( c . Param ( "chunk_id" ) )
if chunkID == "" {
2025-08-05 07:08:07 +00:00
logger . Error ( ctx , "Chunk ID is empty" )
c . Error ( errors . NewBadRequestError ( "Chunk ID cannot be empty" ) )
return
}
2026-01-30 08:28:01 +00:00
_ , effCtx , err := h . resolveKnowledgeAndValidateKBAccess ( c , id , types . OrgRoleEditor )
if err != nil {
c . Error ( err )
return
}
2025-08-05 07:08:07 +00:00
var request struct {
ImageInfo string ` json:"image_info" `
}
if err := c . ShouldBindJSON ( & request ) ; err != nil {
logger . Error ( ctx , "Failed to parse request parameters" , err )
c . Error ( errors . NewBadRequestError ( err . Error ( ) ) )
return
}
2025-11-25 16:19:26 +00:00
logger . Infof ( ctx , "Updating knowledge chunk, knowledge ID: %s, chunk ID: %s" , id , chunkID )
2026-01-30 08:28:01 +00:00
err = h . kgService . UpdateImageInfo ( effCtx , id , chunkID , secutils . SanitizeForLog ( request . ImageInfo ) )
2025-08-05 07:08:07 +00:00
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
2026-09-08 07:36:33 +00:00
var appErr * errors . AppError
if goerrors . As ( err , & appErr ) {
_ = c . Error ( appErr )
} else {
_ = c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
}
2025-08-05 07:08:07 +00:00
return
}
2025-11-25 16:19:26 +00:00
logger . Infof ( ctx , "Knowledge chunk updated successfully, knowledge ID: %s, chunk ID: %s" , id , chunkID )
2025-08-05 07:08:07 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"message" : "Knowledge chunk image updated successfully" ,
} )
}
2025-12-19 14:17:56 +00:00
// SearchKnowledge godoc
// @Summary Search knowledge
2026-06-24 12:30:56 +00:00
// @Description Search knowledge files by keyword. Pass recent=true without a keyword to browse recent files. When agent_id is set (shared agent), scope is the agent's configured knowledge bases.
2025-12-19 14:17:56 +00:00
// @Tags Knowledge
// @Accept json
// @Produce json
2026-02-04 12:12:29 +00:00
// @Param keyword query string false "Keyword to search"
2026-08-22 09:46:47 +00:00
// @Param offset query int false "Offset for pagination (minimum 0)" minimum(0)
// @Param limit query int false "Limit for pagination (default 20, maximum 100)" minimum(1) maximum(100)
2026-02-04 12:12:29 +00:00
// @Param file_types query string false "Comma-separated file extensions to filter (e.g., csv,xlsx)"
// @Param agent_id query string false "Shared agent ID (search within agent's KB scope)"
2026-06-24 12:30:56 +00:00
// @Param recent query bool false "Return recent files when keyword is empty"
2025-12-31 10:18:22 +00:00
// @Success 200 {object} map[string]interface{} "Search results"
// @Failure 400 {object} errors.AppError "Invalid request"
2025-12-19 14:17:56 +00:00
// @Security Bearer
2025-12-22 06:36:29 +00:00
// @Security ApiKeyAuth
2025-12-19 14:17:56 +00:00
// @Router /knowledge/search [get]
func ( h * KnowledgeHandler ) SearchKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
2026-01-30 08:28:01 +00:00
if userID , ok := c . Get ( types . UserIDContextKey . String ( ) ) ; ok {
ctx = context . WithValue ( ctx , types . UserIDContextKey , userID )
}
2026-06-05 00:01:02 +00:00
// Accept both ?keyword= (legacy / upstream name) and ?query= (what most
2026-06-24 12:30:56 +00:00
// MCP / agent integrations send). Empty input is only valid for an explicit
// recent-file browse request; ordinary callers still get a clear 400 instead
// of silently receiving the same newest cards for every missing query.
2025-12-19 14:17:56 +00:00
keyword := c . Query ( "keyword" )
2026-06-05 00:01:02 +00:00
if keyword == "" {
keyword = c . Query ( "query" )
}
2026-06-24 12:30:56 +00:00
recent , _ := strconv . ParseBool ( c . DefaultQuery ( "recent" , "false" ) )
if strings . TrimSpace ( keyword ) == "" && ! recent {
2026-06-05 00:01:02 +00:00
c . Error ( errors . NewBadRequestError ( "missing search keyword: pass ?keyword=... or ?query=..." ) )
return
}
2026-06-24 12:30:56 +00:00
keyword = strings . TrimSpace ( keyword )
2026-08-22 09:46:47 +00:00
offset , limit , ok := parseOffsetPagination ( c )
if ! ok {
return
}
2025-12-19 14:17:56 +00:00
2025-12-31 10:18:22 +00:00
var fileTypes [ ] string
if fileTypesStr := c . Query ( "file_types" ) ; fileTypesStr != "" {
for _ , ft := range strings . Split ( fileTypesStr , "," ) {
ft = strings . TrimSpace ( ft )
if ft != "" {
fileTypes = append ( fileTypes , ft )
}
}
}
2026-02-04 12:12:29 +00:00
agentID := c . Query ( "agent_id" )
if agentID != "" {
2026-09-08 07:36:33 +00:00
agent , err := resolveSharedAgentForRequest ( c , agentID , h . agentShareService )
2026-02-04 12:12:29 +00:00
if err != nil {
2026-09-08 07:36:33 +00:00
_ = c . Error ( err )
2026-02-04 12:12:29 +00:00
return
}
sourceTenantID := agent . TenantID
2026-09-08 07:36:33 +00:00
ctx = access . WithSharedAgent ( ctx , agent )
scope := types . NewSharedAgentKBScope ( agent )
if scope . IsEmpty ( ) {
2026-02-04 12:12:29 +00:00
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : [ ] interface { } { } ,
"has_more" : false ,
2026-06-29 08:01:58 +00:00
"total" : 0 ,
2026-02-04 12:12:29 +00:00
} )
return
}
var scopes [ ] types . KnowledgeSearchScope
2026-09-08 07:36:33 +00:00
if ! scope . IsAll ( ) {
for _ , kbID := range scope . IDs ( ) {
2026-02-04 12:12:29 +00:00
if kbID != "" {
scopes = append ( scopes , types . KnowledgeSearchScope { TenantID : sourceTenantID , KBID : kbID } )
}
}
}
2026-09-08 07:36:33 +00:00
if scope . IsAll ( ) {
2026-02-04 12:12:29 +00:00
kbs , err := h . kbService . ListKnowledgeBasesByTenantID ( ctx , sourceTenantID )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( "Failed to list knowledge bases" ) . WithDetails ( err . Error ( ) ) )
return
}
2026-09-08 07:36:33 +00:00
for _ , kb := range filterKnowledgeBasesForSharedAgent ( kbs , agent ) {
if kb . Type == types . KnowledgeBaseTypeDocument {
scopes = append ( scopes , types . KnowledgeSearchScope { TenantID : kb . TenantID , KBID : kb . ID } )
2026-04-23 09:22:18 +00:00
}
2026-02-04 12:12:29 +00:00
}
}
2026-07-05 02:55:00 +00:00
scopes = filterKnowledgeSearchScopesForAPIKey ( ctx , scopes )
if len ( scopes ) == 0 {
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : [ ] interface { } { } ,
"has_more" : false ,
"total" : 0 ,
} )
return
}
knowledges , hasMore , total , err := h . kgService . SearchKnowledgeForScopes ( ctx , scopes , keyword , offset , limit , fileTypes )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( "Failed to search knowledge" ) . WithDetails ( err . Error ( ) ) )
return
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : knowledges ,
"has_more" : hasMore ,
"total" : total ,
} )
return
}
if scopes , restricted := tenantAPIKeySearchScopes ( ctx ) ; restricted {
if len ( scopes ) == 0 {
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : [ ] interface { } { } ,
"has_more" : false ,
"total" : 0 ,
} )
return
}
2026-06-29 08:01:58 +00:00
knowledges , hasMore , total , err := h . kgService . SearchKnowledgeForScopes ( ctx , scopes , keyword , offset , limit , fileTypes )
2026-02-04 12:12:29 +00:00
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( "Failed to search knowledge" ) . WithDetails ( err . Error ( ) ) )
return
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : knowledges ,
"has_more" : hasMore ,
2026-06-29 08:01:58 +00:00
"total" : total ,
2026-02-04 12:12:29 +00:00
} )
return
}
// Default: own + shared KBs
2026-06-29 08:01:58 +00:00
knowledges , hasMore , total , err := h . kgService . SearchKnowledge ( ctx , keyword , offset , limit , fileTypes )
2025-12-19 14:17:56 +00:00
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( errors . NewInternalServerError ( "Failed to search knowledge" ) . WithDetails ( err . Error ( ) ) )
return
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : knowledges ,
"has_more" : hasMore ,
2026-06-29 08:01:58 +00:00
"total" : total ,
2025-12-19 14:17:56 +00:00
} )
}
2026-03-09 05:14:42 +00:00
// MoveKnowledgeRequest defines the request for moving knowledge items
type MoveKnowledgeRequest struct {
KnowledgeIDs [ ] string ` json:"knowledge_ids" binding:"required,min=1" `
SourceKBID string ` json:"source_kb_id" binding:"required" `
TargetKBID string ` json:"target_kb_id" binding:"required" `
Mode string ` json:"mode" binding:"required,oneof=reuse_vectors reparse" `
}
// MoveKnowledgeResponse defines the response for move knowledge
type MoveKnowledgeResponse struct {
TaskID string ` json:"task_id" `
SourceKBID string ` json:"source_kb_id" `
TargetKBID string ` json:"target_kb_id" `
KnowledgeCount int ` json:"knowledge_count" `
Message string ` json:"message" `
}
// MoveKnowledge moves knowledge items from one knowledge base to another (async task).
2026-05-11 09:37:14 +00:00
//
// MoveKnowledge godoc
// @Summary 移动知识到其他知识库
// @Description 将一条或多条知识从源知识库移动到目标知识库(异步),返回任务 ID 用于查询进度
// @Tags 知识
// @Accept json
// @Produce json
// @Param request body handler.MoveKnowledgeRequest true "{source_kb_id, target_kb_id, knowledge_ids}"
// @Success 200 {object} handler.MoveKnowledgeResponse "任务信息"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge/move [post]
2026-03-09 05:14:42 +00:00
func ( h * KnowledgeHandler ) MoveKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
var req MoveKnowledgeRequest
if err := c . ShouldBindJSON ( & req ) ; err != nil {
logger . Error ( ctx , "MoveKnowledge: failed to parse request" , err )
c . Error ( errors . NewBadRequestError ( "Invalid request parameters: " + err . Error ( ) ) )
return
}
// Validate source != target
if req . SourceKBID == req . TargetKBID {
c . Error ( errors . NewBadRequestError ( "Source and target knowledge base cannot be the same" ) )
return
}
tenantID , exists := c . Get ( types . TenantIDContextKey . String ( ) )
if ! exists {
c . Error ( errors . NewUnauthorizedError ( "Unauthorized" ) )
return
}
2026-07-05 02:55:00 +00:00
if err := requireTenantAPIKeyKnowledgeBases ( ctx , req . SourceKBID , req . TargetKBID ) ; err != nil {
c . Error ( err )
return
}
2026-03-09 05:14:42 +00:00
// Validate source KB
sourceKB , err := h . kbService . GetKnowledgeBaseByID ( ctx , req . SourceKBID )
if err != nil {
if goerrors . Is ( err , repository . ErrKnowledgeBaseNotFound ) {
c . Error ( errors . NewNotFoundError ( "Source knowledge base not found" ) )
return
}
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
if sourceKB . TenantID != tenantID . ( uint64 ) {
c . Error ( errors . NewForbiddenError ( "No permission to access source knowledge base" ) )
return
}
2026-07-02 06:49:27 +00:00
if err := h . requireKBOwnershipOrAdmin ( c , req . SourceKBID ) ; err != nil {
c . Error ( err )
return
}
2026-03-09 05:14:42 +00:00
// Validate target KB
targetKB , err := h . kbService . GetKnowledgeBaseByID ( ctx , req . TargetKBID )
if err != nil {
if goerrors . Is ( err , repository . ErrKnowledgeBaseNotFound ) {
c . Error ( errors . NewNotFoundError ( "Target knowledge base not found" ) )
return
}
c . Error ( errors . NewInternalServerError ( err . Error ( ) ) )
return
}
if targetKB . TenantID != tenantID . ( uint64 ) {
c . Error ( errors . NewForbiddenError ( "No permission to access target knowledge base" ) )
return
}
2026-07-02 06:49:27 +00:00
if err := h . requireKBOwnershipOrAdmin ( c , req . TargetKBID ) ; err != nil {
c . Error ( err )
return
}
2026-03-09 05:14:42 +00:00
2026-09-08 07:36:33 +00:00
// Resolve explicit write grants independently for both body IDs.
if _ , err := resolveHandlerKBAccessFor ( c , req . SourceKBID , h . kbService , nil , nil , types . OrgRoleEditor ) ; err != nil {
_ = c . Error ( err )
2026-03-09 05:14:42 +00:00
return
}
2026-09-08 07:36:33 +00:00
if _ , err := resolveHandlerKBAccessFor ( c , req . TargetKBID , h . kbService , nil , nil , types . OrgRoleEditor ) ; err != nil {
_ = c . Error ( err )
2026-03-09 05:14:42 +00:00
return
}
2026-09-08 07:36:33 +00:00
taskID := utils . GenerateTaskID ( "kg_move" , tenantID . ( uint64 ) , req . SourceKBID )
ctx , err = access . WithKBTransfer ( c . Request . Context ( ) , sourceKB , targetKB , access . KBTransferMove , taskID , false )
if err != nil {
_ = c . Error ( kbAccessHTTPError ( err ) )
return
}
tenant , _ := ctx . Value ( types . TenantInfoContextKey ) . ( * types . Tenant )
if err := access . ValidateKBTransferCompatibility ( sourceKB ,
targetKB ,
access . KBTransferMove ,
req . Mode ,
tenant ) ; err != nil {
_ = c . Error ( errors . NewBadRequestError ( err . Error ( ) ) )
2026-06-05 01:13:45 +00:00
return
}
2026-09-08 07:36:33 +00:00
uniqueIDs := make ( [ ] string , 0 , len ( req . KnowledgeIDs ) )
seen := make ( map [ string ] bool , len ( req . KnowledgeIDs ) )
for _ , id := range req . KnowledgeIDs {
if strings . TrimSpace ( id ) == "" {
_ = c . Error ( errors . NewBadRequestError ( "Knowledge ID cannot be empty" ) )
return
}
if ! seen [ id ] {
seen [ id ] = true
uniqueIDs = append ( uniqueIDs , id )
}
}
req . KnowledgeIDs = uniqueIDs
2026-06-05 01:13:45 +00:00
2026-03-09 05:14:42 +00:00
// Validate all knowledge IDs belong to source KB and are in completed status
for _ , kID := range req . KnowledgeIDs {
knowledge , err := h . kgService . GetKnowledgeByID ( ctx , kID )
if err != nil {
c . Error ( errors . NewBadRequestError ( fmt . Sprintf ( "Knowledge item %s not found" , kID ) ) )
return
}
2026-09-08 07:36:33 +00:00
if knowledge == nil || knowledge . ID != kID || knowledge . TenantID != tenantID . ( uint64 ) ||
knowledge . KnowledgeBaseID != req . SourceKBID {
_ = c . Error (
errors . NewBadRequestError (
fmt . Sprintf ( "Knowledge item %s does not belong to the source knowledge base" , kID ) ,
) ,
)
2026-03-09 05:14:42 +00:00
return
}
if knowledge . ParseStatus != types . ParseStatusCompleted {
c . Error ( errors . NewBadRequestError ( fmt . Sprintf ( "Knowledge item %s is not in completed status (current: %s)" , kID , knowledge . ParseStatus ) ) )
return
}
}
// Create move payload
payload := types . KnowledgeMovePayload {
TenantID : tenantID . ( uint64 ) ,
TaskID : taskID ,
KnowledgeIDs : req . KnowledgeIDs ,
SourceKBID : req . SourceKBID ,
TargetKBID : req . TargetKBID ,
Mode : req . Mode ,
2026-07-22 11:15:27 +00:00
Initiator : types . TaskInitiatorFromContext ( ctx ) ,
2026-03-09 05:14:42 +00:00
}
feat(observability): extend Langfuse tracing across asynq pipeline
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.
This change threads one trace end-to-end:
- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
worker can attach to an upstream trace without emitting a duplicate
trace-create; StartGeneration now auto-picks parentObservationId from
ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
payloads so trace_id / parent_obs_id / user_id / session_id serialise
into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
resumes the upstream trace or opens a standalone asynq.<type> trace
for scheduled jobs, and wraps the handler in a SPAN with task metadata
(id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
ResumeTrace no-trace-create guarantee; asynq_test.go covers
InjectTracing round-trip, middleware resume path, and standalone
trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
and documents the cross-process propagation model.
No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
2026-04-24 04:23:30 +00:00
langfuse . InjectTracing ( ctx , & payload )
2026-03-09 05:14:42 +00:00
payloadBytes , err := json . Marshal ( payload )
if err != nil {
logger . Errorf ( ctx , "MoveKnowledge: failed to marshal payload: %v" , err )
c . Error ( errors . NewInternalServerError ( "Failed to create task" ) )
return
}
// Enqueue move task
task := asynq . NewTask ( types . TypeKnowledgeMove , payloadBytes ,
2026-07-10 15:19:41 +00:00
asynq . TaskID ( taskID ) , asynq . Queue ( types . QueueMaintenance ) ,
asynq . MaxRetry ( 3 ) , asynq . Timeout ( 2 * time . Hour ) )
2026-03-09 05:14:42 +00:00
info , err := h . asynqClient . Enqueue ( task )
if err != nil {
logger . Errorf ( ctx , "MoveKnowledge: failed to enqueue task: %v" , err )
c . Error ( errors . NewInternalServerError ( "Failed to enqueue task" ) )
return
}
logger . Infof ( ctx , "MoveKnowledge: task enqueued: %s, asynq_id: %s, source: %s, target: %s, count: %d" ,
taskID , info . ID , secutils . SanitizeForLog ( req . SourceKBID ) , secutils . SanitizeForLog ( req . TargetKBID ) , len ( req . KnowledgeIDs ) )
// Save initial progress
initialProgress := & types . KnowledgeMoveProgress {
TaskID : taskID ,
SourceKBID : req . SourceKBID ,
TargetKBID : req . TargetKBID ,
Status : types . KBCloneStatusPending ,
Total : len ( req . KnowledgeIDs ) ,
Progress : 0 ,
Message : "Task queued, waiting to start..." ,
CreatedAt : time . Now ( ) . Unix ( ) ,
UpdatedAt : time . Now ( ) . Unix ( ) ,
}
if err := h . kgService . SaveKnowledgeMoveProgress ( ctx , initialProgress ) ; err != nil {
logger . Warnf ( ctx , "MoveKnowledge: failed to save initial progress: %v" , err )
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : MoveKnowledgeResponse {
TaskID : taskID ,
SourceKBID : req . SourceKBID ,
TargetKBID : req . TargetKBID ,
KnowledgeCount : len ( req . KnowledgeIDs ) ,
Message : "Knowledge move task started" ,
} ,
} )
}
// GetKnowledgeMoveProgress retrieves the progress of a knowledge move task.
2026-05-11 09:37:14 +00:00
//
// GetKnowledgeMoveProgress godoc
// @Summary 获取知识移动进度
// @Description 按任务 ID 查询移动进度
// @Tags 知识
// @Produce json
// @Param task_id path string true "移动任务 ID"
// @Success 200 {object} types.KnowledgeMoveProgress "进度信息"
// @Failure 404 {object} errors.AppError "任务不存在"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge/move/progress/{task_id} [get]
2026-03-09 05:14:42 +00:00
func ( h * KnowledgeHandler ) GetKnowledgeMoveProgress ( c * gin . Context ) {
ctx := c . Request . Context ( )
taskID := c . Param ( "task_id" )
if taskID == "" {
c . Error ( errors . NewBadRequestError ( "Task ID cannot be empty" ) )
return
}
2026-07-02 03:59:50 +00:00
if err := requireTaskProgressTenant ( ctx , taskID ) ; err != nil {
c . Error ( err )
return
}
2026-03-09 05:14:42 +00:00
progress , err := h . kgService . GetKnowledgeMoveProgress ( ctx , taskID )
if err != nil {
logger . ErrorWithFields ( ctx , err , nil )
c . Error ( err )
return
}
c . JSON ( http . StatusOK , gin . H {
"success" : true ,
"data" : progress ,
} )
}
2026-04-01 13:45:35 +00:00
feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-25 16:18:48 +00:00
// parseCommaSeparatedTagIDs splits a comma-separated string of tag IDs and
// filters out empty strings and the "__untagged__" sentinel value.
func parseCommaSeparatedTagIDs ( raw string ) [ ] string {
if raw == "" {
return nil
}
parts := strings . Split ( raw , "," )
result := make ( [ ] string , 0 , len ( parts ) )
for _ , p := range parts {
p = strings . TrimSpace ( p )
if p == "" || p == "__untagged__" {
continue
}
result = append ( result , p )
}
return result
}
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
2026-05-12 10:16:34 +00:00
// parseFilterTime parses a query-string timestamp accepted by knowledge list
// filters. It supports RFC3339, RFC3339 with milliseconds, and the date-only
// "2006-01-02" form (interpreted at start of day in the local timezone).
func parseFilterTime ( raw string ) ( time . Time , error ) {
raw = strings . TrimSpace ( raw )
if raw == "" {
return time . Time { } , nil
}
layouts := [ ] string { time . RFC3339Nano , time . RFC3339 , "2006-01-02 15:04:05" , "2006-01-02" }
var lastErr error
for _ , layout := range layouts {
if t , err := time . ParseInLocation ( layout , raw , time . Local ) ; err == nil {
return t , nil
} else {
lastErr = err
}
}
return time . Time { } , lastErr
}
2026-06-25 13:03:11 +00:00
type batchReparseKnowledgeRequest struct {
KBID string ` json:"kb_id" binding:"required" `
IDs [ ] string ` json:"ids" binding:"required" `
ProcessConfig * types . KnowledgeProcessOverrides ` json:"process_config,omitempty" `
}
// BatchReparseKnowledge godoc
// @Summary 批量重新解析知识
// @Description 按 ID 列表批量重新解析单个知识库下的多个知识条目
// @Tags 知识管理
// @Accept json
// @Produce json
// @Param request body batchReparseKnowledgeRequest true "批量重解析请求"
// @Success 200 {object} map[string]interface{} "任务已提交"
// @Failure 400 {object} errors.AppError "请求参数错误"
// @Failure 403 {object} errors.AppError "权限不足"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /knowledge/batch-reparse [post]
func ( h * KnowledgeHandler ) BatchReparseKnowledge ( c * gin . Context ) {
ctx := c . Request . Context ( )
var req batchReparseKnowledgeRequest
if err := c . ShouldBindJSON ( & req ) ; err != nil {
logger . Errorf ( ctx , "failed to parse batch reparse knowledge request: %v" , err )
c . Error ( errors . NewBadRequestError ( "invalid batch reparse knowledge request parameters" ) )
return
}
2026-06-25 15:51:38 +00:00
seen := make ( map [ string ] struct { } , len ( req . IDs ) )
ids := make ( [ ] string , 0 , len ( req . IDs ) )
for _ , raw := range req . IDs {
id := strings . TrimSpace ( raw )
if id == "" {
continue
}
if _ , ok := seen [ id ] ; ok {
continue
2026-06-25 13:03:11 +00:00
}
2026-06-25 15:51:38 +00:00
seen [ id ] = struct { } { }
ids = append ( ids , id )
2026-06-25 13:03:11 +00:00
}
if len ( ids ) == 0 {
c . Error ( errors . NewBadRequestError ( "no knowledge IDs provided for batch reparse" ) )
return
}
2026-06-25 15:51:38 +00:00
const maxBatch = 200
if len ( ids ) > maxBatch {
c . Error ( errors . NewBadRequestError ( fmt . Sprintf ( "too many ids (max %d per batch)" , maxBatch ) ) )
2026-06-25 13:03:11 +00:00
return
}
2026-09-08 07:36:33 +00:00
_ , kbID , effectiveTenantID , permission , err := h . validateKnowledgeBaseWriteAccessWithKBID ( c , req . KBID )
2026-06-25 13:03:11 +00:00
if err != nil {
c . Error ( err )
return
}
if permission != types . OrgRoleAdmin && permission != types . OrgRoleEditor {
c . Error ( errors . NewForbiddenError ( "no permission to reparse knowledge in this kb" ) )
return
}
2026-09-08 07:36:33 +00:00
if err := h . requireKBOwnershipOrAdmin ( c , kbID ) ; err != nil {
_ = c . Error ( err )
return
}
ctx = types . WithExecutionTenant ( c . Request . Context ( ) , effectiveTenantID )
2026-06-25 15:51:38 +00:00
knowledgeList , err := h . kgService . GetKnowledgeBatch ( ctx , effectiveTenantID , ids )
2026-06-25 13:03:11 +00:00
if err != nil {
2026-06-25 15:51:38 +00:00
logger . Errorf ( ctx , "failed to get knowledge batch, kb_id: %s, size: %d, err: %v" , kbID , len ( ids ) , err )
2026-06-25 13:03:11 +00:00
c . Error ( errors . NewInternalServerError ( "failed to get knowledge batch" ) )
return
}
2026-06-25 15:51:38 +00:00
if len ( knowledgeList ) != len ( ids ) {
2026-06-25 13:03:11 +00:00
c . Error ( errors . NewBadRequestError ( "some knowledge entries were not found" ) )
return
}
2026-06-25 15:51:38 +00:00
for _ , k := range knowledgeList {
2026-09-08 07:36:33 +00:00
if err := access . RejectMovingKnowledge ( k ) ; err != nil {
_ = c . Error ( err )
return
}
2026-06-25 15:51:38 +00:00
if k . KnowledgeBaseID != kbID {
c . Error ( errors . NewBadRequestError (
fmt . Sprintf ( "Knowledge %s does not belong to knowledge base %s" ,
secutils . SanitizeForLog ( k . ID ) , secutils . SanitizeForLog ( kbID ) ) ) )
2026-06-25 13:03:11 +00:00
return
}
}
2026-09-08 07:36:33 +00:00
taskID , err := h . enqueueKnowledgeListReparse ( ctx , effectiveTenantID , kbID , ids , req . ProcessConfig )
2026-06-25 15:51:38 +00:00
if err != nil {
logger . Errorf ( ctx , "Failed to enqueue batch knowledge reparse task: %v" , err )
c . Error ( errors . NewInternalServerError ( "Failed to enqueue batch reparse task" ) )
2026-06-25 13:03:11 +00:00
return
}
2026-06-25 15:51:38 +00:00
logger . Infof ( ctx , "Batch knowledge reparse task enqueued: %s, kb_id: %s, count: %d" ,
taskID , secutils . SanitizeForLog ( kbID ) , len ( ids ) )
c . JSON ( http . StatusOK , gin . H {
2026-06-25 13:03:11 +00:00
"success" : true ,
2026-06-25 15:51:38 +00:00
"message" : "Batch reparse task submitted" ,
"data" : gin . H {
"task_id" : taskID ,
"reparse_count" : len ( ids ) ,
} ,
2026-06-25 13:03:11 +00:00
} )
}
2026-07-05 02:55:00 +00:00
func requireTenantAPIKeyKnowledgeBase ( ctx context . Context , kbID string ) error {
return requireTenantAPIKeyKnowledgeBases ( ctx , kbID )
}
func requireTenantAPIKeyKnowledgeBases ( ctx context . Context , kbIDs ... string ) error {
2026-07-06 05:23:55 +00:00
return types . AuthorizeTenantAPIKeyKnowledgeBases ( ctx , kbIDs ... )
2026-07-05 02:55:00 +00:00
}
func tenantAPIKeyAllowedKBSet ( ctx context . Context ) map [ string ] bool {
scope , ok := types . TenantAPIKeyScopeFromContext ( ctx )
if ! ok || ! scope . IsKnowledgeBaseRestricted ( ) {
return nil
}
allowed := make ( map [ string ] bool , len ( scope . KnowledgeBaseIDs ) )
for _ , id := range scope . KnowledgeBaseIDs {
allowed [ id ] = true
}
return allowed
}
func intersectKBAllowSet ( base , restrict map [ string ] bool ) map [ string ] bool {
if restrict == nil {
return base
}
if base == nil {
return restrict
}
out := make ( map [ string ] bool )
for id := range base {
if restrict [ id ] {
out [ id ] = true
}
}
return out
}
func filterKnowledgesByKBAllowSet ( knowledges [ ] * types . Knowledge , allowed map [ string ] bool ) [ ] * types . Knowledge {
if allowed == nil {
return knowledges
}
filtered := make ( [ ] * types . Knowledge , 0 , len ( knowledges ) )
for _ , k := range knowledges {
if k != nil && allowed [ k . KnowledgeBaseID ] {
filtered = append ( filtered , k )
}
}
return filtered
}
func filterKnowledgeSearchScopesForAPIKey ( ctx context . Context , scopes [ ] types . KnowledgeSearchScope ) [ ] types . KnowledgeSearchScope {
allowed := tenantAPIKeyAllowedKBSet ( ctx )
if allowed == nil {
return scopes
}
filtered := make ( [ ] types . KnowledgeSearchScope , 0 , len ( scopes ) )
for _ , scope := range scopes {
if allowed [ scope . KBID ] {
filtered = append ( filtered , scope )
}
}
return filtered
}
func tenantAPIKeySearchScopes ( ctx context . Context ) ( [ ] types . KnowledgeSearchScope , bool ) {
scope , ok := types . TenantAPIKeyScopeFromContext ( ctx )
if ! ok || ! scope . IsKnowledgeBaseRestricted ( ) {
return nil , false
}
tenantID , ok := types . TenantIDFromContext ( ctx )
if ! ok || tenantID == 0 {
return nil , true
}
scopes := make ( [ ] types . KnowledgeSearchScope , 0 , len ( scope . KnowledgeBaseIDs ) )
for _ , kbID := range scope . KnowledgeBaseIDs {
scopes = append ( scopes , types . KnowledgeSearchScope { TenantID : tenantID , KBID : kbID } )
}
return scopes , true
}