WeKnora/internal/application/service
wizardchen 492e92580b feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).

Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
  LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
  the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
  so logger.CloneContext can preserve it across handler / goroutine
  boundaries (otherwise each LLM call auto-created an orphan trace,
  fragmenting one request into many)

Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes

Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.

Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
  reuses WeKnora's existing PostgreSQL (separate database via an idempotent
  init container that fixes ICU collation drift) and Redis (separate DB
  number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
  to avoid Prisma P1013 when passwords contain @ / # / etc.

Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-24 10:29:19 +08:00
..
chat_pipeline feat: Enhance wiki API and repository with slug encoding and improved search functionality 2026-04-22 21:18:09 +08:00
file fix(storage): auto-detect content-type to resolve image rendering issues 2026-04-20 21:59:54 +08:00
llmcontext feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
memory feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
metric fix: resolve failing test and build errors 2026-04-22 21:17:29 +08:00
retriever feat: add registry dual-map and engine factory for VectorStore 2026-04-17 00:15:37 +08:00
agent_service.go feat: Enhance agent retrieval strategies and regex handling 2026-04-22 21:18:21 +08:00
agent_share.go fix(agent-share): don't require rerank model for wiki-only agents 2026-04-24 10:21:05 +08:00
chunk.go feat: enhance document loading and state management in doc-content component 2026-03-02 21:21:49 +08:00
custom_agent.go feat: Enhance suggested questions functionality with wiki integration 2026-04-22 21:18:20 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource_service.go feat(notion): implement Connector interface, markdown renderer, and DI registration 2026-04-08 16:31:25 +08:00
evaluation.go feat(knowledge): add channel support to knowledge entries and related components 2026-03-25 22:08:29 +08:00
extract.go feat(prompts): update assistant descriptions and add intent-specific templates 2026-03-25 22:08:29 +08:00
graph.go feat(graph): enhance graph extraction prompt rendering and language context handling 2026-03-25 22:08:29 +08:00
image_multimodal.go feat: Implement conditional indexing for multimodal chunks based on embedding model availability 2026-04-22 21:18:29 +08:00
kbshare.go feat(api): integrate per-organization resource counts into organization listing 2026-02-09 11:40:25 +08:00
knowledge.go feat: Enhance wiki cleanup and reparse logic for improved consistency 2026-04-23 10:47:53 +08:00
knowledge_image_cleanup_test.go fix(knowledge): delete extracted images from storage when knowledge is removed 2026-04-02 12:04:04 +08:00
knowledge_manual_test.go fix: sanitize tab chars and avoid double .md extension in manual download filename 2026-03-16 18:18:23 +08:00
knowledge_post_process.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
knowledgebase.go feat: Refactor knowledge base service and router for improved wiki handling 2026-04-22 21:18:32 +08:00
knowledgebase_search.go feat: Update agent type presets and prompt templates for improved knowledge base handling 2026-04-22 21:18:34 +08:00
knowledgebase_search_faq.go feat: enhance input field and agent stream display with improved handling and sanitization 2026-03-16 02:26:47 +08:00
knowledgebase_search_fusion.go feat(repository): update keyword retrieval logic to use ParadeDB's matching operator 2026-04-15 20:16:02 +08:00
knowledgebase_search_results.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
knowledgebase_search_shared.go feat: enhance input field and agent stream display with improved handling and sanitization 2026-03-16 02:26:47 +08:00
mcp_service.go feat: add built-in MCP service support 2026-03-05 21:57:01 +08:00
message.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
metric_hook.go refactor: Clean up code by removing unnecessary blank lines and improving comments for better readability and maintainability 2025-11-28 16:10:37 +08:00
model.go feat(models): support custom HTTP headers across all remote model calls 2026-04-23 23:36:40 +08:00
ocr_sanitizer.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
ocr_sanitizer_test.go feat: introduce WeKnora Lite edition with lightweight configuration and deployment 2026-03-02 21:21:49 +08:00
organization.go feat(i18n): update localization for knowledge bases and organization features 2026-02-09 11:40:25 +08:00
session.go feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
session_agent_qa.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
session_knowledge_qa.go feat: Enhance agent tool compatibility and knowledge base filtering 2026-04-23 17:28:00 +08:00
session_qa_helpers.go fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
skill_service.go feat: adapt preloaded skills to smart-reasoning agent 2026-02-04 20:08:49 +08:00
tag.go feat: enhance document loading and state management in doc-content component 2026-03-02 21:21:49 +08:00
tenant.go feat(tenant): add storage bucket uniqueness validation for tenant creation and updates 2026-04-16 18:13:20 +08:00
user.go simplify oidc callback result payload 2026-04-01 20:47:18 +08:00
vectorstore.go feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
vectorstore_healthcheck.go feat: add VectorStore service layer with types and connection testing 2026-04-13 22:57:17 +08:00
vectorstore_test.go feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
web_search.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_provider.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_state.go fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_ingest.go feat: Implement auto-linked content updates in wiki page management 2026-04-23 11:16:47 +08:00
wiki_ingest_batch.go feat: Enhance wiki cleanup and reparse logic for improved consistency 2026-04-23 10:47:53 +08:00
wiki_ingest_cite.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_cite_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_test.go feat: Implement comprehensive wiki feature for knowledge bases 2026-04-22 21:17:18 +08:00
wiki_linkify.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_linkify_test.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_lint.go feat: Implement auto-linked content updates in wiki page management 2026-04-23 11:16:47 +08:00
wiki_page.go feat: Implement auto-linked content updates in wiki page management 2026-04-23 11:16:47 +08:00
wiki_page_test.go refactor: Update wiki link syntax and rendering in WikiBrowser component 2026-04-22 21:17:26 +08:00