WeKnora/internal/types
wizardchen 492e92580b feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).

Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
  LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
  the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
  so logger.CloneContext can preserve it across handler / goroutine
  boundaries (otherwise each LLM call auto-created an orphan trace,
  fragmenting one request into many)

Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes

Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.

Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
  reuses WeKnora's existing PostgreSQL (separate database via an idempotent
  init container that fixes ICU collation drift) and Redis (separate DB
  number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
  to avoid Prisma P1013 when passwords contain @ / # / etc.

Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-24 10:29:19 +08:00
..
interfaces feat: Implement auto-linked content updates in wiki page management 2026-04-23 11:16:47 +08:00
agent.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
agent_test.go Fix system prompt not being used when flag is false 2026-04-15 20:21:12 +08:00
agent_type_preset.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
builtin_agent_config.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
chat.go add oidc 2026-03-30 11:13:44 +08:00
chat_history_config.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
chat_manage.go fix: missing deep copy entity 2026-04-15 20:35:48 +08:00
chunk.go feat: Implement comprehensive wiki feature for knowledge bases 2026-04-22 21:17:18 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
const.go feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
context_helpers.go fix(i18n): use WEKNORA_LANGUAGE env for prompt language fallback instead of hardcoded zh-CN 2026-04-02 12:01:25 +08:00
context_helpers_test.go refactor: Replace hardcoded language logic with middleware infrastructure 2026-04-22 21:17:17 +08:00
custom_agent.go feat: Update built-in agent display order and clarify internal agent usage 2026-04-22 21:18:30 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource.go feat(notion): add Notion API client and type definitions 2026-04-08 16:31:25 +08:00
docparser.go feat(video): remove video upload support and update related terminology 2026-04-14 22:59:14 +08:00
embedding.go perf: 优化FAQ索引更新和删除性能 2026-01-14 17:30:04 +08:00
errors.go init commit 2025-08-05 15:08:07 +08:00
evaluation.go refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
event_bus.go refactor: Implement event-driven streaming and enhance session management 2025-11-07 22:41:41 +08:00
extract_graph.go Refactor knowledge post-processing into orchestrator task 2026-04-16 18:13:20 +08:00
faq.go feat(types): enhance FAQ metadata normalization and sanitization 2026-03-19 18:20:09 +08:00
faq_test.go feat(chunk): implement batch deletion for chunks to avoid MySQL placeholder limit 2026-03-19 18:20:09 +08:00
graph.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
indexing_strategy.go feat: Add comprehensive documentation for WeKnora knowledge base and wiki features 2026-04-22 21:18:10 +08:00
json.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
json_test.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
knowledge.go feat(datasource): add Yuque (语雀) connector 2026-04-22 16:39:55 +08:00
knowledgebase.go feat: Refactor knowledge base service and router for improved wiki handling 2026-04-22 21:18:32 +08:00
knowledgebase_test.go feat: add vector_store_id column to knowledge_bases for per-KB vector store binding 2026-04-20 19:53:44 +08:00
mcp.go fix: use service name for MCP tool naming with collision protection (#715) 2026-03-21 10:22:05 +08:00
memory.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
message.go feat(chat_pipeline): enhance query handling with image and attachment metadata 2026-04-14 18:21:12 +08:00
model.go feat(models): support custom HTTP headers across all remote model calls 2026-04-23 23:36:40 +08:00
organization.go feat(api): integrate per-organization resource counts into organization listing 2026-02-09 11:40:25 +08:00
placeholder.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
qa_request.go feat: attachment_support 2026-04-10 12:49:20 +08:00
retrieval_config.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
retriever.go feat: support weaviate vectordb for knowledge retrieve 2026-03-09 10:36:16 +08:00
search.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
session.go feat: 添加思考模式配置支持 2025-12-31 15:36:20 +08:00
tag.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
task.go feat: Add pending tasks and active status to WikiStats 2026-04-22 21:17:31 +08:00
tenant.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
user.go add oidc 2026-03-30 11:13:44 +08:00
vectorstore.go feat: add VectorStore management UI settings page 2026-04-17 21:51:25 +08:00
vectorstore_test.go feat: add VectorStore management UI settings page 2026-04-17 21:51:25 +08:00
web_search.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_provider.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_page.go feat: Refactor wiki page update logic for improved version control 2026-04-22 21:28:20 +08:00
wiki_page_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00