WeKnora/docker-compose.dev.yml

534 lines
19 KiB
YAML
Raw Normal View History

# 开发环境配置 - 只启动基础设施服务,app 和 frontend 在本地运行
services:
# 只启动依赖的基础设施服务
postgres:
image: paradedb/paradedb:v0.22.2-pg17
container_name: WeKnora-postgres-dev
ports:
- "${DB_PORT:-5432}:5432"
environment:
- POSTGRES_USER=${DB_USER}
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_DB=${DB_NAME}
volumes:
- postgres-data-dev:/var/lib/postgresql/data
networks:
- WeKnora-network-dev
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"]
interval: 10s
timeout: 10s
retries: 3
start_period: 30s
restart: unless-stopped
stop_grace_period: 1m
redis:
image: redis:7.0-alpine
container_name: WeKnora-redis-dev
ports:
- "${REDIS_PORT:-6379}:6379"
volumes:
- redis_data_dev:/data
command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
restart: always
networks:
- WeKnora-network-dev
# See docker-compose.yml searxng-init for rationale (avoid in-place rewrite of
# the repo-tracked settings.yml by the searxng entrypoint).
searxng-init:
image: busybox:1.36
container_name: WeKnora-searxng-init-dev
command: ["sh", "-c", "cp /template/settings.yml /etc/searxng/settings.yml && chmod 0644 /etc/searxng/settings.yml"]
volumes:
- ./docker/searxng/settings.yml:/template/settings.yml:ro
- searxng_config_dev:/etc/searxng
restart: "no"
networks:
- WeKnora-network-dev
profiles:
- searxng
- full
searxng:
image: searxng/searxng:latest
container_name: WeKnora-searxng-dev
# Local dev: bind to loopback by default; the host-side WeKnora process
# reaches it at http://127.0.0.1:${SEARXNG_PORT}.
ports:
- "${SEARXNG_BIND:-127.0.0.1}:${SEARXNG_PORT:-8888}:8080"
volumes:
- searxng_config_dev:/etc/searxng
environment:
- SEARXNG_BASE_URL=http://localhost:${SEARXNG_PORT:-8888}/
- INSTANCE_NAME=weknora-searxng-dev
# See docker-compose.yml for rationale on the default secret.
- SEARXNG_SECRET=${SEARXNG_SECRET:-weknora-default-searxng-secret-rotate-before-exposing-publicly}
cap_drop:
- ALL
cap_add:
- CHOWN
- SETGID
- SETUID
restart: unless-stopped
depends_on:
searxng-init:
condition: service_completed_successfully
networks:
- WeKnora-network-dev
profiles:
- searxng
- full
minio:
image: minio/minio:latest
container_name: WeKnora-minio-dev
ports:
- "${MINIO_PORT:-9000}:9000"
- "${MINIO_CONSOLE_PORT:-9001}:9001"
environment:
- MINIO_ROOT_USER=${MINIO_ACCESS_KEY_ID:-minioadmin}
- MINIO_ROOT_PASSWORD=${MINIO_SECRET_ACCESS_KEY:-minioadmin}
command: server --console-address ":9001" /data
volumes:
- minio_data_dev:/data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
networks:
- WeKnora-network-dev
profiles:
- minio
- full
qdrant:
image: qdrant/qdrant:v1.16.2
container_name: WeKnora-qdrant-dev
ports:
- "${QDRANT_REST_PORT:-6333}:6333"
- "${QDRANT_PORT:-6334}:6334"
volumes:
- qdrant_data_dev:/qdrant/storage
networks:
- WeKnora-network-dev
restart: unless-stopped
profiles:
- qdrant
- full
feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482) laid the type prep + driver skeleton + read/write paths as gated dead code; this PR wires every activation surface so opensearch becomes a registerable VectorStore engine. Activation wiring - internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping / buildEnvStoreForDriver — every gated surface now recognises "opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM / HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines' serialised config byte-identical. - internal/container: createOpenSearchEngine + the switch case in createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path in initRetrieveEngineRegistry; NewEngineFactory now closes over the AuditLogService (the EngineFactory type itself is unchanged). - internal/application/service/vectorstore_healthcheck.go: a testOpenSearchConnection case so CreateStore's connectivity probe accepts opensearch instead of returning 400. - internal/application/repository/retriever/opensearch/transport.go: NewOpenSearchClient is exported so the factory and env path can build the TLS-hardened client; healthcheck.go reuses the unexported probeVersion / probeKNNPlugin for the service-layer probe. Service-layer validation - validateOpenSearchIndexConfig validates the HNSW caps (m 2-100, ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss). Shards/replicas continue to be enforced by the flat ValidateIndexConfig. Create-only: UpdateStore mutates the name only. - validateConnectionConfig requires addr for opensearch. Sync implementations (stubs.go shrinks) - CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern — search → BatchSave with the source_id remap for generated questions — so dim/keyword routing and the source_id contract come from BatchSave for free. embeddingMap is keyed by the *target* SourceID because OpenSearch's BatchSave looks up embeddings by SourceID (lookupEmbedding), not by chunk_id (the ES driver's convention). Pagination is from/size; copies larger than max_result_window (default 10000) need the scroll-based async path that lands later. - BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go) group the input by target value and issue one _update_by_query per group over the cross-dim <base>_* pattern. Caller values flow through bound script params only — never string-interpolated into the Painless source — closing the script-injection surface. - inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the full failure reason goes to the debug log only; the returned error carries the bounded id + type. - UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same shape as DeleteByQuery's quirk), so refresh=wait_for is not expressible; we use refresh=true. Driver-owned audit (DIP) - A new opensearch.AuditSink interface (with nopSink + WithAuditSink functional option) lets the driver emit opensearch.index_created and opensearch.reindex_executed events without importing any service package — the service layer implements the interface. NewRepository takes opts, so existing 4-arg test call sites keep compiling unchanged. - internal/container/audit_sink.go bridges AuditSink to AuditLogService. When the context carries no tenant (the env-path registration ctx during boot, for example) the adapter skips the emit with a warning rather than silently writing tenant_id=0, which would collide with the system-scope sentinel. Frontend + polish - FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/ immutable. VectorStoreSettings.vue is now schema-driven: a closed `enum` renders a t-select; number inputs use the schema's `:min`/`:max` and fall back to the legacy replica-vs-shard heuristic only when the schema does not pin them; a danger-coloured warning fires when insecure_skip_verify is toggled on (the switch and warning are wrapped in a vertical stack so the warning sits on its own row below the switch). - i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search / knn_engine / insecure_skip_verify plus the warning copy in en-US, ko-KR, zh-CN, ru-RU. - docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with security plugin disabled for dev only). OpenSearch Dashboards lives in a separate, opt-in opensearch-ui profile so the heavy UI container is not forced up alongside the cluster (the driver e2e is fully curl-verifiable against :9200). The new docs/dev/opensearch-integration-test.md covers the end-to-end exercise and the single-node guidance (set replicas=0 to keep the cluster Green). Gating-guard tests flipped - The "OpenSearch is NOT in validEngineTypes / mapping / types list / env builder / stubs" guard tests from PR 1 / PR 2 are replaced by their positive counterparts in this PR. The test suite was the activation checklist; the activation flip is its diff. Backward compatibility - Additive everywhere. IndexConfig's new HNSW fields are omitempty so other engines' serialised config is byte-identical. Existing Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB stores are untouched. No migrations. Test plan - go build ./... clean - go vet ./... clean - gofmt -l clean on touched files - go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS endpoint), the docreader gRPC tests, and the doris SQL-shape tests fail; all three are pre-existing on upstream/main and untouched by this PR. - New tests across internal/types, opensearch, service and container — including a full end-to-end env-path test that exercises initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an httptest cluster.
2026-05-28 09:01:17 +00:00
# OpenSearch k-NN (Phase 3 driver). Single-node dev profile with the
# security plugin disabled → plain HTTP on :9200, no auth/TLS. The image
# bundles the opensearch-knn plugin. For production use a secured,
# multi-node cluster. See docs/dev/opensearch-integration-test.md.
opensearch:
image: opensearchproject/opensearch:3.3.2
container_name: WeKnora-opensearch-dev
environment:
- discovery.type=single-node
# dev only: plain HTTP on :9200, no TLS/auth. The entrypoint script
# honours DISABLE_SECURITY_PLUGIN (env var) to skip both the demo
# install and the OPENSEARCH_INITIAL_ADMIN_PASSWORD requirement.
- DISABLE_SECURITY_PLUGIN=true
- DISABLE_INSTALL_DEMO_CONFIG=true
- OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
- bootstrap.memory_lock=true
ulimits:
memlock:
soft: -1
hard: -1
ports:
- "${OPENSEARCH_PORT:-9200}:9200"
volumes:
- opensearch_data_dev:/usr/share/opensearch/data
networks:
- WeKnora-network-dev
restart: unless-stopped
profiles:
- opensearch
- full
# Also a member of opensearch-ui so the Dashboards depends_on resolves
# when only that profile is active (`--profile opensearch-ui up`).
- opensearch-ui
# Optional UI for visual index/mapping/query inspection. Decoupled from the
# "opensearch" / "full" profiles so the heavy Dashboards container is never
# forced up alongside the cluster — the driver e2e is fully curl-verifiable
# against :9200. Start it on demand with `--profile opensearch-ui up -d`
# (depends_on pulls the cluster in automatically).
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:3.3.0
container_name: WeKnora-opensearch-dashboards-dev
environment:
- OPENSEARCH_HOSTS=["http://opensearch:9200"]
- DISABLE_SECURITY_DASHBOARDS_PLUGIN=true
ports:
- "${OPENSEARCH_DASHBOARDS_PORT:-5601}:5601"
networks:
- WeKnora-network-dev
depends_on:
- opensearch
restart: unless-stopped
profiles:
- opensearch-ui
milvus:
image: milvusdb/milvus:v2.6.11
container_name: WeKnora-milvus-dev
security_opt:
- seccomp:unconfined
command: ["milvus", "run", "standalone"]
environment:
- ETCD_USE_EMBED=true
- ETCD_DATA_DIR=/var/lib/milvus/etcd
- COMMON_STORAGETYPE=local
- DEPLOY_MODE=STANDALONE
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
interval: 30s
start_period: 90s
timeout: 20s
retries: 3
ports:
- "${MILVUS_PORT:-19530}:19530"
- "${MILVUS_HEALTH_PORT:-9091}:9091"
volumes:
- milvus_data_dev:/var/lib/milvus
networks:
- WeKnora-network-dev
restart: unless-stopped
profiles:
- milvus
- full
neo4j:
image: neo4j:latest
container_name: WeKnora-neo4j-dev
volumes:
- neo4j-data-dev:/data
environment:
- NEO4J_AUTH=${NEO4J_USERNAME:-neo4j}/${NEO4J_PASSWORD:-password}
- NEO4J_apoc_export_file_enabled=true
- NEO4J_apoc_import_file_enabled=true
- NEO4J_apoc_import_file_use__neo4j__config=true
- NEO4JLABS_PLUGINS=["apoc"]
ports:
- "7474:7474"
- "7687:7687"
restart: always
networks:
- WeKnora-network-dev
profiles:
- neo4j
- full
# Sandbox 镜像:仅用于 build/pull,非常驻服务;本地 app 执行 Skills 时按需 docker run 该镜像,用毕即释
sandbox:
image: wechatopenai/weknora-sandbox:${WEKNORA_VERSION:-latest}
container_name: WeKnora-sandbox-dev
build:
context: .
dockerfile: docker/Dockerfile.sandbox
profiles:
- full
command: ["true"]
restart: "no"
docreader:
build:
context: .
dockerfile: docker/Dockerfile.docreader
image: wechatopenai/weknora-docreader:${WEKNORA_VERSION:-latest}
container_name: WeKnora-docreader-dev
ports:
- "${DOCREADER_PORT:-50051}:50051"
volumes:
- docreader-tmp-dev:/tmp/docreader
environment:
- DOCREADER_IMAGE_OUTPUT_DIR=/tmp/docreader
- DOCREADER_PDF_FORCE_SCANNED=${DOCREADER_PDF_FORCE_SCANNED:-false}
- MAX_FILE_SIZE_MB=${MAX_FILE_SIZE_MB:-}
- DOCREADER_ODL_MAX_WORKERS=${DOCREADER_ODL_MAX_WORKERS:-1}
- DOCREADER_ODL_HYBRID=${DOCREADER_ODL_HYBRID:-off}
- DOCREADER_ODL_HYBRID_URL=${DOCREADER_ODL_HYBRID_URL:-http://odl-hybrid:5002}
- DOCREADER_ODL_HYBRID_MODE=${DOCREADER_ODL_HYBRID_MODE:-auto}
- DOCREADER_ODL_HYBRID_FALLBACK=${DOCREADER_ODL_HYBRID_FALLBACK:-false}
- DOCREADER_ODL_MARKDOWN_WITH_HTML=${DOCREADER_ODL_MARKDOWN_WITH_HTML:-false}
- DOCREADER_MARKITDOWN_MAX_WORKERS=${DOCREADER_MARKITDOWN_MAX_WORKERS:-1}
- DOCREADER_PDF_RENDER_MAX_WORKERS=${DOCREADER_PDF_RENDER_MAX_WORKERS:-1}
- DOCREADER_PDF_RENDER_DPI=${DOCREADER_PDF_RENDER_DPI:-200}
- DOCREADER_PDF_JPEG_QUALITY=${DOCREADER_PDF_JPEG_QUALITY:-90}
2026-05-16 09:49:31 +00:00
- GRPC_TLS_ENABLED=${GRPC_TLS_ENABLED:-false}
- GRPC_TLS_CERT=${GRPC_TLS_CERT:-}
- GRPC_TLS_KEY=${GRPC_TLS_KEY:-}
- GRPC_TLS_CA=${GRPC_TLS_CA:-}
fix(docreader/auth): harden gRPC TLS/Token rollout from #1359 Follow-up to #1359. Addresses a set of correctness and security gaps in the initial docreader auth implementation. - docker-compose: inject GRPC_TLS_*/GRPC_TLS_SERVER_NAME/GRPC_AUTH_TOKEN into the WeKnora-app service. Without this the Go client never saw the knobs, so enabling token auth on the server broke every RPC. - client: bind tokenAuth.RequireTransportSecurity() to TLSEnabled so a bearer token cannot be sent over an insecure channel once TLS is on. - server: load_tls_credentials now raises TLSConfigError on misconfig (cert/key missing, file unreadable, mTLS without CA); main.py exits 1 instead of silently downgrading to insecure. - server: replace endswith("/Check"|"/Watch") health bypass with exact match against /grpc.health.v1.Health/{Check,Watch}. - server: compare tokens with hmac.compare_digest, warn on tokens < 16B. - server: AuthInterceptor now returns an abort handler matching the original RPC kind (unary/stream) and uses context.abort, so streaming RPCs surface UNAUTHENTICATED instead of INTERNAL. - internal/infrastructure/docparser/grpc_parser.go: drop the duplicated TLS/tokenAuth block and reuse docreader/client.LoadAuthConfigFromEnv + BuildDialOptions. Single source of truth for client-side auth. - Add GRPC_TLS_SERVER_NAME (client SNI override) and GRPC_MTLS_REQUIRE_CLIENT_CERT (server explicit mTLS toggle); document the differing CA semantics between client and server in .env*.example. - Reject half-configured client mTLS (cert XOR key) loudly. - Fix missing trailing newline in .env.lite.example. Verified locally: go build ./... and go vet ./... clean; auth.py fail-fast / token paths smoke-tested.
2026-05-16 13:39:10 +00:00
- GRPC_TLS_SERVER_NAME=${GRPC_TLS_SERVER_NAME:-}
2026-05-16 09:49:31 +00:00
- GRPC_AUTH_TOKEN=${GRPC_AUTH_TOKEN:-}
healthcheck:
test: ["CMD", "grpc_health_probe", "-addr=localhost:50051"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
networks:
- WeKnora-network-dev
restart: unless-stopped
extra_hosts:
- "host.docker.internal:host-gateway"
# OpenDataLoader hybrid backend (optional). Enable profile "odl-hybrid" and set
# DOCREADER_ODL_HYBRID=docling-fast on docreader. Default --no-ocr (no EasyOCR).
# Local build only — not published to Docker Hub.
odl-hybrid:
build:
context: .
dockerfile: docker/Dockerfile.odl-hybrid
image: weknora-odl-hybrid:local
container_name: WeKnora-odl-hybrid
profiles:
- odl-hybrid
ports:
- "${ODL_HYBRID_PORT:-5002}:5002"
environment:
# Default --no-ocr (digital PDFs). Scanned PDFs: use builtin OCR / MinerU, or
# ODL_HYBRID_EXTRA_ARGS="--force-ocr" (needs EasyOCR + libGL in image).
- ODL_HYBRID_EXTRA_ARGS=${ODL_HYBRID_EXTRA_ARGS:---no-ocr}
networks:
- WeKnora-network-dev
restart: unless-stopped
2026-03-30 03:13:44 +00:00
dex:
image: dexidp/dex:latest
container_name: WeKnora-dex-dev
ports:
- "5556:5556"
volumes:
- ./misc/dex-config.yaml:/etc/dex/config.yaml
command: ["dex", "serve", "/etc/dex/config.yaml"]
profiles:
- dex
- full
feat(observability): integrate Langfuse for LLM token tracking and tracing Closes #620 #497. Add opt-in Langfuse observability covering all five model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped traces and Docker Compose support (both cloud and self-hosted). Core package internal/tracing/langfuse: - HTTP client with batched async ingestion (non-blocking in request path) - Sampling, environment / release tagging, and graceful fallback when LANGFUSE_* env vars are absent (wrappers become no-ops) - Gin middleware opens one trace per traced request and finishes it after the handler chain returns, attaching method / path / user / session - Trace context is stored under a typed key exported from internal/types so logger.CloneContext can preserve it across handler / goroutine boundaries (otherwise each LLM call auto-created an orphan trace, fragmenting one request into many) Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...): - chat: captures prompt, streaming output, token usage + TTFT - embedding: approximates tokens when the provider omits usage - rerank: previews query/docs, summarizes results to keep payload small - vlm: records image count and total bytes, never uploads raw pixels - asr: records file size and audio duration, never uploads audio bytes Async title generation (GenerateTitleAsync) now forwards the trace key into the goroutine so title calls appear under the parent chat trace. Docker Compose: - LANGFUSE_* env passthrough on the `app` service for cloud deployments - Optional `langfuse` profile spins up a self-hosted Langfuse stack that reuses WeKnora's existing PostgreSQL (separate database via an idempotent init container that fixes ICU collation drift) and Redis (separate DB number), adding only ClickHouse, MinIO, web and worker containers - web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start to avoid Prisma P1013 when passwords contain @ / # / etc. Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage strategy, code map, and resource footprint.
2026-04-23 17:57:12 +00:00
# ---------------------------------------------------------------------------
# Langfuse 自建栈(dev 对称版)
#
# 用法:
# docker compose -f docker-compose.dev.yml --profile langfuse up -d
#
# 本地 app (go run) 需要的环境变量:
# export LANGFUSE_HOST=http://localhost:3000
# export LANGFUSE_PUBLIC_KEY=pk-lf-xxx
# export LANGFUSE_SECRET_KEY=sk-lf-xxx
#
# 复用 dev 已有的 postgres(独立 langfuse 数据库)和 redis(DB 1),
# 新增:clickhouse、minio、web、worker + 一次性 db-init,和生产版结构一致。
# ---------------------------------------------------------------------------
# 复用 dev 已有的 ParadeDB 镜像,不额外拉 postgres 镜像
langfuse-db-init:
image: paradedb/paradedb:v0.22.2-pg17
container_name: WeKnora-langfuse-db-init-dev
depends_on:
postgres:
condition: service_healthy
environment:
PGPASSWORD: ${DB_PASSWORD}
# ${LANGFUSE_DB_NAME:-langfuse} / ${DB_USER} 由 compose 解析成字面量后再传给 shell。
entrypoint: ["sh", "-c"]
command:
- |
set -e
echo "[langfuse-db-init] ensuring database '${LANGFUSE_DB_NAME:-langfuse}' exists..."
# 先刷新现有库的 collation(镜像 ICU 2.36 与宿主 2.41 不匹配时必须做),否则 CREATE DATABASE 会失败
psql -h postgres -U ${DB_USER} -d postgres -v ON_ERROR_STOP=0 -c "ALTER DATABASE template1 REFRESH COLLATION VERSION;" >/dev/null 2>&1 || true
psql -h postgres -U ${DB_USER} -d postgres -v ON_ERROR_STOP=0 -c "ALTER DATABASE postgres REFRESH COLLATION VERSION;" >/dev/null 2>&1 || true
# 幂等创建:已存在则跳过;不存在则从 template0 克隆(template0 永远不会有 collation 漂移)
if psql -h postgres -U ${DB_USER} -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname='${LANGFUSE_DB_NAME:-langfuse}'" | grep -q 1; then
echo "[langfuse-db-init] database '${LANGFUSE_DB_NAME:-langfuse}' already exists, skipping."
else
psql -h postgres -U ${DB_USER} -d postgres -v ON_ERROR_STOP=1 -c "CREATE DATABASE \"${LANGFUSE_DB_NAME:-langfuse}\" TEMPLATE template0;"
echo "[langfuse-db-init] database '${LANGFUSE_DB_NAME:-langfuse}' created."
fi
echo "[langfuse-db-init] done."
networks:
- WeKnora-network-dev
restart: "no"
profiles:
- langfuse
- full
langfuse-clickhouse:
image: clickhouse/clickhouse-server:24.8
container_name: WeKnora-langfuse-clickhouse-dev
restart: unless-stopped
user: "101:101"
environment:
CLICKHOUSE_DB: default
CLICKHOUSE_USER: ${LANGFUSE_CLICKHOUSE_USER:-clickhouse}
CLICKHOUSE_PASSWORD: ${LANGFUSE_CLICKHOUSE_PASSWORD:-clickhouse}
volumes:
- langfuse_clickhouse_data_dev:/var/lib/clickhouse
- langfuse_clickhouse_logs_dev:/var/log/clickhouse-server
healthcheck:
test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
interval: 5s
timeout: 5s
retries: 10
start_period: 10s
networks:
- WeKnora-network-dev
profiles:
- langfuse
- full
langfuse-minio:
image: minio/minio:RELEASE.2025-09-07T16-13-09Z
container_name: WeKnora-langfuse-minio-dev
restart: unless-stopped
entrypoint: sh
command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
environment:
MINIO_ROOT_USER: ${LANGFUSE_MINIO_USER:-langfuseminio}
MINIO_ROOT_PASSWORD: ${LANGFUSE_MINIO_PASSWORD:-langfuseminiosecret}
ports:
- "${LANGFUSE_MINIO_S3_PORT:-9100}:9000"
- "${LANGFUSE_MINIO_CONSOLE_PORT:-9101}:9001"
volumes:
- langfuse_minio_data_dev:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 10s
retries: 5
networks:
- WeKnora-network-dev
profiles:
- langfuse
- full
langfuse-worker:
image: langfuse/langfuse-worker:3
container_name: WeKnora-langfuse-worker-dev
restart: unless-stopped
depends_on: &langfuse-dev-depends-on
langfuse-db-init:
condition: service_completed_successfully
redis:
condition: service_started
langfuse-clickhouse:
condition: service_healthy
langfuse-minio:
condition: service_healthy
# wrapper entrypoint 把 DB_PASSWORD / REDIS_PASSWORD URL 编码后再拼 URL,
# 避免密码含 '@' / '#' 等字符导致 Prisma P1013 解析失败。
# 注意:compose 覆盖 entrypoint 会清空镜像默认 CMD,所以末尾写死原始命令。
entrypoint:
- /bin/sh
- -ec
- |
_enc() { node -e 'process.stdout.write(encodeURIComponent(process.argv[1]))' "$$1"; }
DU=$$(_enc "$$_LF_DB_USER")
DP=$$(_enc "$$_LF_DB_PASSWORD")
RP=$$(_enc "$$_LF_REDIS_PASSWORD")
export DATABASE_URL="postgresql://$$DU:$$DP@postgres:5432/$$_LF_DB_NAME"
export REDIS_CONNECTION_STRING="redis://:$$RP@redis:6379/$$_LF_REDIS_DB"
unset _LF_DB_USER _LF_DB_PASSWORD _LF_REDIS_PASSWORD
exec dumb-init -- ./worker/entrypoint.sh node worker/dist/index.js
environment: &langfuse-dev-env
# 原始凭证(未 URL 编码),由 entrypoint wrapper 读取并组装
_LF_DB_USER: ${DB_USER}
_LF_DB_PASSWORD: ${DB_PASSWORD}
_LF_DB_NAME: ${LANGFUSE_DB_NAME:-langfuse}
_LF_REDIS_PASSWORD: ${REDIS_PASSWORD}
_LF_REDIS_DB: ${LANGFUSE_REDIS_DB:-1}
SALT: ${LANGFUSE_SALT:-weknora-langfuse-dev-salt-change-me}
ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY:-0000000000000000000000000000000000000000000000000000000000000000}
NEXTAUTH_URL: ${LANGFUSE_NEXTAUTH_URL:-http://localhost:3000}
NEXTAUTH_SECRET: ${LANGFUSE_NEXTAUTH_SECRET:-weknora-langfuse-dev-nextauth-secret-change-me}
TELEMETRY_ENABLED: ${LANGFUSE_TELEMETRY_ENABLED:-false}
LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: "false"
CLICKHOUSE_URL: http://langfuse-clickhouse:8123
CLICKHOUSE_MIGRATION_URL: clickhouse://langfuse-clickhouse:9000
CLICKHOUSE_USER: ${LANGFUSE_CLICKHOUSE_USER:-clickhouse}
CLICKHOUSE_PASSWORD: ${LANGFUSE_CLICKHOUSE_PASSWORD:-clickhouse}
CLICKHOUSE_CLUSTER_ENABLED: "false"
LANGFUSE_S3_EVENT_UPLOAD_BUCKET: langfuse
LANGFUSE_S3_EVENT_UPLOAD_REGION: auto
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_MINIO_USER:-langfuseminio}
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_MINIO_PASSWORD:-langfuseminiosecret}
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: http://langfuse-minio:9000
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: "true"
LANGFUSE_S3_EVENT_UPLOAD_PREFIX: events/
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: langfuse
LANGFUSE_S3_MEDIA_UPLOAD_REGION: auto
LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_MINIO_USER:-langfuseminio}
LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_MINIO_PASSWORD:-langfuseminiosecret}
LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: ${LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT:-http://localhost:9100}
LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: "true"
LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: media/
networks:
- WeKnora-network-dev
profiles:
- langfuse
- full
langfuse-web:
image: langfuse/langfuse:3
container_name: WeKnora-langfuse-web-dev
restart: unless-stopped
depends_on: *langfuse-dev-depends-on
ports:
- "${LANGFUSE_WEB_PORT:-3000}:3000"
entrypoint:
- /bin/sh
- -ec
- |
_enc() { node -e 'process.stdout.write(encodeURIComponent(process.argv[1]))' "$$1"; }
DU=$$(_enc "$$_LF_DB_USER")
DP=$$(_enc "$$_LF_DB_PASSWORD")
RP=$$(_enc "$$_LF_REDIS_PASSWORD")
export DATABASE_URL="postgresql://$$DU:$$DP@postgres:5432/$$_LF_DB_NAME"
export REDIS_CONNECTION_STRING="redis://:$$RP@redis:6379/$$_LF_REDIS_DB"
unset _LF_DB_USER _LF_DB_PASSWORD _LF_REDIS_PASSWORD
if [ -n "$$NEXT_PUBLIC_LANGFUSE_CLOUD_REGION" ]; then
exec dumb-init -- ./web/entrypoint.sh node --import dd-trace/initialize.mjs ./web/server.js --keepAliveTimeout 110000
else
exec dumb-init -- ./web/entrypoint.sh node ./web/server.js --keepAliveTimeout 110000
fi
environment:
<<: *langfuse-dev-env
LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-}
LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-}
LANGFUSE_INIT_PROJECT_ID: ${LANGFUSE_INIT_PROJECT_ID:-}
LANGFUSE_INIT_PROJECT_NAME: ${LANGFUSE_INIT_PROJECT_NAME:-}
LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY:-}
LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LANGFUSE_INIT_PROJECT_SECRET_KEY:-}
LANGFUSE_INIT_USER_EMAIL: ${LANGFUSE_INIT_USER_EMAIL:-}
LANGFUSE_INIT_USER_NAME: ${LANGFUSE_INIT_USER_NAME:-}
LANGFUSE_INIT_USER_PASSWORD: ${LANGFUSE_INIT_USER_PASSWORD:-}
networks:
- WeKnora-network-dev
profiles:
- langfuse
- full
2026-03-30 03:13:44 +00:00
networks:
WeKnora-network-dev:
driver: bridge
volumes:
postgres-data-dev:
redis_data_dev:
minio_data_dev:
neo4j-data-dev:
qdrant_data_dev:
feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482) laid the type prep + driver skeleton + read/write paths as gated dead code; this PR wires every activation surface so opensearch becomes a registerable VectorStore engine. Activation wiring - internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping / buildEnvStoreForDriver — every gated surface now recognises "opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM / HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines' serialised config byte-identical. - internal/container: createOpenSearchEngine + the switch case in createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path in initRetrieveEngineRegistry; NewEngineFactory now closes over the AuditLogService (the EngineFactory type itself is unchanged). - internal/application/service/vectorstore_healthcheck.go: a testOpenSearchConnection case so CreateStore's connectivity probe accepts opensearch instead of returning 400. - internal/application/repository/retriever/opensearch/transport.go: NewOpenSearchClient is exported so the factory and env path can build the TLS-hardened client; healthcheck.go reuses the unexported probeVersion / probeKNNPlugin for the service-layer probe. Service-layer validation - validateOpenSearchIndexConfig validates the HNSW caps (m 2-100, ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss). Shards/replicas continue to be enforced by the flat ValidateIndexConfig. Create-only: UpdateStore mutates the name only. - validateConnectionConfig requires addr for opensearch. Sync implementations (stubs.go shrinks) - CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern — search → BatchSave with the source_id remap for generated questions — so dim/keyword routing and the source_id contract come from BatchSave for free. embeddingMap is keyed by the *target* SourceID because OpenSearch's BatchSave looks up embeddings by SourceID (lookupEmbedding), not by chunk_id (the ES driver's convention). Pagination is from/size; copies larger than max_result_window (default 10000) need the scroll-based async path that lands later. - BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go) group the input by target value and issue one _update_by_query per group over the cross-dim <base>_* pattern. Caller values flow through bound script params only — never string-interpolated into the Painless source — closing the script-injection surface. - inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the full failure reason goes to the debug log only; the returned error carries the bounded id + type. - UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same shape as DeleteByQuery's quirk), so refresh=wait_for is not expressible; we use refresh=true. Driver-owned audit (DIP) - A new opensearch.AuditSink interface (with nopSink + WithAuditSink functional option) lets the driver emit opensearch.index_created and opensearch.reindex_executed events without importing any service package — the service layer implements the interface. NewRepository takes opts, so existing 4-arg test call sites keep compiling unchanged. - internal/container/audit_sink.go bridges AuditSink to AuditLogService. When the context carries no tenant (the env-path registration ctx during boot, for example) the adapter skips the emit with a warning rather than silently writing tenant_id=0, which would collide with the system-scope sentinel. Frontend + polish - FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/ immutable. VectorStoreSettings.vue is now schema-driven: a closed `enum` renders a t-select; number inputs use the schema's `:min`/`:max` and fall back to the legacy replica-vs-shard heuristic only when the schema does not pin them; a danger-coloured warning fires when insecure_skip_verify is toggled on (the switch and warning are wrapped in a vertical stack so the warning sits on its own row below the switch). - i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search / knn_engine / insecure_skip_verify plus the warning copy in en-US, ko-KR, zh-CN, ru-RU. - docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with security plugin disabled for dev only). OpenSearch Dashboards lives in a separate, opt-in opensearch-ui profile so the heavy UI container is not forced up alongside the cluster (the driver e2e is fully curl-verifiable against :9200). The new docs/dev/opensearch-integration-test.md covers the end-to-end exercise and the single-node guidance (set replicas=0 to keep the cluster Green). Gating-guard tests flipped - The "OpenSearch is NOT in validEngineTypes / mapping / types list / env builder / stubs" guard tests from PR 1 / PR 2 are replaced by their positive counterparts in this PR. The test suite was the activation checklist; the activation flip is its diff. Backward compatibility - Additive everywhere. IndexConfig's new HNSW fields are omitempty so other engines' serialised config is byte-identical. Existing Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB stores are untouched. No migrations. Test plan - go build ./... clean - go vet ./... clean - gofmt -l clean on touched files - go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS endpoint), the docreader gRPC tests, and the doris SQL-shape tests fail; all three are pre-existing on upstream/main and untouched by this PR. - New tests across internal/types, opensearch, service and container — including a full end-to-end env-path test that exercises initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an httptest cluster.
2026-05-28 09:01:17 +00:00
opensearch_data_dev:
milvus_data_dev:
docreader-tmp-dev:
feat(observability): integrate Langfuse for LLM token tracking and tracing Closes #620 #497. Add opt-in Langfuse observability covering all five model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped traces and Docker Compose support (both cloud and self-hosted). Core package internal/tracing/langfuse: - HTTP client with batched async ingestion (non-blocking in request path) - Sampling, environment / release tagging, and graceful fallback when LANGFUSE_* env vars are absent (wrappers become no-ops) - Gin middleware opens one trace per traced request and finishes it after the handler chain returns, attaching method / path / user / session - Trace context is stored under a typed key exported from internal/types so logger.CloneContext can preserve it across handler / goroutine boundaries (otherwise each LLM call auto-created an orphan trace, fragmenting one request into many) Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...): - chat: captures prompt, streaming output, token usage + TTFT - embedding: approximates tokens when the provider omits usage - rerank: previews query/docs, summarizes results to keep payload small - vlm: records image count and total bytes, never uploads raw pixels - asr: records file size and audio duration, never uploads audio bytes Async title generation (GenerateTitleAsync) now forwards the trace key into the goroutine so title calls appear under the parent chat trace. Docker Compose: - LANGFUSE_* env passthrough on the `app` service for cloud deployments - Optional `langfuse` profile spins up a self-hosted Langfuse stack that reuses WeKnora's existing PostgreSQL (separate database via an idempotent init container that fixes ICU collation drift) and Redis (separate DB number), adding only ClickHouse, MinIO, web and worker containers - web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start to avoid Prisma P1013 when passwords contain @ / # / etc. Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage strategy, code map, and resource footprint.
2026-04-23 17:57:12 +00:00
langfuse_clickhouse_data_dev:
langfuse_clickhouse_logs_dev:
langfuse_minio_data_dev:
searxng_config_dev: