2025-11-07 07:52:52 +00:00
|
|
|
|
# 开发环境配置 - 只启动基础设施服务,app 和 frontend 在本地运行
|
|
|
|
|
|
services:
|
|
|
|
|
|
# 只启动依赖的基础设施服务
|
|
|
|
|
|
postgres:
|
fix(deploy): bump paradedb image to v0.25.11-pg17 (#4120)
* fix(deploy): bump paradedb image to v0.25.11-pg17
Batch-importing large knowledge bases on the pinned
paradedb/paradedb:v0.22.6-pg17 image drives PostgreSQL into a
PANIC/crash-recovery loop (corrupted page pointers in shared buffers,
amcheck clean, no OOM), taking every ingest task down with it.
Three independent fixes upstream cover this exact write path, and the
pinned image predates all of them:
- pgvector 0.8.2 fixed a buffer overflow in parallel HNSW index
builds, 0.8.3 fixed possible index corruption during HNSW vacuuming,
and 0.8.4 fixed inserts during HNSW vacuuming; the image ships
pgvector 0.8.1, and the embeddings table maintains three HNSW
halfvec indexes on import.
- pg_search gained WAL-based crash recovery in 0.24.0 (paradedb#4901):
without it, any backend crash corrupts BM25 segment metadata and
turns a single crash into a recovery loop (paradedb#4935).
- PostgreSQL core moved 17.9 -> 17.11; its release notes contain no
fix matching this failure signature, ruling the core out.
Bump the compose image tags to v0.25.11-pg17 (postgres and
langfuse-db-init, prod and dev) and update the docs that pin the old
tag. Fresh installs are covered by the existing schema SQL (verified
against 0.25.11: extensions, BM25 index and HNSW halfvec indexes all
create cleanly). Existing data volumes keep working across the image
swap without any action (keyword and vector queries byte-identical in
testing); ALTER EXTENSION pg_search/vector UPDATE aligns the SQL
catalog afterwards, and migration 000099 is a no-op for 0.25.x by
design. The Helm chart pins a different image line and is left alone.
Fixes #4077
* fix(migrations): upgrade pg_search catalog to 0.25.11 with the image
Swapping in the v0.25.11 image leaves the extension catalog at 0.22.x.
Calling extension functions such as paradedb.version_info() against the
mismatched catalog segfaults the backend and sends the whole server into
crash recovery, so the catalog upgrade is required rather than optional.
- Add migration 000115 to update pg_search 0.22.0-0.25.10 to 0.25.11 and
pgvector 0.8.0-0.8.3 to 0.8.4 when the server ships those versions.
- Skip 000099 on newer images instead of advising UPDATE TO '0.22.6',
which leaves the catalog behind the shared library and crashes.
- Rewrite the upgrade guide as one 0.22.x -> 0.25.11 path with a required
ALTER EXTENSION UPDATE and a note that rollback needs a backup.
- Extend scripts/test_paradedb_upgrade.py to verify 0.22.2 -> 0.25.11 by
default (--to 0.22.6 keeps the old path) and run CONCURRENTLY
migrations outside a transaction like golang-migrate does.
---------
Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-12 03:50:11 +00:00
|
|
|
|
image: paradedb/paradedb:v0.25.11-pg17
|
2025-11-07 07:52:52 +00:00
|
|
|
|
container_name: WeKnora-postgres-dev
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${DB_PORT:-5432}:5432"
|
|
|
|
|
|
environment:
|
|
|
|
|
|
- POSTGRES_USER=${DB_USER}
|
|
|
|
|
|
- POSTGRES_PASSWORD=${DB_PASSWORD}
|
|
|
|
|
|
- POSTGRES_DB=${DB_NAME}
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- postgres-data-dev:/var/lib/postgresql/data
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
healthcheck:
|
2026-09-15 15:50:38 +00:00
|
|
|
|
test: ["CMD-SHELL", "pg_isready -U ${DB_USER} -d ${DB_NAME}"]
|
2025-11-07 07:52:52 +00:00
|
|
|
|
interval: 10s
|
|
|
|
|
|
timeout: 10s
|
|
|
|
|
|
retries: 3
|
|
|
|
|
|
start_period: 30s
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
stop_grace_period: 1m
|
|
|
|
|
|
|
|
|
|
|
|
redis:
|
|
|
|
|
|
image: redis:7.0-alpine
|
|
|
|
|
|
container_name: WeKnora-redis-dev
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${REDIS_PORT:-6379}:6379"
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- redis_data_dev:/data
|
|
|
|
|
|
command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
|
|
|
|
|
|
restart: always
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
|
2026-05-10 14:51:24 +00:00
|
|
|
|
# See docker-compose.yml searxng-init for rationale (avoid in-place rewrite of
|
|
|
|
|
|
# the repo-tracked settings.yml by the searxng entrypoint).
|
|
|
|
|
|
searxng-init:
|
|
|
|
|
|
image: busybox:1.36
|
|
|
|
|
|
container_name: WeKnora-searxng-init-dev
|
|
|
|
|
|
command: ["sh", "-c", "cp /template/settings.yml /etc/searxng/settings.yml && chmod 0644 /etc/searxng/settings.yml"]
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- ./docker/searxng/settings.yml:/template/settings.yml:ro
|
|
|
|
|
|
- searxng_config_dev:/etc/searxng
|
|
|
|
|
|
restart: "no"
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- searxng
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
2026-05-10 14:01:28 +00:00
|
|
|
|
searxng:
|
|
|
|
|
|
image: searxng/searxng:latest
|
|
|
|
|
|
container_name: WeKnora-searxng-dev
|
2026-05-10 14:51:24 +00:00
|
|
|
|
# Local dev: bind to loopback by default; the host-side WeKnora process
|
|
|
|
|
|
# reaches it at http://127.0.0.1:${SEARXNG_PORT}.
|
2026-05-10 14:01:28 +00:00
|
|
|
|
ports:
|
2026-05-10 14:51:24 +00:00
|
|
|
|
- "${SEARXNG_BIND:-127.0.0.1}:${SEARXNG_PORT:-8888}:8080"
|
2026-05-10 14:01:28 +00:00
|
|
|
|
volumes:
|
2026-05-10 14:51:24 +00:00
|
|
|
|
- searxng_config_dev:/etc/searxng
|
2026-05-10 14:01:28 +00:00
|
|
|
|
environment:
|
|
|
|
|
|
- SEARXNG_BASE_URL=http://localhost:${SEARXNG_PORT:-8888}/
|
|
|
|
|
|
- INSTANCE_NAME=weknora-searxng-dev
|
2026-05-11 08:50:07 +00:00
|
|
|
|
# See docker-compose.yml for rationale on the default secret.
|
|
|
|
|
|
- SEARXNG_SECRET=${SEARXNG_SECRET:-weknora-default-searxng-secret-rotate-before-exposing-publicly}
|
2026-05-10 14:01:28 +00:00
|
|
|
|
cap_drop:
|
|
|
|
|
|
- ALL
|
|
|
|
|
|
cap_add:
|
|
|
|
|
|
- CHOWN
|
|
|
|
|
|
- SETGID
|
|
|
|
|
|
- SETUID
|
|
|
|
|
|
restart: unless-stopped
|
2026-05-10 14:51:24 +00:00
|
|
|
|
depends_on:
|
|
|
|
|
|
searxng-init:
|
|
|
|
|
|
condition: service_completed_successfully
|
2026-05-10 14:01:28 +00:00
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- searxng
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
2025-11-07 07:52:52 +00:00
|
|
|
|
minio:
|
2026-09-26 07:26:09 +00:00
|
|
|
|
image: pgsty/minio:latest
|
2025-11-07 07:52:52 +00:00
|
|
|
|
container_name: WeKnora-minio-dev
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${MINIO_PORT:-9000}:9000"
|
|
|
|
|
|
- "${MINIO_CONSOLE_PORT:-9001}:9001"
|
|
|
|
|
|
environment:
|
|
|
|
|
|
- MINIO_ROOT_USER=${MINIO_ACCESS_KEY_ID:-minioadmin}
|
|
|
|
|
|
- MINIO_ROOT_PASSWORD=${MINIO_SECRET_ACCESS_KEY:-minioadmin}
|
|
|
|
|
|
command: server --console-address ":9001" /data
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- minio_data_dev:/data
|
|
|
|
|
|
healthcheck:
|
|
|
|
|
|
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
|
|
|
|
|
|
interval: 30s
|
|
|
|
|
|
timeout: 20s
|
|
|
|
|
|
retries: 3
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
2025-12-08 11:32:46 +00:00
|
|
|
|
profiles:
|
|
|
|
|
|
- minio
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
|
|
|
|
|
qdrant:
|
|
|
|
|
|
image: qdrant/qdrant:v1.16.2
|
|
|
|
|
|
container_name: WeKnora-qdrant-dev
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${QDRANT_REST_PORT:-6333}:6333"
|
|
|
|
|
|
- "${QDRANT_PORT:-6334}:6334"
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- qdrant_data_dev:/qdrant/storage
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- qdrant
|
|
|
|
|
|
- full
|
2025-11-07 07:52:52 +00:00
|
|
|
|
|
feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3)
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482)
laid the type prep + driver skeleton + read/write paths as gated dead
code; this PR wires every activation surface so opensearch becomes a
registerable VectorStore engine.
Activation wiring
- internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW
bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping /
buildEnvStoreForDriver — every gated surface now recognises
"opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM /
HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines'
serialised config byte-identical.
- internal/container: createOpenSearchEngine + the switch case in
createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path
in initRetrieveEngineRegistry; NewEngineFactory now closes over the
AuditLogService (the EngineFactory type itself is unchanged).
- internal/application/service/vectorstore_healthcheck.go: a
testOpenSearchConnection case so CreateStore's connectivity probe
accepts opensearch instead of returning 400.
- internal/application/repository/retriever/opensearch/transport.go:
NewOpenSearchClient is exported so the factory and env path can build
the TLS-hardened client; healthcheck.go reuses the unexported
probeVersion / probeKNNPlugin for the service-layer probe.
Service-layer validation
- validateOpenSearchIndexConfig validates the HNSW caps (m 2-100,
ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss).
Shards/replicas continue to be enforced by the flat ValidateIndexConfig.
Create-only: UpdateStore mutates the name only.
- validateConnectionConfig requires addr for opensearch.
Sync implementations (stubs.go shrinks)
- CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern —
search → BatchSave with the source_id remap for generated questions —
so dim/keyword routing and the source_id contract come from BatchSave
for free. embeddingMap is keyed by the *target* SourceID because
OpenSearch's BatchSave looks up embeddings by SourceID
(lookupEmbedding), not by chunk_id (the ES driver's convention).
Pagination is from/size; copies larger than max_result_window
(default 10000) need the scroll-based async path that lands later.
- BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go)
group the input by target value and issue one _update_by_query per
group over the cross-dim <base>_* pattern. Caller values flow through
bound script params only — never string-interpolated into the Painless
source — closing the script-injection surface.
- inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the
full failure reason goes to the debug log only; the returned error
carries the bounded id + type.
- UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same
shape as DeleteByQuery's quirk), so refresh=wait_for is not
expressible; we use refresh=true.
Driver-owned audit (DIP)
- A new opensearch.AuditSink interface (with nopSink + WithAuditSink
functional option) lets the driver emit opensearch.index_created and
opensearch.reindex_executed events without importing any service
package — the service layer implements the interface. NewRepository
takes opts, so existing 4-arg test call sites keep compiling unchanged.
- internal/container/audit_sink.go bridges AuditSink to AuditLogService.
When the context carries no tenant (the env-path registration ctx
during boot, for example) the adapter skips the emit with a warning
rather than silently writing tenant_id=0, which would collide with the
system-scope sentinel.
Frontend + polish
- FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/
immutable. VectorStoreSettings.vue is now schema-driven: a closed
`enum` renders a t-select; number inputs use the schema's `:min`/`:max`
and fall back to the legacy replica-vs-shard heuristic only when the
schema does not pin them; a danger-coloured warning fires when
insecure_skip_verify is toggled on (the switch and warning are wrapped
in a vertical stack so the warning sits on its own row below the switch).
- i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search /
knn_engine / insecure_skip_verify plus the warning copy in en-US,
ko-KR, zh-CN, ru-RU.
- docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with
security plugin disabled for dev only). OpenSearch Dashboards lives in a
separate, opt-in opensearch-ui profile so the heavy UI container is not
forced up alongside the cluster (the driver e2e is fully curl-verifiable
against :9200). The new docs/dev/opensearch-integration-test.md covers the
end-to-end exercise and the single-node guidance (set replicas=0 to keep
the cluster Green).
Gating-guard tests flipped
- The "OpenSearch is NOT in validEngineTypes / mapping / types list /
env builder / stubs" guard tests from PR 1 / PR 2 are replaced by
their positive counterparts in this PR. The test suite was the
activation checklist; the activation flip is its diff.
Backward compatibility
- Additive everywhere. IndexConfig's new HNSW fields are omitempty so
other engines' serialised config is byte-identical. Existing
Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB
stores are untouched. No migrations.
Test plan
- go build ./... clean
- go vet ./... clean
- gofmt -l clean on touched files
- go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS
endpoint), the docreader gRPC tests, and the doris SQL-shape tests
fail; all three are pre-existing on upstream/main and untouched by
this PR.
- New tests across internal/types, opensearch, service and container —
including a full end-to-end env-path test that exercises
initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an
httptest cluster.
2026-05-28 09:01:17 +00:00
|
|
|
|
# OpenSearch k-NN (Phase 3 driver). Single-node dev profile with the
|
|
|
|
|
|
# security plugin disabled → plain HTTP on :9200, no auth/TLS. The image
|
|
|
|
|
|
# bundles the opensearch-knn plugin. For production use a secured,
|
2026-09-22 15:25:40 +00:00
|
|
|
|
# multi-node cluster. See website-docs/03-features/05-retrieval-engines.md.
|
feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3)
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482)
laid the type prep + driver skeleton + read/write paths as gated dead
code; this PR wires every activation surface so opensearch becomes a
registerable VectorStore engine.
Activation wiring
- internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW
bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping /
buildEnvStoreForDriver — every gated surface now recognises
"opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM /
HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines'
serialised config byte-identical.
- internal/container: createOpenSearchEngine + the switch case in
createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path
in initRetrieveEngineRegistry; NewEngineFactory now closes over the
AuditLogService (the EngineFactory type itself is unchanged).
- internal/application/service/vectorstore_healthcheck.go: a
testOpenSearchConnection case so CreateStore's connectivity probe
accepts opensearch instead of returning 400.
- internal/application/repository/retriever/opensearch/transport.go:
NewOpenSearchClient is exported so the factory and env path can build
the TLS-hardened client; healthcheck.go reuses the unexported
probeVersion / probeKNNPlugin for the service-layer probe.
Service-layer validation
- validateOpenSearchIndexConfig validates the HNSW caps (m 2-100,
ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss).
Shards/replicas continue to be enforced by the flat ValidateIndexConfig.
Create-only: UpdateStore mutates the name only.
- validateConnectionConfig requires addr for opensearch.
Sync implementations (stubs.go shrinks)
- CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern —
search → BatchSave with the source_id remap for generated questions —
so dim/keyword routing and the source_id contract come from BatchSave
for free. embeddingMap is keyed by the *target* SourceID because
OpenSearch's BatchSave looks up embeddings by SourceID
(lookupEmbedding), not by chunk_id (the ES driver's convention).
Pagination is from/size; copies larger than max_result_window
(default 10000) need the scroll-based async path that lands later.
- BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go)
group the input by target value and issue one _update_by_query per
group over the cross-dim <base>_* pattern. Caller values flow through
bound script params only — never string-interpolated into the Painless
source — closing the script-injection surface.
- inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the
full failure reason goes to the debug log only; the returned error
carries the bounded id + type.
- UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same
shape as DeleteByQuery's quirk), so refresh=wait_for is not
expressible; we use refresh=true.
Driver-owned audit (DIP)
- A new opensearch.AuditSink interface (with nopSink + WithAuditSink
functional option) lets the driver emit opensearch.index_created and
opensearch.reindex_executed events without importing any service
package — the service layer implements the interface. NewRepository
takes opts, so existing 4-arg test call sites keep compiling unchanged.
- internal/container/audit_sink.go bridges AuditSink to AuditLogService.
When the context carries no tenant (the env-path registration ctx
during boot, for example) the adapter skips the emit with a warning
rather than silently writing tenant_id=0, which would collide with the
system-scope sentinel.
Frontend + polish
- FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/
immutable. VectorStoreSettings.vue is now schema-driven: a closed
`enum` renders a t-select; number inputs use the schema's `:min`/`:max`
and fall back to the legacy replica-vs-shard heuristic only when the
schema does not pin them; a danger-coloured warning fires when
insecure_skip_verify is toggled on (the switch and warning are wrapped
in a vertical stack so the warning sits on its own row below the switch).
- i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search /
knn_engine / insecure_skip_verify plus the warning copy in en-US,
ko-KR, zh-CN, ru-RU.
- docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with
security plugin disabled for dev only). OpenSearch Dashboards lives in a
separate, opt-in opensearch-ui profile so the heavy UI container is not
forced up alongside the cluster (the driver e2e is fully curl-verifiable
against :9200). The new docs/dev/opensearch-integration-test.md covers the
end-to-end exercise and the single-node guidance (set replicas=0 to keep
the cluster Green).
Gating-guard tests flipped
- The "OpenSearch is NOT in validEngineTypes / mapping / types list /
env builder / stubs" guard tests from PR 1 / PR 2 are replaced by
their positive counterparts in this PR. The test suite was the
activation checklist; the activation flip is its diff.
Backward compatibility
- Additive everywhere. IndexConfig's new HNSW fields are omitempty so
other engines' serialised config is byte-identical. Existing
Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB
stores are untouched. No migrations.
Test plan
- go build ./... clean
- go vet ./... clean
- gofmt -l clean on touched files
- go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS
endpoint), the docreader gRPC tests, and the doris SQL-shape tests
fail; all three are pre-existing on upstream/main and untouched by
this PR.
- New tests across internal/types, opensearch, service and container —
including a full end-to-end env-path test that exercises
initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an
httptest cluster.
2026-05-28 09:01:17 +00:00
|
|
|
|
opensearch:
|
|
|
|
|
|
image: opensearchproject/opensearch:3.3.2
|
|
|
|
|
|
container_name: WeKnora-opensearch-dev
|
|
|
|
|
|
environment:
|
|
|
|
|
|
- discovery.type=single-node
|
|
|
|
|
|
# dev only: plain HTTP on :9200, no TLS/auth. The entrypoint script
|
|
|
|
|
|
# honours DISABLE_SECURITY_PLUGIN (env var) to skip both the demo
|
|
|
|
|
|
# install and the OPENSEARCH_INITIAL_ADMIN_PASSWORD requirement.
|
|
|
|
|
|
- DISABLE_SECURITY_PLUGIN=true
|
|
|
|
|
|
- DISABLE_INSTALL_DEMO_CONFIG=true
|
|
|
|
|
|
- OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
|
|
|
|
|
|
- bootstrap.memory_lock=true
|
|
|
|
|
|
ulimits:
|
|
|
|
|
|
memlock:
|
|
|
|
|
|
soft: -1
|
|
|
|
|
|
hard: -1
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${OPENSEARCH_PORT:-9200}:9200"
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- opensearch_data_dev:/usr/share/opensearch/data
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- opensearch
|
|
|
|
|
|
- full
|
|
|
|
|
|
# Also a member of opensearch-ui so the Dashboards depends_on resolves
|
|
|
|
|
|
# when only that profile is active (`--profile opensearch-ui up`).
|
|
|
|
|
|
- opensearch-ui
|
|
|
|
|
|
|
|
|
|
|
|
# Optional UI for visual index/mapping/query inspection. Decoupled from the
|
|
|
|
|
|
# "opensearch" / "full" profiles so the heavy Dashboards container is never
|
|
|
|
|
|
# forced up alongside the cluster — the driver e2e is fully curl-verifiable
|
|
|
|
|
|
# against :9200. Start it on demand with `--profile opensearch-ui up -d`
|
|
|
|
|
|
# (depends_on pulls the cluster in automatically).
|
|
|
|
|
|
opensearch-dashboards:
|
|
|
|
|
|
image: opensearchproject/opensearch-dashboards:3.3.0
|
|
|
|
|
|
container_name: WeKnora-opensearch-dashboards-dev
|
|
|
|
|
|
environment:
|
|
|
|
|
|
- OPENSEARCH_HOSTS=["http://opensearch:9200"]
|
|
|
|
|
|
- DISABLE_SECURITY_DASHBOARDS_PLUGIN=true
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${OPENSEARCH_DASHBOARDS_PORT:-5601}:5601"
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
depends_on:
|
|
|
|
|
|
- opensearch
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- opensearch-ui
|
|
|
|
|
|
|
2026-03-19 14:38:35 +00:00
|
|
|
|
milvus:
|
|
|
|
|
|
image: milvusdb/milvus:v2.6.11
|
|
|
|
|
|
container_name: WeKnora-milvus-dev
|
|
|
|
|
|
security_opt:
|
|
|
|
|
|
- seccomp:unconfined
|
|
|
|
|
|
command: ["milvus", "run", "standalone"]
|
|
|
|
|
|
environment:
|
|
|
|
|
|
- ETCD_USE_EMBED=true
|
|
|
|
|
|
- ETCD_DATA_DIR=/var/lib/milvus/etcd
|
|
|
|
|
|
- COMMON_STORAGETYPE=local
|
|
|
|
|
|
- DEPLOY_MODE=STANDALONE
|
|
|
|
|
|
healthcheck:
|
|
|
|
|
|
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
|
|
|
|
|
|
interval: 30s
|
|
|
|
|
|
start_period: 90s
|
|
|
|
|
|
timeout: 20s
|
|
|
|
|
|
retries: 3
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${MILVUS_PORT:-19530}:19530"
|
|
|
|
|
|
- "${MILVUS_HEALTH_PORT:-9091}:9091"
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- milvus_data_dev:/var/lib/milvus
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- milvus
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
2025-11-07 07:52:52 +00:00
|
|
|
|
neo4j:
|
|
|
|
|
|
image: neo4j:latest
|
|
|
|
|
|
container_name: WeKnora-neo4j-dev
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- neo4j-data-dev:/data
|
|
|
|
|
|
environment:
|
|
|
|
|
|
- NEO4J_AUTH=${NEO4J_USERNAME:-neo4j}/${NEO4J_PASSWORD:-password}
|
|
|
|
|
|
- NEO4J_apoc_export_file_enabled=true
|
|
|
|
|
|
- NEO4J_apoc_import_file_enabled=true
|
|
|
|
|
|
- NEO4J_apoc_import_file_use__neo4j__config=true
|
|
|
|
|
|
- NEO4JLABS_PLUGINS=["apoc"]
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "7474:7474"
|
|
|
|
|
|
- "7687:7687"
|
|
|
|
|
|
restart: always
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
2025-12-08 11:32:46 +00:00
|
|
|
|
profiles:
|
|
|
|
|
|
- neo4j
|
|
|
|
|
|
- full
|
2025-11-07 07:52:52 +00:00
|
|
|
|
|
2026-02-04 12:47:03 +00:00
|
|
|
|
# Sandbox 镜像:仅用于 build/pull,非常驻服务;本地 app 执行 Skills 时按需 docker run 该镜像,用毕即释
|
|
|
|
|
|
sandbox:
|
|
|
|
|
|
image: wechatopenai/weknora-sandbox:${WEKNORA_VERSION:-latest}
|
2026-02-04 13:31:16 +00:00
|
|
|
|
container_name: WeKnora-sandbox-dev
|
2026-02-04 12:47:03 +00:00
|
|
|
|
build:
|
|
|
|
|
|
context: .
|
|
|
|
|
|
dockerfile: docker/Dockerfile.sandbox
|
|
|
|
|
|
profiles:
|
2026-02-04 13:31:16 +00:00
|
|
|
|
- full
|
2026-02-04 12:47:03 +00:00
|
|
|
|
command: ["true"]
|
|
|
|
|
|
restart: "no"
|
|
|
|
|
|
|
2025-11-07 07:52:52 +00:00
|
|
|
|
docreader:
|
2026-02-28 05:22:07 +00:00
|
|
|
|
build:
|
|
|
|
|
|
context: .
|
|
|
|
|
|
dockerfile: docker/Dockerfile.docreader
|
2026-03-20 04:17:49 +00:00
|
|
|
|
image: wechatopenai/weknora-docreader:${WEKNORA_VERSION:-latest}
|
2025-11-07 07:52:52 +00:00
|
|
|
|
container_name: WeKnora-docreader-dev
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${DOCREADER_PORT:-50051}:50051"
|
2026-02-28 05:22:07 +00:00
|
|
|
|
volumes:
|
|
|
|
|
|
- docreader-tmp-dev:/tmp/docreader
|
2025-11-07 07:52:52 +00:00
|
|
|
|
environment:
|
feat(compose): align env vars with code + fix helm GraphRAG bug (#2164)
* feat(compose): align env vars in docker-compose and .env.example with code
docker-compose.yml / docker-compose.dev.yml
- app.environment: remove deprecated vars Go app no longer reads
(CRYPTO_MASTER_KEY, CRYPTO_SALT, TENANT_AES_KEY, ENABLE_GRAPH_RAG);
add missing vars the code actually consumes (OSS/S3/TOS/COS completion,
OpenSearch, Tencent VectorDB, Milvus credentials, Redis TLS/namespace,
OIDC, audit/invitation/quota, LLM tuning, doc timeouts, etc.)
- docreader.environment: complete PDF tuning knobs, SSRF/LOG_LEVEL/gRPC
params, GRPC_MTLS_REQUIRE_CLIENT_CERT; fix DOCREADER_PDF_JPEG_QUALITY
default 90 -> 85 to match code (config.py)
- mcp.environment: add MCP_SERVER_AUTH_TOKEN (required by HTTP transport),
WEKNORA_CHAT_TIMEOUT, WEKNORA_VERIFY_SSL, MCP_ALLOWED_UPLOAD_DIRS
- dev compose docreader.environment kept symmetric with prod
.env.example
- reorganize into A-J sections (deploy / data / retrieval / models /
parsing / auth / agent / integrations / observability / security)
- remove the four deprecated vars, document why
- add missing vars, fix NEO4J_URI to bolt:// (matches compose + single-node),
fix DORIS_COMPAT_MODE typo, complete docreader PDF tuning defaults
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(helm): inject NEO4J_ENABLE and drop deprecated TENANT_AES_KEY
helm graph bug fix
- NEO4J_ENABLE is the sole graph switch the Go app reads (since v0.1.6,
ENABLE_GRAPH_RAG is no longer read). helm never injected NEO4J_ENABLE,
so GraphRAG was silently broken: setting neo4j.enabled=true gave the
app NEO4J_URI/USERNAME/PASSWORD but not the switch, and the UI rejected
Node Extractor config with "请正确配置环境变量NEO4J_ENABLE".
- app.yaml: inject NEO4J_ENABLE=true inside the neo4j.enabled block;
remove the dead ENABLE_GRAPH_RAG injection
- NOTES.txt: update GraphRAG instructions (just enable neo4j, no need
to set the deprecated ENABLE_GRAPH_RAG)
- values.yaml: drop ENABLE_GRAPH_RAG, fix neo4j section comment
TENANT_AES_KEY cleanup (deprecated since v0.4.0, encryption now uses
SYSTEM_AES_KEY exclusively; Go app no longer reads it)
- secrets.yaml: stop generating/looking-up/writing TENANT_AES_KEY,
fix stale comment about tenants.api_key rotation
- app.yaml: remove TENANT_AES_KEY secretKeyRef
- values.yaml: drop tenantAesKey, fix existingSecret key list + comments
- cloud-image/firstboot.sh: stop generating/writing/printing TENANT_AES
- cloud-image/README.md: drop TENANT_AES_KEY from credential list
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-21 07:55:52 +00:00
|
|
|
|
# ========== 基础 ==========
|
2026-07-09 05:24:38 +00:00
|
|
|
|
- TZ=${TZ:-Asia/Shanghai}
|
feat(compose): align env vars with code + fix helm GraphRAG bug (#2164)
* feat(compose): align env vars in docker-compose and .env.example with code
docker-compose.yml / docker-compose.dev.yml
- app.environment: remove deprecated vars Go app no longer reads
(CRYPTO_MASTER_KEY, CRYPTO_SALT, TENANT_AES_KEY, ENABLE_GRAPH_RAG);
add missing vars the code actually consumes (OSS/S3/TOS/COS completion,
OpenSearch, Tencent VectorDB, Milvus credentials, Redis TLS/namespace,
OIDC, audit/invitation/quota, LLM tuning, doc timeouts, etc.)
- docreader.environment: complete PDF tuning knobs, SSRF/LOG_LEVEL/gRPC
params, GRPC_MTLS_REQUIRE_CLIENT_CERT; fix DOCREADER_PDF_JPEG_QUALITY
default 90 -> 85 to match code (config.py)
- mcp.environment: add MCP_SERVER_AUTH_TOKEN (required by HTTP transport),
WEKNORA_CHAT_TIMEOUT, WEKNORA_VERIFY_SSL, MCP_ALLOWED_UPLOAD_DIRS
- dev compose docreader.environment kept symmetric with prod
.env.example
- reorganize into A-J sections (deploy / data / retrieval / models /
parsing / auth / agent / integrations / observability / security)
- remove the four deprecated vars, document why
- add missing vars, fix NEO4J_URI to bolt:// (matches compose + single-node),
fix DORIS_COMPAT_MODE typo, complete docreader PDF tuning defaults
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(helm): inject NEO4J_ENABLE and drop deprecated TENANT_AES_KEY
helm graph bug fix
- NEO4J_ENABLE is the sole graph switch the Go app reads (since v0.1.6,
ENABLE_GRAPH_RAG is no longer read). helm never injected NEO4J_ENABLE,
so GraphRAG was silently broken: setting neo4j.enabled=true gave the
app NEO4J_URI/USERNAME/PASSWORD but not the switch, and the UI rejected
Node Extractor config with "请正确配置环境变量NEO4J_ENABLE".
- app.yaml: inject NEO4J_ENABLE=true inside the neo4j.enabled block;
remove the dead ENABLE_GRAPH_RAG injection
- NOTES.txt: update GraphRAG instructions (just enable neo4j, no need
to set the deprecated ENABLE_GRAPH_RAG)
- values.yaml: drop ENABLE_GRAPH_RAG, fix neo4j section comment
TENANT_AES_KEY cleanup (deprecated since v0.4.0, encryption now uses
SYSTEM_AES_KEY exclusively; Go app no longer reads it)
- secrets.yaml: stop generating/looking-up/writing TENANT_AES_KEY,
fix stale comment about tenants.api_key rotation
- app.yaml: remove TENANT_AES_KEY secretKeyRef
- values.yaml: drop tenantAesKey, fix existingSecret key list + comments
- cloud-image/firstboot.sh: stop generating/writing/printing TENANT_AES
- cloud-image/README.md: drop TENANT_AES_KEY from credential list
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-21 07:55:52 +00:00
|
|
|
|
- LOG_LEVEL=${LOG_LEVEL:-}
|
2026-02-28 05:22:07 +00:00
|
|
|
|
- DOCREADER_IMAGE_OUTPUT_DIR=/tmp/docreader
|
|
|
|
|
|
- MAX_FILE_SIZE_MB=${MAX_FILE_SIZE_MB:-}
|
feat(compose): align env vars with code + fix helm GraphRAG bug (#2164)
* feat(compose): align env vars in docker-compose and .env.example with code
docker-compose.yml / docker-compose.dev.yml
- app.environment: remove deprecated vars Go app no longer reads
(CRYPTO_MASTER_KEY, CRYPTO_SALT, TENANT_AES_KEY, ENABLE_GRAPH_RAG);
add missing vars the code actually consumes (OSS/S3/TOS/COS completion,
OpenSearch, Tencent VectorDB, Milvus credentials, Redis TLS/namespace,
OIDC, audit/invitation/quota, LLM tuning, doc timeouts, etc.)
- docreader.environment: complete PDF tuning knobs, SSRF/LOG_LEVEL/gRPC
params, GRPC_MTLS_REQUIRE_CLIENT_CERT; fix DOCREADER_PDF_JPEG_QUALITY
default 90 -> 85 to match code (config.py)
- mcp.environment: add MCP_SERVER_AUTH_TOKEN (required by HTTP transport),
WEKNORA_CHAT_TIMEOUT, WEKNORA_VERIFY_SSL, MCP_ALLOWED_UPLOAD_DIRS
- dev compose docreader.environment kept symmetric with prod
.env.example
- reorganize into A-J sections (deploy / data / retrieval / models /
parsing / auth / agent / integrations / observability / security)
- remove the four deprecated vars, document why
- add missing vars, fix NEO4J_URI to bolt:// (matches compose + single-node),
fix DORIS_COMPAT_MODE typo, complete docreader PDF tuning defaults
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(helm): inject NEO4J_ENABLE and drop deprecated TENANT_AES_KEY
helm graph bug fix
- NEO4J_ENABLE is the sole graph switch the Go app reads (since v0.1.6,
ENABLE_GRAPH_RAG is no longer read). helm never injected NEO4J_ENABLE,
so GraphRAG was silently broken: setting neo4j.enabled=true gave the
app NEO4J_URI/USERNAME/PASSWORD but not the switch, and the UI rejected
Node Extractor config with "请正确配置环境变量NEO4J_ENABLE".
- app.yaml: inject NEO4J_ENABLE=true inside the neo4j.enabled block;
remove the dead ENABLE_GRAPH_RAG injection
- NOTES.txt: update GraphRAG instructions (just enable neo4j, no need
to set the deprecated ENABLE_GRAPH_RAG)
- values.yaml: drop ENABLE_GRAPH_RAG, fix neo4j section comment
TENANT_AES_KEY cleanup (deprecated since v0.4.0, encryption now uses
SYSTEM_AES_KEY exclusively; Go app no longer reads it)
- secrets.yaml: stop generating/looking-up/writing TENANT_AES_KEY,
fix stale comment about tenants.api_key rotation
- app.yaml: remove TENANT_AES_KEY secretKeyRef
- values.yaml: drop tenantAesKey, fix existingSecret key list + comments
- cloud-image/firstboot.sh: stop generating/writing/printing TENANT_AES
- cloud-image/README.md: drop TENANT_AES_KEY from credential list
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-21 07:55:52 +00:00
|
|
|
|
# docreader 对外 HTTP 抓取的 SSRF 白名单(逗号分隔;与 app 共用同名变量)
|
|
|
|
|
|
- SSRF_WHITELIST=${SSRF_WHITELIST:-}
|
|
|
|
|
|
- SSRF_WHITELIST_EXTRA=${SSRF_WHITELIST_EXTRA:-}
|
feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393)
* feat(security): add a whitelist-only outbound mode that refuses before DNS
A penetration test of a private deployment found that a host outside
SSRF_WHITELIST still produces a DNS query: every outbound guard resolves
the name first and judges the addresses afterwards, which is what makes
it immune to DNS rebinding, but it also means an attacker-chosen name
reaches the resolver and can carry data out of an otherwise offline
network (#3378).
SSRF_DNS_WHITELIST_ONLY makes the whitelist the whole egress policy, and
the check is placed wherever a name is still in hand.
Where a URL is available, CheckSSRFWhitelistOnly refuses a host the
whitelist does not name, IP literals included: URL validation, the
sandbox outbound policy, and the web-fetch target resolver.
Where only a dial address is available, CheckDialWhitelistOnly judges
hostnames and leaves addresses to the checks that already exist, because
an IP arriving at a dialer is either a name something else resolved or a
literal the URL layer already refused. That covers the shared SSRF-safe
dialer, the web-fetch dialer, the Yunzhijia webhook dialer, and every
sandbox transport: the HTTP and websocket data planes, the guarded
transport and the Docker engine now share one GuardedDialContext instead
of each building its own dialer.
Where the client resolves its own target, the name is judged before the
client exists. gRPC's resolver runs before any dialer it was given, so
the env-configured qdrant and milvus clients are built through small
constructors that refuse a non-whitelisted host outright; a whitelisted
one is resolved by gRPC as before.
docreader enforces the same switch in its own validator, and compose
passes the variable to both services.
Matching is by name only in this mode: the whitelist's CIDR entries
still match an IP literal, but a hostname is no longer resolved to see
whether its addresses fall inside one, because that lookup is the query
the mode exists to prevent.
The value is parsed with ParseBool, and a non-empty value that parses as
neither enables the mode: silently leaving an egress lockdown off
because an operator wrote "yes" is the one failure mode a security
control must not have.
Still not covered, and documented as such: the Langfuse OTLP exporter
builds its own client and never reaches these guards; and with
HTTP(S)_PROXY set the proxy host is dialled unguarded when the dial
address equals the proxy URL's host exactly, while a mismatch makes the
proxy host itself subject to the whitelist.
The switch defaults to off, so nothing changes for deployments that do
not set it.
* docs(security): document SSRF_DNS_WHITELIST_ONLY in website-docs
docs/ has been retired in favour of website-docs/ (#3580), so the notes that
lived in docs/QA.md move to the configuration table: the new switch, what to
whitelist before turning it on, and the three outbound paths it still does
not cover.
The existing SSRF_WHITELIST row was also stale: compose gives only the app its
SSRF_WHITELIST_EXTRA default (which now includes minio), and docreader's is
empty, so "shared by app and docreader" held for SSRF_WHITELIST alone.
2026-09-23 04:02:06 +00:00
|
|
|
|
- SSRF_DNS_WHITELIST_ONLY=${SSRF_DNS_WHITELIST_ONLY:-}
|
feat(compose): align env vars with code + fix helm GraphRAG bug (#2164)
* feat(compose): align env vars in docker-compose and .env.example with code
docker-compose.yml / docker-compose.dev.yml
- app.environment: remove deprecated vars Go app no longer reads
(CRYPTO_MASTER_KEY, CRYPTO_SALT, TENANT_AES_KEY, ENABLE_GRAPH_RAG);
add missing vars the code actually consumes (OSS/S3/TOS/COS completion,
OpenSearch, Tencent VectorDB, Milvus credentials, Redis TLS/namespace,
OIDC, audit/invitation/quota, LLM tuning, doc timeouts, etc.)
- docreader.environment: complete PDF tuning knobs, SSRF/LOG_LEVEL/gRPC
params, GRPC_MTLS_REQUIRE_CLIENT_CERT; fix DOCREADER_PDF_JPEG_QUALITY
default 90 -> 85 to match code (config.py)
- mcp.environment: add MCP_SERVER_AUTH_TOKEN (required by HTTP transport),
WEKNORA_CHAT_TIMEOUT, WEKNORA_VERIFY_SSL, MCP_ALLOWED_UPLOAD_DIRS
- dev compose docreader.environment kept symmetric with prod
.env.example
- reorganize into A-J sections (deploy / data / retrieval / models /
parsing / auth / agent / integrations / observability / security)
- remove the four deprecated vars, document why
- add missing vars, fix NEO4J_URI to bolt:// (matches compose + single-node),
fix DORIS_COMPAT_MODE typo, complete docreader PDF tuning defaults
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(helm): inject NEO4J_ENABLE and drop deprecated TENANT_AES_KEY
helm graph bug fix
- NEO4J_ENABLE is the sole graph switch the Go app reads (since v0.1.6,
ENABLE_GRAPH_RAG is no longer read). helm never injected NEO4J_ENABLE,
so GraphRAG was silently broken: setting neo4j.enabled=true gave the
app NEO4J_URI/USERNAME/PASSWORD but not the switch, and the UI rejected
Node Extractor config with "请正确配置环境变量NEO4J_ENABLE".
- app.yaml: inject NEO4J_ENABLE=true inside the neo4j.enabled block;
remove the dead ENABLE_GRAPH_RAG injection
- NOTES.txt: update GraphRAG instructions (just enable neo4j, no need
to set the deprecated ENABLE_GRAPH_RAG)
- values.yaml: drop ENABLE_GRAPH_RAG, fix neo4j section comment
TENANT_AES_KEY cleanup (deprecated since v0.4.0, encryption now uses
SYSTEM_AES_KEY exclusively; Go app no longer reads it)
- secrets.yaml: stop generating/looking-up/writing TENANT_AES_KEY,
fix stale comment about tenants.api_key rotation
- app.yaml: remove TENANT_AES_KEY secretKeyRef
- values.yaml: drop tenantAesKey, fix existingSecret key list + comments
- cloud-image/firstboot.sh: stop generating/writing/printing TENANT_AES
- cloud-image/README.md: drop TENANT_AES_KEY from credential list
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-21 07:55:52 +00:00
|
|
|
|
# ========== gRPC 服务端 TLS / 认证 ==========
|
|
|
|
|
|
- GRPC_TLS_ENABLED=${GRPC_TLS_ENABLED:-false}
|
|
|
|
|
|
- GRPC_TLS_CERT=${GRPC_TLS_CERT:-}
|
|
|
|
|
|
- GRPC_TLS_KEY=${GRPC_TLS_KEY:-}
|
|
|
|
|
|
- GRPC_TLS_CA=${GRPC_TLS_CA:-}
|
|
|
|
|
|
- GRPC_TLS_SERVER_NAME=${GRPC_TLS_SERVER_NAME:-}
|
|
|
|
|
|
# true 时强制要求客户端证书(mTLS)
|
|
|
|
|
|
- GRPC_MTLS_REQUIRE_CLIENT_CERT=${GRPC_MTLS_REQUIRE_CLIENT_CERT:-false}
|
|
|
|
|
|
- GRPC_AUTH_TOKEN=${GRPC_AUTH_TOKEN:-}
|
|
|
|
|
|
# ========== gRPC 服务参数 ==========
|
|
|
|
|
|
- DOCREADER_GRPC_MAX_WORKERS=${DOCREADER_GRPC_MAX_WORKERS:-4}
|
|
|
|
|
|
- DOCREADER_GRPC_MAX_FILE_SIZE_MB=${DOCREADER_GRPC_MAX_FILE_SIZE_MB:-${MAX_FILE_SIZE_MB:-50}}
|
|
|
|
|
|
- DOCREADER_GRPC_PORT=${DOCREADER_GRPC_PORT:-50051}
|
|
|
|
|
|
# ========== 解析通用 ==========
|
|
|
|
|
|
- DOCREADER_DOCX_MAX_PAGES=${DOCREADER_DOCX_MAX_PAGES:-0}
|
|
|
|
|
|
- DOCREADER_MARKITDOWN_MAX_WORKERS=${DOCREADER_MARKITDOWN_MAX_WORKERS:-1}
|
|
|
|
|
|
# 出站 HTTP/HTTPS 代理(抓取网页 / 远程图片时)
|
|
|
|
|
|
- DOCREADER_EXTERNAL_HTTP_PROXY=${DOCREADER_EXTERNAL_HTTP_PROXY:-}
|
|
|
|
|
|
- DOCREADER_EXTERNAL_HTTPS_PROXY=${DOCREADER_EXTERNAL_HTTPS_PROXY:-}
|
|
|
|
|
|
# ========== OpenDataLoader hybrid ==========
|
2026-06-03 04:00:09 +00:00
|
|
|
|
- DOCREADER_ODL_MAX_WORKERS=${DOCREADER_ODL_MAX_WORKERS:-1}
|
|
|
|
|
|
- DOCREADER_ODL_HYBRID=${DOCREADER_ODL_HYBRID:-off}
|
|
|
|
|
|
- DOCREADER_ODL_HYBRID_URL=${DOCREADER_ODL_HYBRID_URL:-http://odl-hybrid:5002}
|
|
|
|
|
|
- DOCREADER_ODL_HYBRID_MODE=${DOCREADER_ODL_HYBRID_MODE:-auto}
|
|
|
|
|
|
- DOCREADER_ODL_HYBRID_FALLBACK=${DOCREADER_ODL_HYBRID_FALLBACK:-false}
|
|
|
|
|
|
- DOCREADER_ODL_MARKDOWN_WITH_HTML=${DOCREADER_ODL_MARKDOWN_WITH_HTML:-false}
|
feat(compose): align env vars with code + fix helm GraphRAG bug (#2164)
* feat(compose): align env vars in docker-compose and .env.example with code
docker-compose.yml / docker-compose.dev.yml
- app.environment: remove deprecated vars Go app no longer reads
(CRYPTO_MASTER_KEY, CRYPTO_SALT, TENANT_AES_KEY, ENABLE_GRAPH_RAG);
add missing vars the code actually consumes (OSS/S3/TOS/COS completion,
OpenSearch, Tencent VectorDB, Milvus credentials, Redis TLS/namespace,
OIDC, audit/invitation/quota, LLM tuning, doc timeouts, etc.)
- docreader.environment: complete PDF tuning knobs, SSRF/LOG_LEVEL/gRPC
params, GRPC_MTLS_REQUIRE_CLIENT_CERT; fix DOCREADER_PDF_JPEG_QUALITY
default 90 -> 85 to match code (config.py)
- mcp.environment: add MCP_SERVER_AUTH_TOKEN (required by HTTP transport),
WEKNORA_CHAT_TIMEOUT, WEKNORA_VERIFY_SSL, MCP_ALLOWED_UPLOAD_DIRS
- dev compose docreader.environment kept symmetric with prod
.env.example
- reorganize into A-J sections (deploy / data / retrieval / models /
parsing / auth / agent / integrations / observability / security)
- remove the four deprecated vars, document why
- add missing vars, fix NEO4J_URI to bolt:// (matches compose + single-node),
fix DORIS_COMPAT_MODE typo, complete docreader PDF tuning defaults
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(helm): inject NEO4J_ENABLE and drop deprecated TENANT_AES_KEY
helm graph bug fix
- NEO4J_ENABLE is the sole graph switch the Go app reads (since v0.1.6,
ENABLE_GRAPH_RAG is no longer read). helm never injected NEO4J_ENABLE,
so GraphRAG was silently broken: setting neo4j.enabled=true gave the
app NEO4J_URI/USERNAME/PASSWORD but not the switch, and the UI rejected
Node Extractor config with "请正确配置环境变量NEO4J_ENABLE".
- app.yaml: inject NEO4J_ENABLE=true inside the neo4j.enabled block;
remove the dead ENABLE_GRAPH_RAG injection
- NOTES.txt: update GraphRAG instructions (just enable neo4j, no need
to set the deprecated ENABLE_GRAPH_RAG)
- values.yaml: drop ENABLE_GRAPH_RAG, fix neo4j section comment
TENANT_AES_KEY cleanup (deprecated since v0.4.0, encryption now uses
SYSTEM_AES_KEY exclusively; Go app no longer reads it)
- secrets.yaml: stop generating/looking-up/writing TENANT_AES_KEY,
fix stale comment about tenants.api_key rotation
- app.yaml: remove TENANT_AES_KEY secretKeyRef
- values.yaml: drop tenantAesKey, fix existingSecret key list + comments
- cloud-image/firstboot.sh: stop generating/writing/printing TENANT_AES
- cloud-image/README.md: drop TENANT_AES_KEY from credential list
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-21 07:55:52 +00:00
|
|
|
|
# ========== PDF 渲染 ==========
|
2026-05-07 08:05:21 +00:00
|
|
|
|
- DOCREADER_PDF_RENDER_MAX_WORKERS=${DOCREADER_PDF_RENDER_MAX_WORKERS:-1}
|
2026-07-21 08:01:16 +00:00
|
|
|
|
# 单个 PDF 内部扫描页渲染并行进程数(pdfium 非线程安全)。留空走代码默认 min(4,cpu),
|
|
|
|
|
|
# 避免在低核容器上过度并行(写死数值会覆盖 CPU 感知逻辑)。
|
|
|
|
|
|
- DOCREADER_PDF_RENDER_PARALLELISM=${DOCREADER_PDF_RENDER_PARALLELISM:-}
|
2026-05-07 08:05:21 +00:00
|
|
|
|
- DOCREADER_PDF_RENDER_DPI=${DOCREADER_PDF_RENDER_DPI:-200}
|
feat(compose): align env vars with code + fix helm GraphRAG bug (#2164)
* feat(compose): align env vars in docker-compose and .env.example with code
docker-compose.yml / docker-compose.dev.yml
- app.environment: remove deprecated vars Go app no longer reads
(CRYPTO_MASTER_KEY, CRYPTO_SALT, TENANT_AES_KEY, ENABLE_GRAPH_RAG);
add missing vars the code actually consumes (OSS/S3/TOS/COS completion,
OpenSearch, Tencent VectorDB, Milvus credentials, Redis TLS/namespace,
OIDC, audit/invitation/quota, LLM tuning, doc timeouts, etc.)
- docreader.environment: complete PDF tuning knobs, SSRF/LOG_LEVEL/gRPC
params, GRPC_MTLS_REQUIRE_CLIENT_CERT; fix DOCREADER_PDF_JPEG_QUALITY
default 90 -> 85 to match code (config.py)
- mcp.environment: add MCP_SERVER_AUTH_TOKEN (required by HTTP transport),
WEKNORA_CHAT_TIMEOUT, WEKNORA_VERIFY_SSL, MCP_ALLOWED_UPLOAD_DIRS
- dev compose docreader.environment kept symmetric with prod
.env.example
- reorganize into A-J sections (deploy / data / retrieval / models /
parsing / auth / agent / integrations / observability / security)
- remove the four deprecated vars, document why
- add missing vars, fix NEO4J_URI to bolt:// (matches compose + single-node),
fix DORIS_COMPAT_MODE typo, complete docreader PDF tuning defaults
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(helm): inject NEO4J_ENABLE and drop deprecated TENANT_AES_KEY
helm graph bug fix
- NEO4J_ENABLE is the sole graph switch the Go app reads (since v0.1.6,
ENABLE_GRAPH_RAG is no longer read). helm never injected NEO4J_ENABLE,
so GraphRAG was silently broken: setting neo4j.enabled=true gave the
app NEO4J_URI/USERNAME/PASSWORD but not the switch, and the UI rejected
Node Extractor config with "请正确配置环境变量NEO4J_ENABLE".
- app.yaml: inject NEO4J_ENABLE=true inside the neo4j.enabled block;
remove the dead ENABLE_GRAPH_RAG injection
- NOTES.txt: update GraphRAG instructions (just enable neo4j, no need
to set the deprecated ENABLE_GRAPH_RAG)
- values.yaml: drop ENABLE_GRAPH_RAG, fix neo4j section comment
TENANT_AES_KEY cleanup (deprecated since v0.4.0, encryption now uses
SYSTEM_AES_KEY exclusively; Go app no longer reads it)
- secrets.yaml: stop generating/looking-up/writing TENANT_AES_KEY,
fix stale comment about tenants.api_key rotation
- app.yaml: remove TENANT_AES_KEY secretKeyRef
- values.yaml: drop tenantAesKey, fix existingSecret key list + comments
- cloud-image/firstboot.sh: stop generating/writing/printing TENANT_AES
- cloud-image/README.md: drop TENANT_AES_KEY from credential list
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-21 07:55:52 +00:00
|
|
|
|
# JPEG 质量(代码默认 85;compose 此前误写 90,已对齐)
|
|
|
|
|
|
- DOCREADER_PDF_JPEG_QUALITY=${DOCREADER_PDF_JPEG_QUALITY:-85}
|
|
|
|
|
|
# 渲染页图最大长边像素(防超大页撑爆 gRPC 消息上限;密集中文建议 >=1600)
|
|
|
|
|
|
- DOCREADER_PDF_RENDER_MAX_EDGE=${DOCREADER_PDF_RENDER_MAX_EDGE:-2000}
|
|
|
|
|
|
# 强制 PDF 全页按扫描件解析(文本层差但可 OCR 时;速度/成本上升)
|
|
|
|
|
|
- DOCREADER_PDF_FORCE_SCANNED=${DOCREADER_PDF_FORCE_SCANNED:-false}
|
|
|
|
|
|
# ========== PDF 版式 / 文本调优 ==========
|
|
|
|
|
|
# glyph 间距超过 median 字宽的多少倍时插空格(默认 0.4)
|
|
|
|
|
|
- DOCREADER_PDF_WORD_GAP_WIDTH_RATIO=${DOCREADER_PDF_WORD_GAP_WIDTH_RATIO:-0.4}
|
|
|
|
|
|
# 丢弃窄边栏(arXiv 侧栏)的页宽比例阈值(默认 0.12)
|
|
|
|
|
|
- DOCREADER_PDF_MARGIN_COL_WIDTH_RATIO=${DOCREADER_PDF_MARGIN_COL_WIDTH_RATIO:-0.12}
|
|
|
|
|
|
- DOCREADER_PDF_MIN_HEADING_LINE_CHARS=${DOCREADER_PDF_MIN_HEADING_LINE_CHARS:-8}
|
|
|
|
|
|
# 清理 U+FFFE/软连字符、剥离矢量图轴文字、图表区按 JPEG 渲染
|
|
|
|
|
|
- DOCREADER_PDF_SANITIZE_TEXT=${DOCREADER_PDF_SANITIZE_TEXT:-true}
|
|
|
|
|
|
- DOCREADER_PDF_STRIP_CHART_DEBRIS=${DOCREADER_PDF_STRIP_CHART_DEBRIS:-true}
|
|
|
|
|
|
- DOCREADER_PDF_RENDER_VECTOR_FIGURES=${DOCREADER_PDF_RENDER_VECTOR_FIGURES:-true}
|
|
|
|
|
|
- DOCREADER_PDF_DETECT_HEADINGS=${DOCREADER_PDF_DETECT_HEADINGS:-}
|
|
|
|
|
|
- DOCREADER_PDF_LAYOUT_ORDERING=${DOCREADER_PDF_LAYOUT_ORDERING:-}
|
|
|
|
|
|
- DOCREADER_PDF_FILTER_HIDDEN_TEXT=${DOCREADER_PDF_FILTER_HIDDEN_TEXT:-}
|
|
|
|
|
|
# 扫描件判定:图片面积占比 >= 此值视为扫描页(默认留空走代码默认)
|
|
|
|
|
|
- DOCREADER_PDF_SCAN_IMAGE_RATIO=${DOCREADER_PDF_SCAN_IMAGE_RATIO:-}
|
|
|
|
|
|
- DOCREADER_PDF_SCAN_MIN_CHARS=${DOCREADER_PDF_SCAN_MIN_CHARS:-}
|
|
|
|
|
|
# 嵌入图片抽取
|
|
|
|
|
|
- DOCREADER_PDF_EXTRACT_EMBEDDED_IMAGES=${DOCREADER_PDF_EXTRACT_EMBEDDED_IMAGES:-}
|
|
|
|
|
|
- DOCREADER_PDF_EMBED_MIN_PIXELS=${DOCREADER_PDF_EMBED_MIN_PIXELS:-}
|
|
|
|
|
|
- DOCREADER_PDF_EMBED_MIN_AREA_RATIO=${DOCREADER_PDF_EMBED_MIN_AREA_RATIO:-}
|
|
|
|
|
|
- DOCREADER_PDF_EMBED_REPEAT_PAGE_FRAC=${DOCREADER_PDF_EMBED_REPEAT_PAGE_FRAC:-}
|
|
|
|
|
|
- DOCREADER_PDF_EMBED_MAX_IMAGES=${DOCREADER_PDF_EMBED_MAX_IMAGES:-}
|
|
|
|
|
|
# 图表区识别
|
|
|
|
|
|
- DOCREADER_PDF_MIN_CHART_REGION_AREA=${DOCREADER_PDF_MIN_CHART_REGION_AREA:-}
|
|
|
|
|
|
- DOCREADER_PDF_MIN_CHART_REGION_CHARS=${DOCREADER_PDF_MIN_CHART_REGION_CHARS:-}
|
|
|
|
|
|
- DOCREADER_PDF_MAX_CHART_REGION_AREA=${DOCREADER_PDF_MAX_CHART_REGION_AREA:-}
|
|
|
|
|
|
- DOCREADER_PDF_MAX_FIGURE_HEIGHT_RATIO=${DOCREADER_PDF_MAX_FIGURE_HEIGHT_RATIO:-}
|
2025-11-07 07:52:52 +00:00
|
|
|
|
healthcheck:
|
2026-04-16 06:09:36 +00:00
|
|
|
|
test: ["CMD", "grpc_health_probe", "-addr=localhost:50051"]
|
2025-11-07 07:52:52 +00:00
|
|
|
|
interval: 30s
|
|
|
|
|
|
timeout: 10s
|
|
|
|
|
|
retries: 3
|
|
|
|
|
|
start_period: 60s
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
extra_hosts:
|
|
|
|
|
|
- "host.docker.internal:host-gateway"
|
|
|
|
|
|
|
2026-06-03 04:00:09 +00:00
|
|
|
|
# OpenDataLoader hybrid backend (optional). Enable profile "odl-hybrid" and set
|
|
|
|
|
|
# DOCREADER_ODL_HYBRID=docling-fast on docreader. Default --no-ocr (no EasyOCR).
|
|
|
|
|
|
# Local build only — not published to Docker Hub.
|
|
|
|
|
|
odl-hybrid:
|
|
|
|
|
|
build:
|
|
|
|
|
|
context: .
|
|
|
|
|
|
dockerfile: docker/Dockerfile.odl-hybrid
|
|
|
|
|
|
image: weknora-odl-hybrid:local
|
|
|
|
|
|
container_name: WeKnora-odl-hybrid
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- odl-hybrid
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${ODL_HYBRID_PORT:-5002}:5002"
|
|
|
|
|
|
environment:
|
|
|
|
|
|
# Default --no-ocr (digital PDFs). Scanned PDFs: use builtin OCR / MinerU, or
|
|
|
|
|
|
# ODL_HYBRID_EXTRA_ARGS="--force-ocr" (needs EasyOCR + libGL in image).
|
|
|
|
|
|
- ODL_HYBRID_EXTRA_ARGS=${ODL_HYBRID_EXTRA_ARGS:---no-ocr}
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
|
2026-03-30 03:13:44 +00:00
|
|
|
|
dex:
|
|
|
|
|
|
image: dexidp/dex:latest
|
|
|
|
|
|
container_name: WeKnora-dex-dev
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "5556:5556"
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- ./misc/dex-config.yaml:/etc/dex/config.yaml
|
|
|
|
|
|
command: ["dex", "serve", "/etc/dex/config.yaml"]
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- dex
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-23 17:57:12 +00:00
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
# Langfuse 自建栈(dev 对称版)
|
|
|
|
|
|
#
|
|
|
|
|
|
# 用法:
|
|
|
|
|
|
# docker compose -f docker-compose.dev.yml --profile langfuse up -d
|
|
|
|
|
|
#
|
|
|
|
|
|
# 本地 app (go run) 需要的环境变量:
|
|
|
|
|
|
# export LANGFUSE_HOST=http://localhost:3000
|
|
|
|
|
|
# export LANGFUSE_PUBLIC_KEY=pk-lf-xxx
|
|
|
|
|
|
# export LANGFUSE_SECRET_KEY=sk-lf-xxx
|
|
|
|
|
|
#
|
|
|
|
|
|
# 复用 dev 已有的 postgres(独立 langfuse 数据库)和 redis(DB 1),
|
|
|
|
|
|
# 新增:clickhouse、minio、web、worker + 一次性 db-init,和生产版结构一致。
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
# 复用 dev 已有的 ParadeDB 镜像,不额外拉 postgres 镜像
|
|
|
|
|
|
langfuse-db-init:
|
fix(deploy): bump paradedb image to v0.25.11-pg17 (#4120)
* fix(deploy): bump paradedb image to v0.25.11-pg17
Batch-importing large knowledge bases on the pinned
paradedb/paradedb:v0.22.6-pg17 image drives PostgreSQL into a
PANIC/crash-recovery loop (corrupted page pointers in shared buffers,
amcheck clean, no OOM), taking every ingest task down with it.
Three independent fixes upstream cover this exact write path, and the
pinned image predates all of them:
- pgvector 0.8.2 fixed a buffer overflow in parallel HNSW index
builds, 0.8.3 fixed possible index corruption during HNSW vacuuming,
and 0.8.4 fixed inserts during HNSW vacuuming; the image ships
pgvector 0.8.1, and the embeddings table maintains three HNSW
halfvec indexes on import.
- pg_search gained WAL-based crash recovery in 0.24.0 (paradedb#4901):
without it, any backend crash corrupts BM25 segment metadata and
turns a single crash into a recovery loop (paradedb#4935).
- PostgreSQL core moved 17.9 -> 17.11; its release notes contain no
fix matching this failure signature, ruling the core out.
Bump the compose image tags to v0.25.11-pg17 (postgres and
langfuse-db-init, prod and dev) and update the docs that pin the old
tag. Fresh installs are covered by the existing schema SQL (verified
against 0.25.11: extensions, BM25 index and HNSW halfvec indexes all
create cleanly). Existing data volumes keep working across the image
swap without any action (keyword and vector queries byte-identical in
testing); ALTER EXTENSION pg_search/vector UPDATE aligns the SQL
catalog afterwards, and migration 000099 is a no-op for 0.25.x by
design. The Helm chart pins a different image line and is left alone.
Fixes #4077
* fix(migrations): upgrade pg_search catalog to 0.25.11 with the image
Swapping in the v0.25.11 image leaves the extension catalog at 0.22.x.
Calling extension functions such as paradedb.version_info() against the
mismatched catalog segfaults the backend and sends the whole server into
crash recovery, so the catalog upgrade is required rather than optional.
- Add migration 000115 to update pg_search 0.22.0-0.25.10 to 0.25.11 and
pgvector 0.8.0-0.8.3 to 0.8.4 when the server ships those versions.
- Skip 000099 on newer images instead of advising UPDATE TO '0.22.6',
which leaves the catalog behind the shared library and crashes.
- Rewrite the upgrade guide as one 0.22.x -> 0.25.11 path with a required
ALTER EXTENSION UPDATE and a note that rollback needs a backup.
- Extend scripts/test_paradedb_upgrade.py to verify 0.22.2 -> 0.25.11 by
default (--to 0.22.6 keeps the old path) and run CONCURRENTLY
migrations outside a transaction like golang-migrate does.
---------
Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-12 03:50:11 +00:00
|
|
|
|
image: paradedb/paradedb:v0.25.11-pg17
|
feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-23 17:57:12 +00:00
|
|
|
|
container_name: WeKnora-langfuse-db-init-dev
|
|
|
|
|
|
depends_on:
|
|
|
|
|
|
postgres:
|
|
|
|
|
|
condition: service_healthy
|
|
|
|
|
|
environment:
|
|
|
|
|
|
PGPASSWORD: ${DB_PASSWORD}
|
|
|
|
|
|
# ${LANGFUSE_DB_NAME:-langfuse} / ${DB_USER} 由 compose 解析成字面量后再传给 shell。
|
|
|
|
|
|
entrypoint: ["sh", "-c"]
|
|
|
|
|
|
command:
|
|
|
|
|
|
- |
|
|
|
|
|
|
set -e
|
|
|
|
|
|
echo "[langfuse-db-init] ensuring database '${LANGFUSE_DB_NAME:-langfuse}' exists..."
|
|
|
|
|
|
# 先刷新现有库的 collation(镜像 ICU 2.36 与宿主 2.41 不匹配时必须做),否则 CREATE DATABASE 会失败
|
|
|
|
|
|
psql -h postgres -U ${DB_USER} -d postgres -v ON_ERROR_STOP=0 -c "ALTER DATABASE template1 REFRESH COLLATION VERSION;" >/dev/null 2>&1 || true
|
|
|
|
|
|
psql -h postgres -U ${DB_USER} -d postgres -v ON_ERROR_STOP=0 -c "ALTER DATABASE postgres REFRESH COLLATION VERSION;" >/dev/null 2>&1 || true
|
|
|
|
|
|
# 幂等创建:已存在则跳过;不存在则从 template0 克隆(template0 永远不会有 collation 漂移)
|
|
|
|
|
|
if psql -h postgres -U ${DB_USER} -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname='${LANGFUSE_DB_NAME:-langfuse}'" | grep -q 1; then
|
|
|
|
|
|
echo "[langfuse-db-init] database '${LANGFUSE_DB_NAME:-langfuse}' already exists, skipping."
|
|
|
|
|
|
else
|
|
|
|
|
|
psql -h postgres -U ${DB_USER} -d postgres -v ON_ERROR_STOP=1 -c "CREATE DATABASE \"${LANGFUSE_DB_NAME:-langfuse}\" TEMPLATE template0;"
|
|
|
|
|
|
echo "[langfuse-db-init] database '${LANGFUSE_DB_NAME:-langfuse}' created."
|
|
|
|
|
|
fi
|
|
|
|
|
|
echo "[langfuse-db-init] done."
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
restart: "no"
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- langfuse
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
|
|
|
|
|
langfuse-clickhouse:
|
|
|
|
|
|
image: clickhouse/clickhouse-server:24.8
|
|
|
|
|
|
container_name: WeKnora-langfuse-clickhouse-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
user: "101:101"
|
|
|
|
|
|
environment:
|
|
|
|
|
|
CLICKHOUSE_DB: default
|
|
|
|
|
|
CLICKHOUSE_USER: ${LANGFUSE_CLICKHOUSE_USER:-clickhouse}
|
|
|
|
|
|
CLICKHOUSE_PASSWORD: ${LANGFUSE_CLICKHOUSE_PASSWORD:-clickhouse}
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- langfuse_clickhouse_data_dev:/var/lib/clickhouse
|
|
|
|
|
|
- langfuse_clickhouse_logs_dev:/var/log/clickhouse-server
|
|
|
|
|
|
healthcheck:
|
|
|
|
|
|
test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
|
|
|
|
|
|
interval: 5s
|
|
|
|
|
|
timeout: 5s
|
|
|
|
|
|
retries: 10
|
|
|
|
|
|
start_period: 10s
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- langfuse
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
|
|
|
|
|
langfuse-minio:
|
2026-09-26 07:26:09 +00:00
|
|
|
|
image: pgsty/minio:RELEASE.2026-08-04T00-00-00Z
|
feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-23 17:57:12 +00:00
|
|
|
|
container_name: WeKnora-langfuse-minio-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
entrypoint: sh
|
|
|
|
|
|
command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
|
|
|
|
|
|
environment:
|
|
|
|
|
|
MINIO_ROOT_USER: ${LANGFUSE_MINIO_USER:-langfuseminio}
|
|
|
|
|
|
MINIO_ROOT_PASSWORD: ${LANGFUSE_MINIO_PASSWORD:-langfuseminiosecret}
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${LANGFUSE_MINIO_S3_PORT:-9100}:9000"
|
|
|
|
|
|
- "${LANGFUSE_MINIO_CONSOLE_PORT:-9101}:9001"
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
- langfuse_minio_data_dev:/data
|
|
|
|
|
|
healthcheck:
|
|
|
|
|
|
test: ["CMD", "mc", "ready", "local"]
|
|
|
|
|
|
interval: 5s
|
|
|
|
|
|
timeout: 10s
|
|
|
|
|
|
retries: 5
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- langfuse
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
|
|
|
|
|
langfuse-worker:
|
|
|
|
|
|
image: langfuse/langfuse-worker:3
|
|
|
|
|
|
container_name: WeKnora-langfuse-worker-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
depends_on: &langfuse-dev-depends-on
|
|
|
|
|
|
langfuse-db-init:
|
|
|
|
|
|
condition: service_completed_successfully
|
|
|
|
|
|
redis:
|
|
|
|
|
|
condition: service_started
|
|
|
|
|
|
langfuse-clickhouse:
|
|
|
|
|
|
condition: service_healthy
|
|
|
|
|
|
langfuse-minio:
|
|
|
|
|
|
condition: service_healthy
|
|
|
|
|
|
# wrapper entrypoint 把 DB_PASSWORD / REDIS_PASSWORD URL 编码后再拼 URL,
|
|
|
|
|
|
# 避免密码含 '@' / '#' 等字符导致 Prisma P1013 解析失败。
|
|
|
|
|
|
# 注意:compose 覆盖 entrypoint 会清空镜像默认 CMD,所以末尾写死原始命令。
|
|
|
|
|
|
entrypoint:
|
|
|
|
|
|
- /bin/sh
|
|
|
|
|
|
- -ec
|
|
|
|
|
|
- |
|
|
|
|
|
|
_enc() { node -e 'process.stdout.write(encodeURIComponent(process.argv[1]))' "$$1"; }
|
|
|
|
|
|
DU=$$(_enc "$$_LF_DB_USER")
|
|
|
|
|
|
DP=$$(_enc "$$_LF_DB_PASSWORD")
|
|
|
|
|
|
RP=$$(_enc "$$_LF_REDIS_PASSWORD")
|
|
|
|
|
|
export DATABASE_URL="postgresql://$$DU:$$DP@postgres:5432/$$_LF_DB_NAME"
|
|
|
|
|
|
export REDIS_CONNECTION_STRING="redis://:$$RP@redis:6379/$$_LF_REDIS_DB"
|
|
|
|
|
|
unset _LF_DB_USER _LF_DB_PASSWORD _LF_REDIS_PASSWORD
|
|
|
|
|
|
exec dumb-init -- ./worker/entrypoint.sh node worker/dist/index.js
|
|
|
|
|
|
environment: &langfuse-dev-env
|
|
|
|
|
|
# 原始凭证(未 URL 编码),由 entrypoint wrapper 读取并组装
|
|
|
|
|
|
_LF_DB_USER: ${DB_USER}
|
|
|
|
|
|
_LF_DB_PASSWORD: ${DB_PASSWORD}
|
|
|
|
|
|
_LF_DB_NAME: ${LANGFUSE_DB_NAME:-langfuse}
|
|
|
|
|
|
_LF_REDIS_PASSWORD: ${REDIS_PASSWORD}
|
|
|
|
|
|
_LF_REDIS_DB: ${LANGFUSE_REDIS_DB:-1}
|
|
|
|
|
|
SALT: ${LANGFUSE_SALT:-weknora-langfuse-dev-salt-change-me}
|
|
|
|
|
|
ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY:-0000000000000000000000000000000000000000000000000000000000000000}
|
|
|
|
|
|
NEXTAUTH_URL: ${LANGFUSE_NEXTAUTH_URL:-http://localhost:3000}
|
|
|
|
|
|
NEXTAUTH_SECRET: ${LANGFUSE_NEXTAUTH_SECRET:-weknora-langfuse-dev-nextauth-secret-change-me}
|
|
|
|
|
|
TELEMETRY_ENABLED: ${LANGFUSE_TELEMETRY_ENABLED:-false}
|
|
|
|
|
|
LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: "false"
|
|
|
|
|
|
CLICKHOUSE_URL: http://langfuse-clickhouse:8123
|
|
|
|
|
|
CLICKHOUSE_MIGRATION_URL: clickhouse://langfuse-clickhouse:9000
|
|
|
|
|
|
CLICKHOUSE_USER: ${LANGFUSE_CLICKHOUSE_USER:-clickhouse}
|
|
|
|
|
|
CLICKHOUSE_PASSWORD: ${LANGFUSE_CLICKHOUSE_PASSWORD:-clickhouse}
|
|
|
|
|
|
CLICKHOUSE_CLUSTER_ENABLED: "false"
|
|
|
|
|
|
LANGFUSE_S3_EVENT_UPLOAD_BUCKET: langfuse
|
|
|
|
|
|
LANGFUSE_S3_EVENT_UPLOAD_REGION: auto
|
|
|
|
|
|
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_MINIO_USER:-langfuseminio}
|
|
|
|
|
|
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_MINIO_PASSWORD:-langfuseminiosecret}
|
|
|
|
|
|
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: http://langfuse-minio:9000
|
|
|
|
|
|
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: "true"
|
|
|
|
|
|
LANGFUSE_S3_EVENT_UPLOAD_PREFIX: events/
|
|
|
|
|
|
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: langfuse
|
|
|
|
|
|
LANGFUSE_S3_MEDIA_UPLOAD_REGION: auto
|
|
|
|
|
|
LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_MINIO_USER:-langfuseminio}
|
|
|
|
|
|
LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_MINIO_PASSWORD:-langfuseminiosecret}
|
|
|
|
|
|
LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: ${LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT:-http://localhost:9100}
|
|
|
|
|
|
LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: "true"
|
|
|
|
|
|
LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: media/
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- langfuse
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
|
|
|
|
|
langfuse-web:
|
|
|
|
|
|
image: langfuse/langfuse:3
|
|
|
|
|
|
container_name: WeKnora-langfuse-web-dev
|
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
depends_on: *langfuse-dev-depends-on
|
|
|
|
|
|
ports:
|
|
|
|
|
|
- "${LANGFUSE_WEB_PORT:-3000}:3000"
|
|
|
|
|
|
entrypoint:
|
|
|
|
|
|
- /bin/sh
|
|
|
|
|
|
- -ec
|
|
|
|
|
|
- |
|
|
|
|
|
|
_enc() { node -e 'process.stdout.write(encodeURIComponent(process.argv[1]))' "$$1"; }
|
|
|
|
|
|
DU=$$(_enc "$$_LF_DB_USER")
|
|
|
|
|
|
DP=$$(_enc "$$_LF_DB_PASSWORD")
|
|
|
|
|
|
RP=$$(_enc "$$_LF_REDIS_PASSWORD")
|
|
|
|
|
|
export DATABASE_URL="postgresql://$$DU:$$DP@postgres:5432/$$_LF_DB_NAME"
|
|
|
|
|
|
export REDIS_CONNECTION_STRING="redis://:$$RP@redis:6379/$$_LF_REDIS_DB"
|
|
|
|
|
|
unset _LF_DB_USER _LF_DB_PASSWORD _LF_REDIS_PASSWORD
|
|
|
|
|
|
if [ -n "$$NEXT_PUBLIC_LANGFUSE_CLOUD_REGION" ]; then
|
|
|
|
|
|
exec dumb-init -- ./web/entrypoint.sh node --import dd-trace/initialize.mjs ./web/server.js --keepAliveTimeout 110000
|
|
|
|
|
|
else
|
|
|
|
|
|
exec dumb-init -- ./web/entrypoint.sh node ./web/server.js --keepAliveTimeout 110000
|
|
|
|
|
|
fi
|
|
|
|
|
|
environment:
|
|
|
|
|
|
<<: *langfuse-dev-env
|
|
|
|
|
|
LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-}
|
|
|
|
|
|
LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-}
|
|
|
|
|
|
LANGFUSE_INIT_PROJECT_ID: ${LANGFUSE_INIT_PROJECT_ID:-}
|
|
|
|
|
|
LANGFUSE_INIT_PROJECT_NAME: ${LANGFUSE_INIT_PROJECT_NAME:-}
|
|
|
|
|
|
LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY:-}
|
|
|
|
|
|
LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LANGFUSE_INIT_PROJECT_SECRET_KEY:-}
|
|
|
|
|
|
LANGFUSE_INIT_USER_EMAIL: ${LANGFUSE_INIT_USER_EMAIL:-}
|
|
|
|
|
|
LANGFUSE_INIT_USER_NAME: ${LANGFUSE_INIT_USER_NAME:-}
|
|
|
|
|
|
LANGFUSE_INIT_USER_PASSWORD: ${LANGFUSE_INIT_USER_PASSWORD:-}
|
|
|
|
|
|
networks:
|
|
|
|
|
|
- WeKnora-network-dev
|
|
|
|
|
|
profiles:
|
|
|
|
|
|
- langfuse
|
|
|
|
|
|
- full
|
|
|
|
|
|
|
2026-03-30 03:13:44 +00:00
|
|
|
|
|
2025-11-07 07:52:52 +00:00
|
|
|
|
networks:
|
|
|
|
|
|
WeKnora-network-dev:
|
|
|
|
|
|
driver: bridge
|
|
|
|
|
|
|
|
|
|
|
|
volumes:
|
|
|
|
|
|
postgres-data-dev:
|
|
|
|
|
|
redis_data_dev:
|
|
|
|
|
|
minio_data_dev:
|
|
|
|
|
|
neo4j-data-dev:
|
2025-12-08 11:32:46 +00:00
|
|
|
|
qdrant_data_dev:
|
feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3)
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482)
laid the type prep + driver skeleton + read/write paths as gated dead
code; this PR wires every activation surface so opensearch becomes a
registerable VectorStore engine.
Activation wiring
- internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW
bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping /
buildEnvStoreForDriver — every gated surface now recognises
"opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM /
HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines'
serialised config byte-identical.
- internal/container: createOpenSearchEngine + the switch case in
createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path
in initRetrieveEngineRegistry; NewEngineFactory now closes over the
AuditLogService (the EngineFactory type itself is unchanged).
- internal/application/service/vectorstore_healthcheck.go: a
testOpenSearchConnection case so CreateStore's connectivity probe
accepts opensearch instead of returning 400.
- internal/application/repository/retriever/opensearch/transport.go:
NewOpenSearchClient is exported so the factory and env path can build
the TLS-hardened client; healthcheck.go reuses the unexported
probeVersion / probeKNNPlugin for the service-layer probe.
Service-layer validation
- validateOpenSearchIndexConfig validates the HNSW caps (m 2-100,
ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss).
Shards/replicas continue to be enforced by the flat ValidateIndexConfig.
Create-only: UpdateStore mutates the name only.
- validateConnectionConfig requires addr for opensearch.
Sync implementations (stubs.go shrinks)
- CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern —
search → BatchSave with the source_id remap for generated questions —
so dim/keyword routing and the source_id contract come from BatchSave
for free. embeddingMap is keyed by the *target* SourceID because
OpenSearch's BatchSave looks up embeddings by SourceID
(lookupEmbedding), not by chunk_id (the ES driver's convention).
Pagination is from/size; copies larger than max_result_window
(default 10000) need the scroll-based async path that lands later.
- BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go)
group the input by target value and issue one _update_by_query per
group over the cross-dim <base>_* pattern. Caller values flow through
bound script params only — never string-interpolated into the Painless
source — closing the script-injection surface.
- inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the
full failure reason goes to the debug log only; the returned error
carries the bounded id + type.
- UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same
shape as DeleteByQuery's quirk), so refresh=wait_for is not
expressible; we use refresh=true.
Driver-owned audit (DIP)
- A new opensearch.AuditSink interface (with nopSink + WithAuditSink
functional option) lets the driver emit opensearch.index_created and
opensearch.reindex_executed events without importing any service
package — the service layer implements the interface. NewRepository
takes opts, so existing 4-arg test call sites keep compiling unchanged.
- internal/container/audit_sink.go bridges AuditSink to AuditLogService.
When the context carries no tenant (the env-path registration ctx
during boot, for example) the adapter skips the emit with a warning
rather than silently writing tenant_id=0, which would collide with the
system-scope sentinel.
Frontend + polish
- FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/
immutable. VectorStoreSettings.vue is now schema-driven: a closed
`enum` renders a t-select; number inputs use the schema's `:min`/`:max`
and fall back to the legacy replica-vs-shard heuristic only when the
schema does not pin them; a danger-coloured warning fires when
insecure_skip_verify is toggled on (the switch and warning are wrapped
in a vertical stack so the warning sits on its own row below the switch).
- i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search /
knn_engine / insecure_skip_verify plus the warning copy in en-US,
ko-KR, zh-CN, ru-RU.
- docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with
security plugin disabled for dev only). OpenSearch Dashboards lives in a
separate, opt-in opensearch-ui profile so the heavy UI container is not
forced up alongside the cluster (the driver e2e is fully curl-verifiable
against :9200). The new docs/dev/opensearch-integration-test.md covers the
end-to-end exercise and the single-node guidance (set replicas=0 to keep
the cluster Green).
Gating-guard tests flipped
- The "OpenSearch is NOT in validEngineTypes / mapping / types list /
env builder / stubs" guard tests from PR 1 / PR 2 are replaced by
their positive counterparts in this PR. The test suite was the
activation checklist; the activation flip is its diff.
Backward compatibility
- Additive everywhere. IndexConfig's new HNSW fields are omitempty so
other engines' serialised config is byte-identical. Existing
Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB
stores are untouched. No migrations.
Test plan
- go build ./... clean
- go vet ./... clean
- gofmt -l clean on touched files
- go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS
endpoint), the docreader gRPC tests, and the doris SQL-shape tests
fail; all three are pre-existing on upstream/main and untouched by
this PR.
- New tests across internal/types, opensearch, service and container —
including a full end-to-end env-path test that exercises
initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an
httptest cluster.
2026-05-28 09:01:17 +00:00
|
|
|
|
opensearch_data_dev:
|
2026-03-19 14:38:35 +00:00
|
|
|
|
milvus_data_dev:
|
2026-02-28 05:22:07 +00:00
|
|
|
|
docreader-tmp-dev:
|
feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-23 17:57:12 +00:00
|
|
|
|
langfuse_clickhouse_data_dev:
|
|
|
|
|
|
langfuse_clickhouse_logs_dev:
|
|
|
|
|
|
langfuse_minio_data_dev:
|
2026-05-10 14:51:24 +00:00
|
|
|
|
searxng_config_dev:
|