WeKnora/internal/handler/chunker_debug.go

273 lines
9.8 KiB
Go
Raw Normal View History

// Package handler — chunker_debug.go exposes a read-only preview endpoint
// that runs the adaptive chunker on supplied text without touching the DB
// or generating embeddings. Used by the KB editor's debug panel so users
// can experiment with chunking parameters before committing to a re-index.
package handler
import (
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
"context"
"math"
"net/http"
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
"strings"
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
"time"
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
"unicode/utf8"
"github.com/Tencent/WeKnora/internal/infrastructure/chunker"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/gin-gonic/gin"
)
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
// previewMaxChars caps the input text size so a single preview request
// cannot tie up the splitter for long. Chosen to bound worst-case CPU
// well under the previewTimeout: at 64k runes even the heaviest tier
// chain finishes in well under a second on commodity hardware.
//
// SECURITY NOTE: the splitter is CPU-bound and does NOT accept a
// context.Context. When previewTimeout fires the handler returns to the
// caller, but the worker goroutine keeps running until the splitter
// finishes naturally. The 64k ceiling is the primary mitigation against
// goroutine pile-up under repeated authenticated requests. If the
// splitter ever gains context-awareness, the goroutine-wrapper in
// PreviewChunking should switch to it for true cancellation.
const previewMaxChars = 64 * 1024
// previewMaxChunks caps the number of chunks returned in a single preview
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
// response so the UI doesn't choke on pathological splits. Stats are
// computed over the full chunk set before truncation so the displayed
// avg/min/max/stddev stay representative.
const previewMaxChunks = 500
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
// previewTimeout caps how long the handler waits for the splitter
// goroutine before returning a 504. See note above on previewMaxChars.
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
const previewTimeout = 5 * time.Second
// PreviewChunkingRequest is the body shape accepted by /chunker/preview.
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
// Text is checked manually below so we can return a friendlier error than
// gin's default "Field validation for 'Text' failed on the 'required' tag".
type PreviewChunkingRequest struct {
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
Text string `json:"text"`
ChunkingConfig PreviewChunkingPayload `json:"chunking_config"`
}
// PreviewChunkingPayload mirrors the snake_case JSON the rest of the API
// uses for ChunkingConfig fields. We don't reuse types.ChunkingConfig
// directly because it carries a lot of unrelated fields (parser engine
// rules, parent-child sizes, etc.) that the preview path doesn't need.
type PreviewChunkingPayload struct {
ChunkSize int `json:"chunk_size"`
ChunkOverlap int `json:"chunk_overlap"`
Separators []string `json:"separators"`
Strategy string `json:"strategy"`
TokenLimit int `json:"token_limit"`
Languages []string `json:"languages"`
}
// PreviewChunkResult describes one chunk emitted during preview.
type PreviewChunkResult struct {
Seq int `json:"seq"`
Start int `json:"start"`
End int `json:"end"`
SizeChars int `json:"size_chars"`
SizeTokensApprox int `json:"size_tokens_approx"`
ContextHeader string `json:"context_header,omitempty"`
Content string `json:"content"`
}
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
// PreviewChunkingStats summarizes chunk-size distribution. Computed over
// the FULL chunk set, even when the response truncates to previewMaxChunks
// items, so avg/min/max/stddev reflect the real distribution, not just
// the first N.
type PreviewChunkingStats struct {
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
Count int `json:"count"`
AvgChars int `json:"avg_chars"`
MinChars int `json:"min_chars"`
MaxChars int `json:"max_chars"`
StddevChars int `json:"stddev_chars"`
// TruncatedTo, when set, is the original chunk count before the
// response was truncated to previewMaxChunks for transport.
TruncatedTo int `json:"truncated_to,omitempty"`
}
// PreviewChunkingResponse is the body returned by /chunker/preview.
type PreviewChunkingResponse struct {
SelectedTier chunker.StrategyTier `json:"selected_tier"`
TierChain []chunker.StrategyTier `json:"tier_chain"`
Rejected []chunker.TierRejection `json:"rejected"`
Profile *chunker.DocProfile `json:"profile"`
Chunks []PreviewChunkResult `json:"chunks"`
Stats PreviewChunkingStats `json:"stats"`
}
// PreviewChunking handles POST /chunker/preview. It runs the supplied text
// through the adaptive chunker and returns the chunks plus diagnostic
// information about which tier won. Read-only: no DB writes, no embedding
// calls, no logging of the supplied text.
docs(api): add and fix swag annotations on handlers 将 swag 注解覆盖率从 ~88% 提升到 100%,并修复审计中发现的若干既有 注解 bug。 新增注解(24 个): - im.go: 4 (UpdateIMChannel/DeleteIMChannel/ToggleIMChannel/IMCallback) 其中 IMCallback 同时注册 GET 和 POST,两个方法都加了 @Router 声明。 - web_search.go: 1 (GetProviders) - web_search_provider.go: 6 - wechat_qrcode.go: 2 - weknoracloud.go: 2 - knowledge.go: 2 (MoveKnowledge/GetKnowledgeMoveProgress) - knowledgebase.go: 1 (ListMoveTargets) - organization.go: 3 (ListSharedAgents/ListOrgAgentShares/RemoveAgentShare) - mcp_service.go: 2 (SetMCPToolApproval/ResolveToolApproval) - chunker_debug.go: 1 (PreviewChunking) 修复既有注解 bug(12 处): - initialization.go: 10 个 @Router 路径与真实路由不符 (e.g. /initialization/kb/{kbId}/config → /initialization/config/{kbId} /initialization/fabri/tag GET → /initialization/extract/fabri-tag POST) - session/stream.go: ContinueStream 路径 /sessions/{session_id}/continue → /sessions/continue-stream/{session_id} - mcp_service.go: ResolveToolApproval 请求体 {approve: bool} → {decision: "approve"|"reject", reason?, modified_args?} 收敛响应 schema 到已有命名 struct(之前用 map[string]interface{}): - knowledge.go MoveKnowledge → handler.MoveKnowledgeResponse - knowledge.go GetKnowledgeMoveProgress → types.KnowledgeMoveProgress - web_search_provider.go GetProvider/UpdateProvider → types.WebSearchProviderEntity - initialization.go InitializeByKB → handler.InitializationRequest - initialization.go 三个 model-test endpoint → handler.ModelTestRequest 修正 organization.go 中 errors.AppError 应使用 apperrors 别名(该文件 stdlib errors 未别名化)。 不补 /files 和 /api/v1/files/presigned 内联闭包路由(文件服务,非业务 REST)。 Refs #890 #1049 #1168
2026-05-11 09:37:14 +00:00
//
// PreviewChunking godoc
// @Summary 预览分块结果
// @Description 对提交的文本运行自适应分块器并返回分块预览,不写入数据库不生成 embedding。文本最大 64k 字符
// @Tags 分块
// @Accept json
// @Produce json
// @Param request body handler.PreviewChunkingRequest true "{text, chunking_config}"
// @Success 200 {object} handler.PreviewChunkingResponse "分块结果"
// @Failure 400 {object} map[string]interface{} "请求参数错误"
// @Failure 413 {object} map[string]interface{} "文本超过预览限制"
// @Failure 504 {object} map[string]interface{} "分块超时"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /chunker/preview [post]
func PreviewChunking(c *gin.Context) {
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
ctx, cancel := context.WithTimeout(c.Request.Context(), previewTimeout)
defer cancel()
var req PreviewChunkingRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"success": false, "error": "invalid request body: " + err.Error()})
return
}
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
if strings.TrimSpace(req.Text) == "" {
c.JSON(http.StatusBadRequest, gin.H{
"success": false,
"error": "text is empty — paste a sample to preview chunking",
})
return
}
if utf8.RuneCountInString(req.Text) > previewMaxChars {
c.JSON(http.StatusRequestEntityTooLarge, gin.H{
"success": false,
"error": "text exceeds preview limit",
"limit": previewMaxChars,
})
return
}
cfg := chunker.SplitterConfig{
ChunkSize: req.ChunkingConfig.ChunkSize,
ChunkOverlap: req.ChunkingConfig.ChunkOverlap,
Separators: req.ChunkingConfig.Separators,
Strategy: req.ChunkingConfig.Strategy,
TokenLimit: req.ChunkingConfig.TokenLimit,
Languages: req.ChunkingConfig.Languages,
}
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
// Run the splitter on a goroutine so we can honor the request timeout.
// The splitter is CPU-bound and doesn't accept a context — wrapping
// here is the cheapest cancellation we can offer.
type splitResult struct {
chunks []chunker.Chunk
diag *chunker.Diagnostics
}
resCh := make(chan splitResult, 1)
go func() {
chunks, diag := chunker.SplitWithDiagnostics(req.Text, cfg)
resCh <- splitResult{chunks: chunks, diag: diag}
}()
var sr splitResult
select {
case sr = <-resCh:
case <-ctx.Done():
c.JSON(http.StatusGatewayTimeout, gin.H{
"success": false,
"error": "chunker preview timed out",
})
return
}
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
chunks, diag := sr.chunks, sr.diag
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
// Diagnostics carries the profile when auto-strategy ran; for explicit
// strategies the profile is nil and we materialize it here so the UI
// can still show document stats. Avoids the previous double-pass.
profile := diag.Profile
if profile == nil {
profile = chunker.ProfileDocument(req.Text)
}
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
logger.Debugf(ctx, "chunker preview: tier=%s chunks=%d", diag.SelectedTier, len(chunks))
lang := chunker.LangMixed
if len(profile.DetectedLangs) > 0 {
lang = profile.DetectedLangs[0]
}
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
// Compute rune lengths once per chunk; reused for stats and result
// payload below. Avoids the previous triple-pass over each chunk's
// content (stats + result + ApproxTokenCount each rune-counted).
runeLens := make([]int, len(chunks))
for i, ch := range chunks {
runeLens[i] = utf8.RuneCountInString(ch.Content)
}
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
// Compute stats over the FULL chunk set first so the metrics stay
// representative even when we trim the response to previewMaxChunks.
totalCount := len(chunks)
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
stats := computeChunkSizeStats(runeLens)
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
if totalCount > previewMaxChunks {
stats.TruncatedTo = totalCount
chunks = chunks[:previewMaxChunks]
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
runeLens = runeLens[:previewMaxChunks]
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
}
results := make([]PreviewChunkResult, 0, len(chunks))
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
for i, ch := range chunks {
results = append(results, PreviewChunkResult{
Seq: ch.Seq,
Start: ch.Start,
End: ch.End,
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
SizeChars: runeLens[i],
SizeTokensApprox: chunker.ApproxTokenCountFromRuneLen(runeLens[i], lang),
ContextHeader: ch.ContextHeader,
Content: ch.Content,
})
}
resp := PreviewChunkingResponse{
SelectedTier: diag.SelectedTier,
TierChain: diag.TierChain,
Rejected: diag.Rejected,
Profile: profile,
Chunks: results,
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
Stats: stats,
}
c.JSON(http.StatusOK, gin.H{"success": true, "data": resp})
}
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
// computeChunkSizeStats summarizes count / avg / min / max / stddev from
// a pre-computed rune-length slice. Decoupling from chunker.Chunk lets
// the caller compute rune lengths once and reuse them for the response
// payload (avoids a second []rune allocation per chunk).
func computeChunkSizeStats(runeLens []int) PreviewChunkingStats {
stats := PreviewChunkingStats{Count: len(runeLens)}
if len(runeLens) == 0 {
return stats
}
var sum, sumSq float64
minLen, maxLen := math.MaxInt32, 0
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
for _, l := range runeLens {
sum += float64(l)
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
sumSq += float64(l) * float64(l)
if l < minLen {
minLen = l
}
if l > maxLen {
maxLen = l
}
}
fix(chunker): post-review fixes for preview endpoint robustness Resolves issues from the review of be326aa..119f5e4. Each fix has a regression test attached and overclaimed findings (parserEngineRules defensive copy, runTier dead-code branch, init-vars architecture) were intentionally not touched after re-evaluation. Goroutine leak mitigation - previewMaxChars dropped from 256k to 64k runes. The splitter does not accept a context.Context, so when previewTimeout fires the worker keeps running. Bounding input size keeps worst-case CPU per request well under a second on commodity hardware. Sized so 10 concurrent timeouts don't pile up faster than they finish. - Frontend MAX_CHARS lowered to match. - Comment in handler explains the trade-off and points at the follow-up: real cancellation needs the splitter to take a ctx. Performance - ApproxTokenCountFromRuneLen variant lets the preview handler reuse a single rune-count per chunk for stats + size + token estimation. Eliminates the previous triple []rune allocation per chunk in the response loop. - computeChunkSizeStats now takes []int (pre-computed rune lens) instead of []chunker.Chunk; sumSq computed in float64 to avoid the int*int overflow at l > ~46k. Correctness / UX - Preview panel sends strategy / token_limit / languages unconditionally, mirroring the buildSubmitData convention so the preview faithfully reflects what would happen on save. - Empty-text returns a friendly 400 ("paste a sample…") instead of gin's cryptic 'Field validation failed on the required tag'. Tests - TestSplit_DelegatesToSplitWithDiagnostics renamed to TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor made the original name a misnomer; the test still asserts the right invariant under the new name). - New TestSplitWithDiagnostics_ProfileSetForAuto and TestSplitWithDiagnostics_ProfileNilForExplicit lock in the profile-reuse contract that the preview endpoint depends on. - New chunker_debug_test.go covers computeChunkSizeStats edge cases (empty / single / varying / no-variance underflow) plus PreviewChunking httptest scenarios (auto path, legacy strategy, empty-text rejection, oversize rejection, chunk truncation with full-set stats). Doc cleanup - runTier comment updated; the "stubbed in this scaffold" line was obsolete since the heading and heuristic splitters shipped. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:16:47 +00:00
avg := sum / float64(len(runeLens))
variance := sumSq/float64(len(runeLens)) - avg*avg
if variance < 0 {
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-03 07:01:20 +00:00
// Float precision can push the variance slightly below zero on
// near-uniform inputs; clamp so sqrt doesn't return NaN.
variance = 0
}
stats.AvgChars = int(avg + 0.5)
stats.MinChars = minLen
stats.MaxChars = maxLen
stats.StddevChars = int(math.Sqrt(variance) + 0.5)
return stats
}