diff --git a/AGENTS.md b/AGENTS.md index c14abd8d..b0a1731a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,6 +17,7 @@ alwaysApply: true - [ZA1a-c] Zero Tolerance Policy (zero assumptions, validation workflow, forbidden practices) - [GT1a-l] Git, history safety, hooks/signing, lock files, and clean commits - [CC1a-d] Clean Code & DDD (Mandatory) +- [SS1a-j] Single Semantic Owner and Java API catalog ownership - [ID1a-d] Idiomatic Patterns & Defaults - [EV1a-f] Secrets and Env Vars (no secrets in properties; OpenAI/Qdrant via .env/env) - [RC1a-f] Root Cause Resolution (single implementation, no fallbacks, no shims/workarounds) @@ -78,6 +79,8 @@ alwaysApply: true - [SS1f] **Stop-Work Trigger**: If an implementation requires listing the same governed fields/keys/rules in a second place, stop and redesign before editing. - [SS1g] **No Positional-Null Sludge**: Constructor/factory calls with repeated placeholder nulls or low-legibility optional argument trains are prohibited; use named factories/builders, parameter objects, or bind/import the canonical owner. - [SS1h] **Whole-List Smell**: If a file "knows the whole list" of a governed concept and it is not the canonical owner, the design is presumed wrong and must be reduced or removed. +- [SS1i] **Java API Catalog Owner**: Define complete Java API mirror records only in `src/main/resources/java-api-documentation-sources.manifest`; consume that manifest from Java, shell, tests, and documentation. +- [SS1j] **Java API Catalog Projection**: Never recreate Java API releases, URLs, mirror paths, display names, or fetch parameters in properties, constants, arrays, fixtures, or documentation; verify projections with `--list-java-api-sources` and `JavaApiDocumentationSourceParityTest`. ## [ID1] Idiomatic Patterns & Defaults diff --git a/docs/configuration.md b/docs/configuration.md index 2b889dc5..3dfaf6ab 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -27,8 +27,7 @@ Common variables: - `OPENAI_BASE_URL` (default `https://api.openai.com/v1`) - `OPENAI_MODEL` (application fallback default `gpt-5.2`; shared-gateway alias `gemma-4-26b-a4b`) - `OPENAI_REASONING_EFFORT` (optional, GPT‑5 family) -- `OPENAI_STREAMING_REQUEST_TIMEOUT_SECONDS` (default `600`) -- `OPENAI_STREAMING_READ_TIMEOUT_SECONDS` (default `75`) +- `OPENAI_STREAMING_REQUEST_TIMEOUT_SECONDS` (default `600`; bounds the complete SDK call while provider gateways own first-output and inter-output deadlines) ### Researchly shared gateway diff --git a/docs/contracts/code-change.md b/docs/contracts/code-change.md index 3d328adf..1a1a5063 100644 --- a/docs/contracts/code-change.md +++ b/docs/contracts/code-change.md @@ -6,7 +6,7 @@ description: "Evergreen contract for change decisions (new file vs edit), reposi # Code Change Policy Contract -See `AGENTS.md` ([FS1a-k], [FS1l], [AR1a-f], [ND1a-c], [CC1a-d], [AB1a-d], [RC1a-f], [CS1a-h]). +See `AGENTS.md` ([SS1a-j], [FS1a-k], [FS1l], [AR1a-f], [ND1a-c], [CC1a-d], [AB1a-d], [RC1a-f], [CS1a-h]). ## Non-negotiables (applies to every change) @@ -28,6 +28,7 @@ Use this as a hard rule, not a suggestion. | Logic change in stable code | Extract/replace via composition; keep stable code stable ([AB1c], [CC1a]) | Add flags, shims, or “compat” paths to hide uncertainty ([RC1b]) | | Touching a large/overloaded file | Extract at least one seam (new type + typed contract) ([FS1f], [FS1b]) | Grow the file further ([FS1f]) | | Reuse needed across features | Add a domain value object / explicit port / explicit service with intent-revealing name ([AB1b]) | Add `*Utils/*Helper/*Common/*Base*` grab bags ([FS1e]) | +| Governed catalog change | Edit its single canonical owner and project it everywhere else ([SS1a-c]) | Restate the inventory in code, scripts, tests, fixtures, or docs ([SS1b], [SS1h]) | ### When adding a method is allowed diff --git a/docs/domains/all-parsing-and-markdown-logic.md b/docs/domains/all-parsing-and-markdown-logic.md index 9d5a59bf..18bd889e 100644 --- a/docs/domains/all-parsing-and-markdown-logic.md +++ b/docs/domains/all-parsing-and-markdown-logic.md @@ -41,7 +41,7 @@ This document provides a comprehensive analysis of all parsing and markdown proc │ ├── ChatController.stream() → SSE events │ ├── normalizeDelta() - token joining/cleanup │ ├── UnifiedMarkdownService.process() - final markdown processing -│ └── ChatMemoryService.addAssistant() - persistence +│ └── ChatMemoryService.addExchange() - atomic user/assistant persistence │ └── 🔄 KEY TRANSITIONS & ISSUES ├── Regex → AST migration (incomplete) diff --git a/docs/ingestion.md b/docs/ingestion.md index 98b5c6d8..40c1cf76 100644 --- a/docs/ingestion.md +++ b/docs/ingestion.md @@ -23,11 +23,9 @@ Fetch all configured sources: make fetch-all ``` -This runs `scripts/fetch_all_docs.sh` (requires `wget`). Source URLs live in: - -- `src/main/resources/docs-sources.properties` - -See [pipeline-commands.md](pipeline-commands.md#scrape-fetch-html-mirrors) for flags (`--force`, `--include-quick`, `--no-clean`). +This runs `scripts/fetch_all_docs.sh` (requires `wget`). See the canonical source ownership and edit +workflow in [pipeline-commands.md](pipeline-commands.md#scrape-fetch-html-mirrors), along with flags +(`--force`, `--include-quick`, `--no-clean`). ## Process + upload to Qdrant @@ -46,14 +44,14 @@ This runs `scripts/process_all_to_qdrant.sh`, which: ### Doc set filtering (CLI) -Limit ingestion to specific doc sets: +List canonical complete Java API paths, then limit ingestion to a selected doc set: ```bash -DOCS_SETS=java25-complete make process-doc-sets -./scripts/process_all_to_qdrant.sh --doc-sets=java25-complete,spring-boot-complete +./scripts/fetch_all_docs.sh --list-java-api-sources +DOCS_SETS=relative/path/from/listing make process-doc-sets ``` -See [pipeline-commands.md](pipeline-commands.md#doc-set-filtering) for the full doc set ID table. +See [pipeline-commands.md](pipeline-commands.md#doc-set-filtering) for filtering and the canonical Java API listing command. ## Hybrid vector storage diff --git a/docs/pipeline-commands.md b/docs/pipeline-commands.md index 72e493ef..7d28611b 100644 --- a/docs/pipeline-commands.md +++ b/docs/pipeline-commands.md @@ -14,7 +14,7 @@ Incremental runs are the default — unchanged content is skipped via SHA-256 ha | **Scrape** (mirror HTML) | `make fetch-all` | `make fetch-force` | | **Ingest** (chunk → embed → upload) | `make process-all` | Clear state, then `make process-all` ([details](#full-re-ingest)) | | **Both** | `make full-pipeline` | `make fetch-force`, then clear state, then `make process-all` | -| **Ingest subset** | `DOCS_SETS=java25-complete make process-doc-sets` | — | +| **Ingest subset** | Set `DOCS_SETS` to a catalog path, then run `make process-doc-sets` | — | | **Ingest GitHub repo** | `REPO_URL=https://github.com/owner/repo make process-github-repo` | — | --- @@ -22,7 +22,19 @@ Incremental runs are the default — unchanged content is skipped via SHA-256 ha ## Scrape (fetch HTML mirrors) The scrape phase mirrors upstream documentation into `data/docs/` using `wget`. -Source URLs are defined in `src/main/resources/docs-sources.properties`. + +### Java API source catalog + +Complete Java API mirror records are defined in +`src/main/resources/java-api-documentation-sources.manifest`; source URLs outside complete Java API mirrors are defined in +`src/main/resources/docs-sources.properties`. + +The manifest is the single semantic owner of each complete Java API release, remote URL, mirror path, +display name, and fetch policy. Add, change, or remove a complete Java API source by editing one manifest +row only. Do not add `JAVA*_API_BASE` properties or environment overrides, and do not duplicate the +manifest's rows or field values in properties, Java constants, shell arrays, tests, or docs. +Run `./scripts/fetch_all_docs.sh --list-java-api-sources` to inspect the exact rows consumed by the fetch +script; the CLI catalog and citation registry project the same rows and parity is enforced by tests. ### Make targets @@ -35,7 +47,7 @@ make fetch-force # Full: force refetch even if mirrors look complete ### Script flags ```bash -./scripts/fetch_all_docs.sh [--include-quick] [--no-clean] [--force] +./scripts/fetch_all_docs.sh [--include-quick] [--no-clean] [--force] [--list-java-api-sources] ``` | Flag | Effect | @@ -43,12 +55,14 @@ make fetch-force # Full: force refetch even if mirrors look complete | `--include-quick` | Also fetch small landing-page mirrors (Spring Boot/Framework/AI quick sets) | | `--no-clean` | Do not quarantine incomplete mirrors before refetching | | `--force` | Refresh all sources even if they look complete | +| `--list-java-api-sources` | Print configured Java API source projections without fetching | | `--help` | Show usage | ### What "incremental" means for scraping - `wget --mirror --timestamping` skips files that haven't changed on the server. -- Sources with fewer HTML files than their configured minimum are quarantined and re-fetched. +- Sources with fewer HTML files than their configured minimum are quarantined and re-fetched when `allowPartial=false`. +- Sources with `allowPartial=true` retain nonempty, below-minimum mirrors for incremental reruns, but the fetch exits nonzero until they reach the configured minimum. Consequently, `make full-pipeline` stops before Qdrant ingestion while any retained mirror remains partial. - Oracle Javadoc uses a deterministic Python seed generator (`scripts/oracle_javadoc_seed.py`) to avoid incomplete recursive crawls. --- @@ -107,23 +121,18 @@ GitHub ingestion runs in headless CLI mode (`spring.main.web-application-type=no Limit ingestion to specific doc sets by ID or path: ```bash -# Single doc set -DOCS_SETS=java25-complete make process-doc-sets -./scripts/process_all_to_qdrant.sh --doc-sets=java25-complete - -# Multiple doc sets -./scripts/process_all_to_qdrant.sh --doc-sets=java25-complete,spring-boot-complete +# Complete Java API paths: copy relativeMirrorPath from the listing command +./scripts/fetch_all_docs.sh --list-java-api-sources +DOCS_SETS=relative/path/from/listing make process-doc-sets -# Path-style IDs -./scripts/process_all_to_qdrant.sh --doc-sets=java/java25-complete +# Non-Java example +./scripts/process_all_to_qdrant.sh --doc-sets=spring-boot-complete ``` -Available doc set IDs are defined in `DocumentationSetCatalog.java`. Common ones: +Complete Java API doc set paths are projected from the manifest. Common non-Java doc sets are: | ID | Content | |---|---| -| `java24-complete` | Java 24 complete API (Oracle Javadoc) | -| `java25-complete` | Java 25 complete API (Oracle Javadoc) | | `spring-boot-complete` | Spring Boot reference + API docs | | `spring-framework-complete` | Spring Framework reference + Javadoc | | `spring-ai-complete` | Spring AI reference + API (stable + 2.0) | @@ -230,7 +239,7 @@ To run the CLI directly: app_jar=$(ls -1 build/libs/*.jar | grep -v -- "-plain.jar" | head -1) # With doc set filtering -DOCS_SETS=java25-complete java -Dspring.profiles.active=cli -jar "$app_jar" +DOCS_SETS=spring-boot-complete java -Dspring.profiles.active=cli -jar "$app_jar" ``` The docs root defaults to `data/docs` unless `DOCS_DIR` is set. diff --git a/frontend/src/lib/components/ChatView.test.ts b/frontend/src/lib/components/ChatView.test.ts index 25ba4831..605ed91c 100644 --- a/frontend/src/lib/components/ChatView.test.ts +++ b/frontend/src/lib/components/ChatView.test.ts @@ -70,4 +70,25 @@ describe("ChatView streaming stability", () => { expect(assistantMessageElementAfter).toBe(assistantMessageElement); expect(container.querySelector(".message.assistant .cursor.visible")).toBeNull(); }); + + it("shows the active fallback provider from structured stream status", async () => { + streamChatMock.mockImplementation(async (_sessionId, _message, _onChunk, options) => { + options?.onStatus?.({ + message: "Retrying stream with provider", + details: "The first provider failed before response text.", + provider: "fallback-provider", + }); + return new Promise(() => {}); + }); + + const { getByLabelText, getByRole, findByText } = await renderChatView(); + const messageInput = getByLabelText("Message input"); + if (!(messageInput instanceof HTMLTextAreaElement)) { + throw new Error("Expected message input element to be a textarea"); + } + await fireEvent.input(messageInput, { target: { value: "Explain records" } }); + await fireEvent.click(getByRole("button", { name: "Send message" })); + + expect(await findByText(/Provider: fallback-provider/)).toBeTruthy(); + }); }); diff --git a/frontend/src/lib/components/GuidedLessonHeader.svelte b/frontend/src/lib/components/GuidedLessonHeader.svelte new file mode 100644 index 00000000..3bd2e4a6 --- /dev/null +++ b/frontend/src/lib/components/GuidedLessonHeader.svelte @@ -0,0 +1,106 @@ + + +
+
+ +

{lessonTitle}

+
+
+ + diff --git a/frontend/src/lib/components/LearnView.svelte b/frontend/src/lib/components/LearnView.svelte index c07340d3..da6e5ca6 100644 --- a/frontend/src/lib/components/LearnView.svelte +++ b/frontend/src/lib/components/LearnView.svelte @@ -14,6 +14,7 @@ import { applyJavaLanguageDetection } from "../services/javaLanguageDetection"; import { parseMarkdown } from "../services/markdown"; import CitationPanel from "./CitationPanel.svelte"; + import GuidedLessonHeader from "./GuidedLessonHeader.svelte"; import GuidedLessonChatPanel from "./GuidedLessonChatPanel.svelte"; import LessonCitations from "./LessonCitations.svelte"; import MessageBubble from "./MessageBubble.svelte"; @@ -606,31 +607,10 @@ {:else}
- -
-
- -

{selectedLesson.title}

-
-
+
@@ -909,62 +889,6 @@ position: relative; } - .lesson-header { - flex-shrink: 0; - padding: var(--space-4) var(--space-6); - border-bottom: 1px solid var(--color-border-subtle); - background: var(--color-bg-secondary); - } - - .lesson-header-inner { - display: flex; - align-items: center; - gap: var(--space-4); - max-width: 1400px; - margin: 0 auto; - width: 100%; - } - - .back-btn { - flex-shrink: 0; - display: flex; - align-items: center; - gap: var(--space-2); - padding: var(--space-2) var(--space-3); - background: transparent; - border: 1px solid var(--color-border-default); - border-radius: var(--radius-md); - font-size: var(--text-sm); - line-height: 1; - color: var(--color-text-secondary); - cursor: pointer; - transition: all var(--duration-fast) var(--ease-out); - } - - .back-btn:hover { - background: var(--color-bg-tertiary); - color: var(--color-text-primary); - } - - .back-btn svg { - width: 16px; - height: 16px; - } - - .lesson-title-header { - font-family: var(--font-serif); - font-size: var(--text-xl); - font-weight: 500; - color: var(--color-text-primary); - letter-spacing: var(--tracking-tight); - white-space: nowrap; - overflow: hidden; - text-overflow: ellipsis; - line-height: 1; - margin: 0; - padding-bottom: 2px; - } - /* Two-column Layout */ .lesson-layout { flex: 1; @@ -1099,14 +1023,6 @@ font-size: var(--text-base); } - .lesson-header { - padding: var(--space-3) var(--space-4); - } - - .lesson-title-header { - font-size: var(--text-lg); - } - .lesson-content-panel { padding: var(--space-4); } @@ -1134,9 +1050,5 @@ .lesson-summary { font-size: var(--text-xs); } - - .back-btn span { - display: none; - } } diff --git a/frontend/src/lib/components/LearnView.test.ts b/frontend/src/lib/components/LearnView.test.ts index 05fe28a8..6242998b 100644 --- a/frontend/src/lib/components/LearnView.test.ts +++ b/frontend/src/lib/components/LearnView.test.ts @@ -53,6 +53,9 @@ describe("LearnView guided chat streaming stability", () => { const allLessonsButton = await findByRole("button", { name: "All Lessons" }); expect(allLessonsButton).toHaveAttribute("aria-label", "All Lessons"); expect(allLessonsButton).toHaveTextContent("All Lessons"); + + await fireEvent.click(allLessonsButton); + expect(await findByRole("button", { name: /test lesson/i })).toBeInTheDocument(); }); it("keeps the guided assistant message DOM node stable when the stream completes", async () => { diff --git a/frontend/src/lib/composables/createStreamingState.svelte.ts b/frontend/src/lib/composables/createStreamingState.svelte.ts index 5a384d59..4f94d19f 100644 --- a/frontend/src/lib/composables/createStreamingState.svelte.ts +++ b/frontend/src/lib/composables/createStreamingState.svelte.ts @@ -133,7 +133,10 @@ export function createStreamingState(options: StreamingStateOptions = {}): Strea updateStatus(status: StreamStatus) { statusMessage = status.message; - statusDetails = status.details ?? ""; + const providerDescription = status.provider ? `Provider: ${status.provider}` : ""; + statusDetails = [status.details ?? "", providerDescription] + .filter((statusDescription) => statusDescription.length > 0) + .join(" · "); }, finishStream() { diff --git a/frontend/src/lib/services/chat.test.ts b/frontend/src/lib/services/chat.test.ts index c05f620a..327bc0af 100644 --- a/frontend/src/lib/services/chat.test.ts +++ b/frontend/src/lib/services/chat.test.ts @@ -10,43 +10,56 @@ vi.mock("./sse", () => { import { streamChat } from "./chat"; -describe("streamChat recovery", () => { +describe("streamChat", () => { beforeEach(() => { streamSseMock.mockReset(); }); - it("retries once for recoverable overflow failure before any streamed chunk", async () => { - streamSseMock.mockRejectedValueOnce(new Error("OverflowException: malformed response frame")); + it("surfaces a recoverable stream failure without replaying the POST", async () => { + const streamFailure = new Error("OverflowException: malformed response frame"); + const streamError = { + message: streamFailure.message, + details: "Malformed response frame at byte 512", + }; streamSseMock.mockImplementationOnce(async (_url, _body, callbacks) => { - callbacks.onText("Recovered response"); + callbacks.onError?.(streamError); + throw streamFailure; }); const onChunk = vi.fn(); const onStatus = vi.fn(); const onError = vi.fn(); + const streamAbortController = new AbortController(); await expect( - streamChat("session-1", "What is new in Java 25?", onChunk, { onStatus, onError }), - ).resolves.toBeUndefined(); - - expect(streamSseMock).toHaveBeenCalledTimes(2); - expect(onStatus).toHaveBeenCalledWith( - expect.objectContaining({ - message: "Temporary stream issue detected", + streamChat("session-1", "What is new in Java 25?", onChunk, { + onStatus, + onError, + signal: streamAbortController.signal, }), - ); - expect(onStatus).toHaveBeenCalledWith( + ).rejects.toBe(streamFailure); + + expect(streamSseMock).toHaveBeenCalledTimes(1); + expect(streamSseMock).toHaveBeenCalledWith( + "/api/chat/stream", + { sessionId: "session-1", latest: "What is new in Java 25?" }, expect.objectContaining({ - message: "Streaming recovered", + onText: onChunk, + onStatus, + onError, }), + "chat.ts", + { signal: streamAbortController.signal }, ); - expect(onChunk).toHaveBeenCalledWith("Recovered response"); - expect(onError).not.toHaveBeenCalled(); + expect(onStatus).not.toHaveBeenCalled(); + expect(onChunk).not.toHaveBeenCalled(); + expect(onError).toHaveBeenCalledWith(streamError); }); - it("does not retry when a chunk already streamed to the UI", async () => { + it("does not replay after a stream has emitted a chunk", async () => { streamSseMock.mockImplementationOnce(async (_url, _body, callbacks) => { callbacks.onText("Partial answer"); + callbacks.onError?.({ message: "OverflowException: malformed response frame" }); throw new Error("OverflowException: malformed response frame"); }); @@ -60,17 +73,12 @@ describe("streamChat recovery", () => { expect(streamSseMock).toHaveBeenCalledTimes(1); expect(onChunk).toHaveBeenCalledWith("Partial answer"); - expect(onStatus).not.toHaveBeenCalledWith( - expect.objectContaining({ - message: "Temporary stream issue detected", - }), - ); expect(onError).toHaveBeenCalledWith({ message: "OverflowException: malformed response frame", }); }); - it("honors backend non-retryable stream status metadata", async () => { + it("forwards backend non-retryable stream status metadata without replaying", async () => { streamSseMock.mockImplementationOnce(async (_url, _body, callbacks) => { callbacks.onStatus?.({ message: "Provider returned fatal stream error", @@ -78,6 +86,7 @@ describe("streamChat recovery", () => { retryable: false, stage: "stream", }); + callbacks.onError?.({ message: "Unexpected provider failure" }); throw new Error("Unexpected provider failure"); }); @@ -96,5 +105,6 @@ describe("streamChat recovery", () => { retryable: false, }), ); + expect(onError).toHaveBeenCalledWith({ message: "Unexpected provider failure" }); }); }); diff --git a/frontend/src/lib/services/chat.ts b/frontend/src/lib/services/chat.ts index 6e13d6bf..7236327e 100644 --- a/frontend/src/lib/services/chat.ts +++ b/frontend/src/lib/services/chat.ts @@ -13,7 +13,7 @@ import { } from "../validation/schemas"; import { validateFetchJson } from "../validation/validate"; import { csrfHeader, extractApiErrorMessage, fetchWithCsrfRetry } from "./csrf"; -import { streamWithRetry } from "./streamRecovery"; +import { streamSse } from "./sse"; export type { StreamStatus, StreamError, Citation }; @@ -52,17 +52,17 @@ export async function streamChat( onChunk: (chunk: string) => void, options: StreamChatOptions = {}, ): Promise { - return streamWithRetry( + return streamSse( "/api/chat/stream", { sessionId, latest: message }, { - onChunk, + onText: onChunk, onStatus: options.onStatus, onError: options.onError, onCitations: options.onCitations, - signal: options.signal, }, "chat.ts", + { signal: options.signal }, ); } diff --git a/frontend/src/lib/services/guided.test.ts b/frontend/src/lib/services/guided.test.ts index 42a4a36a..77064d6a 100644 --- a/frontend/src/lib/services/guided.test.ts +++ b/frontend/src/lib/services/guided.test.ts @@ -10,22 +10,28 @@ vi.mock("./sse", () => { import { streamGuidedChat, streamLessonContent } from "./guided"; -describe("streamGuidedChat recovery", () => { +describe("streamGuidedChat", () => { beforeEach(() => { streamSseMock.mockReset(); streamSseGetMock.mockReset(); }); - it("retries once for recoverable invalid stream errors before any chunk", async () => { - streamSseMock.mockRejectedValueOnce(new Error("OverflowException: malformed response frame")); + it("surfaces a recoverable stream failure without replaying the POST", async () => { + const streamFailure = new Error("OverflowException: malformed response frame"); + const streamError = { + message: streamFailure.message, + details: "Malformed response frame at byte 512", + }; streamSseMock.mockImplementationOnce(async (_url, _body, callbacks) => { - callbacks.onText("Recovered guided response"); + callbacks.onError?.(streamError); + throw streamFailure; }); const onChunk = vi.fn(); const onStatus = vi.fn(); const onError = vi.fn(); const onCitations = vi.fn(); + const streamAbortController = new AbortController(); await expect( streamGuidedChat("guided-session-1", "intro", "Teach me streams", { @@ -33,26 +39,33 @@ describe("streamGuidedChat recovery", () => { onStatus, onError, onCitations, + signal: streamAbortController.signal, }), - ).resolves.toBeUndefined(); + ).rejects.toBe(streamFailure); - expect(streamSseMock).toHaveBeenCalledTimes(2); - expect(onStatus).toHaveBeenCalledWith( - expect.objectContaining({ - message: "Temporary stream issue detected", - }), - ); - expect(onStatus).toHaveBeenCalledWith( + expect(streamSseMock).toHaveBeenCalledTimes(1); + expect(streamSseMock).toHaveBeenCalledWith( + "/api/guided/stream", + { sessionId: "guided-session-1", slug: "intro", latest: "Teach me streams" }, expect.objectContaining({ - message: "Streaming recovered", + onText: onChunk, + onStatus, + onError, + onCitations, }), + "guided.ts", + { signal: streamAbortController.signal }, ); - expect(onChunk).toHaveBeenCalledWith("Recovered guided response"); - expect(onError).not.toHaveBeenCalled(); + expect(onStatus).not.toHaveBeenCalled(); + expect(onChunk).not.toHaveBeenCalled(); + expect(onError).toHaveBeenCalledWith(streamError); }); it("does not retry for non-recoverable rate-limit errors", async () => { - streamSseMock.mockRejectedValueOnce(new Error("429 rate limit exceeded")); + streamSseMock.mockImplementationOnce(async (_url, _body, callbacks) => { + callbacks.onError?.({ message: "429 rate limit exceeded" }); + throw new Error("429 rate limit exceeded"); + }); const onChunk = vi.fn(); const onStatus = vi.fn(); @@ -87,6 +100,7 @@ describe("streamGuidedChat recovery", () => { retryable: false, stage: "stream", }); + callbacks.onError?.({ message: "Provider stream unavailable" }); throw new Error("Provider stream unavailable"); }); diff --git a/frontend/src/lib/services/guided.ts b/frontend/src/lib/services/guided.ts index f9be3b86..77ae7931 100644 --- a/frontend/src/lib/services/guided.ts +++ b/frontend/src/lib/services/guided.ts @@ -17,8 +17,7 @@ import { } from "../validation/schemas"; import { validateFetchJson } from "../validation/validate"; import { fetchCitationsByEndpoint, type CitationFetchResult } from "./chat"; -import { streamSseGet } from "./sse"; -import { streamWithRetry } from "./streamRecovery"; +import { streamSse, streamSseGet } from "./sse"; export type { StreamStatus, GuidedLesson, LessonContentResponse }; @@ -144,16 +143,16 @@ export async function streamGuidedChat( message: string, callbacks: GuidedStreamCallbacks, ): Promise { - return streamWithRetry( + return streamSse( "/api/guided/stream", { sessionId, slug, latest: message }, { - onChunk: callbacks.onChunk, + onText: callbacks.onChunk, onStatus: callbacks.onStatus, onError: callbacks.onError, onCitations: callbacks.onCitations, - signal: callbacks.signal, }, "guided.ts", + { signal: callbacks.signal }, ); } diff --git a/frontend/src/lib/services/javaLanguageDetection.ts b/frontend/src/lib/services/javaLanguageDetection.ts index bb85f40c..a92c7962 100644 --- a/frontend/src/lib/services/javaLanguageDetection.ts +++ b/frontend/src/lib/services/javaLanguageDetection.ts @@ -15,7 +15,7 @@ const JAVA_IDENTIFIER_IGNORABLE_CONTROL_RANGES = "\\u0000-\\u0008\\u000E-\\u001B /** Matches Java identifier parts, including identifier-ignorable characters. */ const JAVA_IDENTIFIER_PART_PATTERN = new RegExp( - `[\\p{ID_Continue}\\p{Sc}\\p{Cf}${JAVA_IDENTIFIER_IGNORABLE_CONTROL_RANGES}]`, + `[\\p{L}\\p{Nl}\\p{Sc}\\p{Pc}\\p{Nd}\\p{Mc}\\p{Mn}\\p{Cf}${JAVA_IDENTIFIER_IGNORABLE_CONTROL_RANGES}]`, "u", ); diff --git a/frontend/src/lib/services/markdown.test.ts b/frontend/src/lib/services/markdown.test.ts index ccbf742e..aa228786 100644 --- a/frontend/src/lib/services/markdown.test.ts +++ b/frontend/src/lib/services/markdown.test.ts @@ -193,6 +193,10 @@ describe("applyJavaLanguageDetection", () => { const javaKeywordBoundarySnippets = [ "int[] greetingCounts = { 1 };", 'String.valueOf("hello");', + "public\u00B7ation", + "public\u0387ation", + "public\u1369ation", + "public\u19DAation", ]; for (const javaKeywordBoundarySnippet of javaKeywordBoundarySnippets) { @@ -216,6 +220,8 @@ describe("applyJavaLanguageDetection", () => { 'const hint = "tip";', 'const publication = "article";', 'const republic = "state";', + 'const public$ation = "article";', + 'const public\u0001ation = "article";', 'const public\uFEFFation = "article";', ]; diff --git a/frontend/src/lib/services/sse.test.ts b/frontend/src/lib/services/sse.test.ts index d7a64bd1..5299189c 100644 --- a/frontend/src/lib/services/sse.test.ts +++ b/frontend/src/lib/services/sse.test.ts @@ -1,7 +1,14 @@ import { afterEach, describe, expect, it, vi } from "vitest"; -import { streamSse } from "./sse"; +import { streamSse, streamSseGet } from "./sse"; const SSE_STREAM_RESPONSE_STATUS = 200; +const HTTP_SERVICE_UNAVAILABLE_STATUS = 503; +const FETCH_FAILURE_MESSAGE = "Network request failed"; +const STREAM_READ_FAILURE_MESSAGE = "Unable to read the SSE stream"; +const SERVER_EVENT_ERROR_MESSAGE = "The provider ended the stream"; +const STATUS_EVENT_MESSAGE = "Switching to an available provider"; +const STATUS_EVENT_PROVIDER = "openai"; +const MISSING_STREAM_BODY_MESSAGE = "No response body"; function createSseStreamResponse(sseWireText: string): Response { const encoder = new TextEncoder(); @@ -17,7 +24,7 @@ function createSseStreamResponse(sseWireText: string): Response { }); } -describe("streamSse abort handling", () => { +describe("streamSse transport handling", () => { afterEach(() => { vi.unstubAllGlobals(); vi.restoreAllMocks(); @@ -43,6 +50,113 @@ describe("streamSse abort handling", () => { expect(onError).not.toHaveBeenCalled(); }); + it("reports and rejects a non-abort fetch failure exactly once", async () => { + const fetchFailure = new Error(FETCH_FAILURE_MESSAGE); + const fetchMock = vi.fn().mockRejectedValue(fetchFailure); + vi.stubGlobal("fetch", fetchMock); + + const onText = vi.fn(); + const onError = vi.fn(); + + await expect( + streamSse("/api/test/stream", { hello: "world" }, { onText, onError }, "sse.test.ts"), + ).rejects.toBe(fetchFailure); + + expect(fetchMock).toHaveBeenCalledOnce(); + expect(onText).not.toHaveBeenCalled(); + expect(onError).toHaveBeenCalledOnce(); + expect(onError).toHaveBeenCalledWith({ message: FETCH_FAILURE_MESSAGE }); + }); + + it("reports and rejects a non-abort GET fetch failure exactly once", async () => { + const fetchFailure = new Error(FETCH_FAILURE_MESSAGE); + const fetchMock = vi.fn().mockRejectedValue(fetchFailure); + vi.stubGlobal("fetch", fetchMock); + + const onText = vi.fn(); + const onError = vi.fn(); + + await expect(streamSseGet("/api/test/stream", { onText, onError }, "sse.test.ts")).rejects.toBe( + fetchFailure, + ); + + expect(fetchMock).toHaveBeenCalledOnce(); + expect(onText).not.toHaveBeenCalled(); + expect(onError).toHaveBeenCalledOnce(); + expect(onError).toHaveBeenCalledWith({ message: FETCH_FAILURE_MESSAGE }); + }); + + it("reports a non-OK GET response exactly once", async () => { + const serviceUnavailableMessage = `HTTP ${HTTP_SERVICE_UNAVAILABLE_STATUS}: Service Unavailable`; + vi.stubGlobal( + "fetch", + vi.fn().mockResolvedValue( + new Response(null, { + status: HTTP_SERVICE_UNAVAILABLE_STATUS, + statusText: "Service Unavailable", + }), + ), + ); + + const onText = vi.fn(); + const onError = vi.fn(); + + await expect( + streamSseGet("/api/test/stream", { onText, onError }, "sse.test.ts"), + ).rejects.toThrow(serviceUnavailableMessage); + + expect(onText).not.toHaveBeenCalled(); + expect(onError).toHaveBeenCalledOnce(); + expect(onError).toHaveBeenCalledWith({ message: serviceUnavailableMessage }); + }); + + it("reports a missing GET response body exactly once", async () => { + vi.stubGlobal( + "fetch", + vi.fn().mockResolvedValue( + new Response(null, { + status: SSE_STREAM_RESPONSE_STATUS, + statusText: "OK", + }), + ), + ); + + const onText = vi.fn(); + const onError = vi.fn(); + + await expect( + streamSseGet("/api/test/stream", { onText, onError }, "sse.test.ts"), + ).rejects.toThrow(MISSING_STREAM_BODY_MESSAGE); + + expect(onText).not.toHaveBeenCalled(); + expect(onError).toHaveBeenCalledOnce(); + expect(onError).toHaveBeenCalledWith({ message: MISSING_STREAM_BODY_MESSAGE }); + }); + + it("reports a valid server error event exactly once", async () => { + vi.stubGlobal( + "fetch", + vi + .fn() + .mockResolvedValue( + createSseStreamResponse( + `event: error\ndata: {"message":"${SERVER_EVENT_ERROR_MESSAGE}"}\n\n`, + ), + ), + ); + + const onText = vi.fn(); + const onError = vi.fn(); + + await expect( + streamSseGet("/api/test/stream", { onText, onError }, "sse.test.ts"), + ).rejects.toThrow(SERVER_EVENT_ERROR_MESSAGE); + + expect(onText).not.toHaveBeenCalled(); + expect(onError).toHaveBeenCalledOnce(); + expect(onError).toHaveBeenCalledWith({ message: SERVER_EVENT_ERROR_MESSAGE }); + }); + it("treats AbortError during read as a cancellation (no onError)", async () => { const encoder = new TextEncoder(); const abortError = Object.assign(new Error("Aborted"), { name: "AbortError" }); @@ -72,6 +186,34 @@ describe("streamSse abort handling", () => { expect(onText).toHaveBeenCalledWith("Hello"); expect(onError).not.toHaveBeenCalled(); }); + + it("reports and rejects a stream-read failure exactly once", async () => { + const streamReadFailure = new Error(STREAM_READ_FAILURE_MESSAGE); + const sseStreamBody = new ReadableStream({ + start(streamController) { + streamController.error(streamReadFailure); + }, + }); + const fetchMock = vi.fn().mockResolvedValue( + new Response(sseStreamBody, { + status: SSE_STREAM_RESPONSE_STATUS, + statusText: "OK", + }), + ); + vi.stubGlobal("fetch", fetchMock); + + const onText = vi.fn(); + const onError = vi.fn(); + + await expect( + streamSse("/api/test/stream", { hello: "world" }, { onText, onError }, "sse.test.ts"), + ).rejects.toBe(streamReadFailure); + + expect(fetchMock).toHaveBeenCalledOnce(); + expect(onText).not.toHaveBeenCalled(); + expect(onError).toHaveBeenCalledOnce(); + expect(onError).toHaveBeenCalledWith({ message: STREAM_READ_FAILURE_MESSAGE }); + }); }); describe("streamSse payload validation", () => { @@ -93,6 +235,30 @@ describe("streamSse payload validation", () => { expect(onText).toHaveBeenCalledWith("Hello"); }); + it("preserves provider from a valid wire-format status event", async () => { + vi.stubGlobal( + "fetch", + vi + .fn() + .mockResolvedValue( + createSseStreamResponse( + `event: status\ndata: {"message":"${STATUS_EVENT_MESSAGE}","provider":"${STATUS_EVENT_PROVIDER}"}\n\n`, + ), + ), + ); + const onText = vi.fn(); + const onStatus = vi.fn(); + + await streamSse("/api/test/stream", {}, { onText, onStatus }, "sse.test.ts"); + + expect(onText).not.toHaveBeenCalled(); + expect(onStatus).toHaveBeenCalledOnce(); + expect(onStatus).toHaveBeenCalledWith({ + message: STATUS_EVENT_MESSAGE, + provider: STATUS_EVENT_PROVIDER, + }); + }); + it("rejects legacy raw text payloads", async () => { vi.spyOn(console, "error").mockImplementation(() => undefined); vi.stubGlobal( diff --git a/frontend/src/lib/services/sse.ts b/frontend/src/lib/services/sse.ts index 4b5a46c1..451ec42f 100644 --- a/frontend/src/lib/services/sse.ts +++ b/frontend/src/lib/services/sse.ts @@ -46,6 +46,11 @@ function isAbortError(error: unknown): boolean { return error instanceof Error && error.name === "AbortError"; } +function reportSseFailure(callbacks: SseCallbacks, sseFailure: unknown): void { + const sseFailureMessage = sseFailure instanceof Error ? sseFailure.message : String(sseFailure); + callbacks.onError?.({ message: sseFailureMessage }); +} + function throwInvalidSseEvent(callbacks: SseCallbacks): never { const streamError: StreamError = { message: INVALID_SSE_EVENT_MESSAGE }; callbacks.onError?.(streamError); @@ -213,6 +218,7 @@ export async function streamSse( if (abortSignal?.aborted || isAbortError(fetchError)) { return; } + reportSseFailure(callbacks, fetchError); throw fetchError; } @@ -237,6 +243,7 @@ export async function streamSseGet( if (abortSignal?.aborted || isAbortError(fetchError)) { return; } + reportSseFailure(callbacks, fetchError); throw fetchError; } @@ -289,7 +296,14 @@ async function consumeSseStream( try { while (true) { - const { done: streamEnded, value: byteSegment } = await sseReader.read(); + const { done: streamEnded, value: byteSegment } = await sseReader + .read() + .catch((streamReadFailure: unknown): never => { + if (!abortSignal?.aborted && !isAbortError(streamReadFailure)) { + reportSseFailure(callbacks, streamReadFailure); + } + throw streamReadFailure; + }); if (streamEnded) { streamCompletedNormally = true; diff --git a/frontend/src/lib/services/streamRecovery.test.ts b/frontend/src/lib/services/streamRecovery.test.ts deleted file mode 100644 index 0e78d958..00000000 --- a/frontend/src/lib/services/streamRecovery.test.ts +++ /dev/null @@ -1,166 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { - buildStreamRecoverySucceededStatus, - buildStreamRetryStatus, - resolveStreamRecoveryRetryCount, - shouldRetryStreamRequest, - StreamFailureError, - toStreamError, - toStreamFailureException, -} from "./streamRecovery"; - -describe("streamRecovery", () => { - it("allows one retry for overflow failures before any chunk is rendered", () => { - const retryDecision = shouldRetryStreamRequest( - new Error("OverflowException while decoding response"), - null, - null, - false, - 0, - 1, - ); - - expect(retryDecision).toBe(true); - }); - - it("refuses retry after content has started streaming", () => { - const retryDecision = shouldRetryStreamRequest( - new Error("OverflowException while decoding response"), - null, - null, - true, - 0, - 1, - ); - - expect(retryDecision).toBe(false); - }); - - it("refuses retry for non-recoverable quota errors", () => { - const retryDecision = shouldRetryStreamRequest( - new Error("HTTP 429 rate limit exceeded"), - null, - null, - false, - 0, - 1, - ); - - expect(retryDecision).toBe(false); - }); - - it("builds a user-visible retry status message", () => { - const retryStatus = buildStreamRetryStatus(1, 1); - - expect(retryStatus.message).toBe("Temporary stream issue detected"); - expect(retryStatus.details).toContain("Retrying your request (1/1)"); - }); - - it("builds a user-visible recovery status message after retry succeeds", () => { - const recoveryStatus = buildStreamRecoverySucceededStatus(1); - - expect(recoveryStatus.message).toBe("Streaming recovered"); - expect(recoveryStatus.details).toContain("Recovered after retry (1)"); - }); - - it("maps thrown errors into StreamError shape", () => { - const mappedStreamError = toStreamError(new Error("OverflowException"), null); - - expect(mappedStreamError).toEqual({ message: "OverflowException" }); - }); - - it("retries once for recoverable network failures before any chunk is rendered", () => { - const retryDecision = shouldRetryStreamRequest( - new Error("TypeError: Failed to fetch"), - null, - null, - false, - 0, - 1, - ); - - expect(retryDecision).toBe(true); - }); - - it("respects backend retry metadata when stage is stream", () => { - const retryDecision = shouldRetryStreamRequest( - new Error("Some fatal backend error"), - null, - { - message: "Provider fallback succeeded", - retryable: true, - stage: "stream", - }, - false, - 0, - 1, - ); - - expect(retryDecision).toBe(true); - }); - - it("retries a generic terminal stream error when the backend marks it retryable", () => { - const retryDecision = shouldRetryStreamRequest( - new Error("Streaming error"), - { - message: "Streaming error", - code: "stream.provider.retryable-error", - retryable: true, - stage: "stream", - }, - null, - false, - 0, - 1, - ); - - expect(retryDecision).toBe(true); - }); - - it("refuses retry when a terminal error overrides an earlier retryable status", () => { - const retryDecision = shouldRetryStreamRequest( - new Error("Streaming error"), - { - message: "Streaming error", - code: "stream.provider.fatal-error", - retryable: false, - stage: "stream", - }, - { - message: "Provider fallback succeeded", - retryable: true, - stage: "stream", - }, - false, - 0, - 1, - ); - - expect(retryDecision).toBe(false); - }); - - it("preserves stream error details in thrown stream failure exception", () => { - const streamFailureException = toStreamFailureException(new Error("Transport failed"), { - message: "OverflowException", - details: "Malformed response frame at byte 512", - }); - - expect(streamFailureException).toBeInstanceOf(StreamFailureError); - expect(streamFailureException.message).toBe("OverflowException"); - expect(streamFailureException.details).toBe("Malformed response frame at byte 512"); - }); - - it("uses default retry count for missing config", () => { - expect(resolveStreamRecoveryRetryCount(undefined)).toBe(1); - }); - - it("clamps configured retry count into safe range", () => { - expect(resolveStreamRecoveryRetryCount(-1)).toBe(0); - expect(resolveStreamRecoveryRetryCount(9)).toBe(3); - }); - - it("falls back to default retry count for non-numeric config", () => { - expect(resolveStreamRecoveryRetryCount("abc")).toBe(1); - expect(resolveStreamRecoveryRetryCount("")).toBe(1); - }); -}); diff --git a/frontend/src/lib/services/streamRecovery.ts b/frontend/src/lib/services/streamRecovery.ts deleted file mode 100644 index 91257c43..00000000 --- a/frontend/src/lib/services/streamRecovery.ts +++ /dev/null @@ -1,289 +0,0 @@ -import type { StreamError, StreamStatus, Citation } from "../validation/schemas"; -import { streamSse } from "./sse"; - -const GENERIC_STREAM_FAILURE_MESSAGE = "Streaming request failed"; - -/** - * Error subclass carrying structured SSE stream failure details. - * Replaces unsafe `as` casts that monkey-patched `details` onto plain Error objects. - */ -export class StreamFailureError extends Error { - readonly details?: string; - - constructor(message: string, details?: string) { - super(message); - this.name = "StreamFailureError"; - this.details = details; - } -} - -const RECOVERABLE_STREAM_ERROR_PATTERNS = [ - /overflowexception/i, - /invalid\s+(stream|response)/i, - /malformed\s+(stream|response|json)/i, - /sseexception/i, - /unexpected end of json input/i, - /failed to fetch/i, - /networkerror/i, - /connection (reset|closed|refused)/i, - /socket hang up/i, - /\btimeout\b/i, - /\btimed out\b/i, - /\bhttp\s*5\d{2}\b/i, - /\b5\d{2}\s+(internal|bad gateway|service unavailable|gateway timeout)\b/i, -]; - -const NON_RECOVERABLE_STREAM_ERROR_PATTERNS = [ - /rate limit/i, - /\b429\b/, - /\b401\b/, - /\b403\b/, - /providers unavailable/i, -]; -const ABORT_STREAM_ERROR_PATTERNS = [/aborterror/i, /\baborted\b/i, /\bcancelled\b/i]; - -const DEFAULT_STREAM_RECOVERY_RETRY_COUNT = 1; -const MIN_STREAM_RECOVERY_RETRY_COUNT = 0; -const MAX_STREAM_RECOVERY_RETRY_COUNT = 3; - -export const MAX_STREAM_RECOVERY_RETRIES = resolveStreamRecoveryRetryCount( - import.meta.env.VITE_STREAM_RECOVERY_MAX_RETRIES, -); - -/** - * Decides whether a stream request should be retried for a likely recoverable provider response issue. - * - * The retry gate is intentionally strict: - * - bounded retries via MAX_STREAM_RECOVERY_RETRIES, - * - only before any assistant chunk has been rendered, - * - terminal backend metadata takes precedence over earlier status events, - * - unstructured failures require a known recoverable signature. - */ -export function shouldRetryStreamRequest( - streamFailure: unknown, - streamErrorEvent: StreamError | null, - latestStreamStatus: StreamStatus | null, - hasStreamedAnyChunk: boolean, - attemptedRetries: number, - maxRecoveryRetries: number, -): boolean { - if (hasStreamedAnyChunk) { - return false; - } - if (attemptedRetries >= maxRecoveryRetries) { - return false; - } - if (typeof streamErrorEvent?.retryable === "boolean") { - return streamErrorEvent.retryable; - } - if (latestStreamStatus?.stage === "stream" && latestStreamStatus.retryable === false) { - return false; - } - if (latestStreamStatus?.stage === "stream" && latestStreamStatus.retryable === true) { - return true; - } - - const failureDescription = describeStreamFailure(streamFailure, streamErrorEvent); - if (ABORT_STREAM_ERROR_PATTERNS.some((pattern) => pattern.test(failureDescription))) { - return false; - } - if (NON_RECOVERABLE_STREAM_ERROR_PATTERNS.some((pattern) => pattern.test(failureDescription))) { - return false; - } - return RECOVERABLE_STREAM_ERROR_PATTERNS.some((pattern) => pattern.test(failureDescription)); -} - -/** - * Shows users that the client detected a transient stream fault and is retrying automatically. - */ -export function buildStreamRetryStatus( - nextAttemptNumber: number, - maxRecoveryRetries: number, -): StreamStatus { - return { - message: "Temporary stream issue detected", - details: `The API response or network stream was temporarily invalid. Retrying your request (${nextAttemptNumber}/${maxRecoveryRetries}).`, - }; -} - -/** - * Reassures users that the retry path succeeded and normal streaming has resumed. - */ -export function buildStreamRecoverySucceededStatus(recoveryAttemptCount: number): StreamStatus { - return { - message: "Streaming recovered", - details: `Recovered after retry (${recoveryAttemptCount}). Continuing your response.`, - }; -} - -/** - * Converts thrown stream failures into the canonical StreamError shape used by UI components. - */ -export function toStreamError( - streamFailure: unknown, - streamErrorEvent: StreamError | null, -): StreamError { - if (streamErrorEvent) { - return streamErrorEvent; - } - if (streamFailure instanceof Error) { - return { message: streamFailure.message }; - } - return { message: GENERIC_STREAM_FAILURE_MESSAGE }; -} - -/** - * Converts stream failures into a typed StreamFailureError that preserves structured details. - */ -export function toStreamFailureException( - streamFailure: unknown, - streamErrorEvent: StreamError | null, -): StreamFailureError { - const mappedStreamError = toStreamError(streamFailure, streamErrorEvent); - return new StreamFailureError(mappedStreamError.message, mappedStreamError.details ?? undefined); -} - -export function resolveStreamRecoveryRetryCount(rawRetrySetting: unknown): number { - if (rawRetrySetting === null || rawRetrySetting === undefined || rawRetrySetting === "") { - return DEFAULT_STREAM_RECOVERY_RETRY_COUNT; - } - - const parsedRetryCount = Number(rawRetrySetting); - if (!Number.isInteger(parsedRetryCount)) { - return DEFAULT_STREAM_RECOVERY_RETRY_COUNT; - } - if (parsedRetryCount < MIN_STREAM_RECOVERY_RETRY_COUNT) { - return MIN_STREAM_RECOVERY_RETRY_COUNT; - } - if (parsedRetryCount > MAX_STREAM_RECOVERY_RETRY_COUNT) { - return MAX_STREAM_RECOVERY_RETRY_COUNT; - } - return parsedRetryCount; -} - -/** Callbacks for the stream-with-retry wrapper. */ -export interface StreamWithRetryCallbacks { - onChunk: (chunk: string) => void; - onStatus?: (status: StreamStatus) => void; - onError?: (error: StreamError) => void; - onCitations?: (citations: Citation[]) => void; - signal?: AbortSignal; -} - -/** - * Streams an SSE endpoint with automatic retry on recoverable failures. - * Retries only before the first assistant chunk is emitted to the UI. - */ -export async function streamWithRetry( - endpoint: string, - body: object, - callbacks: StreamWithRetryCallbacks, - sourceLabel: string, -): Promise { - const { onChunk, onStatus, onError, onCitations, signal } = callbacks; - let attemptedRecoveryRetries = 0; - let hasPendingRecoverySuccessNotice = false; - - while (true) { - let hasStreamedAnyChunk = false; - let streamErrorEvent: StreamError | null = null; - let latestStreamStatus: StreamStatus | null = null; - - try { - await streamSse( - endpoint, - body, - { - onText: (chunk) => { - hasStreamedAnyChunk = true; - if (hasPendingRecoverySuccessNotice) { - onStatus?.(buildStreamRecoverySucceededStatus(attemptedRecoveryRetries)); - hasPendingRecoverySuccessNotice = false; - } - onChunk(chunk); - }, - onStatus: (status) => { - latestStreamStatus = status; - onStatus?.(status); - }, - onError: (streamError) => { - streamErrorEvent = streamError; - }, - onCitations, - }, - sourceLabel, - { signal }, - ); - return; - } catch (streamFailure) { - if ( - shouldRetryStreamRequest( - streamFailure, - streamErrorEvent, - latestStreamStatus, - hasStreamedAnyChunk, - attemptedRecoveryRetries, - MAX_STREAM_RECOVERY_RETRIES, - ) - ) { - attemptedRecoveryRetries++; - hasPendingRecoverySuccessNotice = true; - onStatus?.(buildStreamRetryStatus(attemptedRecoveryRetries, MAX_STREAM_RECOVERY_RETRIES)); - continue; - } - const mappedStreamError = toStreamError(streamFailure, streamErrorEvent); - onError?.(mappedStreamError); - throw toStreamFailureException(streamFailure, streamErrorEvent); - } - } -} - -function describeStreamFailure( - streamFailure: unknown, - streamErrorEvent: StreamError | null, -): string { - const diagnosticTokens: string[] = []; - if (streamErrorEvent?.message) { - diagnosticTokens.push(streamErrorEvent.message); - } - if (streamErrorEvent?.details) { - diagnosticTokens.push(streamErrorEvent.details); - } - if (streamErrorEvent?.code) { - diagnosticTokens.push(streamErrorEvent.code); - } - if (streamFailure instanceof Error) { - diagnosticTokens.push(streamFailure.message); - if ("details" in streamFailure && typeof streamFailure.details === "string") { - diagnosticTokens.push(streamFailure.details); - } - } else if (streamFailure !== null && streamFailure !== undefined) { - diagnosticTokens.push(formatUnknownDiagnostic(streamFailure)); - } - return diagnosticTokens.join(" ").trim(); -} - -function formatUnknownDiagnostic(value: unknown): string { - if (typeof value === "string") { - return value; - } - if (typeof value === "number" || typeof value === "boolean") { - return String(value); - } - if (typeof value === "bigint") { - return value.toString(); - } - if (typeof value === "symbol") { - return value.description ?? "Symbol"; - } - if (typeof value === "function") { - return value.name ? `[function ${value.name}]` : "[function]"; - } - - try { - const json = JSON.stringify(value); - return json === undefined ? "" : json; - } catch { - return Object.prototype.toString.call(value); - } -} diff --git a/scripts/fetch_all_docs.sh b/scripts/fetch_all_docs.sh index 12fd7ad6..17e5fcec 100755 --- a/scripts/fetch_all_docs.sh +++ b/scripts/fetch_all_docs.sh @@ -1,23 +1,23 @@ #!/bin/bash # Consolidated Documentation Fetcher with Deduplication -# This script fetches all required documentation (Java 24/25, Spring ecosystem) +# This script fetches all required documentation (configured Java APIs, Spring ecosystem) # and ensures no redundant downloads by checking existing files -set -euo pipefail - SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck source=lib/shell_bootstrap.sh source "$SCRIPT_DIR/lib/shell_bootstrap.sh" # shellcheck source=lib/env_loader.sh source "$SCRIPT_DIR/lib/env_loader.sh" +# shellcheck source=lib/documentation_sources.sh +source "$SCRIPT_DIR/lib/documentation_sources.sh" +# shellcheck source=lib/documentation_fetch_metadata.sh +source "$SCRIPT_DIR/lib/documentation_fetch_metadata.sh" -# Centralized source URLs (single source of truth) +# Centralized source definitions RES_PROPS="$SCRIPT_DIR/../src/main/resources/docs-sources.properties" -if [ -f "$RES_PROPS" ]; then - preserve_process_env_then_source_file "$RES_PROPS" -fi +JAVA_API_SOURCES_MANIFEST="$SCRIPT_DIR/../src/main/resources/java-api-documentation-sources.manifest" DOCS_ROOT="$SCRIPT_DIR/../data/docs" LOG_FILE="$SCRIPT_DIR/../fetch_all_docs.log" @@ -25,79 +25,40 @@ LOG_FILE="$SCRIPT_DIR/../fetch_all_docs.log" INCLUDE_QUICK="${INCLUDE_QUICK:-false}" CLEAN_INCOMPLETE="${CLEAN_INCOMPLETE:-true}" FORCE_REFRESH="${FORCE_REFRESH:-false}" -for arg in "$@"; do - case $arg in - --include-quick) - INCLUDE_QUICK="true" - ;; - --no-clean) - CLEAN_INCOMPLETE="false" - ;; - --force) - FORCE_REFRESH="true" - ;; - --help|-h) - echo "Usage: $0 [--include-quick] [--no-clean] [--force]" - echo " --include-quick : Also refresh small 'quick' doc mirrors" - echo " --no-clean : Do not quarantine incomplete mirrors before refetch" - echo " --force : Refresh even when mirrors look complete" - exit 0 - ;; - *) - echo "Unknown option: $arg" - echo "Use --help for usage information" - exit 1 - ;; - esac -done - -echo "==============================================" -echo "Consolidated Documentation Fetcher" -echo "==============================================" -echo "Docs root: $DOCS_ROOT" -echo "Log file: $LOG_FILE" -echo "" - -# Initialize log -echo "[$(date)] Starting consolidated documentation fetch" > "$LOG_FILE" - -extract_meta_version() { - local file_path="$1" - if [ -z "$file_path" ] || [ ! -f "$file_path" ]; then - echo "" - return 0 - fi - local line - line="$(grep -E '/dev/null | head -n 1 || true)" - if [ -z "$line" ]; then - echo "" - return 0 - fi - echo "$line" | sed -E 's/.*content="([^"]+)".*/\1/' -} - -extract_javadoc_comment_version() { - local file_path="$1" - if [ -z "$file_path" ] || [ ! -f "$file_path" ]; then - echo "" - return 0 - fi - local line - line="$(grep -E '