Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
d76dabc
refactor(ingestion): make Java API manifest canonical
WilliamAGH Jul 13, 2026
f24081c
fix(retrieval): preserve same-page chunks before citation dedupe
WilliamAGH Jul 13, 2026
8152d79
fix(streaming): prevent stateful chat replay
WilliamAGH Jul 13, 2026
54e39b6
fix(markdown): match Java identifier boundaries
WilliamAGH Jul 13, 2026
9374046
refactor(learning): extract guided lesson header
WilliamAGH Jul 13, 2026
ae4b64f
fix(streaming): commit exchanges after completion
WilliamAGH Jul 13, 2026
5a2beb1
fix(ingestion): align manifest line ending parsing
WilliamAGH Jul 13, 2026
111a320
test(retrieval): derive Javadoc URLs from catalog
WilliamAGH Jul 13, 2026
a97a5e3
style(ingestion): name manifest record threshold
WilliamAGH Jul 13, 2026
faf1393
feat(ingestion): honor per-source partial mirror policy
WilliamAGH Jul 13, 2026
f9f71b1
test(ingestion): assert partial mirror policy behavior
WilliamAGH Jul 13, 2026
287def4
fix(security): keep CSRF protection stateless
WilliamAGH Jul 13, 2026
f204efc
fix(ingestion): isolate manifest fetch execution
WilliamAGH Jul 13, 2026
6ab4682
test(ingestion): cover oracle javadoc partial mirror dispatch
WilliamAGH Jul 13, 2026
0154de2
fix(ingestion): block incomplete documentation runs
WilliamAGH Jul 13, 2026
f0c2ddb
test(ingestion): cover partial run exit
WilliamAGH Jul 13, 2026
8259404
fix(streaming): let request timeout govern SDK reads
WilliamAGH Jul 13, 2026
9b4bb5e
fix(security): preserve CSRF cookie deletion
WilliamAGH Jul 13, 2026
e474671
fix(citations): anchor guided PDF chunks before dedupe
WilliamAGH Jul 13, 2026
4f1adc5
fix(streaming): report SSE transport failures
WilliamAGH Jul 13, 2026
3392b95
fix(ingestion): remove stale Java quick docsets
WilliamAGH Jul 13, 2026
1f8489e
test(retrieval): preserve distinct unmapped local sources
WilliamAGH Jul 13, 2026
3f4d567
test(streaming): separate provider routing coverage
WilliamAGH Jul 13, 2026
ce3a9ef
fix(config): validate Java API manifest locations
WilliamAGH Jul 13, 2026
45a9a0f
fix(config): protect Java API source projections
WilliamAGH Jul 13, 2026
7b81b34
fix(ingestion): enforce canonical Java API selectors
WilliamAGH Jul 13, 2026
3259bfa
fix(ingestion): preserve canonical selector case
WilliamAGH Jul 13, 2026
5fe43c5
fix(provider): classify OkHttp call timeouts
WilliamAGH Jul 13, 2026
2a84071
fix(retrieval): retain distinct redacted citations
WilliamAGH Jul 13, 2026
2631849
fix(docs): validate canonical Java API source rows
WilliamAGH Jul 13, 2026
2a120a5
fix(docs): dispatch exact Java API source rows
WilliamAGH Jul 13, 2026
e08ed9d
test(docs): enforce complete Java API fetch parity
WilliamAGH Jul 13, 2026
e61968f
test(config): reject unsafe Java API manifest values
WilliamAGH Jul 13, 2026
b95c521
test(ingestion): reject Java API selector aliases
WilliamAGH Jul 13, 2026
502bc8a
test(provider): cover wrapped OkHttp call timeouts
WilliamAGH Jul 13, 2026
da3a1aa
test(retrieval): preserve redacted citation identity
WilliamAGH Jul 13, 2026
aaec1e1
test(security): verify secure CSRF cookie lifecycle
WilliamAGH Jul 13, 2026
d8f8183
test(frontend): cover SSE failure contracts
WilliamAGH Jul 13, 2026
4665088
test(frontend): preserve chat stream error identity
WilliamAGH Jul 13, 2026
c11049b
test(frontend): preserve guided stream error identity
WilliamAGH Jul 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ alwaysApply: true
- [ZA1a-c] Zero Tolerance Policy (zero assumptions, validation workflow, forbidden practices)
- [GT1a-l] Git, history safety, hooks/signing, lock files, and clean commits
- [CC1a-d] Clean Code & DDD (Mandatory)
- [SS1a-j] Single Semantic Owner and Java API catalog ownership
- [ID1a-d] Idiomatic Patterns & Defaults
- [EV1a-f] Secrets and Env Vars (no secrets in properties; OpenAI/Qdrant via .env/env)
- [RC1a-f] Root Cause Resolution (single implementation, no fallbacks, no shims/workarounds)
Expand Down Expand Up @@ -78,6 +79,8 @@ alwaysApply: true
- [SS1f] **Stop-Work Trigger**: If an implementation requires listing the same governed fields/keys/rules in a second place, stop and redesign before editing.
- [SS1g] **No Positional-Null Sludge**: Constructor/factory calls with repeated placeholder nulls or low-legibility optional argument trains are prohibited; use named factories/builders, parameter objects, or bind/import the canonical owner.
- [SS1h] **Whole-List Smell**: If a file "knows the whole list" of a governed concept and it is not the canonical owner, the design is presumed wrong and must be reduced or removed.
- [SS1i] **Java API Catalog Owner**: Define complete Java API mirror records only in `src/main/resources/java-api-documentation-sources.manifest`; consume that manifest from Java, shell, tests, and documentation.
- [SS1j] **Java API Catalog Projection**: Never recreate Java API releases, URLs, mirror paths, display names, or fetch parameters in properties, constants, arrays, fixtures, or documentation; verify projections with `--list-java-api-sources` and `JavaApiDocumentationSourceParityTest`.

## [ID1] Idiomatic Patterns & Defaults

Expand Down
3 changes: 1 addition & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,7 @@ Common variables:
- `OPENAI_BASE_URL` (default `https://api.openai.com/v1`)
- `OPENAI_MODEL` (application fallback default `gpt-5.2`; shared-gateway alias `gemma-4-26b-a4b`)
- `OPENAI_REASONING_EFFORT` (optional, GPT‑5 family)
- `OPENAI_STREAMING_REQUEST_TIMEOUT_SECONDS` (default `600`)
- `OPENAI_STREAMING_READ_TIMEOUT_SECONDS` (default `75`)
- `OPENAI_STREAMING_REQUEST_TIMEOUT_SECONDS` (default `600`; bounds the complete SDK call while provider gateways own first-output and inter-output deadlines)

### Researchly shared gateway

Expand Down
3 changes: 2 additions & 1 deletion docs/contracts/code-change.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: "Evergreen contract for change decisions (new file vs edit), reposi

# Code Change Policy Contract

See `AGENTS.md` ([FS1a-k], [FS1l], [AR1a-f], [ND1a-c], [CC1a-d], [AB1a-d], [RC1a-f], [CS1a-h]).
See `AGENTS.md` ([SS1a-j], [FS1a-k], [FS1l], [AR1a-f], [ND1a-c], [CC1a-d], [AB1a-d], [RC1a-f], [CS1a-h]).

## Non-negotiables (applies to every change)

Expand All @@ -28,6 +28,7 @@ Use this as a hard rule, not a suggestion.
| Logic change in stable code | Extract/replace via composition; keep stable code stable ([AB1c], [CC1a]) | Add flags, shims, or “compat” paths to hide uncertainty ([RC1b]) |
| Touching a large/overloaded file | Extract at least one seam (new type + typed contract) ([FS1f], [FS1b]) | Grow the file further ([FS1f]) |
| Reuse needed across features | Add a domain value object / explicit port / explicit service with intent-revealing name ([AB1b]) | Add `*Utils/*Helper/*Common/*Base*` grab bags ([FS1e]) |
| Governed catalog change | Edit its single canonical owner and project it everywhere else ([SS1a-c]) | Restate the inventory in code, scripts, tests, fixtures, or docs ([SS1b], [SS1h]) |

### When adding a method is allowed

Expand Down
2 changes: 1 addition & 1 deletion docs/domains/all-parsing-and-markdown-logic.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ This document provides a comprehensive analysis of all parsing and markdown proc
│ ├── ChatController.stream() → SSE events
│ ├── normalizeDelta() - token joining/cleanup
│ ├── UnifiedMarkdownService.process() - final markdown processing
│ └── ChatMemoryService.addAssistant() - persistence
│ └── ChatMemoryService.addExchange() - atomic user/assistant persistence
└── 🔄 KEY TRANSITIONS & ISSUES
├── Regex → AST migration (incomplete)
Expand Down
16 changes: 7 additions & 9 deletions docs/ingestion.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,9 @@ Fetch all configured sources:
make fetch-all
```

This runs `scripts/fetch_all_docs.sh` (requires `wget`). Source URLs live in:

- `src/main/resources/docs-sources.properties`

See [pipeline-commands.md](pipeline-commands.md#scrape-fetch-html-mirrors) for flags (`--force`, `--include-quick`, `--no-clean`).
This runs `scripts/fetch_all_docs.sh` (requires `wget`). See the canonical source ownership and edit
workflow in [pipeline-commands.md](pipeline-commands.md#scrape-fetch-html-mirrors), along with flags
(`--force`, `--include-quick`, `--no-clean`).

## Process + upload to Qdrant

Expand All @@ -46,14 +44,14 @@ This runs `scripts/process_all_to_qdrant.sh`, which:

### Doc set filtering (CLI)

Limit ingestion to specific doc sets:
List canonical complete Java API paths, then limit ingestion to a selected doc set:

```bash
DOCS_SETS=java25-complete make process-doc-sets
./scripts/process_all_to_qdrant.sh --doc-sets=java25-complete,spring-boot-complete
./scripts/fetch_all_docs.sh --list-java-api-sources
DOCS_SETS=relative/path/from/listing make process-doc-sets
```

See [pipeline-commands.md](pipeline-commands.md#doc-set-filtering) for the full doc set ID table.
See [pipeline-commands.md](pipeline-commands.md#doc-set-filtering) for filtering and the canonical Java API listing command.

## Hybrid vector storage

Expand Down
41 changes: 25 additions & 16 deletions docs/pipeline-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,27 @@ Incremental runs are the default — unchanged content is skipped via SHA-256 ha
| **Scrape** (mirror HTML) | `make fetch-all` | `make fetch-force` |
| **Ingest** (chunk → embed → upload) | `make process-all` | Clear state, then `make process-all` ([details](#full-re-ingest)) |
| **Both** | `make full-pipeline` | `make fetch-force`, then clear state, then `make process-all` |
| **Ingest subset** | `DOCS_SETS=java25-complete make process-doc-sets` | — |
| **Ingest subset** | Set `DOCS_SETS` to a catalog path, then run `make process-doc-sets` | — |
| **Ingest GitHub repo** | `REPO_URL=https://github.com/owner/repo make process-github-repo` | — |

---

## Scrape (fetch HTML mirrors)

The scrape phase mirrors upstream documentation into `data/docs/` using `wget`.
Source URLs are defined in `src/main/resources/docs-sources.properties`.

### Java API source catalog

Complete Java API mirror records are defined in
`src/main/resources/java-api-documentation-sources.manifest`; source URLs outside complete Java API mirrors are defined in
`src/main/resources/docs-sources.properties`.

The manifest is the single semantic owner of each complete Java API release, remote URL, mirror path,
display name, and fetch policy. Add, change, or remove a complete Java API source by editing one manifest
row only. Do not add `JAVA*_API_BASE` properties or environment overrides, and do not duplicate the
manifest's rows or field values in properties, Java constants, shell arrays, tests, or docs.
Run `./scripts/fetch_all_docs.sh --list-java-api-sources` to inspect the exact rows consumed by the fetch
script; the CLI catalog and citation registry project the same rows and parity is enforced by tests.

### Make targets

Expand All @@ -35,20 +47,22 @@ make fetch-force # Full: force refetch even if mirrors look complete
### Script flags

```bash
./scripts/fetch_all_docs.sh [--include-quick] [--no-clean] [--force]
./scripts/fetch_all_docs.sh [--include-quick] [--no-clean] [--force] [--list-java-api-sources]
```

| Flag | Effect |
|---|---|
| `--include-quick` | Also fetch small landing-page mirrors (Spring Boot/Framework/AI quick sets) |
| `--no-clean` | Do not quarantine incomplete mirrors before refetching |
| `--force` | Refresh all sources even if they look complete |
| `--list-java-api-sources` | Print configured Java API source projections without fetching |
| `--help` | Show usage |

### What "incremental" means for scraping

- `wget --mirror --timestamping` skips files that haven't changed on the server.
- Sources with fewer HTML files than their configured minimum are quarantined and re-fetched.
- Sources with fewer HTML files than their configured minimum are quarantined and re-fetched when `allowPartial=false`.
- Sources with `allowPartial=true` retain nonempty, below-minimum mirrors for incremental reruns, but the fetch exits nonzero until they reach the configured minimum. Consequently, `make full-pipeline` stops before Qdrant ingestion while any retained mirror remains partial.
- Oracle Javadoc uses a deterministic Python seed generator (`scripts/oracle_javadoc_seed.py`) to avoid incomplete recursive crawls.

---
Expand Down Expand Up @@ -107,23 +121,18 @@ GitHub ingestion runs in headless CLI mode (`spring.main.web-application-type=no
Limit ingestion to specific doc sets by ID or path:

```bash
# Single doc set
DOCS_SETS=java25-complete make process-doc-sets
./scripts/process_all_to_qdrant.sh --doc-sets=java25-complete

# Multiple doc sets
./scripts/process_all_to_qdrant.sh --doc-sets=java25-complete,spring-boot-complete
# Complete Java API paths: copy relativeMirrorPath from the listing command
./scripts/fetch_all_docs.sh --list-java-api-sources
DOCS_SETS=relative/path/from/listing make process-doc-sets

# Path-style IDs
./scripts/process_all_to_qdrant.sh --doc-sets=java/java25-complete
# Non-Java example
./scripts/process_all_to_qdrant.sh --doc-sets=spring-boot-complete
```

Available doc set IDs are defined in `DocumentationSetCatalog.java`. Common ones:
Complete Java API doc set paths are projected from the manifest. Common non-Java doc sets are:

| ID | Content |
|---|---|
| `java24-complete` | Java 24 complete API (Oracle Javadoc) |
| `java25-complete` | Java 25 complete API (Oracle Javadoc) |
| `spring-boot-complete` | Spring Boot reference + API docs |
| `spring-framework-complete` | Spring Framework reference + Javadoc |
| `spring-ai-complete` | Spring AI reference + API (stable + 2.0) |
Expand Down Expand Up @@ -230,7 +239,7 @@ To run the CLI directly:
app_jar=$(ls -1 build/libs/*.jar | grep -v -- "-plain.jar" | head -1)

# With doc set filtering
DOCS_SETS=java25-complete java -Dspring.profiles.active=cli -jar "$app_jar"
DOCS_SETS=spring-boot-complete java -Dspring.profiles.active=cli -jar "$app_jar"
```

The docs root defaults to `data/docs` unless `DOCS_DIR` is set.
Expand Down
21 changes: 21 additions & 0 deletions frontend/src/lib/components/ChatView.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,25 @@ describe("ChatView streaming stability", () => {
expect(assistantMessageElementAfter).toBe(assistantMessageElement);
expect(container.querySelector(".message.assistant .cursor.visible")).toBeNull();
});

it("shows the active fallback provider from structured stream status", async () => {
streamChatMock.mockImplementation(async (_sessionId, _message, _onChunk, options) => {
options?.onStatus?.({
message: "Retrying stream with provider",
details: "The first provider failed before response text.",
provider: "fallback-provider",
});
return new Promise<void>(() => {});
});

const { getByLabelText, getByRole, findByText } = await renderChatView();
const messageInput = getByLabelText("Message input");
if (!(messageInput instanceof HTMLTextAreaElement)) {
throw new Error("Expected message input element to be a textarea");
}
await fireEvent.input(messageInput, { target: { value: "Explain records" } });
await fireEvent.click(getByRole("button", { name: "Send message" }));

expect(await findByText(/Provider: fallback-provider/)).toBeTruthy();
});
});
106 changes: 106 additions & 0 deletions frontend/src/lib/components/GuidedLessonHeader.svelte
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
<script lang="ts">
const ALL_LESSONS_LABEL = "All Lessons";

let {
lessonTitle,
onReturnToLessons,
}: {
lessonTitle: string;
onReturnToLessons: () => void;
} = $props();
</script>

<div class="lesson-header">
<div class="lesson-header-inner">
<button
type="button"
class="back-button"
aria-label={ALL_LESSONS_LABEL}
onclick={onReturnToLessons}
>
<svg viewBox="0 0 20 20" fill="currentColor" aria-hidden="true">
<path
fill-rule="evenodd"
d="M17 10a.75.75 0 0 1-.75.75H5.612l4.158 3.96a.75.75 0 1 1-1.04 1.08l-5.5-5.25a.75.75 0 0 1 0-1.08l5.5-5.25a.75.75 0 1 1 1.04 1.08L5.612 9.25H16.25A.75.75 0 0 1 17 10Z"
clip-rule="evenodd"
/>
</svg>
<span>{ALL_LESSONS_LABEL}</span>
</button>
<h2 class="lesson-title">{lessonTitle}</h2>
</div>
</div>

<style>
.lesson-header {
flex-shrink: 0;
padding: var(--space-4) var(--space-6);
border-bottom: 1px solid var(--color-border-subtle);
background: var(--color-bg-secondary);
}

.lesson-header-inner {
display: flex;
align-items: center;
gap: var(--space-4);
max-width: 1400px;
margin: 0 auto;
width: 100%;
}

.back-button {
flex-shrink: 0;
display: flex;
align-items: center;
gap: var(--space-2);
padding: var(--space-2) var(--space-3);
background: transparent;
border: 1px solid var(--color-border-default);
border-radius: var(--radius-md);
font-size: var(--text-sm);
line-height: 1;
color: var(--color-text-secondary);
cursor: pointer;
transition: all var(--duration-fast) var(--ease-out);
}

.back-button:hover {
background: var(--color-bg-tertiary);
color: var(--color-text-primary);
}

.back-button svg {
width: 16px;
height: 16px;
}

.lesson-title {
font-family: var(--font-serif);
font-size: var(--text-xl);
font-weight: 500;
color: var(--color-text-primary);
letter-spacing: var(--tracking-tight);
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
line-height: 1;
margin: 0;
padding-bottom: 2px;
}

@media (max-width: 768px) {
.lesson-header {
padding: var(--space-3) var(--space-4);
}

.lesson-title {
font-size: var(--text-lg);
}
}

@media (max-width: 640px) {
.back-button span {
display: none;
}
}
</style>
Loading