Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 41 additions & 62 deletions docs/domains/all-parsing-and-markdown-logic.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,10 @@ This document provides a comprehensive analysis of all parsing and markdown proc
│ └── postProcessHtml() - DOM manipulation
├── 🌐 CLIENT-SIDE PROCESSING
│ ├── chat.html streaming logic
│ │ ├── formatText() - server-first, client-fallback
│ │ ├── clientMarkdownFallback() - minimal parser
│ │ └── preserveEnrichments() - mirror server placeholders
│ │
│ └── markdown-utils.js (MU namespace)
│ ├── normalizeInlineOrderedLists() - list marker fixing
│ ├── promoteLikelyJavaBlocks() - code fence promotion
│ ├── applyInlineEnrichments() - DOM enrichment rendering
│ └── createCitationPill() - citation UI components
│ └── Svelte frontend (frontend/src/lib/)
│ ├── markdown.ts - parseMarkdown() with DOMPurify sanitization
│ ├── javaLanguageDetection.ts - auto-detect Java code blocks
│ └── AssistantMarkdownBody.svelte - streaming-aware markdown rendering
├── 📊 STREAMING FLOW (GPT-5.2 → User)
│ ├── ChatService.streamAnswer() → Flux<String> (uses OpenAIStreamingService)
Expand Down Expand Up @@ -146,52 +140,36 @@ Return ProcessedMarkdown

### 2. Client-Side Processing Components

#### chat.html Streaming Logic
#### Svelte Frontend Markdown Processing

**File**: `src/main/resources/static/chat.html`
**File**: `frontend/src/lib/services/markdown.ts`

**Streaming Flow**:
```
User Input
fetch('/api/chat/stream')
SSE Event Processing
Buffer tokens (10 tokens/100ms)
normalizeDelta() - clean token joins
formatText() - markdown processing
DOM updates with debouncing
Final UnifiedMarkdownService.process()
```
**Purpose**: Client-side markdown parsing with DOMPurify sanitization for the Svelte frontend.

**Key Functions**:
- `formatText()` - Tries server processing first, falls back to client
- `clientMarkdownFallback()` - Minimal client parser
- `preserveEnrichments()` - Mirrors server placeholder system
- `upgradeCodeBlocks()` - Safe code block enhancement
- `parseMarkdown(markdownText, isStreaming)` - Main entry point, parses markdown to sanitized HTML
- `escapeHtml()` - Safe HTML escaping for inline code
- `createEnrichmentExtension()` - Custom marked extension for `{{type:content}}` markers

#### markdown-utils.js (MU)
**Streaming-Aware Parsing**:
- Uses separate `COMPLETE_MARKDOWN_PARSER` and `STREAMING_MARKDOWN_PARSER` instances
- `isStreaming` flag enables graceful handling of incomplete enrichment markers
- DOMPurify sanitization for XSS protection

**File**: `src/main/resources/static/js/markdown-utils.js`
**Java Language Detection**:

**Purpose**: Shared utilities for consistent markdown processing across views.
**File**: `frontend/src/lib/services/javaLanguageDetection.ts`

**Key Functions**:
- `normalizeInlineOrderedLists()` - Fixes list markers in prose
- `promoteLikelyJavaBlocks()` - Promotes Java code to fenced blocks
- `applyInlineEnrichments()` - Renders enrichment cards
- `createCitationPill()` - Citation UI components
- `createCitationsRow()` - Citation collections
- `applyJavaLanguageDetection()` - Auto-detects Java code in unmarked blocks
- Adds `language-java` class for syntax highlighting

**Component Integration**:

**File**: `frontend/src/lib/components/AssistantMarkdownBody.svelte`

**List Processing**:
- Supports: `1. 2. 3.`, `i. ii. iii.`, `a. b. c.`, `- * + • → ▸ ◆ □ ▪`
- Requires trigger phrases: `:`, `such as`, `include`, etc.
- Handles nested lists with colon notation
- Renders assistant message markdown with streaming support
- Applies Java language detection and syntax highlighting
- Shows streaming cursor animation during message assembly

### 3. Streaming and GPT-5.2 Processing

Expand Down Expand Up @@ -233,11 +211,11 @@ ChatMemory persistence
4. Structured citations and enrichments extracted
5. HTML rendering with proper escaping

**Client-Side**:
**Client-Side (Svelte)**:
1. SSE event processing
2. Progressive markdown rendering
3. Enrichment card injection
4. Code syntax highlighting
2. Progressive markdown rendering via `parseMarkdown()`
3. Enrichment card injection (client-side enrichment extension)
4. Code syntax highlighting (Prism.js)
5. Citation pill rendering

### 4. Data Structures and Types
Expand Down Expand Up @@ -503,20 +481,21 @@ GPT-5.2 (tokens)

### What processes what, where, and when

- Markdown: server authoritative (`UnifiedMarkdownService.process`) during streaming flushes and once at completion; client fallback only if server unavailable.
- Code blocks: server pre-normalizes/AST → `<pre><code class="language-...">`; client adds classes if missing + copy + highlight.
- HTML: server escapes raw HTML and adds structural classes (no regex); client only injects returned HTML and creates DOM components.
- Paragraphs: soft=`\n` (space in paragraphs), hard=`<br />`; server may split long paragraphs; client doesn’t re-paragraph.
- Streaming: server buffers (10/100ms) → SSE; client debounces and calls structured render; server processes once more on completion for persistence.
- Markdown (server-side endpoints): server processes via `UnifiedMarkdownService.process` for REST endpoints; returns HTML with citations/enrichments.
- Markdown (Svelte frontend): client-side `parseMarkdown()` with DOMPurify; streaming-aware enrichment handling; no server round-trip during streaming.
- Code blocks: server pre-normalizes/AST → `<pre><code class="language-...">`; client adds Java language detection + Prism highlight.
- HTML: server escapes raw HTML and adds structural classes; client sanitizes with DOMPurify.
- Paragraphs: soft=`\n` (space in paragraphs), hard=`<br />`; server may split long paragraphs; client doesn't re-paragraph.
- Streaming: server buffers (10/100ms) → SSE; Svelte client renders progressively with `parseMarkdown(markdownText, true)`; server processes once more on completion for persistence.

### Server vs Client boundaries (single source of truth)
### Server vs Client boundaries

- Server (authoritative)
- Markdown-to-HTML rendering, enrichment card generation, inline list normalization, code fence normalization, XSS escaping, final persisted representation.
- Server (REST endpoints)
- Markdown-to-HTML rendering for API consumers, enrichment card generation, inline list normalization, code fence normalization, XSS escaping, final persisted representation.

- Client (presentation-only)
- Streaming assembly, debounced asks to server for HTML, syntax highlighting, copy buttons, citation pills for inline anchors, gentle UX flourishes (cursor, loading dots).
- Minimal, conservative fallback markdown shaping only when server endpoints fail.
- Svelte Client (frontend)
- Streaming SSE assembly, client-side markdown rendering via `parseMarkdown()`, enrichment extension processing, syntax highlighting, copy buttons, citation pills, UX flourishes (cursor, loading dots).
- Two parser modes: streaming (graceful incomplete markers) and complete (strict parsing).

### Known issues, duplications, and rough edges

Expand Down
4 changes: 4 additions & 0 deletions frontend/src/lib/services/markdown.ts
Original file line number Diff line number Diff line change
Expand Up @@ -424,6 +424,10 @@ const STREAMING_MARKDOWN_PARSER = createMarkdownParser(true);
* Parse markdown to sanitized HTML. SSR-safe - no DOM APIs used.
* Uses DOMPurify for sanitization. Use this in `$derived` for reactive markdown rendering.
*
* @param markdownText - The markdown content to parse. Null/undefined returns empty string.
* @param isStreaming - When true, handles incomplete enrichment markers gracefully during
* streaming (e.g., "{{hint: some text}" without closing braces). Defaults to false.
* @returns Sanitized HTML string, or empty string on null/undefined input or parse failure.
* @throws Never throws - returns empty string on parse failure and logs error in dev mode
*/
export function parseMarkdown(
Expand Down