fix(anthropic): default max_tokens to the model's output ceiling (#849)#853
Open
tombeckenham wants to merge 2 commits into
Conversation
Anthropic's Messages API requires `max_tokens`, so the text adapter must always send a value. It previously hard-coded `?? 1024` when the caller didn't pass one, silently truncating any non-trivial generation mid-stream with `stop_reason: "max_tokens"`. Now default to the resolved model's real `max_output_tokens` from model-meta (e.g. 64K Sonnet, 128K Opus), falling back to 64K for unrecognized ids. `max_tokens` is a ceiling, not a reservation, so this costs nothing extra. Also log a warning when a response is truncated while using the defaulted cap, so it isn't silently read as the model "doing nothing"; callers that set `max_tokens` explicitly are unaffected. The new id -> max_output_tokens map is kept in lockstep with ANTHROPIC_MODELS by `scripts/sync-provider-models.ts`, so a freshly-synced model resolves to its real ceiling rather than the fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughThe Anthropic adapter now defaults ChangesAnthropic max token defaulting
Sequence Diagram(s)sequenceDiagram
participant Caller
participant text_ts as text.ts
participant model_meta_ts as model-meta.ts
participant Anthropic_Messages_API as Anthropic Messages API
participant logger
Caller->>text_ts: omit modelOptions.max_tokens
text_ts->>model_meta_ts: getAnthropicDefaultMaxTokens(this.model, { stream })
text_ts->>Anthropic_Messages_API: send max_tokens
Anthropic_Messages_API->>text_ts: stop_reason = "max_tokens"
text_ts->>logger: warn when max_tokens was defaulted
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 markdownlint-cli2 (0.22.1).changeset/anthropic-max-tokens-default.mdmarkdownlint-cli2 wrapper config was not available before execution docs/adapters/anthropic.mdmarkdownlint-cli2 wrapper config was not available before execution Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🚀 Changeset Version Preview8 package(s) bumped directly, 4 bumped as dependents. 🟥 Major bumps
🟨 Minor bumps
🟩 Patch bumps
|
|
View your CI Pipeline Execution ↗ for commit aee6dd1
☁️ Nx Cloud last updated this comment at |
@tanstack/ai
@tanstack/ai-angular
@tanstack/ai-anthropic
@tanstack/ai-client
@tanstack/ai-code-mode
@tanstack/ai-code-mode-skills
@tanstack/ai-devtools-core
@tanstack/ai-elevenlabs
@tanstack/ai-event-client
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-groq
@tanstack/ai-isolate-cloudflare
@tanstack/ai-isolate-node
@tanstack/ai-isolate-quickjs
@tanstack/ai-mcp
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-openrouter
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-utils
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/openai-base
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/adapters/anthropic.md`:
- Around line 139-142: The `max_tokens` default section in the Anthropic adapter
docs skips a heading level and breaks the document hierarchy. Update the heading
used for this section in `docs/adapters/anthropic.md` from the current `####`
level to `###` so it sits correctly between `Model Options` and `Thinking
(Extended Thinking)`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c9a19c83-f27b-450b-a500-404090524534
📒 Files selected for processing (9)
.changeset/anthropic-max-tokens-default.mddocs/adapters/anthropic.mddocs/config.jsonpackages/ai-anthropic/src/adapters/text.tspackages/ai-anthropic/src/model-meta.tspackages/ai-anthropic/tests/anthropic-adapter.test.tspackages/ai-anthropic/tests/model-meta.test.tspackages/ai/skills/ai-core/adapter-configuration/SKILL.mdscripts/sync-provider-models.ts
Comment on lines
+139
to
+142
| #### `max_tokens` default | ||
|
|
||
| Anthropic's Messages API _requires_ `max_tokens` on every request, so the adapter always sends a value. When you don't set `modelOptions.max_tokens`, it defaults to the selected model's full output ceiling (`max_output_tokens` from the model metadata — e.g. 64K for Sonnet, 128K for Opus), falling back to a safe constant for unrecognized models. `max_tokens` is a ceiling, not a reservation — billing is on tokens actually generated — so this default costs nothing extra and avoids the silent mid-response truncation (`stop_reason: "max_tokens"`) that a low default would cause. Set `max_tokens` explicitly only when you want to _cap_ output below the model ceiling. If a response is truncated while using the default cap, the adapter logs a warning (visible with [debug logging](../advanced/debug-logging) enabled). | ||
|
|
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Fix heading level to maintain document hierarchy.
The new #### max_tokens default heading skips a level. It follows ## Model Options and precedes ### Thinking (Extended Thinking), so it should be ### max_tokens default to increment by one level at a time.
📝 Proposed fix
-#### `max_tokens` default
+### `max_tokens` default📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| #### `max_tokens` default | |
| Anthropic's Messages API _requires_ `max_tokens` on every request, so the adapter always sends a value. When you don't set `modelOptions.max_tokens`, it defaults to the selected model's full output ceiling (`max_output_tokens` from the model metadata — e.g. 64K for Sonnet, 128K for Opus), falling back to a safe constant for unrecognized models. `max_tokens` is a ceiling, not a reservation — billing is on tokens actually generated — so this default costs nothing extra and avoids the silent mid-response truncation (`stop_reason: "max_tokens"`) that a low default would cause. Set `max_tokens` explicitly only when you want to _cap_ output below the model ceiling. If a response is truncated while using the default cap, the adapter logs a warning (visible with [debug logging](../advanced/debug-logging) enabled). | |
| ### `max_tokens` default |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 139-139: Heading levels should only increment by one level at a time
Expected: h3; Actual: h4
(MD001, heading-increment)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/adapters/anthropic.md` around lines 139 - 142, The `max_tokens` default
section in the Anthropic adapter docs skips a heading level and breaks the
document hierarchy. Update the heading used for this section in
`docs/adapters/anthropic.md` from the current `####` level to `###` so it sits
correctly between `Model Options` and `Thinking (Extended Thinking)`.
Source: Linters/SAST tools
…ult (#849) The #849 default of the model's full output ceiling broke the non-streaming `structuredOutput()` path: the Anthropic SDK refuses a non-streaming request whose `max_tokens` could exceed its 10-minute timeout (~21,333 tokens), so `chat({ outputSchema })` on any fallback-path model threw "Streaming is required for operations that may take longer than 10 minutes". `getAnthropicDefaultMaxTokens(model, { stream })` now clamps the default to `ANTHROPIC_MAX_NONSTREAMING_TOKENS` when `stream: false`; the streaming chat path keeps the model's full ceiling. An explicit oversized `max_tokens` still surfaces the SDK's "use streaming" error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/ai-anthropic/tests/anthropic-adapter.test.ts`:
- Line 5: The imports in anthropic-adapter.test.ts are out of the required order
and will fail the import/order lint rule. Reorder the value import from
model-meta so it comes before the type-only import from adapters/text, keeping
the existing symbols like ANTHROPIC_MAX_NONSTREAMING_TOKENS and the TextAdapter
type import intact.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: cfac1478-f2ab-4ae6-ada8-6fcb471edf34
📒 Files selected for processing (6)
.changeset/anthropic-max-tokens-default.mddocs/adapters/anthropic.mdpackages/ai-anthropic/src/adapters/text.tspackages/ai-anthropic/src/model-meta.tspackages/ai-anthropic/tests/anthropic-adapter.test.tspackages/ai-anthropic/tests/model-meta.test.ts
✅ Files skipped from review due to trivial changes (2)
- .changeset/anthropic-max-tokens-default.md
- docs/adapters/anthropic.md
| import { chat, type Tool, type StreamChunk } from '@tanstack/ai' | ||
| import { AnthropicTextAdapter } from '../src/adapters/text' | ||
| import type { AnthropicTextProviderOptions } from '../src/adapters/text' | ||
| import { ANTHROPIC_MAX_NONSTREAMING_TOKENS } from '../src/model-meta' |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Reorder import to satisfy import/order.
ESLint flags this: the value import of ../src/model-meta should precede the type import of ../src/adapters/text. This will fail lint in CI.
🧰 Tools
🪛 ESLint
[error] 5-5: ../src/model-meta import should occur before type import of ../src/adapters/text
(import/order)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-anthropic/tests/anthropic-adapter.test.ts` at line 5, The imports
in anthropic-adapter.test.ts are out of the required order and will fail the
import/order lint rule. Reorder the value import from model-meta so it comes
before the type-only import from adapters/text, keeping the existing symbols
like ANTHROPIC_MAX_NONSTREAMING_TOKENS and the TextAdapter type import intact.
Source: Linters/SAST tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Changes
Closes #849.
@tanstack/ai-anthropic's text adapter defaulted the Anthropicmax_tokensrequest field to 1024 when the caller didn't pass one. Anthropic's Messages API requiresmax_tokens, so the adapter must send some value — but 1024 is far below what the targeted models can produce, so any non-trivial generation (codegen, agentic tool flows, long-form output) silently truncated mid-stream withstop_reason: "max_tokens", looking like the run "did nothing" rather than "ran out of output budget".modelOptions.max_tokens, default to the resolved model's realmax_output_tokensfrommodel-meta.ts(e.g. 64K Sonnet, 128K Opus) via a newgetAnthropicDefaultMaxTokens(model).max_tokensis a ceiling, not a reservation (billing is per token generated), so this costs nothing unless the model genuinely produces more. Unrecognized model ids fall back to 64K (the current mainstream Claude tier's ceiling). The shared mapper meansstructuredOutputbenefits too.stop_reason: "max_tokens"and the caller didn't setmax_tokens, the adapter logs alogger.warnwith actionable guidance so the truncation isn't silent. Explicit caller caps are left alone.model-meta.tsis auto-synced byscripts/sync-provider-models.ts. That script now also maintains the new id →max_output_tokensmap (newmaxOutputTokensMapNameconfig +addToObjectMaphelper), so a freshly-synced model resolves to its real ceiling rather than the fallback — staying in lockstep withANTHROPIC_MODELS.structuredOutput()path: the Anthropic SDK refuses a non-streaming request whosemax_tokenscould exceed its 10-minute timeout (~21,333 tokens), so everychat({ outputSchema })on a fallback-path model threw "Streaming is required for operations that may take longer than 10 minutes".getAnthropicDefaultMaxTokens(model, { stream })now clamps the default toANTHROPIC_MAX_NONSTREAMING_TOKENS(21K) whenstream: false; the streaming chat path keeps the model's full ceiling. An explicit oversizedmax_tokensstill surfaces the SDK's "use streaming" error, which is the correct signal. This was caught by two failing E2E specs (anthropic-structured-usage,structured-output-middleware).This is Anthropic-specific;
@tanstack/ai-openaiand the other adapters treat token limits as optional and have no equivalent floor.Tests
getAnthropicDefaultMaxTokensunit tests (known models, unknown 64K fallback, never-1024; plus the non-streaming clamp, no-clamp-when-already-below, and streaming-unaffected cases).max_tokens; the non-streaming structured-output request sendsstream: falsewith the clampedmax_tokens.Docs / skill
docs/adapters/anthropic.md(+docs/config.jsonupdatedAt) and theadapter-configurationskill document the new defaulting behavior, including the non-streaming structured-output clamp.✅ Checklist
pnpm run test:pr.🚀 Release Impact
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
max_tokensto the selected model’s metadata-derived output limit when it isn’t provided.Bug Fixes
Documentation
Tests