v0.4.0

Changelog

• Updated the llama.cpp base through upstream commit 32e789fdf. This includes
  EAGLE3 and expanded MTP support; new multimodal models and video input;
  LFM2/LFM2.5 tool parsing; router, API, and Web UI updates; the ET-SOC backend;
  and updates across the existing compute backends.
• Added KVarN target-context KV-cache compression through --cache-type-k and
  --cache-type-v values kvarn2, kvarn3, kvarn4, kvarn5, kvarn6,
  and kvarn8. It supports independent K/V widths, Qwen3.6 and Gemma 4,
  unified and non-unified caches, separate SWA cache types, CPU/CUDA stores,
  and native CUDA FlashAttention. Draft and auxiliary contexts continue to use
  standard cache types; unsupported placements fail closed or use an explicit
  bit-width-matched fallback.
• The standard KV-cache surface now includes q2_0, q2_1, q3_0, q3_1,
  and q6_1 alongside q6_0, with CPU/CUDA quantization,
  SET_ROWS/GET_ROWS, CUDA MMQ/vec-dot, and FlashAttention vector coverage.
  Cache-facing q2_0 uses GGML_TYPE_Q2_0S, keeping it distinct from
  upstream's serialized Q2_0 weight format.
• New KV cache precision tail (KVCPT) controls --kv-tail-tokens and --kv-tail-type keep the newest
  attention-visible entries exact in F16 or BF16 for standard quantized and
  KVarN target caches. Partial tails use compact overlays and fully covered
  groups can use native exact storage. Tail routing respects attention bias and
  cache placement, while state save/restore, prompt-cache reuse, sequence
  operations, and server handoff preserve the same model. Restores are
  transactional, and incompatible v0.3.1 sessions fail closed.
• Scoped --cache-ram to prompt-cache storage instead of active context
  checkpoint policy. Prompt-cache saves are byte-counted against the host
  budget before commit, KV rollback aligns to multimodal chunk boundaries,
  text-only slot save/restore works when an mtmd projector is loaded, and MTP
  prompt-cache reuse no longer requires regular checkpoints.
• Replaced the fork DFlash implementation with upstream draft-dflash.
  The speculative type uses upstream's draft-dflash name, and draft GGUFs
  must use upstream's dflash architecture, metadata keys, tensor names, and
  tokenizer contract. Bee retains a default-on profit-based draft-depth
  controller and reasoning-loop guard while using upstream token, sampler, and
  checkpoint behavior. Without --spec-draft-n-max, the draft limit comes
  from dflash.block_size - 1; an explicit value takes precedence.
• Removed TurboQuant/TCQ cache formats, TQ3/TQ4 weight formats, DDTree,
  CopySpec, the fork DFlash ring/capture/tape and reduced-verifier paths, the
  fringe controller, and their private arguments and environment variables.
  The turbo2, turbo3, and turbo4 cache names, including TCQ variants,
  warn and redirect by width to KVarN for target caches or a standard low-bit
  type for draft caches. Legacy TQ GGUF type IDs fail with a re-quantization
  error; copyspec, suffix, and recycle fail with migration guidance to
  draft-dflash or upstream n-gram modes.
• Replaced the CUDA FlashAttention quant build policy with 50 standard vector
  pairs and 15 balanced KVarN fast-decode pairs by default. Standard quant
  pairs follow the KVarN bit-pair rules; homogeneous F16 and BF16 pairs retain
  native precision-tail support.
  GGML_CUDA_FA_ALL_QUANTS=ON expands these to 169 standard pairs and all 36
  ordered KVarN pairs. The default-on GGML_CUDA_KVARN option is the single
  KVarN compilation gate for CUDA and HIP; disabling it omits the dedicated
  kernels and template instances.
• Hardened server control and router APIs. /v1/chat/completions/control
  reports success only when an active reasoning sampler accepts the transition.
  GET /models is read-only and sanitized; authenticated refresh uses
  POST /models/reload. Hugging Face tokens reach child processes only through
  HF_TOKEN and are excluded from arguments, presets, logs, and responses.
• Updated release packaging with CUDA 12.4/13.1 assets, upstream-DFlash and
  KVarN container metadata, ROCm/HIP shuffle compatibility, Windows CPU OpenMP
  runtime packaging, and updated Intel SYCL images. Every platform build,
  package, container, and publication step uses the same resolved source SHA;
  publication stops if the selected branch or tag moves.

macOS:

• macOS Apple Silicon

Linux:

• Ubuntu x64 CPU
• Ubuntu arm64 CPU
• Ubuntu x64 CUDA 12.4
• Ubuntu x64 CUDA 13.1
• Ubuntu x64 Vulkan
• Ubuntu x64 ROCm 7.2
• Ubuntu x64 SYCL

Windows:

• Windows x64 CPU
• Windows x64 Vulkan
• Windows x64 SYCL
• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs
• Windows x64 HIP

Docker:

• CPU: docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-v0.4.0
• CUDA: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.4.0
• CUDA 12: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-v0.4.0
• CUDA 13: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.4.0
• ROCm: docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-v0.4.0
• Vulkan: docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-v0.4.0
• SYCL: docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-v0.4.0

Browse all container images

v0.4.0 Preview

Warning

Rolling preview build from v0.4.0 at c6f7eef252b2. Assets and moving Docker tags are replaced by newer successful branch builds. Last updated: 18/07/2026 20:49.

Changelog

All changes below are relative to v0.3.1.

• Merged upstream llama.cpp through merge 827bfda66, whose upstream parent is
  32e789fdf, and based BeeLlama's maintained extensions on its current model,
  memory, speculative, and server architecture. Notable inherited upstream
  changes include EAGLE3 speculative decoding with backend sampling (plus
  Bee-added Minimax2 EAGLE3 support), Gemma 4 MTP including E2B/E4B assistants
  and unified conversion fixes, Granite 4 Vision, multimodal video input with
  ffmpeg in the released image, Qwen-VL frame merge support, Cohere2-MoE
  support, Mistral-Medium-3.5 conversion, the unified LFM2/LFM2.5 tool parser,
  the router model management API and -hf preset rework, optional server
  prompt logging, the */input_tokens token-counting API, Web UI improvements
  including PWA support, CUDA 13.3 release images, HIP gfx1152/gfx1153
  support, and broad backend work across CUDA, Vulkan, SYCL, Metal, WebGPU,
  OpenCL, OpenVINO, and CPU.
• Added KVarN target-context KV-cache compression through --cache-type-k /
  --cache-type-v pseudo types kvarn2, kvarn3, kvarn4, kvarn5,
  kvarn6, and kvarn8. K and V widths are selected independently across all
  2/3/4/5/6/8-bit pairs, one-sided KVarN inputs are normalized to a concrete
  pair, and draft/auxiliary contexts stay on normal cache types. Every KVarN
  group keeps the exact 128-token sink and newest 128-token suffix around its
  compressed body.
• Wired KVarN into Qwen3.6 and Gemma 4 target memory paths, including
  128/256/512-dimensional K/V heads, unified and non-unified KV storage,
  SWA-layer coverage for single-stream caches through a sliding-window KVarN
  ring with explicit --cache-type-k-swa / --cache-type-v-swa precision
  overrides, prompt-cache state save/restore with compact group-range
  serialization, and bit-width-matched fallback cache types for layers that
  cannot use KVarN records. Unsupported placements fail closed.
• Added native KVarN runtime support: CPU/CUDA store ops, rotated-domain
  attention, and CUDA FlashAttention that consumes KVarN views directly —
  specialized split and SWA-vector decode kernels with descriptor-native MMA
  fallback, no graph-level F16 materialization — plus bounded windowed
  prefill, parallel streams, ROCm/HIP and guarded Vulkan store support, and
  llama-bench KVarN cache names, route counters, and opt-in --kv-memory
  accounting.
• Added new standard quantized KV-cache types q2_0, q2_1, q3_0, q3_1,
  and q6_1 alongside the existing q6_0, with full CPU/CUDA kernel coverage
  including FlashAttention vector paths. Cache-facing q2_0 is internally
  GGML_TYPE_Q2_0S to avoid a format collision with upstream's Q2_0 weight
  type; saved sessions from earlier Bee releases are not compatible with the
  re-slotted cache type IDs.
• Added exact tails for quantized caches: --kv-tail-tokens and
  --kv-tail-type keep the newest attention-visible entries exact in F16 or
  BF16 over both standard quantized and KVarN target caches, either as a
  compact exact overlay or, when a request covers a group's full visibility
  window, as native exact storage. Mixed-precision tail attention routes are
  implemented across GGML backends and respect per-layer relative position
  bias in models such as T5.
• Made standard and KVarN state restoration transactional: tensor writes and
  metadata are staged until the complete frame validates, then committed once.
  Failed deferred exact-tail copies are rolled back and are never observed as
  successful by save, decode, or server callers.
• Scoped --cache-ram to prompt-cache storage only, so context-checkpoint
  policy no longer derives from it; prompt-cache saves are byte-counted
  against the host budget before commit, KV rollback aligns to multimodal
  chunk boundaries, and MTP prompt-cache reuse works without regular
  checkpoints.
• Replaced the fork DFlash stack with upstream draft-dflash. The speculative
  type uses upstream's draft-dflash name, and draft GGUFs must use upstream's
  dflash architecture, metadata, and tensor names. Corrected the upstream
  draft-context merge to own a normal KV cache, fixing a null-memory crash
  during draft graph reservation at startup.
• The profit-only adaptive draft-depth controller and the server
  reasoning-loop guard now run on upstream token/sampler and checkpoint
  behavior. The profit controller bootstraps at the maximum useful depth,
  settles on the fastest measured depth, and remains default-on. An omitted
  DFlash --spec-draft-n-max uses the drafter's trained block depth
  (dflash.block_size - 1, normally 15) instead of upstream's default 3;
  explicit values still win.
• Removed TurboQuant/TCQ cache types, TQ weight formats, DDTree, CopySpec, the
  DFlash GPU ring/capture/tape paths, the fringe adaptive controller, and all
  fork-only DFlash environment variables and arguments. The turbo2,
  turbo3, and turbo4 cache names and their TCQ variants are redirected
  with warnings to the same-width KVarN presets for target caches or a
  standard low-bit type for draft caches. GGUF files marked as the former
  TQ3/TQ4 weight formats fail early with a re-quantization error, and the
  removed copyspec, suffix, and recycle speculative type names produce
  migration errors pointing to draft-dflash or upstream n-gram modes.
• Replaced the FlashAttention quant build policy with a 103-pair default
  standard vector matrix and a 169-pair GGML_CUDA_FA_ALL_QUANTS matrix.
  KVarN keeps 15 balanced fast decode pairs by default and 36 with ALL. The
  default-on GGML_CUDA_KVARN option is KVarN's only CUDA compilation gate;
  disabling it omits all dedicated KVarN kernels and templates.
• Hardened realtime reasoning control: /v1/chat/completions/control reports
  success only when an active reasoning sampler actually accepts the
  transition; inactive, completed, and unknown completions report failure.
• Hardened router management. GET /models is read-only and returns sanitized
  model identity, capability, source, and status data. Refresh is now
  POST /models/reload, protected by the normal API-key middleware when keys
  are configured. Hugging Face tokens are removed from child arguments,
  presets, logs, and public responses and reach child processes only through
  the HF_TOKEN environment variable.
• Added KLD baseline tooling to llama-perplexity: --save-all-logits writes
  a compressed, versioned base-model log-probability file for later
  --kl-divergence runs, measuring KV-cache format quality under matched
  model, corpus, context, and batching.
• Updated release packaging: CUDA 12.4/13.1 assets, container metadata
  advertising upstream DFlash and KVarN, and publication gated on portable
  CPU, Windows CUDA default, CUDA ALL_QUANTS, and CUDA-without-KVarN
  behavioral checks at one exact source SHA. Shipped builds add ROCm/HIP
  shuffle compatibility, Windows CPU OpenMP runtime packaging, and Intel SYCL
  Docker images on compute runtime 26.18 / IGC v2.34.4.
• RTX 3090 release validation measured Qwen3.6-27B KVarN4 KLD 0.002400,
  Gemma-4-31B KVarN4 KLD 0.402575 versus q5_0 0.415296, and upstream
  DFlash at 73.81 median decode tokens/s with 40.1% draft acceptance —
  12.61% slower than the removed fork DFlash stack in the matched
  three-prompt server gate.

macOS:

• macOS Apple Silicon

Linux:

• Ubuntu x64 CPU
• Ubuntu arm64 CPU
• Ubuntu x64 CUDA 12.4
• Ubuntu x64 CUDA 13.1
• Ubuntu x64 Vulkan
• Ubuntu x64 ROCm 7.2
• Ubuntu x64 SYCL

Windows:

• Windows x64 CPU
• Windows x64 Vulkan
• Windows x64 SYCL
• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs
  -...

v0.3.2 Preview

Warning

Rolling preview build from v0.3.2 at fe67745dbbd5. Assets and moving Docker tags are replaced by newer successful branch builds. Last updated: 10/07/2026 21:29.

Changelog

• Merged a newer upstream llama.cpp master after the current main baseline. Notable inherited changes include Granite 4 Vision, Gemma 4 MTP including E2B/E4B assistants, Gemma 4 unified conversion and audio-projector fixes, multimodal video input with ffmpeg in the released image, Qwen-VL frame merge support, Mistral-Medium-3.5 conversion, the unified LFM2/LFM2.5 tool parser and reasoning round-trip fixes, speculative vocab-compatibility checks, the placeholder-bitmap token counting and */input_tokens API, optional server prompt logging, KV-cache cell-sharing/copy-avoidance fixes, GGML_OP_COL2IM_1D, ggml 0.14.0, CUDA 13.3 release images, HIP gfx1152/gfx1153 support, and backend/UI improvements across WebGPU, Vulkan, SYCL, Metal, CPU, and the Web UI.
• Added experimental KVarN KV-cache compression for target contexts through --cache-type-k / --cache-type-v pseudo types kvarn2, kvarn3, kvarn4, kvarn5, kvarn6, and kvarn8. K and V can be selected independently across all 2/3/4/5/6/8-bit pairs, with one-sided KVarN inputs normalized to a concrete pair and draft/auxiliary contexts kept on normal cache types.
• Wired KVarN into Qwen3.6 and Gemma 4 target memory paths, including 128/256/512-dimensional K/V heads, unified and non-unified KV storage, supported single-stream iSWA/SWA rings, explicit --cache-type-k-swa / --cache-type-v-swa overrides for SWA-layer precision, prompt-cache state save/restore, and bit-width-matched fallback cache types for layers that cannot use KVarN records. Unsupported placements fail closed unless normal-KV fallback is explicitly enabled.
• Added native KVarN runtime support: CPU/CUDA store ops, native CUDA FlashAttention view consumption without graph-level F16 materialization, bounded windowed prefill for Qwen/Gemma/SWA paths, compact group-range prompt-cache state, ROCm/HIP low-shared-memory store support, guarded Vulkan store support, and llama-bench KVarN cache names.
• Added new KV/cache types: turbo4_tcq plus standard quantized q2_0, q2_1, q3_0, q3_1, and q6_1, with CPU/CUDA quantize/dequantize, MMQ/vec-dot, FlashAttention vec coverage, and use as KVarN non-KVarN-layer fallback targets where appropriate.
• Reworked CUDA/HIP/MUSA FlashAttention cache-type coverage. The no-flag vec build now focuses on the recommended q-cache and KVarN fallback pairs, GGML_CUDA_FA_HALF_QUANTS adds the K>=V half matrix for broader Turbo/TCQ experiments, GGML_CUDA_FA_ALL_QUANTS keeps the full matrix, and runtime diagnostics report or optionally ignore uncompiled pairs instead of failing opaquely. Turbo/TCQ route planning now validates effective K/V types, fixes the issue #41 CUDA paths, and keeps Gemma-sized D256/D512 mixed-Turbo routes away from unsupported vec paths.
• Improved DFlash serving relative to v0.3.1: tensor-split Meta target placement keeps auto-detected drafters on compatible placement, reduced verification is gated by backend capability, DFlash mixed with other speculative types is detected order-insensitively for rollback planning, and MTP draft policy is isolated from DFlash-only adaptive logic.
• Reduced DFlash/recurrent and prompt-cache memory pressure. Flat DFlash on recurrent targets now uses recurrent-only rollback state instead of dead attention/KV backup streams; prompt-cache saves are prepared and byte-counted before commit; cached prompt entries copy only the newest fitting context checkpoints; and --cache-ram remains scoped to prompt-cache storage rather than active context-checkpoint policy.
• Improved the DFlash profit adaptive draft-max controller so cold starts hold the maximum useful depth while lower depths are characterized through gated probes, then demote only when measurements show a lower depth is actually faster.
• Hardened shipped builds and release artifacts: ROCm/HIP shuffle compatibility, HIP/MUSA KVarN build fixes, Windows CPU OpenMP runtime packaging, Intel SYCL Docker images updated from compute runtime 25.40 / IGC v2.20.5 to compute runtime 26.18 / IGC v2.34.4, and release downloads/notes filtered to final packages and runtime artifacts.

macOS:

• macOS Apple Silicon

Linux:

• Ubuntu x64 CPU
• Ubuntu arm64 CPU
• Ubuntu x64 CUDA 12.4
• Ubuntu x64 CUDA 13.1
• Ubuntu x64 Vulkan
• Ubuntu x64 ROCm 7.2
• Ubuntu x64 SYCL

Windows:

• Windows x64 CPU
• Windows x64 Vulkan
• Windows x64 SYCL
• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs
• Windows x64 HIP

Docker:

• CPU: docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-preview-v0.3.2
• CUDA: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-preview-v0.3.2
• CUDA 12: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-preview-v0.3.2
• CUDA 13: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-preview-v0.3.2
• ROCm: docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-preview-v0.3.2
• Vulkan: docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-preview-v0.3.2
• SYCL: docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-preview-v0.3.2

Browse all container images

v0.3.1

Changelog

• Merged latest upstream llama.cpp master. This pulls in Gemma 4 12B and Gemma 4 unified multimodal support fixes, including non-causal vision, unified audio/vision projector handling, and FPE fixes; Qwen3.5 post-norm hidden-state behavior for MTP; CUDA KV-cache quantization preallocation and PDL race fixes; WebGPU FlashAttention refactoring with standardized quantization support; CPU backend improvements for RVV/SVE; lower-latency Metal command-buffer status polling; Mermaid diagram rendering and preview support in tools/ui; updated BoringSSL, SYCL documentation, save/load-state tests, Docker docs, and small CI/release maintenance.
• Repaired CUDA fused TurboQuant FlashAttention for same-type turbo2, turbo3, and turbo4 K/V caches. The fused MMA path now loads each supported format correctly, while mixed TurboQuant and TCQ pairs stay on the established non-fused paths; TurboQuant/TCQ partial KV offload now fails early instead of falling back to an incompatible CPU cache and reaching a scheduler crash. Added GGML_TURBO_FA_DEBUG=1 path diagnostics and regression coverage for the supported dispatch matrix.
• Updated release packaging and documentation. HIP/ROCm builds now include all quantized FlashAttention combinations, and the prebuilt binary and Docker image lists reflect the current release outputs.

macOS:

• macOS Apple Silicon

Linux:

• Ubuntu x64 CPU
• Ubuntu arm64 CPU
• Ubuntu x64 CUDA 12.4
• Ubuntu x64 CUDA 13.1
• Ubuntu x64 Vulkan
• Ubuntu x64 ROCm 7.2
• Ubuntu x64 SYCL

Windows:

• Windows x64 CPU
• Windows x64 SYCL
• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs
• Windows x64 HIP

Docker:

• CPU: docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-v0.3.1
• CUDA: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.3.1
• CUDA 12: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-v0.3.1
• CUDA 13: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.3.1
• ROCm: docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-v0.3.1
• Vulkan: docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-v0.3.1
• SYCL: docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-v0.3.1

Browse all container images

v0.3.0

Changelog

• Updated to a much newer llama.cpp base. The upstream refresh brings native MTP speculative decoding, parallel drafting and backend sampling work, the unified llama app, newer server/API behavior, the tools/ui Web UI restructure, model and converter additions, multimodal improvements, and backend gains across CUDA/HIP, Metal, Vulkan, SYCL, OpenCL, WebGPU, Hexagon, ZenDNN, and SpacemiT.
• Added usable MTP serving on the Bee tree. draft-mtp/mtp now has its own speculative path instead of being mixed with DFlash state, uses draft KV cache types, cleans up draft resources on sleep, works with target pre-norm hidden-state capture, and handles text requests when an mmproj is loaded.
• Cleaned up DFlash command-line behavior around canonical --spec-* arguments. DFlash can still be selected with --spec-type dflash or auto-detected from compatible draft GGUF metadata, but stale aliases such as --spec-dflash-default, --draft*, --draft-topk, --draft-model, --tree-budget, --dflash-max-slots, --spec-draft-replace, and --spec-replace were removed.
• Made DFlash defaults match the new upstream speculative surface without changing the practical DFlash defaults. Raw --spec-draft-n-max stays at upstream 3, DFlash raises the effective omitted draft max to 16 only after explicit or auto-detected DFlash, omitted --spec-draft-ctx-size still becomes 256, and DFlash no longer lowers target -b 2048 / -ub 512 unless the user asks for smaller batches.
• Made DFlash-only arguments safe around non-DFlash modes. DFlash-only controls warn and no-op for MTP or other speculative types, while still surviving long enough for server-side DFlash draft-model auto-detection when no explicit --spec-type dflash was passed.
• Expanded multi-slot DFlash serving. DFlash slots now default to server parallelism, --spec-dflash-max-slots caps them below -np when needed, uneven -np target batches are split safely, mixed speculative/non-speculative target batches are avoided, and flat multi-slot DFlash can use shared drafter batching with GGML_DFLASH_SHARED_DRAFT_BATCH=0 as the fallback switch.
• Reworked adaptive DFlash draft depth for live serving. The default profit controller now seeds and periodically remeasures a no-spec baseline, probes shallow/mid/full positive depths, backs off failed wake probes, avoids premature demotion, resets per-request state correctly, preserves compatible continuation state, and gates timing logs behind DFlash profiling.
• Added default-on device-aware DFlash GPU capture/tape/replay for split CUDA/ROCm target placement. Hidden capture, prefill capture, recurrent tape, conv replay, direct GDN replay, rollback copies, and synchronization now follow each layer's backend device, with CPU/eval-callback fallback and GGML_DFLASH_MULTI_GPU_TAPE=0 / GGML_DFLASH_ALLOW_MULTI_GPU_TAPE=0 kill switches.
• Hardened DFlash device placement. Explicit single draft-device placement pins the target output tensor before target load, auto-detected DFlash drafters stay single-device by default unless draft devices are explicit, iGPU backends are accepted for GPU paths, tensor-split/meta placement is guarded, CUDA helper calls preserve the caller device, and peer D2D copies are used where available.
• Hardened DFlash hidden-ring and prompt-cache state. The server guards GPU hidden-ring spans, discards stale DFlash ring checkpoints, skips DFlash checkpoints for uncached prompts, restores shared prefill capture state, and avoids requesting raw prompt logits from DFlash paths that should not read them.
• Reduced DFlash accept and verification overhead while failing closed on bad reduced-logit state. The accept path defers single-slot rollback sync and drafter KV maintenance, keeps the fused GDN 4D state fast path, adds CPU f16 out_prod fallback, and disables DFlash drafting after repeated invalid reduced-logits drafts instead of looping or corrupting state.
• Improved DFlash tool-call and reasoning behavior. DFlash can keep drafting before a lazy tool-call grammar actually constrains output, stale drafter KV state is cleared before long tool-call continuations, stable partial tool-call headers stream earlier, streaming reasoning deltas stay isolated, and streamed title generation suppresses leading thinking syntax.
• Kept flat DFlash available with multimodal serving while making unsupported combinations explicit. With --mmproj, Bee keeps flat DFlash usable, forces DFlash tree branch budget to 0, disables non-DFlash speculative modes, disables unsupported context-shift/cache-reuse paths, and fixes MTP text-only requests when an mmproj is present.
• Updated Qwen 3.5/3.6 speculative paths. Qwen gets per-layer KV heads, final-layer output gathering, stable MTP draft context behavior, and Qwen DDTree conv/GDN paths for tree verification.
• Removed legacy DDTree total-node semantics from the public CLI. Tree DFlash now uses the branch-only --spec-branch-budget model consistently, with --spec-draft-top-k controlling candidates per draft position and flat DFlash forcing top-k back to 1.
• Added new cache and quantization surface beyond v0.2.0. q6_0 is available as a KV/cache type, and Tom's TQ3_1S / TQ4_1S model weight formats are exposed through llama-quantize with non-conflicting serialized GGML type IDs; existing turbo2, turbo3, turbo4, turbo2_tcq, and turbo3_tcq cache types remain available on the newer base.
• Hardened backend behavior used by Bee features. HIP TCQ attention stays on the native vector path, ROCm can probe fused GDN support, D512 Flash Attention selection is available again, unsupported CPU BF16 scale and CPU Flash Attention types are rejected instead of misrunning, CUDA fatal-warning cases were fixed, and the build now requires C++17 for the common base.
• Improved server and API behavior inherited from upstream and Bee integration. The server reports prompt token counts in /slots, supports SSE ping intervals and HTTP ETags, exposes real-time reasoning interruption, handles router/model metadata fixes, adds built-in tools such as datetime, and keeps malformed tool-looking text out of final responses.
• Updated packaging and release workflows. Docker image links and labels were aligned for BeeLlama, SYCL package/release images were added and documented, release builds were split and cached more reliably, stale package builds are cancelled, and obsolete CUDA architecture options are rejected at configure time.
• Updated user documentation for the new release state, including DFlash args/defaults, removed aliases, adaptive Draft-Max, multi-GPU DFlash behavior, Turbo/TQ cache and weight formats, Docker images, and the upstream multi-GPU guide.
• Expanded regression coverage around DFlash default normalization, removed arg aliases, DFlash-only no-op behavior, DFlash auto-detection, multi-GPU policy helpers, per-layer capture/tape allocation, device-aware replay, CUDA device restoration, adaptive Draft-Max, reduced-logit failure handling, and tool-call/speculative boundaries.

macOS:

• macOS Apple Silicon

Linux:

• Ubuntu x64 CPU
• Ubuntu x64 Vulkan
• Ubuntu x64 ROCm 7.2
• Ubuntu x64 SYCL

Windows:

• Windows x64 CPU
• Windows x64 SYCL
• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs
• Windows x64 HIP

Docker:

• CPU: docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-v0.3.0
• CUDA: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.3.0
• CUDA 12: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-v0.3.0
• CUDA 13: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.3.0
• ROCm: docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-v0.3.0
• Vulkan: docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-v0.3.0
• SYCL: docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-v0.3.0

Browse all container images

v0.2.0

Changelog

• Added compatibility with upstream DFlash PR drafter GGUFs that use general.architecture = dflash. Bee now keeps this separate from the older dflash-draft schema, understands upstream metadata keys such as dflash.block_size and dflash.target_layer_ids, reads upstream tensor names, and keeps existing Bee/buun draft GGUF naming intact.
• Tightened DFlash draft model discovery and converter behavior. Bee now prefers exact sibling DFlash draft directories, supports nested dflash_config metadata, scopes Gemma4 tokenizer handling correctly, and logs clearer DFlash metadata warnings and summaries during conversion.
• Hardened recurrent memory, prompt-cache restore, and unified-KV scheduling. Recurrent resize now repairs its metadata after shrink/expand, the server shrinks recurrent state before prompt-cache save/load when it is safe, backup-sequence cleanup is tracked correctly, and non-parent tasks defer unified-KV admission so large pending prompts do not over-commit shared cells.
• Added richer DFlash diagnostics, profiling, and validation. GGML_DFLASH_PROFILE now exposes categorized summary/replay/copy/prefill/verify/trace logging, routine decode timing is hidden behind debug logging instead of always printing, the profit controller now logs when it disables speculative depth, drafter/target contract and input validation are stricter, and Bee also exposes targeted debug envs such as GGML_DFLASH_DEBUG, GGML_DFLASH_INPUT_DEBUG, GGML_DFLASH_CUDA_DEBUG, GGML_DFLASH_FORCE_CPU_CROSS, GGML_DFLASH_VERBOSE_CONTRACT, and GGML_DFLASH_CRASH_TRACE.
• Improved DFlash CUDA ordering and split-buffer correctness. Hidden capture, recurrent replay, backup copies, K/V projection-cache updates, and DFlash stream waits now use explicit ordering helpers and safer backend ownership checks instead of broader synchronization or wrong-buffer access.
• Added DFlash drafter K/V projection caching for the cross-attention window. Bee now keeps ring-backed drafter K/V state for recent target hidden-state windows, supports chronological D2D append/interleave on CUDA, excludes the unsafe parts from graph capture when needed, and falls back more safely on placements that cannot use the fast GPU path.
• Reworked DFlash prefill capture and flush handling. Prefill capture now uses per-slot and per-view plans, GPU staging buffers, source-aware CPU/GPU ring validity, suffix-span tracking across internal ubatches, graph-reuse keys for source/destination offsets, callback suppression for irrelevant ubatches, and fail-closed behavior for partial or mismatched captures.
• Hardened target hidden-state capture across Qwen3.5, Qwen3.5-MoE, Gemma4-ISWA, hidden-only contexts, GPU tape, and multi-slot GPU cross data. Capture layer assignment, token-count derivation, callback routing, and GPU multi-slot cross collection now have explicit correctness checks.
• Reduced greedy DFlash verification overhead and made verifier control stricter. Eligible verify batches can use reduced top-k logits without raw-logit readback, Bee keeps seed-row alignment correct, the flat verify horizon is capped, server-side depth control is authoritative, and the reduced path falls back when grammar, sampler, or reasoning state requires full logits.
• Hardened DFlash reasoning, draft, and suffix handling. Reasoning-end forcing now goes through the normal full-logits path when needed, invalid reduced-logits drafts are rejected instead of crashing or looping, empty drafts fall back safely, accepted-prefix full-KV commits respect the drafter window, explicit --spec-draft-ctx-size overrides are tracked correctly, Bee keeps the DFlash auto--cd 256 default path when no draft ctx is passed, and the drafter stays aligned with the live accepted suffix.
• Improved Gemma 4 support substantially. Bee added Gemma4-ISWA DFlash target plumbing and profiling callbacks, ported the cleaner upstream Gemma4 graph and loader path back onto Bee hooks, restored Bee precision behavior where needed, synced SWA max-position authority and 512-dim FlashAttention selection with upstream, and fixed Gemma multimodal image decode and dynamic resize bounds.
• Extended CUDA kernel coverage and backend hardening. Bee now keeps 512-wide quantized FlashAttention instances for standard and TurboQuant/TCQ KV combinations, syncs upstream Hadamard rotation plumbing, propagates CUDA driver links correctly, and hardens op-table / Gated DeltaNet integration alongside long-context GPU ring stability fixes.
• Reduced peak memory in the perplexity tool and fixed streaming perplexity / KLD cache handling. Streaming perplexity now writes bounded chunks, checks stream errors, avoids retaining unbounded logits for long-context KL runs, and keeps the logits-cache format versioning compatible with the legacy magic.
• Completed the malformed tool-call guard path for non-stream responses. Final OpenAI-compatible responses now quarantine malformed raw tool-looking text the same way streamed tool-parsing responses already did.

Windows:

• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs

v0.1.2

Changelog

• Fixed the adaptive profit controller's no-spec baseline path. Profit mode now seeds baseline samples before positive-depth warmup, can shut DFlash fully off when the measured baseline wins, and no longer makes speculative decisions from draft-only telemetry.
• Fixed profit-controller reset handling across context-bucket and configuration changes so cleared baseline telemetry cannot leave the controller in a stale active or off state.
• Added low-frequency profit-controller baseline reprobes with --spec-dm-profit-baseline-interval / LLAMA_ARG_SPEC_DM_PROFIT_BASELINE_INTERVAL so runs can refresh target-only timing as context grows. The default interval is 1024 active speculative cycles; reprobes resume the previous active draft depth and avoid off-probe counter starvation.
• Hardened active-reasoning EOS handling. When an end-of-generation token appears while reasoning output is still active, the sampler now forces the reasoning-end sequence through the normal full-logits path; reduced DFlash verification rejects that case instead of accepting an unsafe reduced candidate set.
• Hardened DFlash on split CUDA / multi-GPU placement. GPU cross-ring setup, hidden capture, CUDA graph capture, K/V projection cache updates, recurrent replay, conv replay, and async tensor get/set paths now check buffer/backend ownership and fall back to safer CPU or owning-buffer paths instead of reading or writing recurrent state through the wrong CUDA backend.
• Added clearer diagnostics and regression coverage for multi-GPU DFlash fallback decisions, CUDA graph buffer visibility, wrong-device async tensor access, active-reasoning reduced-sampling rejection, adaptive DM defaults, and profit-controller baseline behavior.
• Fixed ROCm 7 build: added cudaPointerAttributes / cudaMemoryType shim aliases to hip.h, extended CUDART_VERSION >= 10000 guards with || defined(GGML_USE_HIP) so the .type field path is taken on HIP, and removed the WIN32 guard around TurboQuant flash-attention instance compilation so Linux ROCm builds include the turbo KV-cache kernels (acerspyro#11).
• Known limitation: the current multi-GPU DFlash path is a correctness fallback, not a performant split-GPU implementation. On split target placement it can be slower than non-speculative decoding because recurrent replay and hidden capture avoid unsafe single-backend GPU fast paths. A performant implementation still needs per-device replay graphs or a scheduler that follows ggml's split-buffer ownership model.

Windows:

• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs

v0.1.1

Changelog

• Improved agentic tool-call reliability with lazy grammars. DFlash now remains enabled before a lazy grammar trigger, but stops speculating once grammar-constrained output or reasoning-budget forcing requires normal token-by-token sampling.
• Fixed DFlash accept bookkeeping at grammar and tool-call boundaries. The server now distinguishes accepted draft tokens from bonus-token-shaped results, updates DFlash hidden-state rows with the root plus accepted draft tokens, and uses the same keep count for rollback.
• Added a DFlash suppression guard for raw tool-call markers. When a tool marker appears while lazy grammar is enabled, the server suppresses DFlash for the rest of that response without steering sampler state; fenced code and embedded marker-like strings are excluded from the guard.
• Made partial OpenAI-compatible tool-call streaming safer. The server can stream a stable tool name/id early so clients can show a pending tool call, while withholding partial arguments until the parser sees a complete call.
• Quarantined malformed raw tool-call text in tool-parsing streams. Unfinished or malformed tool-looking text no longer leaks into visible assistant content or hidden reasoning deltas before the parser can classify it.
• Accepted direct tag-style function starts for Qwen-style tool calls. Lazy grammar triggers now include structural function markers such as <function=, and the tag parser can parse valid direct function calls without the outer <tool_call> wrapper.
• Added regression coverage for Kimi and Qwen tool-call streaming, malformed raw marker quarantine, fenced-code false positives, direct Qwen function calls, lazy grammar triggers, and DFlash speculative boundary plumbing.
• Fixed small build issues found after 0.1.0: the DFlash callback setup now uses an explicit callback type for GCC 15, and tests/server code include the required standard headers for INT_MAX and FLT_MAX.

Windows:

• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs

v0.1.0

Changelog

• DFlash speculative decoding: --spec-type dflash drives a DFlash draft GGUF alongside the target model. The target captures hidden states into a per-layer 4096-slot ring buffer, the drafter cross-attends to the most recent --spec-dflash-cross-ctx hidden-state tokens and proposes drafts for target verification.
• TurboQuant / TCQ KV-cache compression: Five cache types (turbo2, turbo3, turbo4, turbo2_tcq, turbo3_tcq) spanning from 4x to 7.5x compression, with higher-bit options being practically lossless in many cases. Set independently with --cache-type-k and --cache-type-v.
• Adaptive draft-max control: The server adjusts the active draft horizon at runtime instead of using a fixed --spec-draft-n-max. The default profit controller compares speculative throughput against a no-spec baseline; the fringe alternative maps acceptance-rate bands to draft depth. Use --no-spec-dm-adaptive for a static horizon.
• Full multimodal support: When --mmproj is active, the server keeps flat DFlash available for text generation. The model can be fully offloaded to CPU with no problems to reduce VRAM pressure.
• Reasoning-loop protection: The server detects repeated hidden reasoning output and intervenes. Default mode is force-close with --reasoning-loop-window and --reasoning-loop-max-period tuning available.
• Sampled DFlash verification: --spec-draft-temp enables rejection-sampling drafter behavior. Activates when both draft and target temperature exceed zero. Draft log probabilities must be available for rejection sampling to produce correct output.
• DDTree branch verification: optional --spec-branch-budget adds branch nodes beyond the main draft path with GPU parent_ids, tree masks, and recurrent tree kernels. Disabled automatically when the target model spans more than one GPU. This one is very much work in progress!
• Request-level speculative overrides: Draft-max and branch budget can be overridden per-request through JSON fields without restarting the server.
• CopySpec model-free speculation: --spec-type copyspec provides rolling-hash suffix matching over previous tokens without a draft model. Results must be benchmarked per workload.

Windows:

• Windows x64 CUDA 12.4 - DLLs
• Windows x64 CUDA 13.1 - DLLs