Two integer dot products per weight cancel the 2x int8-over-fp16 tensor
core advantage, so at large batch MMQ cannot beat dequantize + fp16
cuBLAS (measured 331 vs 426 t/s pp512 on the 27B, RTX 4060 Ti, while
running at the same ~20% of its int8 ceiling as Q2_K MMQ does of its
own). MMQ still avoids the dequantization round-trip at moderate batch;
the threshold default is provisional until the crossover is measured
(GGML_CUDA_DT3_MMQ_MAX_BATCH overrides it for that measurement).
With MMQ, ncols_dst > 8 is an integer path in the same numerical regime
as MMVQ and is judged against the exact reference at 1e-5 instead of
riding the loose GEMM gate. The regime is told apart by the result
itself; a GEMM fallback (backends without DT3 MMQ) keeps the fp16
reference and the F16 GEMM bit-identity control. Also add ncols_dst=100
to exercise a wide tile with a clamped last column block.
Two decoded ternary planes per SRAM tile row (the planes cannot be fused
into one int8 because d1 != d2), each with its own per-chunk scales, both
multiplied against the same q8_1 y tile: sum = dB*(sumi1*dA1 + sumi2*dA2).
The load decodes each packed byte once with the same base-3 digit
iteration as the MMVQ vec_dot and turns digit bytes {0,1,2} into trit
bytes {-1,0,+1} without cross-byte borrows.
The MMA tile at I=128 takes 592 B/row: 75776 B of x tile plus the y tile,
94720 B at J=128 — fits the 99 KiB opt-in limit of Ampere-class devices
but not e.g. Turing's 64 KiB, so the runtime gate requires the MMA data
layout and enough shared memory for the narrowest tile and declines
otherwise (AMD keeps declining: no config entries select DT3).
The Vulkan backend now forces fp32 accumulators for the DT3 dequant
fallback, so the DT3 GEMM is no longer bit-identical to the backend's
default-precision F16 GEMM (fp16 accumulators on fp16-capable Vulkan
devices). Run the F16 control at both the default and the F32-forced
precision and require bit-identity with either one. CUDA still matches
the default-precision control; Vulkan matches the F32 one - both with
0 mismatches.
The dequant fallback runs DT3 matmuls as fp16 weights through the f16
matmul pipelines, which default to fp16 accumulators when the device
supports fp16. DT3 weights are the sum of two fp16-scaled ternary
planes and are generally not fp16-representable, so the fallback
already pays one fp16 rounding on the weights; accumulating on top of
that in fp16 measurably hurts.
Force GGML_PREC_F32 for the DT3 fallback, which selects the f32acc
pipelines - the same numerics as the CUDA GEMM fallback (fp16 inputs,
fp32 compute).
Measured on Qwen2.5-3B DT3, wiki.test, 4 chunks, --no-mmap, AMD
Radeon 890M (RADV STRIX1), CPU reference 15.0608:
default batch, before 15.5562 (+3.29%)
default batch, after 15.4062 (+2.29%)
GGML_VK_DISABLE_F16=1 15.3348 (+1.82%, floor of this route)
The remaining gap is shared with the mul_mat_vec route (15.3302, which
does not move under GGML_VK_DISABLE_F16) and is under investigation
separately; DT3 dequantization itself is bit-exact vs the CPU
reference on real model tensors (214,695,936 elements, 0 mismatches).
Wires the dual-plane ternary type into the Vulkan backend through three
paths only:
- get_rows and the generic scalar mul_mat_vec use per-element decode in
dequant_funcs.glsl: the byte and base-3 digit are located from the
element index (regions qs[0..16), qs[16..24), qh[0..2)), the byte is
multiplied by 3^n mod 256 and the top digit taken. w = d1*t1 + d2*t2
is accumulated in fp32; both products are exact so the sum carries a
single float rounding and reproduces the CPU reference bit by bit
(verified: 0 mismatches over the 112-block synthetic test and over
214,695,936 elements of real model tensors).
- larger matmuls fall back to dequant_dt3.comp (decode each byte once
with q <- q*3 mod 256, fp32 sum, one rounding at the f16 write) plus
the existing f16 matmul pipelines.
The qh bytes hold only 4 trits; their 5th base-3 digit is packing
padding that decodes to -1, so both decoders stop at 4 digits.
Deliberately NOT implemented, and declined instead of half-supported:
no coopmat/coopmat2/MMQ shaders are generated for DT3, and supports_op
answers false for MUL_MAT_ID (mul_mat_vec_id shaders are not generated
either). GET_ROWS and MUL_MAT answer true.
test-dt3-gpu accepts the Vulkan backend (and IGPU-type devices) and
passes on RADV STRIX1: dequant 0 mismatches, mul_mat n<=8 norm rel err
~1e-8, GEMM fallback bit-identical to an F16 GEMM on fp16-rounded
weights.
The test picked the first GPU device it found and treated an unsupported
op as a failure. Vulkan and SYCL answer supports_op == false for DT3,
which is the right answer for them and not a bug to report, so the test
went red on machines that were behaving correctly. Metal is worse: it
answers true for almost any type but has no DT3 shader, so the run died
in pipeline compilation halfway through.
Pick the backend by name instead — CUDA and HIP (which reports itself as
ROCm) are the only ones implementing DT3 — and skip everything else. A
supports_op failure on those two is still a real failure.
Also document that the n <= 8 gating mirrors MMVQ_MAX_BATCH_SIZE by hand
and goes stale silently if the MMVQ dispatch changes.
The old vec_dot decoded every element with its own pair of multiplications
(256 inlined get_trit per block, each qs byte re-read 5 times). Decode each
byte once instead, iterating q -> (q*3) & 0xFF two bytes at a time in 16-bit
lanes, and accumulate dp4a over base-3 digits in {0, 1, 2}; one extra dp4a
with 0x01010101 per q8_1 int, shared by both planes, turns the digit sums
back into trit sums in exact integer arithmetic, so the result stays
bit-identical to the per-trit decode. The qh bytes keep their own 4-digit
path so the padding digit is never decoded.
Builds without a native DT3 kernel rename the generic symbol to
ggml_vec_dot_dt3_q8_0 (arch-fallback.h), so test-dt3 failed to link on
them. With a weak declaration the test links everywhere and skips,
loudly, when there is no separate generic to compare against. MSVC has
no weak symbols, so there the test is compiled out.
Calls the actual ggml_vec_dot_dt3_q8_0_generic symbol against the
dispatched vec_dot and requires memcmp-equal floats. Blocks exercise
the three regions, the 79/80 and 119/120 boundaries, non-trivial qh
bytes (would expose a vectorization reading their padding 5th digit),
and scales of both and mixed signs; y reaches the full q8_0 range.
Mutation-checked: flipping one bit of a digit blend mask in the
AVX-512 kernel makes 94 of the 96 reps fail.
Decode both planes of a block with VBMI byte permutes: the *3 multiply
chain (wrapping, so it commutes with the permutation) is computed once
on the whole 56-byte block, and masked vpermb picks each element's byte
from the chain vector of its base-3 digit. The qh lanes never see 3^4,
which would read the padding 5th digit of the qh bytes. The trits reach
the integer product as xi in {0, 1, 2} via the same avg trick as
tq1_0, with VNNI dpbusd against the q8_0 bytes and sum(y) subtracted.
The per-q8_0-block sums and the float accumulation keep the exact
operation order of the generic implementation, so the result is
bit-identical to it (checked by test-dt3).
2.2x over the (autovectorized) generic on a Ryzen AI 9 HX 370.
The dequantize + cuBLAS fallback computes in fp16 (CUBLAS_COMPUTE_16F)
on fast-fp16 hardware and in TF32 under
GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32, so no analytic tolerance separates
'correct' from 'broken' there without also tracking cuBLAS numerics.
What IS ours to guarantee: the fallback must behave exactly as if the
weights were an F16 tensor holding fp16(dequant(block)). Gate on that
bit-identity and demote the analytic GEMM errors to INFO.
Elementwise max relative error explodes on cancellation whenever a true
output element is near zero, so the mul_mat checks now gate on the
relative Frobenius norm and keep the max as information. The GEMM
fallback dequantizes to fp16 on fast-fp16 hardware and DT3 weights
(d1*t1 + d2*t2) are generally not fp16-representable, so that path is
judged against a reference computed from fp16-rounded weights (taking
the better of both references so GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 also
passes). Q4_1 (same non-fp16-exact regime) and Q4_0 (fp16-exact weights,
pure GEMM error floor) run through the identical comparison as controls.
Checks the GPU backend against the validated CPU path: GET_ROWS
dequantization must match dequantize_row_dt3 bit by bit on directed
blocks (region boundaries 79/80 and 119/120, qh elements, negative
scales) and on raw random bytes; MUL_MAT must match a double precision
reference from the dequantized weights, with activations whose q8_1
quantization is exact, plus a manual trit-sum check with non-trivial qh.
Skips cleanly when no GPU backend is available.
vec_dot_dt3_q8_1 processes a whole 128-element DT3 block per call
(VDR_DT3_Q8_1_MMVQ = 4, QI_DT3 = 4), i.e. the 4 q8_1 chunks it spans,
with one pair of integer accumulators per chunk:
sum_j d8[j] * (d1*sumi1[j] + d2*sumi2[j])
The trit decode reuses ggml_cuda_dt3_get_trit with fully unrolled loops,
so all indices and pow3 factors fold into constants; no __byte_perm or
other NVIDIA-only intrinsics. Enables MUL_MAT in supports_op: ncols_dst
<= 8 takes MMVQ, larger falls back to dequantization + cuBLAS (no MMQ
tile kernel yet).
Decode one packed ternary plane with the shared ggml_cuda_dt3_get_trit
helper (shifts, masks and a small pow3 table; the uint8_t wrap-around of
the intermediate product is intentional and matches the CPU reference).
The qh bytes hold only 4 trits; their 5th base-3 digit is packing padding
that always decodes to -1 and is never read.
Wires DT3 into the generic dequantize_block templates (to fp32/fp16/bf16,
contiguous and not) and into get_rows, and enables GET_ROWS in
supports_op.
The previous byte-position test packed with the test's own packer on
both sides of the comparison, so it exercised none of the library code.
It now pins hand-computed byte values (43/100/127/42/124...) at the
region boundaries (79/80, 119/120) as ground truth and drives both
directions through the library: to_float must place each literal byte's
trit at the exact element, and from_float must produce the exact literal
byte, for both planes.
Also probes ggml_validate_row_data over all 256 byte values in qs and
qh positions (must accept exactly the 243/81 reachable codes), the
all-0xaa block, and a well-formed packed block.
The reference-quantizer disclaimer only existed in the code and in
llama-quantize --help; now it is also printed where the mistake would
actually be made, at the start of a quantization run targeting DT3.
ggml_validate_row_data now rejects unreachable code bytes: the ceiling
division packing reaches only 243 of the 256 byte values in qs and 81
in qh (4 trits plus an always-zero padding digit), so corruption that
previously loaded and generated garbage silently is caught at load
time. Previously only the two fp16 scales were checked.
quantize_dt3 no longer discards quant_weights silently: an ignored
imatrix now prints a loud warning (once), otherwise an imatrix A/B on
DT3 would come out byte-identical and invite the false conclusion that
the imatrix does nothing.
The two initial trit passes of quantize_row_dt3_ref now clamp like the
refit passes do, so a NaN input cannot push an out-of-range value from
lroundf into the packer.
test-dt3 checks the layout against an independent packer written from
the format spec: single-trit position mapping for all 256 (plane, pos)
pairs, structural byte-position checks at the region boundaries
(79/80, 119/120), exact round-trips with negative scales, byte parity
of the in-tree quantizer on already-ternary inputs, and the vec_dot
against a hand-made sum over the known trits (catches any path that
reads the padding 5th trit of the qh bytes).
test-dt3-rust-parity.py packs known trits with ternaria's pack_dt3 and
verifies that dequantize_row_dt3 (via test-dt3 --dequant) reproduces
d1*t1 + d2*t2 bit-exactly.
Also wires DT3 into the test-quantize-fns thresholds (ternary class).
Registers the type id, file type and block size, and implements numpy
dequantization (verified bit-exact against the C implementation with
gguf-py/tests/test_quants.py, including random byte payloads).
Quantization is intentionally left unimplemented, like the K-quants:
DT3 planes come from an external solver (PTQTP) and are packed
directly, so a from-float numpy path would only invite quantizing
models with the wrong algorithm.
Adds LLAMA_FTYPE_MOSTLY_DT3 at the end of the ftype enum, the loader
name/guess mappings, the quantization fallbacks (same as the other
ternary types), and the llama-quantize table entry. The table entry
warns that the in-tree quantizer is only the reference one: DT3 models
with the measured quality are produced by the external PTQTP pipeline.
The vec_dot pairs DT3 with Q8_0 (4 q8_0 blocks per DT3 block) and keeps
one integer accumulator per plane: sumf += dy * (d1*sumi1 + d2*sumi2).
Q8_0 instead of Q8_K on purpose: the planes are symmetric ternary so the
q8_K bsums are dead weight, and 32-element blocks accept any row size
that is a multiple of 128.
Trit decoding reuses unpack_plane_dt3, which reads only 4 trits per qh
byte; the 5th base-3 digit of those bytes is packer padding that always
decodes to -1 and must never be read.
Add the dual-plane ternary DT3 type to the type registry along with its
reference row functions. Each of the two planes is packed exactly like
tq1_0 with all constants halved (block of 128 elements): qs 48 -> 24
bytes over two passes of 16 and 8 bytes, qh 4 -> 2 bytes.
The trit decoding lives in a single exported helper (unpack_plane_dt3)
so that dequantization and the upcoming CPU vec_dot share it.
The reference quantizer is a greedy two-pass (plane 1 by absolute max,
plane 2 on the residual) plus two rounds of alternating least-squares
refits. It is intentionally NOT the PTQTP solver used to produce the
published DT3 models.
DT3 stores w_i = d[0]*t0_i + d[1]*t1_i with t in {-1,0,+1}, two ternary
planes over a 128-element block: 56 bytes, 3.5 bpw exactly.
Each plane uses the tq1_0 base-3 packing with every constant halved for
the smaller block (qs 48->24 B, qh 4->2 B, qs passes over 16 then 8
bytes instead of 32 then 16), which tiles 128 with no leftover bytes.
Reducing tq1_0 to 128 without halving the passes does not tile: with a
24-byte qs the first pass covers nothing and the second overruns.
* Switch ROCm from 7.2.1 to 7.14
ROCm 7.14 is the first production release using TheRock build system.
It can be installed using multi-arch deliverables from wheels, debs,
rpms, tarballs or runfiles.
Adjust ROCm targets for Linux and Windows to use this instead.
* ci: switch all other Windows ROCm jobs to ROCm 7.14 wheels
Move the shared windows-setup-rocm composite action from the HIP SDK PRO
Edition installer to the multi-arch ROCm wheels (rocm[libraries,devel]).
The wheel-install logic that previously lived inline in release.yml is now
in the shared action, and both build-cache.yml and release.yml call it.
Also migrate the build-cuda-windows.yml hip job to the same wheel-based
layout (cache path/key, rocm-sdk environment setup, llvm/bin compiler
paths) so it keeps working after the action's contract changed; drop its
now-unused ROCm 7.2.1 rocWMMA download and stale include path.
* Enable backend sampling with token speculation
* Clamp the mask sum before converting it into the sampled index
* Add a numeric context parameter declaring the maximum outputs one sequence
* More fixes
* Don't reuse memory for output views.
* Match dist between CPU and GPU
* Fix CPU and backend sampling mismatches
* Simpify some of the changes
* Fix tests on Vulkan
* More test fixes
* Rebase changes
* Rebase and address review comments
* Address review comments
* Address review comments
* Update src/llama-sampler.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a
valid input, but the CUDA and Metal roll kernels index by ne alone and
never read the nb strides. A non-contiguous src therefore produced
silently wrong results. Neither backend declared a contiguity
requirement in supports_op, so the scheduler did not fall back to the
CPU implementation, which does handle strides correctly.
Add the requirement to both backends, matching the existing
GGML_OP_ROPE guard, and add a permuted test_roll case.
* ui: split the markdown rendering setting per surface
User content and thinking get their own toggle again, so turning off
markdown for a message leaves reasoning blocks formatted. Both default
to markdown. A stored renderContentAsRawText unfolds onto the user key
and is dropped from the config.
File mentions render as badges in the raw text path too, through a
narrow pass over [name](file://path) that leaves everything else
untouched.
* ui: let the rich chat input scroll past its max height
The contenteditable renderer caps its height with max-height but had no
overflow rule, so a long buffer overflowed into the input area wrapper
and got clipped by its overflow-hidden, leaving no way to reach the
bottom of the message. The textarea renderer scrolls natively and was
never affected.
* ui: apply the new lint and format config
* ui: move the render keys unfolding into the migration service
Address review from @allozaur: the settings store no longer rewrites
persisted config on load, the raw text toggle now unfolds onto the
per-surface render keys in migration.service.ts, next to the other
config migrations. The mention scanner flag and the directory path
suffix become named constants.
* server: add an ssh transport to the tools runtime
--tools-runtime ssh:<target> runs the built-in tools on a remote host,
where target is whatever ssh already resolves, a user@host or a config
alias, so no credentials live in llama.cpp.
Only build_argv and upload differ from the docker transport: the remote
shell re-parses the command line, so the argv travels through
shell_quote_join, and files go over scp with the same quoting on the
remote path. Authentication is key-based and the host key must already
be trusted, since the tools run without a console and any prompt would
hang them.
The target is validated before use. The spec can reach us from the
x-tool-runtime header, and a leading dash would turn it into an ssh
option, which is enough to run a command back on the host.
Nothing is created and nothing is reclaimed, so an ssh spec goes
straight to the tool call instead of through the container runtime.
Note that this is remoting rather than isolation: the tools can do
whatever the target account can do, and the isolation is whatever runs
them on the far side.
* server: support podman in the tools runtime
docker and podman expose the same run, exec, cp and inspect verbs with the
same argument order, so a single implementation drives both and the engine
is carried by the spec prefix: podman:<image> and podman-container:<id> sit
next to the docker forms.
tools_io_docker becomes tools_io_container and the runtime spawner becomes
server_tools_container_runtime, both holding the client binary chosen at
parse time. A single parse_container_runtime() resolves every spec, so
adding another engine is one string in the table.
make_tools_io() now rejects the spawning forms. The spec also reaches it
from the x-tool-runtime header, which is client controlled, and only the
runtime that owns a container is allowed to create one: a tool call can
attach to a running container, nothing more.
* ./build/bin/llama-gen-docs
* server: simplify the tools runtime and drop the file copy step
A server_tools_runtime base with one virtual spec() replaces the
container runtime and the bare spec string that ssh needed next to it,
so server_tools is back to a single pointer and neither setup nor the
handler tests which of the two is set.
write_file used to spill its content into a temporary file on the host
and copy it in, because run_subprocess had no way to feed a child. It
now takes an optional stdin payload and creates the parent directory
and the file in a single round trip through a shell in the isolate.
That removes the upload virtual and both implementations: no more
container cp or scp, no second binary on the host, no sftp subsystem on
the target, no predictable temporary in a shared tmp, and none of the
content reaching an argv the remote shell re-parses. It also fixes
write_file over ssh, which never worked: scp speaks sftp and takes the
remote path literally, so quoting it kept the quotes in the file name.
Writing the payload before reading the output relies on the child
draining stdin as it goes, which holds for cat, its only user today.
* ./build/bin/llama-gen-docs
* server: harden the tools runtime against argv injection and a stdin stall
Validate the container id from x-tool-runtime and --tools-runtime the
same way the ssh target already is, so an id shaped like an option
(docker-container:--privileged) is rejected before it reaches the
engine's exec command line instead of running against a hardened
container. Feed the child's stdin after the watchdog is armed, so a
transport that stalls mid-write is terminated at the deadline rather
than blocking the request forever.
Cover both guards and fix the unknown-scheme test, which used ssh: as
its example and now names a real runtime.
* tests: exercise the tools runtime tests on podman as well as docker
Follow-up #26507. The container runtime drives docker and podman
through one implementation, so parametrize the availability helper,
the container fixture and the attach test on the engine, and cover
both engine prefixes in the container id injection test. Each engine
skips on its own when it is not installed.
The spawn cleanup test stays docker only: it recovers the spawned id
from the container hostname, which docker sets to the short id and
podman rootless does not guarantee. Podman keeps its coverage through
the attach path.
* server: release the container handle before respawning
Follow-up #26507. create() writes over the handle it is given, so a
respawn after the container died on its own leaked the pipes and the
process handle of the previous one.
* server: trim the tools runtime comments
* server: read tool output as raw bytes and harden the runtime on Windows
The stdout pipe is read with read() instead of fgets(), so a chunk
can hold any byte, including NUL, and still streams as soon as data
is available. Past the size cap the pipe keeps draining so the child
never blocks on a full pipe. Both pipe fds are forced to binary mode
on Windows, where the CRT defaults them to text mode and translates
line endings in both directions. Stdin is now always closed after
the feed: the child reads a deterministic EOF, and the Windows
docker and ssh clients stop outliving their command on a stdin pipe
that never closes.
The attach form of --tools-runtime has no lifecycle to own, so it
becomes a static target validated once at startup. This removes the
subprocess that ran on every tool call and
serialized calls behind a mutex; a stopped container now surfaces
the engine's own error at exec time.
The cidfile path is passed as UTF-8, matching the encoding the
subprocess layer expects for the CreateProcessW command line, so
the spawn form works from a non-ASCII Windows profile.
The SIGPIPE note in server.cpp now names the tools runtime children
as well as the MCP ones.
* clean up comments
* less pollute global scope
* nits
* tests: name the container image after both engines
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* Get started with Onyx
* Add architecture
* Skip keys handled in super()
* Loading tensors
* Shorten
* Graph
* Apply suggestion from @pcuenca
* Remove norm now embedding in transformers weights
* Add eot
* Explicit output_multiplier
* Handle post_norm_eps
* No super call; unhardcode eot.
The pattern `self._set_vocab_gpt2()` seems preferred throughout the
codebase, and it allows `set_vocab()` to be called from a different part
of the Python class hierarchy: the drafter model converter that we may
need eventually.
* Register for drafting
* DFlash: inherit rope type from the linked target.
Another option would be to store it in the gguf file itself.
* mmproj conversion
Note: some fields to be renamed after the implementation works. We are
keeping compatibility with the reference Meta gguf for testing purposes.
* "clip" header declarations
* Load mmproj
* Pre-processing
* Graph
* Go back to using delimiters.
Otherwise our generations are worse.
Transformers does not use them. We need to trace inputs to verify
whether they are equivalent.
* downsample_factor -> merge_size
* Add vision graph
lol, forgot from a previous commit
* Additional renames, align with llama.cpp / transformers
* Prefer _size instead of independent _h and _w
* Fix token layout
Co-authored-by: Young Han <younghan@fb.com>
* onyx: bring the chat parser onto the onyx branch
common/chat.cpp on this branch has no Onyx handling, so a converted model
serves malformed chat: the assistant preamble leaks into content
("to=self<|message|>...") and tool calls fail with
HTTP 500 "The model produced output that does not match the expected
peg-native format"
common_chat_params_init_onyx exists on onyx-fair-patch, added there by
8bb73dd3d. It was never on this branch, so this is not a regression --
the two lines developed independently.
The code here is taken verbatim from that commit. It is the clean side of
`git merge origin/onyx-fair-patch`: chat.cpp is one of the files that
merges without conflict. The full merge is not viable -- it produces 13
conflicts, including add/add on conversion/onyx.py and src/models/onyx.cpp
where the q_norm-folding and metadata-scale approaches contradict each
other, and #4/#7 are stacked on this branch's side of that.
Verified on this branch: builds with 0 errors, converts an Onyx checkpoint,
and serving it gives "4" for "What is 2+2?" plus a correct
get_weather {"city":"Paris"} tool call, where the unported branch gives the
two failures above.
No converter or runtime changes are included, so this should not interact
with the q_norm work.
Co-authored-by: Beto de Paola <betodepaola@meta.com>
* Less params, bilinear pos-emb interpolation as a graph op instead of CPU
* Map to symbolic V_MMPROJ instead of strings
* Make a couple params explicit
* Patchify via build_inp()
* No param for rope_theta
* Small cleanup
* Restore blank line
* Unpermute, to adapt to the latest transformers checkpoint
* Apply norm after token embeddings
This follows the latest transformers approach.
* Remove duplicated function
* build_vit
* onyx: use the model rope theta on sliding-window layers
* DFlash: conversion from transformers drafter
* Revert rope_type derivation from target
NOTE: this breaks compatibility with Meta's distributed DFlash GGUFs, as
the Q/K are stored in "NEOX" (rotated half) format, like in
transformers.
* Apply suggestion from @pcuenca
* Set model type
* Remove comment that will become obsolete
* Hardcode post_norm_rms_eps instead of new param
* Derive SWA+RoPE pattern from gguf array or scalar
* Fix model type <-> number of layers
* Reorder
* Rename
* Fix typo
* DFlash: seed the draft KV cache from multimodal embedding batches
`common_speculative_impl_draft_dflash::process()` returned early on any batch carrying embeddings, so an image prefill never had its target-layer features fused through the DFlash encoder and injected into the draft's KV cache. That left a hole spanning the image's positions, and the next injection at a post-image position failed to initialize its batch:
```
decoding image batch 1/1, n_tokens_batch = 256
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1
process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=17, offset=0)
srv decode: failed to process speculative batch
```
Every image request with `--spec-type draft-dflash` failed with HTTP 500. Text-only was unaffected, since those batches carry token ids and were let through.
Restore the earlier condition, which admits a batch that is either tokens or embeddings and skips only the degenerate neither/both cases. The rest of `process()` is already layout-agnostic -- it gathers features via `llama_get_embeddings_layer_inp()` and indexes `batch_in.pos[]` / `batch_in.seq_id[]`, none of which assume token ids -- so this is the whole fix.
Validated against `muse-glimmer-30B-bf16.gguf` + `mmproj-muse-glimmer-30B-bf16.gguf` + a DFlash draft head, on an image describe-the-shapes request:
- before: HTTP 500, `failed to process speculative batch`
- after: HTTP 200, draft acceptance 0.34012 (167 accepted / 491 generated), mean len 3.04
Output equivalence holds, which is the property that matters: at temperature 0 the drafted response is byte-identical to the same request served with no draft attached (1213/1213 chars), so the draft is drafting correctly through the image context rather than merely not crashing.
* Conversion: prefer rewrite to mapping
* Revert "Conversion: prefer rewrite to mapping"
This reverts commit a92d0ac584d315e876741e85b6dad3dbc8b23bf7.
* fix lint
* sliding_window metadata is not optional
* disable state save/load
* Apply suggestion from @pcuenca
---------
Co-authored-by: Young Han <younghan@fb.com>
Co-authored-by: Beto de Paola <betodepaola@meta.com>
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: ruanrms <ruanslv@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Upstream merged the Windows argument quoting fix, the NetBSD build
fix and the chdir fallback for glibc older than 2.29, so pin the
vendored copy to a commit that carries all three and remove the
patch files along with the apply step in the sync script.
The new pin also brings the exec error report on glibc older than
2.24 and the ENOSYS mapping to a dedicated error code. Both are
additive and no caller inspects those values.
* Restore quantization of mmprojs
This was lost in the refactor undertaken in #22004.
* add noreturn
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ci: Add support for CUDA 13.4 ARM64 builds for Windows
Added an architecture-specific CUDA 13.4 Windows build entry targeting ARM64.
Added a CMake configuration to enable ARM64 CUDA cross-compilation from an x64 Windows environment using the x64-hosted CUDA and MSVC toolchain while linking against the ARM64 CUDA import libraries to produce ggml-cuda.dll.
Validated the self-hosted Windows x64 workflow, including toolkit acquisition, CMake configuration, ARM64 CUDA cross-compilation, and packaging. Runtime validation was performed separately on a native ARM64 RTX Spark system using TinyLlama 1.1B Q4_K_M to verify the generated binaries.
The ARM64 CUDA job builds only the ggml-cuda.dll backend (LLAMA_BUILD_SERVER=OFF). The release consists of two packages: the main ARM64 release package, which combines the existing ARM64 CPU outputs with ggml-cuda.dll, and a separate runtime package containing the required CUDA runtime libraries (cudart64_13.dll, cublas64_13.dll, and cublasLt64_13.dll).
The CUDA 13.4 setup uses NVIDIA Developer Preview component archives instead of the GA component downloads used by the existing CUDA setups and will require updates once CUDA 13.4 reaches GA.
* ci: cleans up to align with x64 CUDA setup
- Moves CUDA-specific CMake options into matrix defines.
- Keeps the CUB 3DOT2 option only for CUDA 12.4.
- Removes runtime argument construction and the unnecessary server option.
- Aligns ARM64 CUDA runtime packaging with the existing robocopy approach.
- Generalizes the ARM64 release label from CUDA 13.4 to CUDA 13.
* ci: Set CUDA job name as version-architecture pair
* mark as preview
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* granite-switch: add llama.cpp backend (POC, CPU)
New "granite-switch" architecture: a dense, all-attention Granite-4.1
model with N embedded LoRA adapters selected per-token by control tokens.
- gguf-py schema (arch, KV keys, stacked LoRA tensor names) + writer helpers
- conversion/granite.py: GraniteSwitchModel converter (stacks N adapters +
zero base slot into per-projection A/B tensors; emits switch metadata)
- C++ arch registration (llama-arch.{h,cpp}, llama-model.{h,cpp})
- src/models/granite_switch.cpp: load + per-token switched-LoRA graph via
ggml_mul_mat_id over stacked tensors; sticky per-token index + control-token
substitution in llm_graph_input_switch::set_input
- llm_graph_input_switch in src/models/models.h
Runs end-to-end on CPU: convert 3b checkpoint (842 tensors, stacked dim 13)
and generate on both base and control-token paths. Sticky switch state is
single-sequence (POC); full multi-sequence machinery is a follow-up.
* granite-switch: add Mac (Metal) build + mid-sequence switch demo script
Self-contained script to build llama.cpp on Apple Silicon (Metal),
convert the composed 3b checkpoint, and run the crisp mid-sequence
adapter-switch demos verified on Vela:
- answerability: <|answerability|> mid-seq -> "unanswerable"
- query_rewrite: <|query_rewrite|> mid-seq -> {"rewritten_question": ...}
Each demo runs the same prompt twice, differing only by a control token
placed before the assistant turn, so the per-token switch is visible.
* granite-switch mac demo: add -no-cnv so each run is one-shot
The composed model ships a chat template, so llama-completion auto-enables
interactive conversation mode and halts at a `>` prompt after generating,
stalling the script. -no-cnv disables conversation mode: generate once from
the raw prompt and exit (also prints special tokens, making the switch visible).
* granite-switch: replace global sticky index with in-graph router attention
The POC computed the per-token adapter index on the CPU and carried it
across ubatches in ONE global `mutable int32_t poc_sticky_index`, reset
only when a ubatch contained sequence position 0. That global had two
problems:
1. Concurrency: with multiple sequences in a batch it was last-writer-
wins — one sequence's adapter leaked into the others.
2. Multi-turn: an interactive `ollama run` chat continues one KV cache,
so turn 2 never saw position 0 and the index never reset — the
adapter stayed stuck on across turns.
Port the vLLM/HF backend mechanism faithfully: a single-head causal
"router" attention recovers the adapter index in-graph. Per token, only
dim 0 carries signal — Q[0]=1, K[0]=+gain for a control token / -gain
otherwise, V[0]=adapter slot / 0 — and the causal softmax over the single
visible control token recovers that adapter's slot (readback =
clamp(round(V[0]), 0, n_adapters)). gain=15 matches config.py and is
F16-safe (no F32 cache).
The router's K/V live in the model KV cache at an extra layer
R == hparams.router_layer (== n_layer). We bump n_layer_all to n_real+1
so the cache allocator gives the router its own per-sequence slot, and
set n_layer_nextn=1 so n_layer() stays n_real — the decoder loop and
tensor loading are untouched and never reference layer R. The router K is
exempted from the k-shift RoPE loop (its dim-0 value is a literal
magnitude, not a rotation).
Because the selection now lives in the per-sequence KV cache, CONCURRENT
requests are isolated for free (problem 1 fixed; verified by
scratch/concurrent_switch_test.cpp). set_input becomes stateless pure
per-token maps; the global is gone.
Single-switch contract / known limitation, identical to vLLM & HF: the
gain is flat (no recency), so within one sequence there is no mechanism to
revert to base mid-sequence — once an adapter fires it stays on until that
sequence ends (problem 2 is therefore NOT fixed by a faithful copy; vLLM/HF
avoid it only because each served request is a fresh sequence). A client
continuing one KV cache across turns must start a fresh sequence per turn,
or opt into a recency-biased router (a deliberate divergence, not done
here). Documented in granite_switch.cpp and asserted by
scratch/multiturn_leak_test.cpp.
Verified (CPU): both demos unchanged (answerability -> "unanswerable",
query_rewrite -> rewritten query); concurrent two-sequence isolation
passes; multi-turn carry-over matches the vLLM/HF contract.
* granite-switch: drop scratch tests and mac demo for upstream PR
Remove the local-only development artifacts that should not ship in the
upstream PR:
- granite-switch-mac-demo.sh (local Metal build + demo driver)
- scratch/concurrent_switch_test.cpp
- scratch/multiturn_leak_test.cpp
Also drop the now-dangling reference to the scratch tests from the
granite_switch.cpp header comment. Leaves only the core architecture
support (conversion, gguf constants, llama-arch/model/kv-cache, and the
granite_switch graph).
* granite-switch: trim comments to match native llama.cpp style
* granite-switch: trim conversion comments to match native style
* granite-switch: drop unused adapter_ranks metadata
* granite-switch: rename arch to graniteswitch and drop obid alias
* granite-switch: fix non-ASCII comments and document router gain assumption
* granite-switch: drop section comments from constants.py to match native style
* granite-switch: add functional tensor block comments matching Granite4 Vision style
* granite-switch: clarify n_expert_used comment
State the actual constraint: mul_mat_id needs n_expert_used == 1, and
since the GGUF carries expert_count = 0 the generic loader's
n_expert == 0 => n_expert_used == 0 assertion has already passed by the
time load_arch_hparams runs, so it is forced to 1 here.
* granite-switch: note n_layer_nextn reuse has no MTP
The router carving reuses n_layer_nextn, normally the MTP/next-token
count. Clarify in the comment that it is borrowed here purely as the
trailing-layers lever and that there is no MTP head, to spare readers
the double-take.
* granite-switch: rename source file and apply review nits
* granite-switch: don't force LoRA tensors to F16, follow --outtype instead
* granite-switch: drop redundant _permute_qk wrapper, call LlamaModel.permute directly
* granite-switch: read router gain from GGUF (control_token_gain) instead of hardcoding 15.0
* granite-switch: derive n_slots()
* granite-switch: move llm_graph_input_switch into granite-switch.cpp
* granite-switch: cut AI-style narration comments
* granite-switch: collapse multi-line comments
* granite-switch: rename control_token_* maps to adapter_token_*
* granite-switch: cut noise comments
* granite-switch: rename embedded LoRA tensors to <base>.lora_a/lora_b
* granite-switch: GGML_ASSERT token input to avoid UB on embeddings
* granite-switch: TODO for raw embedding input support
* granite-switch: collapse LoRA tensor constants to .lora_a/.lora_b suffix
* granite-switch: drop n_expert_used hack, guard mul_mat_id buft probe
* granite-switch: stop forcing dense expert counts, read from config
* granite-switch: renamed control_token_gain metadata key to router_gain
* granite-switch: trim header comments to match native style
* granite-switch: collapse LoRA tensors to base name + suffix
* granite-switch: inline suffix checks in tensor op resolution
* granite-switch: drop switch-lora struct comment
* granite-switch: guard router layer index and inline n_slots
* granite-switch: group adapter metadata under {arch}.adapters.* namespace
* granite-switch: add hparams.has_rope(il) for KV-shift rope skipping
* granite-switch: skip arch in test-llama-archs (adapter fixture missing, TODO)
* granite-switch: Keys.Adapters namespace + simplify n_slots
* granite-switch: validate substitute token ids against n_vocab
* granite-switch: bound adapter count and lora rank from GGUF
* granite-switch: reject MTP context type when router_layer is set
* granite-switch: throw on bad adapter metadata instead of GGML_ASSERT
* granite-switch: use ASCII +/- in router K signal comment
* granite-switch: document n_layer_nextn repurpose and its leak points
* granite-switch: gate lora_a/lora_b op mapping on router_layer
* granite-switch: label all three preview model sizes
docker info only proves the daemon answers, so the Windows CI passes
the check and then dies trying to run a linux image. The hosted
Windows runners cannot run one: GitHub states the VMs are not enabled
for nested virtualization and will not be, since they already sit one
level deep and the hypervisor does not support more levels
(https://github.com/orgs/community/discussions/25491). Probing the
image itself skips those tests there, and pulls it before the server
waits for the container id.
The saver called add_kv with LLM_KV_EXPERT_SHARED_FEED_FORWARD_LENGTH twice, the
second time passing n_ff_chexp. gguf_set_val_u32 removes-then-appends, so the second
call clobbers the first: the saved shared_feed_forward_length ends up as n_ff_chexp
(0 for every arch except GroveMoE), and expert_chunk_feed_forward_length is never
written at all.
So a save->load roundtrip of any MoE model with a shared expert loses n_ff_shexp. On
reload the arch falls back to n_ff for the shexp tensor shape, that no longer matches
the saved tensor, and the model FAILS to load. Hits qwen2moe, qwen3-next, granite-moe,
hunyuan-moe, ernie4.5, bailingmoe2, nemotron-h, and the other shared-expert MoEs.
Fix: the second call writes LLM_KV_EXPERT_CHUNK_FEED_FORWARD_LENGTH.
test-llama-archs: set expert_shared_feed_forward_length to a value distinct from n_ff
in the MoE setup so the roundtrip exercises it. Without the fix the reload fails on a
shexp tensor-shape mismatch; with it, every arch roundtrips clean.
* Update build-sanitize.yml
* make it run on pr
* fix thread
* Update build-sanitize.yml
* Update build-sanitize.yml
* just run thread on github machine
The picker mounts whenever a cwd-aware builtin tool is enabled, so
it can open while file_glob_search is not served or was disabled by
the user. Every typed query then fired a search that could only
fail with a raw error.
Gate the debounced search on the tool state, the same way the
mention picker does, and show a message in place of the results
list that explains why search is unavailable. Manual entry with
Enter still commits a directory. The Browse button and the search
scope footer are hidden as well: Browse resolves the picked folder
name through file_glob_search, and the client-side toggle would not
stop that call.
Millaguie