Two integer dot products per weight cancel the 2x int8-over-fp16 tensor
core advantage, so at large batch MMQ cannot beat dequantize + fp16
cuBLAS (measured 331 vs 426 t/s pp512 on the 27B, RTX 4060 Ti, while
running at the same ~20% of its int8 ceiling as Q2_K MMQ does of its
own). MMQ still avoids the dequantization round-trip at moderate batch;
the threshold default is provisional until the crossover is measured
(GGML_CUDA_DT3_MMQ_MAX_BATCH overrides it for that measurement).
Two decoded ternary planes per SRAM tile row (the planes cannot be fused
into one int8 because d1 != d2), each with its own per-chunk scales, both
multiplied against the same q8_1 y tile: sum = dB*(sumi1*dA1 + sumi2*dA2).
The load decodes each packed byte once with the same base-3 digit
iteration as the MMVQ vec_dot and turns digit bytes {0,1,2} into trit
bytes {-1,0,+1} without cross-byte borrows.
The MMA tile at I=128 takes 592 B/row: 75776 B of x tile plus the y tile,
94720 B at J=128 — fits the 99 KiB opt-in limit of Ampere-class devices
but not e.g. Turing's 64 KiB, so the runtime gate requires the MMA data
layout and enough shared memory for the narrowest tile and declines
otherwise (AMD keeps declining: no config entries select DT3).
The dequant fallback runs DT3 matmuls as fp16 weights through the f16
matmul pipelines, which default to fp16 accumulators when the device
supports fp16. DT3 weights are the sum of two fp16-scaled ternary
planes and are generally not fp16-representable, so the fallback
already pays one fp16 rounding on the weights; accumulating on top of
that in fp16 measurably hurts.
Force GGML_PREC_F32 for the DT3 fallback, which selects the f32acc
pipelines - the same numerics as the CUDA GEMM fallback (fp16 inputs,
fp32 compute).
Measured on Qwen2.5-3B DT3, wiki.test, 4 chunks, --no-mmap, AMD
Radeon 890M (RADV STRIX1), CPU reference 15.0608:
default batch, before 15.5562 (+3.29%)
default batch, after 15.4062 (+2.29%)
GGML_VK_DISABLE_F16=1 15.3348 (+1.82%, floor of this route)
The remaining gap is shared with the mul_mat_vec route (15.3302, which
does not move under GGML_VK_DISABLE_F16) and is under investigation
separately; DT3 dequantization itself is bit-exact vs the CPU
reference on real model tensors (214,695,936 elements, 0 mismatches).
Wires the dual-plane ternary type into the Vulkan backend through three
paths only:
- get_rows and the generic scalar mul_mat_vec use per-element decode in
dequant_funcs.glsl: the byte and base-3 digit are located from the
element index (regions qs[0..16), qs[16..24), qh[0..2)), the byte is
multiplied by 3^n mod 256 and the top digit taken. w = d1*t1 + d2*t2
is accumulated in fp32; both products are exact so the sum carries a
single float rounding and reproduces the CPU reference bit by bit
(verified: 0 mismatches over the 112-block synthetic test and over
214,695,936 elements of real model tensors).
- larger matmuls fall back to dequant_dt3.comp (decode each byte once
with q <- q*3 mod 256, fp32 sum, one rounding at the f16 write) plus
the existing f16 matmul pipelines.
The qh bytes hold only 4 trits; their 5th base-3 digit is packing
padding that decodes to -1, so both decoders stop at 4 digits.
Deliberately NOT implemented, and declined instead of half-supported:
no coopmat/coopmat2/MMQ shaders are generated for DT3, and supports_op
answers false for MUL_MAT_ID (mul_mat_vec_id shaders are not generated
either). GET_ROWS and MUL_MAT answer true.
test-dt3-gpu accepts the Vulkan backend (and IGPU-type devices) and
passes on RADV STRIX1: dequant 0 mismatches, mul_mat n<=8 norm rel err
~1e-8, GEMM fallback bit-identical to an F16 GEMM on fp16-rounded
weights.
The old vec_dot decoded every element with its own pair of multiplications
(256 inlined get_trit per block, each qs byte re-read 5 times). Decode each
byte once instead, iterating q -> (q*3) & 0xFF two bytes at a time in 16-bit
lanes, and accumulate dp4a over base-3 digits in {0, 1, 2}; one extra dp4a
with 0x01010101 per q8_1 int, shared by both planes, turns the digit sums
back into trit sums in exact integer arithmetic, so the result stays
bit-identical to the per-trit decode. The qh bytes keep their own 4-digit
path so the padding digit is never decoded.
Decode both planes of a block with VBMI byte permutes: the *3 multiply
chain (wrapping, so it commutes with the permutation) is computed once
on the whole 56-byte block, and masked vpermb picks each element's byte
from the chain vector of its base-3 digit. The qh lanes never see 3^4,
which would read the padding 5th digit of the qh bytes. The trits reach
the integer product as xi in {0, 1, 2} via the same avg trick as
tq1_0, with VNNI dpbusd against the q8_0 bytes and sum(y) subtracted.
The per-q8_0-block sums and the float accumulation keep the exact
operation order of the generic implementation, so the result is
bit-identical to it (checked by test-dt3).
2.2x over the (autovectorized) generic on a Ryzen AI 9 HX 370.
vec_dot_dt3_q8_1 processes a whole 128-element DT3 block per call
(VDR_DT3_Q8_1_MMVQ = 4, QI_DT3 = 4), i.e. the 4 q8_1 chunks it spans,
with one pair of integer accumulators per chunk:
sum_j d8[j] * (d1*sumi1[j] + d2*sumi2[j])
The trit decode reuses ggml_cuda_dt3_get_trit with fully unrolled loops,
so all indices and pow3 factors fold into constants; no __byte_perm or
other NVIDIA-only intrinsics. Enables MUL_MAT in supports_op: ncols_dst
<= 8 takes MMVQ, larger falls back to dequantization + cuBLAS (no MMQ
tile kernel yet).
Decode one packed ternary plane with the shared ggml_cuda_dt3_get_trit
helper (shifts, masks and a small pow3 table; the uint8_t wrap-around of
the intermediate product is intentional and matches the CPU reference).
The qh bytes hold only 4 trits; their 5th base-3 digit is packing padding
that always decodes to -1 and is never read.
Wires DT3 into the generic dequantize_block templates (to fp32/fp16/bf16,
contiguous and not) and into get_rows, and enables GET_ROWS in
supports_op.
ggml_validate_row_data now rejects unreachable code bytes: the ceiling
division packing reaches only 243 of the 256 byte values in qs and 81
in qh (4 trits plus an always-zero padding digit), so corruption that
previously loaded and generated garbage silently is caught at load
time. Previously only the two fp16 scales were checked.
quantize_dt3 no longer discards quant_weights silently: an ignored
imatrix now prints a loud warning (once), otherwise an imatrix A/B on
DT3 would come out byte-identical and invite the false conclusion that
the imatrix does nothing.
The two initial trit passes of quantize_row_dt3_ref now clamp like the
refit passes do, so a NaN input cannot push an out-of-range value from
lroundf into the packer.
The vec_dot pairs DT3 with Q8_0 (4 q8_0 blocks per DT3 block) and keeps
one integer accumulator per plane: sumf += dy * (d1*sumi1 + d2*sumi2).
Q8_0 instead of Q8_K on purpose: the planes are symmetric ternary so the
q8_K bsums are dead weight, and 32-element blocks accept any row size
that is a multiple of 128.
Trit decoding reuses unpack_plane_dt3, which reads only 4 trits per qh
byte; the 5th base-3 digit of those bytes is packer padding that always
decodes to -1 and must never be read.
Add the dual-plane ternary DT3 type to the type registry along with its
reference row functions. Each of the two planes is packed exactly like
tq1_0 with all constants halved (block of 128 elements): qs 48 -> 24
bytes over two passes of 16 and 8 bytes, qh 4 -> 2 bytes.
The trit decoding lives in a single exported helper (unpack_plane_dt3)
so that dequantization and the upcoming CPU vec_dot share it.
The reference quantizer is a greedy two-pass (plane 1 by absolute max,
plane 2 on the residual) plus two rounds of alternating least-squares
refits. It is intentionally NOT the PTQTP solver used to produce the
published DT3 models.
DT3 stores w_i = d[0]*t0_i + d[1]*t1_i with t in {-1,0,+1}, two ternary
planes over a 128-element block: 56 bytes, 3.5 bpw exactly.
Each plane uses the tq1_0 base-3 packing with every constant halved for
the smaller block (qs 48->24 B, qh 4->2 B, qs passes over 16 then 8
bytes instead of 32 then 16), which tiles 128 with no leftover bytes.
Reducing tq1_0 to 128 without halving the passes does not tile: with a
24-byte qs the first pass covers nothing and the second overruns.
ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a
valid input, but the CUDA and Metal roll kernels index by ne alone and
never read the nb strides. A non-contiguous src therefore produced
silently wrong results. Neither backend declared a contiguity
requirement in supports_op, so the scheduler did not fall back to the
CPU implementation, which does handle strides correctly.
Add the requirement to both backends, matching the existing
GGML_OP_ROPE guard, and add a permuted test_roll case.
ggml_metal_op_norm sized the threadgroup with
`nth = std::min(nth, args.ne00_t)`, which can leave nth not a multiple of
the simdgroup size. The kernels finish their row reduction with a
cross-simdgroup step where each lane of the last simdgroup reads one
per-simdgroup partial sum out of shmem_f32:
if (tiisg == 0) { shmem_f32[sgitg] = sumf; }
threadgroup_barrier(mem_flags::mem_threadgroup);
sumf = shmem_f32[tiisg];
sumf = simd_sum(sumf);
When the last simdgroup is partial it has fewer lanes than the
threadgroup has simdgroups, so the tail of the partial sums is never
read and the row sum is too small. For ne00_t = 33 nth becomes 33: two
simdgroups, but only one lane in the second, so one of the two partial
sums is dropped. The mean and variance are then wrong for the whole row.
Round ne00_t up to a whole number of simdgroups instead. Rounding up
rather than dropping the clamp keeps the threadgroup as small as
possible: deleting the line would raise nth to the next power of two
(ne00_t = 544 -> 1024 instead of 544), which costs idle lanes on 26 row
lengths below 8192 that were already correct, including 1536 and 3584.
GGML_OP_NORM is affected as well as GGML_OP_RMS_NORM - both dispatch
through ggml_metal_op_norm.
No mainstream LLM hidden size hits this: ne00_t is ne00/4 on the
vectorized path, so 4096, 8192, 2048 and friends all give a multiple of
32. It is reachable from other norm shapes, e.g. 320-channel norms.
Add NORM and RMS_NORM cases for ne0 = 33, 132 and 260 across the
existing eps values. 33 exercises the scalar path and 132/260 the
vectorized one, since only those divide by 4.
Before, on M3 Pro:
test-backend-ops test -b MTL0 -o NORM 25/50
test-backend-ops test -b MTL0 -o RMS_NORM 26/51
After:
test-backend-ops test -b MTL0 -o NORM 50/50
test-backend-ops test -b MTL0 -o RMS_NORM 51/51
test-backend-ops test -b MTL0 13943/13943
* ggml : add fallback definitions for missing aarch64 HWCAP bits
* ggml : require HWCAP_ASIMDHP for the aarch64 fp16 cpu variants
Also rename has_fp16_va to has_fp16, the field gates the whole FEAT_FP16
extension, scalar and vector half-precision arithmetic together.
The NVFP4 quantization format stores a scaling factor for every group of
16 weights, packed into a single UE4M3 byte.
The SYCL GPU code was converting these scale values using the E4M3 path,
but that's *signed*, and these are unsigned values.
* tests: add SWIGLU perf cases
perf mode had no GLU coverage. Adds SWIGLU at 17408 columns, 512 and
2048 tokens, f16 and f32, with the operands both fused and split.
* sycl: consolidate fused-GLU kernels
They differed only in which op_* they called, so take the op as an argument and share a common launcher.
Their block sizes were all 256, so launch geometry is unchanged;
SYCL_GELU_BLOCK_SIZE and SYCL_SILU_BLOCK_SIZE lose their last users so are dropped.
* sycl: contiguous fast path for the fused GLU ops
o0 == n and o1 == n collapse the de-interleave index math to the
identity, so dispatch a flat kernel in that case. It fires for
ggml_glu_split with packed operands; a fused [gate|up] tensor keeps the
strided path. test-backend-ops perf -o SWIGLU on an Arc Pro B70: split
+14% f16 and +4% f32, fused unchanged.
- In MSL, declaring an array of matrix types like `threadgroup half4x4` causes
a 'no matching constructor' compilation error because MSL matrix types do not
have zero-argument default constructors and threadgroup variables cannot have
initializers.
- Fix this by declaring a POD `threadgroup half` array instead and casting
to `threadgroup half4x4 *` for matrix indexing.
Signed-off-by: JamePeng <jame_peng@sina.com>
* ci: prepare for amd rocm ci
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: fix editorconfig-checker
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: fix device not recognised
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: rename gpu-amd to gpu-hip
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: gpu-hip to gpu-rocm
haha
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* CUDA: allow integrated-GPU host output buffer in debug assert
On integrated GPUs (APUs), the scheduler can legitimately place a graph
node's output on the host-visible buffer, which ggml_cuda_compute_forward
already handles. The debug assert in ggml_cuda_graph_evaluate_and_capture
required every node output to be on the device buffer, so a debug build
aborts on such a node (e.g. attn_residual ADD -> ROCm_Host on RDNA3.5).
The source-tensor assert directly below already permits this via the
integrated + cuda_host exception; apply the same exception to the node's
own output buffer. Debug-only; no effect on release/compute.
Fixes test-recurrent-state-rollback on gfx1151 (Strix Halo).
* ci: enable unified memory for ROCm gfx1151 job
Work around a coherence issue on integrated RDNA3.5 (gfx1151) where GPU
kernels reading mmap-loaded weights can return incorrect output, which
makes test-llama-archs (and real inference) intermittently wrong.
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 uses managed memory, which restores
coherence. Remove once the underlying ROCm/HIP issue is fixed.
* test-llama-archs: skip jamba on HIP backend
jamba produces incorrect output (~0.55 NMSE vs CPU) on the HIP backend on
RDNA3.5 (gfx1151); the SSM kernels need separate investigation. Skip it
for now, matching the existing per-backend carve-outs (WebGPU), so the
ROCm CI can run the test for the remaining architectures.
* ci: use HIP_LAUNCH_BLOCKING for ROCm gfx1151 job
The gfx1151 ROCm CI job produced incorrect inference output (qwen3 perplexity ~88 vs ~9.4) due to an async-execution correctness issue in the HIP path. Serializing kernel launches with HIP_LAUNCH_BLOCKING=1 restores correctness. This replaces the earlier GGML_CUDA_ENABLE_UNIFIED_MEMORY workaround, which did not fix batched inference.
* test-backend-sampler: skip top-k subtests on HIP backend
The ROCm backend does not support the TOP_K/ARGSORT op at vocab scale (no CUB; bitonic argsort is capped at ncols <= 1024), so top-k/top-p backend samplers cannot be offloaded. The penalties, set_sampler, mixed, and top_p subtests assert that offload happened, so they fail on HIP. Skip them until TOP_K is supported on the ROCm backend.
* Update tests/test-backend-sampler.cpp
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
* Update tests/test-backend-sampler.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Jim Wu <ywu@xilinx.com>
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan: add debug tooling to get more information about a DeviceLost error
* fix submission threshold applied too late
* use logging macros, throw instead of aborting
* clean up circular dependency
* ggml: add ggml_build_forward_order
ggml_build_forward_expand marks the tensor and all its ancestors for
compute, so using it as a pure ordering hint (keeping q, k and v
together) defeats ggml_build_forward_select: the unselected branch is
forced to run with inputs that were never uploaded. In the mtmd audio
graph this makes GEN_WAV calls execute the GEN_CODE branch with a
stale inp_code0, hitting the get_rows bound assert on CPU.
Add ggml_build_forward_order, which inserts nodes without the compute
flag; the flag is restored when the branch is actually selected.
Switch the q/k/v hints in clip_graph::build_attn to it.
* nit: reduce comments (AGENTS.md)
* sycl: parallelize the non-contiguous concat kernel
Launch geometry only: the non-contiguous concat kernel launched a single-lane
work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one.
SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`.
llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV),
on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)
* sycl: cap non-contiguous concat block at ne0
* sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)
* Revert "sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)"
This reverts commit 2709909e790d298cc0746950caff16ad8ecf9d6f.
* sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches
Extends the oneDNN SDPA path (PR #25222) to handle non-F16 KV caches by
dequantizing or converting K/V to dense FP16 on-device before feeding
them into the SDPA graph. The fused systolic kernel then runs identically
to the native FP16 path.
Supported KV types:
- Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl
- F32: cont_to_f16_sycl<float>
- BF16 and IQ types are excluded (no conversion kernel available)
Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only).
F16 KV runs at any length (existing behavior).
Also includes the stream sync fix (stream->wait_and_throw() unconditional,
PR #25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are
always dequantized to separate buffers).
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* add a direct size condition for `large` weights; the original
dimension condition is insufficient -- q6_K lm_head for gemma-4 E2B
has [1536, 262144], which is big enough to slowdown gemv_noshuffle but
does not satisfy the dimension condition (ne0 >= 2048)
* CUDA: Fix data-races when reusing block_reduce
block_reduce currently doesn't resync after reading from SMEM, causing
potential data-races when reusing SMEM for multiple reductions.
One may consider simply always adding this in block_reduce, but this
comes at a potential perf cost
* double-buffering for single-row softmax
* double-buffering for norm as well
* Add comment
* Add explanatory comment to block_reduce
* Specify need for + do memory barrier only in multi-warp scenario
* Implement review-suggestion from @gaugarg-nv
* feat(silu_back): implemented silu_back op for f32
* fix(silu_back): removed redundant asserts in ggml-metal-ops.cpp function ggml_metal_op_silu_back.
- Implement GGML_OP_DSV4_HC_COMB, GGML_OP_DSV4_HC_PRE, and
GGML_OP_DSV4_HC_POST with SIMDgroup register and shuffle optimized kernels.
- Add Metal dispatch and support plumbing and test the production Sinkhorn
iteration count and embedding width.
Assisted-by: Codex
Co-authored-by: Thiago Padilha <thiago@padilha.cc>
Incrementing `ref_count` at the beginning is important later
in the `free()` method of the `ggml_backend_opencl_context` at program end.
If we do not increment the `ref_count`, the result would be -1 here,
and consequently, the profiling data would not be flushed and written.
( #ifdef GGML_OPENCL_PROFILING )
* vulkan : add pool1d push constants and pipeline field
Declared data structures needed for POOL1D OP, which are the vk_op_pool1d_push_constants struct and pipeline_pool1d_f32 field.
* vulkan : add pool1d compute shader
Added pool1d.comp for Vulkan backend mirroring the existing pool2d shader.
* vulkan : add full GGML_OP_POOL_1D support
Added pipeline creation and op dispatch for 1D pooling in the Vulkan backend.
* vulkan : fix pool1d shader logic
Registered pool1d_f32 in vulkan-shaders-gen.cpp and fixed tensor dimension indices and avg pool scale.
* vulkan : fix pool1d end boundary crash and expand test coverage
Fixed an issue where the shader crashed when the end boundary was negative when k0 < p0. Also, added more test cases related to this fix.
Millaguie