Two integer dot products per weight cancel the 2x int8-over-fp16 tensor
core advantage, so at large batch MMQ cannot beat dequantize + fp16
cuBLAS (measured 331 vs 426 t/s pp512 on the 27B, RTX 4060 Ti, while
running at the same ~20% of its int8 ceiling as Q2_K MMQ does of its
own). MMQ still avoids the dequantization round-trip at moderate batch;
the threshold default is provisional until the crossover is measured
(GGML_CUDA_DT3_MMQ_MAX_BATCH overrides it for that measurement).
With MMQ, ncols_dst > 8 is an integer path in the same numerical regime
as MMVQ and is judged against the exact reference at 1e-5 instead of
riding the loose GEMM gate. The regime is told apart by the result
itself; a GEMM fallback (backends without DT3 MMQ) keeps the fp16
reference and the F16 GEMM bit-identity control. Also add ncols_dst=100
to exercise a wide tile with a clamped last column block.
Two decoded ternary planes per SRAM tile row (the planes cannot be fused
into one int8 because d1 != d2), each with its own per-chunk scales, both
multiplied against the same q8_1 y tile: sum = dB*(sumi1*dA1 + sumi2*dA2).
The load decodes each packed byte once with the same base-3 digit
iteration as the MMVQ vec_dot and turns digit bytes {0,1,2} into trit
bytes {-1,0,+1} without cross-byte borrows.
The MMA tile at I=128 takes 592 B/row: 75776 B of x tile plus the y tile,
94720 B at J=128 — fits the 99 KiB opt-in limit of Ampere-class devices
but not e.g. Turing's 64 KiB, so the runtime gate requires the MMA data
layout and enough shared memory for the narrowest tile and declines
otherwise (AMD keeps declining: no config entries select DT3).
The Vulkan backend now forces fp32 accumulators for the DT3 dequant
fallback, so the DT3 GEMM is no longer bit-identical to the backend's
default-precision F16 GEMM (fp16 accumulators on fp16-capable Vulkan
devices). Run the F16 control at both the default and the F32-forced
precision and require bit-identity with either one. CUDA still matches
the default-precision control; Vulkan matches the F32 one - both with
0 mismatches.
The dequant fallback runs DT3 matmuls as fp16 weights through the f16
matmul pipelines, which default to fp16 accumulators when the device
supports fp16. DT3 weights are the sum of two fp16-scaled ternary
planes and are generally not fp16-representable, so the fallback
already pays one fp16 rounding on the weights; accumulating on top of
that in fp16 measurably hurts.
Force GGML_PREC_F32 for the DT3 fallback, which selects the f32acc
pipelines - the same numerics as the CUDA GEMM fallback (fp16 inputs,
fp32 compute).
Measured on Qwen2.5-3B DT3, wiki.test, 4 chunks, --no-mmap, AMD
Radeon 890M (RADV STRIX1), CPU reference 15.0608:
default batch, before 15.5562 (+3.29%)
default batch, after 15.4062 (+2.29%)
GGML_VK_DISABLE_F16=1 15.3348 (+1.82%, floor of this route)
The remaining gap is shared with the mul_mat_vec route (15.3302, which
does not move under GGML_VK_DISABLE_F16) and is under investigation
separately; DT3 dequantization itself is bit-exact vs the CPU
reference on real model tensors (214,695,936 elements, 0 mismatches).
Wires the dual-plane ternary type into the Vulkan backend through three
paths only:
- get_rows and the generic scalar mul_mat_vec use per-element decode in
dequant_funcs.glsl: the byte and base-3 digit are located from the
element index (regions qs[0..16), qs[16..24), qh[0..2)), the byte is
multiplied by 3^n mod 256 and the top digit taken. w = d1*t1 + d2*t2
is accumulated in fp32; both products are exact so the sum carries a
single float rounding and reproduces the CPU reference bit by bit
(verified: 0 mismatches over the 112-block synthetic test and over
214,695,936 elements of real model tensors).
- larger matmuls fall back to dequant_dt3.comp (decode each byte once
with q <- q*3 mod 256, fp32 sum, one rounding at the f16 write) plus
the existing f16 matmul pipelines.
The qh bytes hold only 4 trits; their 5th base-3 digit is packing
padding that decodes to -1, so both decoders stop at 4 digits.
Deliberately NOT implemented, and declined instead of half-supported:
no coopmat/coopmat2/MMQ shaders are generated for DT3, and supports_op
answers false for MUL_MAT_ID (mul_mat_vec_id shaders are not generated
either). GET_ROWS and MUL_MAT answer true.
test-dt3-gpu accepts the Vulkan backend (and IGPU-type devices) and
passes on RADV STRIX1: dequant 0 mismatches, mul_mat n<=8 norm rel err
~1e-8, GEMM fallback bit-identical to an F16 GEMM on fp16-rounded
weights.
The test picked the first GPU device it found and treated an unsupported
op as a failure. Vulkan and SYCL answer supports_op == false for DT3,
which is the right answer for them and not a bug to report, so the test
went red on machines that were behaving correctly. Metal is worse: it
answers true for almost any type but has no DT3 shader, so the run died
in pipeline compilation halfway through.
Pick the backend by name instead — CUDA and HIP (which reports itself as
ROCm) are the only ones implementing DT3 — and skip everything else. A
supports_op failure on those two is still a real failure.
Also document that the n <= 8 gating mirrors MMVQ_MAX_BATCH_SIZE by hand
and goes stale silently if the MMVQ dispatch changes.
The old vec_dot decoded every element with its own pair of multiplications
(256 inlined get_trit per block, each qs byte re-read 5 times). Decode each
byte once instead, iterating q -> (q*3) & 0xFF two bytes at a time in 16-bit
lanes, and accumulate dp4a over base-3 digits in {0, 1, 2}; one extra dp4a
with 0x01010101 per q8_1 int, shared by both planes, turns the digit sums
back into trit sums in exact integer arithmetic, so the result stays
bit-identical to the per-trit decode. The qh bytes keep their own 4-digit
path so the padding digit is never decoded.
Builds without a native DT3 kernel rename the generic symbol to
ggml_vec_dot_dt3_q8_0 (arch-fallback.h), so test-dt3 failed to link on
them. With a weak declaration the test links everywhere and skips,
loudly, when there is no separate generic to compare against. MSVC has
no weak symbols, so there the test is compiled out.
Calls the actual ggml_vec_dot_dt3_q8_0_generic symbol against the
dispatched vec_dot and requires memcmp-equal floats. Blocks exercise
the three regions, the 79/80 and 119/120 boundaries, non-trivial qh
bytes (would expose a vectorization reading their padding 5th digit),
and scales of both and mixed signs; y reaches the full q8_0 range.
Mutation-checked: flipping one bit of a digit blend mask in the
AVX-512 kernel makes 94 of the 96 reps fail.
Decode both planes of a block with VBMI byte permutes: the *3 multiply
chain (wrapping, so it commutes with the permutation) is computed once
on the whole 56-byte block, and masked vpermb picks each element's byte
from the chain vector of its base-3 digit. The qh lanes never see 3^4,
which would read the padding 5th digit of the qh bytes. The trits reach
the integer product as xi in {0, 1, 2} via the same avg trick as
tq1_0, with VNNI dpbusd against the q8_0 bytes and sum(y) subtracted.
The per-q8_0-block sums and the float accumulation keep the exact
operation order of the generic implementation, so the result is
bit-identical to it (checked by test-dt3).
2.2x over the (autovectorized) generic on a Ryzen AI 9 HX 370.
The dequantize + cuBLAS fallback computes in fp16 (CUBLAS_COMPUTE_16F)
on fast-fp16 hardware and in TF32 under
GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32, so no analytic tolerance separates
'correct' from 'broken' there without also tracking cuBLAS numerics.
What IS ours to guarantee: the fallback must behave exactly as if the
weights were an F16 tensor holding fp16(dequant(block)). Gate on that
bit-identity and demote the analytic GEMM errors to INFO.
Elementwise max relative error explodes on cancellation whenever a true
output element is near zero, so the mul_mat checks now gate on the
relative Frobenius norm and keep the max as information. The GEMM
fallback dequantizes to fp16 on fast-fp16 hardware and DT3 weights
(d1*t1 + d2*t2) are generally not fp16-representable, so that path is
judged against a reference computed from fp16-rounded weights (taking
the better of both references so GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 also
passes). Q4_1 (same non-fp16-exact regime) and Q4_0 (fp16-exact weights,
pure GEMM error floor) run through the identical comparison as controls.
Checks the GPU backend against the validated CPU path: GET_ROWS
dequantization must match dequantize_row_dt3 bit by bit on directed
blocks (region boundaries 79/80 and 119/120, qh elements, negative
scales) and on raw random bytes; MUL_MAT must match a double precision
reference from the dequantized weights, with activations whose q8_1
quantization is exact, plus a manual trit-sum check with non-trivial qh.
Skips cleanly when no GPU backend is available.
vec_dot_dt3_q8_1 processes a whole 128-element DT3 block per call
(VDR_DT3_Q8_1_MMVQ = 4, QI_DT3 = 4), i.e. the 4 q8_1 chunks it spans,
with one pair of integer accumulators per chunk:
sum_j d8[j] * (d1*sumi1[j] + d2*sumi2[j])
The trit decode reuses ggml_cuda_dt3_get_trit with fully unrolled loops,
so all indices and pow3 factors fold into constants; no __byte_perm or
other NVIDIA-only intrinsics. Enables MUL_MAT in supports_op: ncols_dst
<= 8 takes MMVQ, larger falls back to dequantization + cuBLAS (no MMQ
tile kernel yet).
Decode one packed ternary plane with the shared ggml_cuda_dt3_get_trit
helper (shifts, masks and a small pow3 table; the uint8_t wrap-around of
the intermediate product is intentional and matches the CPU reference).
The qh bytes hold only 4 trits; their 5th base-3 digit is packing padding
that always decodes to -1 and is never read.
Wires DT3 into the generic dequantize_block templates (to fp32/fp16/bf16,
contiguous and not) and into get_rows, and enables GET_ROWS in
supports_op.
The previous byte-position test packed with the test's own packer on
both sides of the comparison, so it exercised none of the library code.
It now pins hand-computed byte values (43/100/127/42/124...) at the
region boundaries (79/80, 119/120) as ground truth and drives both
directions through the library: to_float must place each literal byte's
trit at the exact element, and from_float must produce the exact literal
byte, for both planes.
Also probes ggml_validate_row_data over all 256 byte values in qs and
qh positions (must accept exactly the 243/81 reachable codes), the
all-0xaa block, and a well-formed packed block.
The reference-quantizer disclaimer only existed in the code and in
llama-quantize --help; now it is also printed where the mistake would
actually be made, at the start of a quantization run targeting DT3.
ggml_validate_row_data now rejects unreachable code bytes: the ceiling
division packing reaches only 243 of the 256 byte values in qs and 81
in qh (4 trits plus an always-zero padding digit), so corruption that
previously loaded and generated garbage silently is caught at load
time. Previously only the two fp16 scales were checked.
quantize_dt3 no longer discards quant_weights silently: an ignored
imatrix now prints a loud warning (once), otherwise an imatrix A/B on
DT3 would come out byte-identical and invite the false conclusion that
the imatrix does nothing.
The two initial trit passes of quantize_row_dt3_ref now clamp like the
refit passes do, so a NaN input cannot push an out-of-range value from
lroundf into the packer.
test-dt3 checks the layout against an independent packer written from
the format spec: single-trit position mapping for all 256 (plane, pos)
pairs, structural byte-position checks at the region boundaries
(79/80, 119/120), exact round-trips with negative scales, byte parity
of the in-tree quantizer on already-ternary inputs, and the vec_dot
against a hand-made sum over the known trits (catches any path that
reads the padding 5th trit of the qh bytes).
test-dt3-rust-parity.py packs known trits with ternaria's pack_dt3 and
verifies that dequantize_row_dt3 (via test-dt3 --dequant) reproduces
d1*t1 + d2*t2 bit-exactly.
Also wires DT3 into the test-quantize-fns thresholds (ternary class).
Registers the type id, file type and block size, and implements numpy
dequantization (verified bit-exact against the C implementation with
gguf-py/tests/test_quants.py, including random byte payloads).
Quantization is intentionally left unimplemented, like the K-quants:
DT3 planes come from an external solver (PTQTP) and are packed
directly, so a from-float numpy path would only invite quantizing
models with the wrong algorithm.
Adds LLAMA_FTYPE_MOSTLY_DT3 at the end of the ftype enum, the loader
name/guess mappings, the quantization fallbacks (same as the other
ternary types), and the llama-quantize table entry. The table entry
warns that the in-tree quantizer is only the reference one: DT3 models
with the measured quality are produced by the external PTQTP pipeline.
The vec_dot pairs DT3 with Q8_0 (4 q8_0 blocks per DT3 block) and keeps
one integer accumulator per plane: sumf += dy * (d1*sumi1 + d2*sumi2).
Q8_0 instead of Q8_K on purpose: the planes are symmetric ternary so the
q8_K bsums are dead weight, and 32-element blocks accept any row size
that is a multiple of 128.
Trit decoding reuses unpack_plane_dt3, which reads only 4 trits per qh
byte; the 5th base-3 digit of those bytes is packer padding that always
decodes to -1 and must never be read.
Add the dual-plane ternary DT3 type to the type registry along with its
reference row functions. Each of the two planes is packed exactly like
tq1_0 with all constants halved (block of 128 elements): qs 48 -> 24
bytes over two passes of 16 and 8 bytes, qh 4 -> 2 bytes.
The trit decoding lives in a single exported helper (unpack_plane_dt3)
so that dequantization and the upcoming CPU vec_dot share it.
The reference quantizer is a greedy two-pass (plane 1 by absolute max,
plane 2 on the residual) plus two rounds of alternating least-squares
refits. It is intentionally NOT the PTQTP solver used to produce the
published DT3 models.
DT3 stores w_i = d[0]*t0_i + d[1]*t1_i with t in {-1,0,+1}, two ternary
planes over a 128-element block: 56 bytes, 3.5 bpw exactly.
Each plane uses the tq1_0 base-3 packing with every constant halved for
the smaller block (qs 48->24 B, qh 4->2 B, qs passes over 16 then 8
bytes instead of 32 then 16), which tiles 128 with no leftover bytes.
Reducing tq1_0 to 128 without halving the passes does not tile: with a
24-byte qs the first pass covers nothing and the second overruns.
Millaguie