With MMQ, ncols_dst > 8 is an integer path in the same numerical regime
as MMVQ and is judged against the exact reference at 1e-5 instead of
riding the loose GEMM gate. The regime is told apart by the result
itself; a GEMM fallback (backends without DT3 MMQ) keeps the fp16
reference and the F16 GEMM bit-identity control. Also add ncols_dst=100
to exercise a wide tile with a clamped last column block.
The Vulkan backend now forces fp32 accumulators for the DT3 dequant
fallback, so the DT3 GEMM is no longer bit-identical to the backend's
default-precision F16 GEMM (fp16 accumulators on fp16-capable Vulkan
devices). Run the F16 control at both the default and the F32-forced
precision and require bit-identity with either one. CUDA still matches
the default-precision control; Vulkan matches the F32 one - both with
0 mismatches.
Wires the dual-plane ternary type into the Vulkan backend through three
paths only:
- get_rows and the generic scalar mul_mat_vec use per-element decode in
dequant_funcs.glsl: the byte and base-3 digit are located from the
element index (regions qs[0..16), qs[16..24), qh[0..2)), the byte is
multiplied by 3^n mod 256 and the top digit taken. w = d1*t1 + d2*t2
is accumulated in fp32; both products are exact so the sum carries a
single float rounding and reproduces the CPU reference bit by bit
(verified: 0 mismatches over the 112-block synthetic test and over
214,695,936 elements of real model tensors).
- larger matmuls fall back to dequant_dt3.comp (decode each byte once
with q <- q*3 mod 256, fp32 sum, one rounding at the f16 write) plus
the existing f16 matmul pipelines.
The qh bytes hold only 4 trits; their 5th base-3 digit is packing
padding that decodes to -1, so both decoders stop at 4 digits.
Deliberately NOT implemented, and declined instead of half-supported:
no coopmat/coopmat2/MMQ shaders are generated for DT3, and supports_op
answers false for MUL_MAT_ID (mul_mat_vec_id shaders are not generated
either). GET_ROWS and MUL_MAT answer true.
test-dt3-gpu accepts the Vulkan backend (and IGPU-type devices) and
passes on RADV STRIX1: dequant 0 mismatches, mul_mat n<=8 norm rel err
~1e-8, GEMM fallback bit-identical to an F16 GEMM on fp16-rounded
weights.
The test picked the first GPU device it found and treated an unsupported
op as a failure. Vulkan and SYCL answer supports_op == false for DT3,
which is the right answer for them and not a bug to report, so the test
went red on machines that were behaving correctly. Metal is worse: it
answers true for almost any type but has no DT3 shader, so the run died
in pipeline compilation halfway through.
Pick the backend by name instead — CUDA and HIP (which reports itself as
ROCm) are the only ones implementing DT3 — and skip everything else. A
supports_op failure on those two is still a real failure.
Also document that the n <= 8 gating mirrors MMVQ_MAX_BATCH_SIZE by hand
and goes stale silently if the MMVQ dispatch changes.
The dequantize + cuBLAS fallback computes in fp16 (CUBLAS_COMPUTE_16F)
on fast-fp16 hardware and in TF32 under
GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32, so no analytic tolerance separates
'correct' from 'broken' there without also tracking cuBLAS numerics.
What IS ours to guarantee: the fallback must behave exactly as if the
weights were an F16 tensor holding fp16(dequant(block)). Gate on that
bit-identity and demote the analytic GEMM errors to INFO.
Elementwise max relative error explodes on cancellation whenever a true
output element is near zero, so the mul_mat checks now gate on the
relative Frobenius norm and keep the max as information. The GEMM
fallback dequantizes to fp16 on fast-fp16 hardware and DT3 weights
(d1*t1 + d2*t2) are generally not fp16-representable, so that path is
judged against a reference computed from fp16-rounded weights (taking
the better of both references so GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 also
passes). Q4_1 (same non-fp16-exact regime) and Q4_0 (fp16-exact weights,
pure GEMM error floor) run through the identical comparison as controls.
Checks the GPU backend against the validated CPU path: GET_ROWS
dequantization must match dequantize_row_dt3 bit by bit on directed
blocks (region boundaries 79/80 and 119/120, qh elements, negative
scales) and on raw random bytes; MUL_MAT must match a double precision
reference from the dequantized weights, with activations whose q8_1
quantization is exact, plus a manual trit-sum check with non-trivial qh.
Skips cleanly when no GPU backend is available.
Millaguie