The test picked the first GPU device it found and treated an unsupported
op as a failure. Vulkan and SYCL answer supports_op == false for DT3,
which is the right answer for them and not a bug to report, so the test
went red on machines that were behaving correctly. Metal is worse: it
answers true for almost any type but has no DT3 shader, so the run died
in pipeline compilation halfway through.
Pick the backend by name instead — CUDA and HIP (which reports itself as
ROCm) are the only ones implementing DT3 — and skip everything else. A
supports_op failure on those two is still a real failure.
Also document that the n <= 8 gating mirrors MMVQ_MAX_BATCH_SIZE by hand
and goes stale silently if the MMVQ dispatch changes.
The dequantize + cuBLAS fallback computes in fp16 (CUBLAS_COMPUTE_16F)
on fast-fp16 hardware and in TF32 under
GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32, so no analytic tolerance separates
'correct' from 'broken' there without also tracking cuBLAS numerics.
What IS ours to guarantee: the fallback must behave exactly as if the
weights were an F16 tensor holding fp16(dequant(block)). Gate on that
bit-identity and demote the analytic GEMM errors to INFO.
Elementwise max relative error explodes on cancellation whenever a true
output element is near zero, so the mul_mat checks now gate on the
relative Frobenius norm and keep the max as information. The GEMM
fallback dequantizes to fp16 on fast-fp16 hardware and DT3 weights
(d1*t1 + d2*t2) are generally not fp16-representable, so that path is
judged against a reference computed from fp16-rounded weights (taking
the better of both references so GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 also
passes). Q4_1 (same non-fp16-exact regime) and Q4_0 (fp16-exact weights,
pure GEMM error floor) run through the identical comparison as controls.
Checks the GPU backend against the validated CPU path: GET_ROWS
dequantization must match dequantize_row_dt3 bit by bit on directed
blocks (region boundaries 79/80 and 119/120, qh elements, negative
scales) and on raw random bytes; MUL_MAT must match a double precision
reference from the dequantized weights, with activations whose q8_1
quantization is exact, plus a manual trit-sum check with non-trivial qh.
Skips cleanly when no GPU backend is available.
Millaguie