Two integer dot products per weight cancel the 2x int8-over-fp16 tensor
core advantage, so at large batch MMQ cannot beat dequantize + fp16
cuBLAS (measured 331 vs 426 t/s pp512 on the 27B, RTX 4060 Ti, while
running at the same ~20% of its int8 ceiling as Q2_K MMQ does of its
own). MMQ still avoids the dequantization round-trip at moderate batch;
the threshold default is provisional until the crossover is measured
(GGML_CUDA_DT3_MMQ_MAX_BATCH overrides it for that measurement).
Millaguie
e9616d415f