BF16 Confusion
#10
by QrusherZA - opened
Hey, i just need some clarity around this one because i dont quiet understand it, so BF16 is not supported on Ampere (RTX 3080 in my case, so its emulated), the weights precision is still BF16, just wondering if it should not have been converted to FP16 first then to int8/int4 convrot.
I always convert BF16 to INT8 and INT4, nothing less than BF16. If you try to quantize FP8, the results will be quite poor
BF16 and FP16 are natively supported on Ampere, just not FP8 or NVFP4. They are "supported" insomuch as they'll work, but they'll be upcast to FP16.
Winnougan changed discussion status to closed