ollama upload

[Xyz00777]

please upload to ollama, there you can also upload different quantization or not quantization versions.

[AHEADer]

Hi, we found the ollama + gguf way behave worse than the cloud deployment. We recommend to use the huggingface inference endpoints currently. We will try uploading to ollama if local deployment matches the online inference in model performance.

[Xyz00777]

Okay, I thought it would be good too because you wrote on the README that you encounter to not be able to upload a not quantized version to gguf, and I already seen many fp16 not quantized version on ollama... (I hope it's correct that fp16 are not quantized, as far as I understood the context)

[AHEADer]

Yes, fp16 is not the quantized version, but we trained with bf16, so fp16 version may not have the expected precision. BF16 support with Ollama is poor and super slow.