Add Quantization Support to Backend for Efficient Model Loading on Smaller GPUs

[abhyudayhari]

Description

Currently, the backend service loads models onto the GPU for inference. However, the models are too large for smaller GPUs like NVIDIA 3050 Ti, NVIDIA 1080 which are present on laptops or low-end devices, leading memory overflows i.e CUDA Error. To improve the service's compatibility and efficiency on these devices, I propose adding quantization to the backend.

The error occurred due to the fact that the model is larger than the GPU's VRAM limit

The GPU usage on NVIDIA 3050 Ti after loading only 2 models out of the Bunch.

Proposed Solution

Integrate Model Quantization:

Add support for model quantization (8bit or 4bit) to the backend service.
Utilize available quantization libraries/frameworks like bitsandbytes / accelerate to lower the GPU usage.

Dynamic Quantization Option:

Allows the system to make the decision whether quantization of model is required or not on the basis of the size of the user's GPU. Also allows users to manually enable or disable quantization through a configuration parameter.

Advantages of Quantization:

Reduced GPU Usage
The Quantization reduces the Model's size. And allows it to be loaded on devices which were earlier incapable of loading the model, due to memory constraint

Terminal Output due to quantization

The GPU usage after quantizing all the models in the backend

Since this leaves plenty of space empty in the GPU, hence, it allows for larger loads that can be handled by the GPU.

Faster Inference

Since Quantization works at the bit level of each parameter of the model, it reduces the size of each parameter, and thus allows more operations to done in a single cycle. Hence making the inference faster.

The link for the mathematical insights of Quantization, for better understanding
https://chautuankien.medium.com/quantization-technique-part-1-d8ab5c0379e9

[abhyudayhari]

@Roaster05 I would love to hear your comments on this issue!!. Please let me know what can be done more on this