Inability to apply qwen2-vl or llama-3.2-vision

[Yoo-Youngjae]

Is your feature request related to a problem? Please describe.

I would like to leverage guidance on vlm models such as qwen2-vl or llama3.2-vision.

However, these are difficult to use because they use the latest transformer's MllamaForConditionalGeneration and Qwen2VLForConditionalGeneration, which causes an error in the current guidance.

Is it possible to update these to support them more widely?

Additional context

Or is it because I am not good at using it?

If anyone has solved it, please let me know the solution.