Tasks: Add image-text-to-text pipeline and inference API to task page #1039
Conversation
merveenoyan
requested review from
SBrandeis,
gary149,
Wauplin,
julien-c,
pcuenca and
ngxson
as code owners
merveenoyan
changed the title
| { | ||
| "role": "assistant", | ||
| "content": [ | ||
| {"type": "text", "text": "There's a pink flower"}, | ||
| ], | ||
| }, |
There was a problem hiding this comment.
It's a bit strange to me that the input ends with an assistant turn. I see in the example later that the model completes the sentence with more details, but I'm not sure this is compatible with all chat VLMs. Can we maybe skip the assistant role from the input and see if the model provides a good description of the image?
There was a problem hiding this comment.
This has not been addressed, I think it's unusual that users supply an assistant turn with the input.
There was a problem hiding this comment.
sorry I thought I answered to this. basically it's to give more control to further align the output during inference. I used the same example here where you can see the output https://huggingface.co/docs/transformers/en/tasks/image_text_to_text
There was a problem hiding this comment.
But that example ends with an user role, while this one ends with an assistant role. I don't think models are expected to be queried with an assistant role in the last turn: they receive a conversation that always ends with an user role, and then they respond with an assistant message.
There was a problem hiding this comment.
sorry I think I should've sent the particular title, here you go https://huggingface.co/docs/transformers/en/tasks/image_text_to_text#pipeline I meant this one
There was a problem hiding this comment.
Still looks weird / confusing to me, but ok if you feel strongly about it.
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
|
ah need to lint |
| { | ||
| "role": "assistant", | ||
| "content": [ | ||
| {"type": "text", "text": "There's a pink flower"}, | ||
| ], | ||
| }, |
There was a problem hiding this comment.
This has not been addressed, I think it's unusual that users supply an assistant turn with the input.
|
@pcuenca I changed it since it looked counterintuitive as an example, merging. thanks for the review |
..and remove the long inference