How to reproduce the result of Figure 6?

[HenryHZY]

Hi @tsb0601 @s9xie , thanks for your great projector.

I would like to inquire about the process of reproducing the result of Figure 6 (i.e., MLLM’s performance on visual patterns). In scripts/evaluate_vlm.py, it seems that we have to obtain the logit of image and text.

However, take LLaVA as an example, we cannot directly obtain its logits. Or do you use the input as "<image> Is it a photo of <statement>" to LLaVA and obtain the result? I am a little confused about it.

Can you provide an instruction or a script to evaluate LLaVA on MMVP_VLM? With this, I can follow these steps to reproduce the results for other models.

[HenryHZY]

Just find that MMVP and MMVP-VLM share the same images but with different imageID.
Do you just obtain the result shown in Figure 4 and then convert the result into Figure 6? As stated in Section 3.3 of the paper, "To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns."
If so, could you please share the imageID mapping file from MMVP to MMVP-VLM (e.g., MMVP 243.jpg -> MMVP-VLM Orientation 2.jpg)?

[HenryHZY]

Any update here? :)

[tsb0601]

Hi,

Thank you for your interest and questions! Yes, we convert the results from Figure.4 to Figure.6 as the questions in MMVp can also be categorized into visual patterns. We will update the repo tomorrow and add a mapping that contains visual patterns for each question in MMVP.

Sincerely

[HenryHZY]

@tsb0601 Hi, is the release schedule going well?

[tsb0601]

Hi, sorry for the delay, we will release it this week shortly after CVPR results

[yeonju7kim]

Hi, Thank you for your great work.
May I know the mapping of questions and visual patterns?

[Haochen-Wang409]

Hi @tsb0601
Any updates about the mapping?

[HashmatShadab]

@tsb0601 ???