blip2_qformer train question

[Codingfarmer-hkl]

##================= Image Captioning ========================##
decoder_input_ids = text_tokens.input_ids.clone()
decoder_input_ids[:, 0] = self.tokenizer.bos_token_id
labels = decoder_input_ids.masked_fill(
decoder_input_ids == self.tokenizer.pad_token_id, -100
)

    query_atts = torch.ones(query_tokens.size()[:-1], dtype=torch.long).to(
        image.device
    )
    attention_mask = torch.cat([query_atts, text_tokens.attention_mask], dim=1)
    lm_output = self.Qformer(
        decoder_input_ids,
        attention_mask=attention_mask,
        past_key_values=query_output.past_key_values,
        return_dict=True,
        labels=labels,
    )

    loss_lm = lm_output.loss

    return BlipOutput(
        loss=loss_itc + loss_itm + loss_lm,
        loss_itc=loss_itc,
        loss_itm=loss_itm,
        loss_lm=loss_lm,
    )
    attention_mask = torch.cat([query_atts, text_tokens.attention_mask], dim=1)，
    attention_mask should be a mask, text attention mask should not be bidirectional？？？？？？？

[parth1313]

Hi @Codingfarmer-hkl ,
I want to train Qformer from scratch, can you guide me step by step?

Currently, I am getting the error :

2025-01-21 05:35:02.641286: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-21 05:35:02.660189: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-21 05:35:02.681263: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-21 05:35:02.687851: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-21 05:35:02.704605: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-21 05:35:03.917998: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
error: XDG_RUNTIME_DIR not set in the environment.
ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM default
ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM default
Traceback (most recent call last):
  File "/content/LAVIS/train.py", line 16, in <module>
    import lavis.tasks as tasks
  File "/content/LAVIS/lavis/__init__.py", line 15, in <module>
    from lavis.datasets.builders import *
  File "/content/LAVIS/lavis/datasets/builders/__init__.py", line 8, in <module>
    from lavis.datasets.builders.base_dataset_builder import load_dataset_config
  File "/content/LAVIS/lavis/datasets/builders/base_dataset_builder.py", line 18, in <module>
    from lavis.processors.base_processor import BaseProcessor
  File "/content/LAVIS/lavis/processors/__init__.py", line 29, in <module>
    from lavis.processors.audio_processors import BeatsAudioProcessor
  File "/content/LAVIS/lavis/processors/audio_processors.py", line 17, in <module>
    from lavis.models.beats.Tokenizers import TokenizersConfig, Tokenizers
  File "/content/LAVIS/lavis/models/__init__.py", line 42, in <module>
    from lavis.models.blip2_models.blip2_vicuna_xinstruct import Blip2VicunaXInstruct
  File "/content/LAVIS/lavis/models/blip2_models/blip2_vicuna_xinstruct.py", line 22, in <module>
    from peft import (
  File "/usr/local/lib/python3.11/dist-packages/peft/__init__.py", line 22, in <module>
    from .auto import (
  File "/usr/local/lib/python3.11/dist-packages/peft/auto.py", line 32, in <module>
    from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING
  File "/usr/local/lib/python3.11/dist-packages/peft/mapping.py", line 25, in <module>
    from .mixed_model import PeftMixedModel
  File "/usr/local/lib/python3.11/dist-packages/peft/mixed_model.py", line 29, in <module>
    from .peft_model import PeftModel
  File "/usr/local/lib/python3.11/dist-packages/peft/peft_model.py", line 37, in <module>
    from transformers import Cache, DynamicCache, EncoderDecoderCache, PreTrainedModel
ImportError: cannot import name 'Cache' from 'transformers' (/usr/local/lib/python3.11/dist-packages/transformers/__init__.py)

When running : !python train.py --cfg-path lavis/projects/blip2/train/pretrain_stage1.yaml

Also, when running: !pip install salesforce-lavis
I got,

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sentence-transformers 3.3.1 requires transformers<5.0.0,>=4.41.0, but you have transformers 4.26.1 which is incompatible.
albucore 0.0.19 requires opencv-python-headless>=4.9.0.80, but you have opencv-python-headless 4.5.5.64 which is incompatible.
albumentations 1.4.20 requires opencv-python-headless>=4.9.0.80, but you have opencv-python-headless 4.5.5.64 which is incompatible.
Successfully installed antlr4-python3-runtime-4.9.3 braceexpand-0.1.7 cfgv-3.4.0 contexttimer-0.3.3 decord-0.6.0 distlib-0.3.9 fairscale-0.4.4 ftfy-6.3.1 identify-2.6.6 iopath-0.1.10 jedi-0.19.2 nodeenv-1.9.1 omegaconf-2.3.0 opencv-python-headless-4.5.5.64 opendatasets-0.1.22 portalocker-3.1.1 pre-commit-4.1.0 pycocoevalcap-1.2 pydeck-0.9.1 python-magic-0.4.27 salesforce-lavis-1.0.2 streamlit-1.41.1 timm-0.4.12 tokenizers-0.13.3 transformers-4.26.1 virtualenv-20.29.1 watchdog-6.0.0 webdataset-0.2.100
WARNING: The following packages were previously imported in this runtime:
  [pydevd_plugins]
You must restart the runtime in order to use newly installed versions.

And, when running: !pip install -r requirements.txt
I got,

ERROR: Could not find a version that satisfies the requirement open3d==0.13.0 (from versions: 0.18.0, 0.19.0)
ERROR: No matching distribution found for open3d==0.13.0

Please note that i am using:

• Python 3.11.11
• Google Colab with A100
• requirements.txt