Iteration speed slows down exponentially

[orionflame]

This is for bugs only

Did you already ask in the discord?

Yes

You verified that this is a bug and not a feature request or question by asking in the discord?

No because no one seems to know why it happens.

Describe the bug

I started running the model, at first it's fast like 3-5s/it, then after 4 it, it's 1200s, then next iteration it's 2500s, etc. It becomes insanely slow. I don't know why.

It's using all of the 24GB VRAM of my RTX 6000 and 100% usage most of the time. Not sure what else I can do.

Now I am trying low VRAM flag.

[ogladkov]

I'm experiencing the same now.

[orionflame]

This is after a windows restart and almost nothing is using the GPU. Laptop screen is using the Intel GPU. No solution has been provided yet.

[NBSTpeterhill]

please show your setting of config, then we can see what happend.

[orionflame]

---
job: extension
config:
  # this name will be the folder and filename name
  name: "cool_v1"
  process:
    - type: 'sd_trainer'
      # root folder to save training sessions/samples/weights
      training_folder: "output"
      # uncomment to see performance stats in the terminal every N steps
#      performance_log_every: 1000
      device: cuda:0
      # if a trigger word is specified, it will be added to captions of training data if it does not already exist
      # alternatively, in your captions you can add [trigger] and it will be replaced with the trigger word
      trigger_word: "pankhg30"
      network:
        type: "lora"
        linear: 16
        linear_alpha: 16
      save:
        dtype: float16 # precision to save
        save_every: 250 # save every this many steps
        max_step_saves_to_keep: 10 # how many intermittent saves to keep
        push_to_hub: false #change this to True to push your trained model to Hugging Face.
        # You can either set up a HF_TOKEN env variable or you'll be prompted to log-in         
#       hf_repo_id: your-username/your-model-slug
#       hf_private: true #whether the repo is private or public
      datasets:
        # datasets are a folder of images. captions need to be txt files with the same name as the image
        # for instance image2.jpg and image2.txt. Only jpg, jpeg, and png are supported currently
        # images will automatically be resized and bucketed into the resolution specified
        # on windows, escape back slashes with another backslash so
        # "C:\\path\\to\\images\\folder"
        - folder_path: "C:\\Users\\alivs\\Downloads\\Pic\\Model"
          caption_ext: "txt"
          caption_dropout_rate: 0.05  # will drop out the caption 5% of time
          shuffle_tokens: false  # shuffle caption order, split by commas
          cache_latents_to_disk: true  # leave this true unless you know what you're doing
          resolution: [ 512, 768, 1024 ]  # flux enjoys multiple resolutions
      train:
        batch_size: 1
        steps: 4000  # total number of steps to train 500 - 4000 is a good range
        gradient_accumulation_steps: 1
        train_unet: true
        train_text_encoder: false  # probably won't work with flux
        gradient_checkpointing: true  # need the on unless you have a ton of vram
        noise_scheduler: "flowmatch" # for training only
        optimizer: "adamw8bit"
        lr: 1e-4
        # uncomment this to skip the pre training sample
#        skip_first_sample: true
        # uncomment to completely disable sampling
#        disable_sampling: true
        # uncomment to use new vell curved weighting. Experimental but may produce better results
#        linear_timesteps: true

        # ema will smooth out learning, but could slow it down. Recommended to leave on.
        ema_config:
          use_ema: true
          ema_decay: 0.99

        # will probably need this if gpu supports it for flux, other dtypes may not work correctly
        dtype: bf16
      model:
        # huggingface model name or path
        name_or_path: "black-forest-labs/FLUX.1-dev"
        is_flux: true
        quantize: true  # run 8bit mixed precision
        low_vram: true  # uncomment this if the GPU is connected to your monitors. It will use less vram to quantize, but is slower.
      sample:
        sampler: "flowmatch" # must match train.noise_scheduler
        sample_every: 250 # sample every this many steps
        width: 1024
        height: 1024
        prompts:
          # you can add [trigger] to the prompts here and it will be replaced with the trigger word
#          - "[trigger] holding a sign that says 'I LOVE PROMPTS!'"\
          - "[trigger] standing confidently in front of a luxurious shopping mall in central Bangkok, wearing a chic summer dress with floral patterns. The bright, bustling city street is filled with people and colorful signs written in Thai, capturing the essence of the city's vibrant atmosphere. [trigger] smiles warmly with a shopping bag in one hand, while the sun glows gently, casting soft shadows on the ground."
          - "[trigger] at a tropical beach during sunset, wearing a stylish bikini in a light pastel color. [trigger] is holding a glass of freshly squeezed orange juice with a tiny umbrella garnish and smiling brightly. The golden sun reflects on the calm waves, with the silhouette of palm trees in the background, while gentle waves lap at [trigger]'s feet, making the scene feel warm and inviting."
          - "[trigger] sitting cross-legged on a grassy field dotted with wildflowers, wearing a casual white tank top and denim shorts. [trigger] holds up a handmade sign that says 'Life is Beautiful' in playful handwriting. The sun shines softly in the sky, casting a golden glow, while a light breeze ruffles [trigger]'s hair, creating a serene, photorealistic atmosphere."
          - "[trigger] sitting at an outdoor café on a charming street lined with European-style buildings. [trigger] is dressed in a cozy, oversized sweater, sipping on a latte with a book open on the table in front of her. [trigger] gazes thoughtfully into the distance with a slight smile, as the afternoon sun casts gentle light across the cobblestone path and the café’s decorative potted plants."
          - "[trigger] at a lively outdoor festival, illuminated by a canopy of fairy lights and lanterns hanging overhead. [trigger] wears a flowy maxi dress with subtle floral designs and holds a small plate of traditional snacks. The scene is full of movement, with people in the background laughing and enjoying the event. [trigger]'s face glows under the warm, festive lighting, eyes twinkling with joy as she stands amid the celebration."
        neg: ""  # not used on flux
        seed: 42
        walk_seed: true
        guidance_scale: 4
        sample_steps: 20
# you can add any additional meta info here. [name] is replaced with config name at top
meta:
  name: "[name]"
  version: '1.0'

[orionflame]

also i tried both low_vram: true and when it was commented out. Same result.

[orionflame]

any ideas? am i doing something wrong?

[poluramus]

I have the same problem. Seems to works fine for 100 to 1000 steps, then the it/s goes up from 20 to 250 s/it. I recreated a fresh new venv 2 or 3 weeks ago. Also tried without samples images, which didn't solve it. Even after a PC restart, starting training(no other application running), close my monitor, the problem occurs. This is a major bug... I have a 3090 and 64g of ram

[orionflame]

It would be nice to have some debugging steps we can perform to narrow down the issue.

[orionflame]

I have the same problem. Seems to works fine for 100 to 1000 steps, then the it/s goes up from 20 to 250 s/it. I recreated a fresh new venv 2 or 3 weeks ago. Also tried without samples images, which didn't solve it. Even after a PC restart, starting training(no other application running), close my monitor, the problem occurs. This is a major bug... I have a 3090 and 64g of ram

Did you figure it out?

[poluramus]

I have the same problem. Seems to works fine for 100 to 1000 steps, then the it/s goes up from 20 to 250 s/it. I recreated a fresh new venv 2 or 3 weeks ago. Also tried without samples images, which didn't solve it. Even after a PC restart, starting training(no other application running), close my monitor, the problem occurs. This is a major bug... I have a 3090 and 64g of ram

Did you figure it out?

No, unfortunately. After having 5 hours of stable performance at normal speed(which is very good lol....), I thought I had resolved the issue by removing everything related to logs in the config file. However, the bug resurfaced. For the first time, I’m seriously considering switching frameworks, especially since the developers have been unresponsive for a while.

[orionflame]

I have the same problem. Seems to works fine for 100 to 1000 steps, then the it/s goes up from 20 to 250 s/it. I recreated a fresh new venv 2 or 3 weeks ago. Also tried without samples images, which didn't solve it. Even after a PC restart, starting training(no other application running), close my monitor, the problem occurs. This is a major bug... I have a 3090 and 64g of ram

Did you figure it out?

No, unfortunately. After having 5 hours of stable performance at normal speed(which is very good lol....), I thought I had resolved the issue by removing everything related to logs in the config file. However, the bug resurfaced. For the first time, I’m seriously considering switching frameworks, especially since the developers have been unresponsive for a while.

Yes the devs are sleeping.

What other frameworks would you recommend?