Finetuning Flux Dev on a 3090! (Local LoRA Training)

[@justinmcdonald1930]

I love how your notepad has a bunch of open text files, too. Glad I'm not the only one with hundreds of text files open in notepad!

[@Larimuss]

your using notepad? I had to upgrade to Onenote + VS code + sticky notes + notepad :P

[@seto007]

Possibly important to note: If you've been pulling your hair out trying to figure out why the training image folder is causing issues when trying to run the script, you might need to put "c:\\" instead of "c:\". Don't know why this is the case for some and not for others, but I'm glad I finally figured it out.

[@promptcrafted]

@@seto007 oh that’s weird, are you in windows? That doesn’t seem typical.

[@seto007]

@@promptcrafted yep. I've had a similar sort of issue happen on very odd occasions with some Python scripts, but it's incredibly rare and the error code has usually told me that it requires two dashes for the disk. This one just said that the directory wasn't valid.

[@mistermcluvin2425]

Interesting, I had a similar issue and I flipped the direction from \ to /....then it ran without errors. I didn't even think about doubling the \\ characters up. I thought it was because python was recognizing the \ and next letter as a control character? I am on windows 11.

[@Bolstered]

@@mistermcluvin2425 This is bringing me back to posting the "shrug" emoji in forums and people saying you lost an arm because you had to double up the slashes to get it to show up

[@mistermcluvin2425]

Just FYI for everyone, I did an update just now before training another Lora and noticed new line items in the sample config file, they added this....... # on windows, escape back slashes with another backslash so # "C:\\path\\to\\images\\folder"

[@krakenunbound]

I literally just downloaded your LoRA randomly from CivitAI and then went browsing on youtube for something to watch. Curious about training my own Lora I came in to check out your video and see that your name is the same as what I downloaded. Cool!

[@benmacauly]

Thanks for this, this came out way better than the secources kohya version

[@EricJamesHurley]

Thanks for all your OS contributions Araminta!

[@cuinheaven3]

One key element missing from this video is how to populate the caption files that accompany the training images. Should it be comma separated words and phrases? New line separated? Any words or phrases to avoid? That sort of thing. Other than that it was a short and helpful video, thanks!

[@erans]

I've found that not using any captions at all works the best (for person training, dont know about styles)

[@dgspitzer4771]

Thank you for the tutorial! That's really helpful!

[@flin4557]

Thanks for the video, easy to follow and helpful.

[@astilen5647]

This is the best guide I would show to a complete beginner that wants to eventually advance to more and more complex workflows, I would encourage you (am prompting you) to wait until you have enough content ready and then return for a part 2, with more nuanced options on training quality and accuracy, and maybe a popular comfyUI setup.

[@doctordiffusion]

Thank you! They added a "examples" folder inside of the "config" so you need to add that to the final command.

[@promptcrafted]

@@doctordiffusion oh actually you’re just supposed to move the file to config, it’s intentional

[@reo4153]

18:19 is there any other online tutorial available showing how to used the trained model

[@81ender]

I trained mine using the instructions then when I test it I can't see the adaptation of imahges so a sample how to use the trained model would be useful to check if I missed something

[@AgustinCaniglia1992]

I can't even run a flux lora on comfy but I am watching a lora training video.

[@promptcrafted]

@@AgustinCaniglia1992 don’t forget to update comfy

[@AgustinCaniglia1992]

@@promptcrafted I have. Mayybe the lora isn't compatible with the flux version I was running. I will try later with the other versions. I also had been generating some very blurri images with flux sometimes randomly. Have you experienced that issue yourself?

[@Gausch2]

Awesome guide! Had problems with the yaml, but Claude helped me fixing it. Tried it with 2000 steps for finetuning on my face (22 photos), but thats not nearly enough. Trying 8000 next time.

[@CardboardBoxed]

The best results (after agonising days of testing) is about 50 images, 15 images front facing close-ups and the rest are all varying angles and distances from the camera. For me it was done it about 4000 steps.

[@promptcrafted]

@@Gausch2 I think 3000 should be fine. You don’t need 55 images imo

[@ronnydaca]

I try for the first time with only six picture x3, total 18 picture (but only six original) and 1000 steps and results is terrific. Second and third try I use 18 different pictures (best quality ) and 4000 steps and results is medium. I don't understand why.

[@CardboardBoxed]

@@ronnydaca it’s over training

[@michaelknight3745]

that would have been good to know what are the other possible optimizer and scheduler. i've read somewhere someone use adafactor and it worked well but i didn't maked that optimizer to work. i may miss setting for it, but what are they...?

[@NateBirkas]

Anyone else having issues when pulling the model? Part 2 of 3 safetensors getting stuck at like 9.93G out of 9.95G. Tried twice, have plenty of HDD space.

[@televizor9300]

Just one question - I noticed that your dataset isn't exclusively 512x512, 1024x1024 (or 1:1 AR in general). I had an understanding that Flux wants square 512 images. Am I wrong? All aspect ratios are fine?

[@Vinz-VYG]

Ostris in Dataset preparation section say : "You do not need to crop/resize your images. The loader will automatically resize them and can handle varying aspect ratios"

[@rakzodya]

@@Vinz-VYG thanks

[@niccolon8095]

followed the steps and im only getting 1 .txt file even though I have many images. In the CMD prompt it describes all my images but only saves 1 .txt file that describes 1 random image .. any idea?

[@nomnomAI]

Thanks. I just got this working. had a pain with the ipadapter. i had to redo all of the steps to get it working. ty!

[@blowit2307]

what did you do? Cuz i keep re doing and it fails every time

[@nomnomAI]

@@blowit2307 what step is it failing on?

[@TalhaKarakoyunlu]

same ipadapter issue, couldn't fix it though

[@henrywang4010]

Is there a way to cache the downloaded flux models so that they don't have to be download again later?

[@zachishot17]

from your ai-toolkit folder git clone the flux dev repo from hugging face. then in the yaml file point it to the flux dev cloned repo directory "FLUX1-dev"

[@hefland]

@@zachishot17 Thank you! This is the answer I was looking for.

[@MisterWealth]

Do you need to change steps or epochs depending on the amount of images you have?

[@wonder111]

Thanks for the tutorial, and looking forward to an update. After I manged to run the script, the models do not load. Loading 1 of 3 tensorfiles, and the first never completes. Using a 3090 with an Eizo monitior that might take more Vram. I think a fast connection speed is manditory. I will be upgrading to that mythical 100mb/s next week so I will try then. I am attempting to make a lora from my own work, which has a unique aesthetic, so I would rather do this locally. I can see potential in offering this possibility to non-AI artists who have a particular painting style. As way to generate ideas in their own visual language.

[@44Beatzz]

thank you so much for this video.

[@promptcrafted]

@@44Beatzz no problem

[@jassimibrahim6535]

hello, is there a method to finetune the model without using lora ? i have 20k images that i wish to use for finetuning flux dev ill appreciate any help

[@netgeek3010]

One issue when I run the script in CMD line it gives me an issue cannot import name 'packaging' from pkg_resources'

[@elite_pencil]

For anyone having issues with the requirements part (i was told no 121 version was findable, or something like that) check your python version. Try 310, 313 was throwing errors left and right.

[@PetitShya]

what is (moduleNotFoundError: No module named"dotenv") ? I setup env file but……look like doesn't work? 🤔

[@grahamulax]

Is there a way to use the fluxdev1 model that ive downloaded for comfyui use instead of referencing it and pulling it from huggingface?!

[@promptcrafted]

No, it’s trained on diffusers so it needs the diffusers model

[@grahamulax]

@@promptcrafted hah I just sat down from training last night (worked amazingly! 6000 steps off of 9 images that were logos) and realized that! I asked because I redo my computer a lot and clean like a freak! BUT heres a pro tip for anyone reading. If you dont want your C drive full of .cache, put that on another drive and symlink it to your user. So if you put the .cache folder on your d drive root lets say, then youd type in: mklink /d "C:\Users\Administrator\.cache" "D:\.cache" (replace Administrator with your user name). That way you dont have to redownload if you keep it in a safe place! :)

[@zachishot17]

from your ai-toolkit folder git clone the flux dev repo from hugging face. then in the yaml file point it to the flux dev cloned repo directory "FLUX1-dev"

[@denismatejcik7033]

you can use "TAB" to autocomplete names in cmd so u dont have to type them and also avoid typos

[@AiHello]

You missed out on explaining how the training imagers and captions are laid out. Overall helpful video though

[@EPK_AI_DUBS]

I'm having the same questions now. I've trained loras with and without image captioning and I would like to know the differences. I also don't know if the captioning is working or not

[@zabique]

where is model stored?

[@francescosetragno694]

How to use the trained lora weights? Should I merge them with the pretrained model?

[@lenny_Videos]

Thanx for the video :-)

[@MisterWealth]

Do we need regularization images? It seems like we might not need it?

[@erans]

no need, also no need for captions. give it a try

[@MisterWealth]

@@erans How are you triggering what you want? :|

[@erans]

@@MisterWealth The trigger word i set in the config files of the training, something like "ohwxman"

[@taezonday]

Amazing! And here I was getting ready to instal Linux to run Simpletuner lol. Speaking of, do you know how Ostris compares with Simpletuner, are they using much different methods?

[@promptcrafted]

@@taezonday I haven’t been a fan of the results of simpletuner.

[@44Beatzz]

i was in the exact same situation lol

[@BikingWIthPanda]

trained like 5 loras now using simpletuner and ostris ai-toolkit ends up leaving artifacts through them. simpletuner converges quickly. looks great

[@max49a]

Which Lora loader works with Flux LoRAs trained with this method? I haven't been successful getting any to work in ComfyUI

[@Doowi92]

does anyone know why some LoRA models fully occupy the RAM when loaded in Forge Web UI for generating, while others don't use up all the memory?

[@espedairsystems]

Gave it a go ... ran out of VRAM ... on a 3090 ... what size images do you have for your training data ?

[@promptcrafted]

@@espedairsystems you need 24 gb, I have a ti but some people have disconnected their monitors while it’s running.

[@espedairsystems]

I think my image files are too big

[@q8evilspirit]

Amazing

[@CemilAL]

I'm using an aws instance with 24gb vram, yet i still got a memory error stating that pytorch tried to allocate 50mb but there was only 30mb left or sth. Since aws instance already doesnt use any vram to output visual inputs to monitors or anything else, what else i can do to free a couple mb of vram?

Hi, thank you so much for the tutorial. I was training a lora and just closed the terminal window before it finished, I wanted to resume it now but when I ran it again it started from the beginning. How do you stop the training to continue later? Edit: nvm I just saw it's stopped with Ctrl + C. I'm dumb. Thanks !

[@promptcrafted]

Glad you found the solution!

[@Gamer4Eire]

Amazing work Thank You! noticed a .dll issue when running to do with Fbgemm, missing step for installation?

[@promptcrafted]

@@Gamer4Eire Not sure - are you doing it locally in a windows environment?

[@Gamer4Eire]

@@promptcrafted yes I am. I found the missing DLL and added is manually but looks like something related to Visual Studio.. I know it’s very beta so I don’t mind hacking it.

[@promptcrafted]

@@Gamer4Eire I haven’t seen a big difference but feel free to submit a pr to Ostris

[@kmdcompelelct]

I had that same problem. If you already have VS 2022 installed, go to modify and install the MSVC v143 - VS 2022 C++ x64/x86 build tools. Or at least, that's what fixed it on my system.

[@VaibhavShewale]

where was training?

[@wndrflx]

Do you have a recommended number of images? I was using 50 and my 250 and 500 samples showed almost no change in style, while yours changed quite a bit early on. Would using less maybe allow things to change faster?

[@rakzodya]

thanks, i'll try it.

[@evgenypuzyrev9461]

hi! Thank you for tutorial When i train lora character it it turns out to be really similar but with absolutely random haircut Dataset is good, haircut is clearly visible from different angles I used from 1000 to 2000 steps Maybe i should use more steps? but I have doubts that it will affect

Hi, do you know if there's a way to change the 'epochs' or I was wondering how does it use the training images, does it cycle through them or use them all at once? And also if the total steps influence something other than how long the training runs? for example is a 600 of 2000 steps save the same as a 600 total steps? Thank you !

[@promptcrafted]

Not sure on the first part, but yes 600 is always 600.

[@TheKoingWolf]

I think including at least one sample caption would have been really good, so we can see how you captioned your dataset. Was it manually done too or run through an LLM for natural language captioning?

[@promptcrafted]

@@TheKoingWolf I’ll do a separate video on captioning, the captions were super lazy - just a string of words.

[@mmm-c9p]

Thanks for the video. I'm training my first lora after spending so much time trying to run simpletuner but I couldn't. Is it possible if you can also make a video on simpletuner, or maybe compare the results?

[@promptcrafted]

No, I don’t like the results from simpletuner so I am sticking with this trainer.

[@rodi4850]

how does the dataset look like? it's image1.jpg and image1.txt ? what does image1.txt have to contain ?

[@thedeepdiveinsights]

I don't put.txt files. Just a folder with images, select all, F2 to rename, put the trigger word as name and enter. Then edit the confog file and run. It works like a charm.

[@_tzk_]

@@thedeepdiveinsights uncomment the txt extension in config?

[@TalhaKarakoyunlu]

Each .txt file contains the caption or annotation for the corresponding image. This caption is usually a text description of what's in the image, for example, if image1.jpg were to be an image of a white wolf jumping on snow, the corresponding txt file should include something like "a white wolf jumping around in the snow" or something. You can also use your trigger word inside these txt files

[@tazztone]

1:50 seeing some errors "filename too long". i have the same

[@KeredaSmile]

What python version u use? i 3.11 and not working :(

[@ERROR0x7CF]

What do the captions you have for the files look like? I understand that Flux prefers more natural-language style prompting, so I'm curious what I should be doing for the images in my set. Thank you

[@promptcrafted]

@@ERROR0x7CF I kind of did trash captions for this test so honestly the captions don’t seem to be having a massive impact. I’m sure natural language would give a slight improvement but the overall results were quite good.

[@Niffelheim]

@@promptcrafted So for prompting in Flux we should use Blip or some LLM like Florence instead of WD14, right?

[@exiacyn4621]

Unfortunatetly the lora does not appear to work in comfyui or forge. Both UI's updated and tested. When I run the trigger word it does nothing however I noticed when I copy past the generation prompt from the yaml file that it used in training, the lora appears to have an effect. There seems to be something wrong here as during training it generated the images fine and was clearly working but the final lora appears useless.

[@promptcrafted]

@@exiacyn4621 if it’s generating the actual outputs from the samples then it’s working, you may have just undertrained it. Flux is a pretty sensitive model.

[@exiacyn4621]

@@promptcrafted ty for the reply, seems odd as I went for 3000 steps and 40 images. Testing civit atm although that's not working out either due to high loads 🤣

[@yiu_ai]

Try using the LoRA model with the FLUX DEV fp8 or Schnell fp8. My LoRA doesn't work on fp16 either. 4090

[@Doowi92]

@@yiu_ai do you know why when i am using the lora that i have created on forge web ui + flux dev bnb nf4, it load 90% of my ram memory ? when i use the lora from xlabs-ai it doesnt takes my memory and it generate picture fast (RTX 3090 24GB)

[@mr.entezaee]

There is no way to not download the ten gig model again? I had already downloaded it. But now in this process it must be downloaded again? It takes a lot of time...

[@EH21UTB]

same question... can't you just put the local location instead of the hugging face one in the yaml?

[@EternalAI-v9b]

you did not show the captions ???? Thats the most imporant info I wanted to see please

[@SRQmedia8.A]

I got this error found unknown escape character 'F' Maybe I missed it. Where to put the image and text files?

[@fusfuru1]

That happened to me, in case you didn't solve it, just use \\ instead of \ with the directories, like "C:\\Photos\\Name" that solves it

[@SRQmedia8.A]

@@fusfuru1 I was able to fix it by adding the extra\. Thanks

[@1m_vital]

Thank you so much for tutorial! One more question: did you make a BLIP for each image in the dataset? If so I want to know if you left the deepbooru tags or made them natural language?

[@promptcrafted]

@@1m_vital I didn’t, this was a previously made dataset and I didn’t mess with caption adjustment from what I had prior.

[@1m_vital]

@@promptcrafted So you created an empty text files only?

[@promptcrafted]

@@1m_vital I used captions they were just very lazy

[@CharlieLee-p3q]

At the 12:57 mark of your video, I encountered a bunch of errors and can't progress any further than that. I followed your video from the beginning and installed it accordingly.

[@InfinantGamers]

I had this too until I realized that my huggingface token wasn't set with write permission. Created a new one with it, ran the command, and it worked for me then.

[@aihszd25]

Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases. ;(

[@jonjoni518]

i find it impossible to download the models. it starts downloading the model at 30mb/s and then it goes down to just a few Kbytes and stays at 99%. i have tried with different hugginface tokens (write, read finegrain....). i also leave the .yaml by default except the path where i indicate the directory of my dataset. by the way i have a 14900k 4090 and 128ram and windows 11

[@hotindianuncle]

i had this issue also where it got stuck at the last few MB, but i solved it by disabling my VPN. huggingface probably blocks some ip ranges. i've had apple updates fail in the same way while using vpn.

[@remy4562]

LORA worked well on the sample but not on CONFYUI. Is there a particular WF used?

[@equilibrium964]

I believe the lora needs to be converted in some way to make it work in comfy. On the huggingface page for flux lora collection are special comfy versions for every lora.

[@fusfuru1]

I'm wondering if training with imagen 512x512 allows me to use my RTX 4060ti 16gb...

[@Mr_Sorus]

I read on Reddit, someone was successful doing this…. I am going to try this

[@stereotyp9991]

Are you the author of the araminta sdxl models?

[@promptcrafted]

@@stereotyp9991 araminta k? Yes

[@ee89199]

thank you, Can i use this to train images of my Dog? I am new

[@promptcrafted]

@@ee89199 yes

[@ee89199]

@@promptcrafted thank you !

[@aslmate]

Has anyone tried doing this on a less VRAM video card? I only have an RTX 3080 12GB......

[@mlnima]

use some IDE not notepad, use higher display scale when you record on 1080p, some stuff are not readable even on 55 inch 4k display, you are welcome

[@researchandbuild1751]

Doesn't work. Always get python errors even when installed from scratch. Training AI is just a big pain in the F'ing ass, none of the trainers work right

[@guijiao]

i like u

[@HestoySeghuro]

Did you use the default validation prompts???