FLUX Fine Tuning with LoRA | Unleash FLUX's Potential

Discover how to unleash the hidden potential of FLUX through fine-tuning with LoRA (Low-Rank Adaptation). Join us as we explore how LoRA can enhance FLUX's capabilities and how to apply this technique to your projects. Get ready to take your AI image generation skills to the next level and unleash the full potential of FLUX!
Promts:
ainxtgen.notion.site/KZbin-Note-c59efa43404f4338ad3158712733512e
LoRA:
huggingface.co/AINxtGen/ScarlettJohansson_LoRA_FLUX
Resource using in this video:
1. fal.ai
2. replicate.com/lucataco/ai-toolkit/train
3. github.com/ostris/ai-toolkit/blob/main/config/examples/train_lora_flux_24gb.yaml
4. github.com/kohya-ss/sd-scripts
5. github.com/Nerogar/OneTrainer
6. github.com/comfyanonymous/ComfyUI_examples/tree/master/flux
#flux #stablediffusion #comfyui #FLUXFineTuning #aiimagegeneration #machinelearning #deeplearning #aitutorialforbeginners

[@TheColonelJJ]

Thank you for adding how much VRAM you have!!! That was helpful! I also have 12.

[@omegablast2002]

to reply to the title: literally no one said it was hard, its just extremely painfully long.

[@steve-g3j6b]

would love a followup video where you learned whats the best way to use those sliders on the fal web.

[@ahtoshkaa]

Great guide. thank you!

[@ee89199]

thank you can i use this to train my dog?

[@AINxtGen8]

yes, of course you can

[@charagga]

@@AINxtGen8 I think ee89199 is trying to be funny 🤔

[@sirishkumar-m5z]

Machine Learning: SmythOS’s pre-configured support for machine learning frameworks accelerates model development and deployment, streamlining the machine learning lifecycle.

[@AINxtGen8]

Fal.ai fal.ai/models/fal-ai/flux-lora-general-training You can also train LoRa on civitai and replicate.com: civitai.com/models/train replicate.com/ostris/flux-dev-lora-trainer/train If your computer has a powerful GPU, you can train locally, script to traning on local machine: github.com/ostris/ai-toolkit/tree/main

[@user-en4ts9mq4t]

@@AINxtGen8 thanks bro

[@steve-g3j6b]

what if I want my generations to be 16:9 should I use that size of pics to train? or 1:1 is best?

[@AINxtGen8]

@@steve-g3j6b Hello, thank you for your question: In fact, you don't need to crop your images to a specific size because I recently learned that fal.ai also uses ai-toolkit script from Ostris for training LoRA. This script supports a technique called 'bucketing', which is an automatic method that groups images of similar aspect ratios together during training. This means you don't need to manually crop your images to a specific size anymore. Bucketing is a technique that allows the model to train on images of various sizes and aspect ratios efficiently. It works by grouping similar-sized images into 'buckets' and processing them together, which helps maintain image quality and reduces the need for excessive resizing or cropping. This approach is particularly useful when working with datasets that contain images of different dimensions, as it preserves the original aspect ratios while still allowing for efficient batch processing during training.

[@steve-g3j6b]

@@AINxtGen8 I would imagine it will make much better backgrounds too (assuming the ai will also learn some of the BG)

[@steve-g3j6b]

@@AINxtGen8 would be a cool vid to have a comprehensive look at this workflow.

[@sankyuubigan]

How do you think when will appear models without censorship, in which will be at once all the celebrities already trained ? I mean communities where publish these models, of course only for introductory viewing, because nsfw content can not be done because it is very bad from the point of view of morality.

[@hellfire3278]

Can I train a LoRA model to control the measurements of a mannequin? The idea is to use trigger words for the waist, chest, and hip measurements, for example: (chest: 94cm; waist: 72cm; hips: 98cm). However, I'm unsure if all of these can be incorporated into a single LoRA model, as it might become complicated. In short, do you know how the trigger words interact with the training dataset?

[@AINxtGen8]

Thank you for your interesting question about controlling mannequin measurements using AI. While training a LoRA model for this purpose is creative, it might be complex and challenging to achieve the desired results. I haven't seen anyone create a LoRA specifically for controlling measurements (possibly due to the difficulty in achieving the desired results). Training such a model to accurately control multiple body measurements simultaneously (chest, waist, hips) would require an extensive and precisely labeled dataset, which could be difficult to create and maintain. Instead, I suggest using ControlNet, a simpler and potentially more effective approach. ControlNet allows for detailed control during image generation using sketches or guide images to control the mannequin's shape and measurements. This method offers several advantages: Precise control: Create a basic sketch with desired measurements. Flexibility: Easily adjust body shape by modifying the input sketch. Consistency: Generate multiple images with the same measurements. Intuitive workflow: Drawing or modifying a sketch is often easier than fine-tuning complex prompts. ControlNet can provide more accurate and consistent results in controlling mannequin measurements compared to the LoRA approach.

[@Reddkomet]

Can you make a tutorial for creating style Loras?

[@Bpmf-g3u]

I used to have same trouble, been using mimicpc lately and I feel pretty good about it, by studying the step-by-step tutorial, the proficiency made for a surprisingly graphic experience and the Lora adjustment produces very realistic visual details

[@sebastianpodesta]

Hi, if I want to make a Lora to give people baby faces or Asian faces, should I make a Lora with many different Asian or baby faces? What would make a good data set?

[@AINxtGen8]

Hi, as I understand, you want to create a baby cute, kawaii style. If you're just creating a general image in this style, Flux can do it. Try some of the prompts below to see. If you want to create this style for a specific face, you'll need to create a LoRA for that face, then combine it with style keywords like those below. Another method that doesn't require LoRA is using IPAdapter Face, but it only works well on SDXL versions. Currently, FLUX doesn't have a well-functioning IPAdapter, although Xlabs has just released an IPAdapter model for FLUX, it's not very good. Reference prompts: "Asian with baby face, cute chibi style, big eyes" "Kawaii Asian portrait, childlike expression" "Cartoon Asian character, baby face, adorable" "Chibi Asian, oversized head, tiny body, playful smile" "Cute Asian portrait, youthful features, cartoon-like eyes" Images created from prompts: imgur.com/a/SQP9Ln5

[@debdutbhadurishorts]

Can I use multiple people lora in same pic ? For example lora of scarlet and Donald Trump , together dancing. And if yes then how

[@AINxtGen8]

Yes, you can train separate LoRAs and then load them together. If you're using ComfyUI, there's a node called 'LoRA Loader Stack' in the rgthree extension (which can be installed via Comfy Manager). You can use that node to load multiple LoRAs, and adjust the strength of each LoRA to achieve good results. imgur.com/a/GldHkqE I understand that Donald Trump was just an example, but if you want to quickly test whether Flux has been trained on a specific keyword, there's a recently launched website called fastflux.ai that can do this. This site uses the Flux Schnell model and generates images at a very high speed. imgur.com/PWOiPMM imgur.com/gubtT0v

[@agnosticatheist4093]

You mean lora lora lora lora.....?

[@quangminhnguyen7834]

Can I use the trained lora to generate images on any free website that has flux?

[@geekyprogrammer4831]

gpus dont come for free

[@charagga]

Thanks for the video! I wonder if one could use it for replacing fashion shoots. I would 1) train on a certain character/person/model (photo realistic ofc) 2) then train a let’s say skirt or fashion piece, maybe a couple of images of the piece 3) then somehow combine it How would you do this, would you also use controlNet for this?

[@AINxtGen8]

Yes, you can, here's a simplified approach: 1. Train a LoRA for the Flux to create your specific character/model. Use ControlNet Pose to control the model's posing accurately. 2. Use ComfyUI's CatVTON node to change dress the AI-generated model in different outfits. This method combines character-specific LoRA models with virtual try-on technology. You can refer to the node below: github.com/chflame163/ComfyUI_CatVTON_Wrapper openart.ai/workflows/HaxcrNaVvjae9pdkut64

[@charagga]

@@AINxtGen8thanks a lot!

[@chrisgg]

I think, taking a celebrity creates out of the box good results without training a model?

[@AINxtGen8]

As I mentioned in this part of the video: 00:00:20 I chose Scarlett Johansson for testing purposes. The reason for this choice is that when I used her name as a keyword, Flux generated images that didn't resemble Johansson. This suggests that her name was likely removed from Flux's training data. I selected Scarlett Johansson for this test because she is a well-known celebrity, which makes it easier to compare the results before and after training.

[@fahimabdulaziz4255]

can I train lora for a consistent streetwear t-shirt design style?

[@AINxtGen8]

Certainly, you can train a LoRA for a consistent streetwear t-shirt design style. Training for a specific style is generally more challenging than training for a character, but it's definitely achievable. Here are some tips to help you succeed: Data preparation: Gather a larger dataset of high-quality images (at least 50 good quality images). There's no need to crop these images due to the bucketing technique which is fal also used Training steps: I recommend increasing the number of training steps to at least 2000. This allows the model more time to learn the nuances of the style. Learning rate: Start with a learning rate of 0.0002. You can adjust this later if needed. Checkpoints: Make use of the new feature on fal called 'Experimental Multi Checkpoints Count'. Set this to save 4 checkpoints during the training process. This is crucial because it allows you to test different stages of the model after training and choose the one that produces the best results. Remember, training for a style requires more attention to detail and experimentation. Don't be discouraged if your first attempt isn't perfect - it often takes some fine-tuning to get the desired results.

[@fahimabdulaziz4255]

@@AINxtGen8 thank you soo much, Ma Sha Allah

[@paulfranco9673]

how did you get it to generate the thumbnail? i'm trying to use Flux to generate multiple views of characters but I'm struggling to do so, if you could give me some guidance pls!

[@AINxtGen8]

The prompt will generally be like below, with the keyword here being "character design sheet". Below is the prompt that I used ChatGPT to create (I input a similar sample image and then asked ChatGPT to generate this prompt): " Character design sheet for Scarlett Johansson as Black Widow in modern 2D animation style. Horizontal layout. Left side: full body front and side views in signature black catsuit with front zipper. Right side: two close-up face views (3/4 and profile) showing detailed features. Add third full body view in dynamic fighting pose. Short wavy red hair, large green eyes with highlights, bold red lips. Exaggerated body proportions for visual appeal. Clean, sharp lines with minimal shading. Flat colors with subtle highlights. Include varied facial expressions: neutral, smiling, serious. Add rear view and close-ups of iconic accessories (e.g. wrist gauntlets, belt). White background with soft shadows. Professional, polished illustration style reminiscent of high-end animated series. "

[@mehmetalirende]

what about combining 2 loras in 1 picture for couples?

[@andrejohnson2079]

A whole romantic getaway to any fictional destination of your imagination

[@AINxtGen8]

yes you can, use Lora Stack node in ComfyUI, refer to this workflow link: openart.ai/workflows/macaque_keen_26/flux-with-multi-lora-loader-workflow/DfB4A8yL27WCwgEGi3YA or try running on replicate: replicate.com/lucataco/flux-dev-multi-lora

[@ronnydaca]

​@@AINxtGen8 It's possibile with forge?

[@AINxtGen8]

@@ronnydaca in forge you can also load multiple lora, and adjust the weights for each lora, but I haven't actually tested the results for lora used for Flux on Forge imgur.com/HYCFTrq

[@frizzfrizz3550]

great video, I want to contact you for a chat or a call, how can I do?

[@user-en4ts9mq4t]

Hi thanks. Where could I get the images I need to use?

[@AINxtGen8]

Hi ! Thank you for your question. Depending on what type of LoRA you want to train - whether it's for a character, object, or style - one of the most commonly used image sources is Google (filtered for large images): images.google.com/advanced_image_search Alternatively, you can also use AI image generators to create a dataset for training. One example of this approach is using ComfyUI. You can refer to this workflow: openart.ai/workflows/serval_quirky_69/one-click-dataset/QoOqXTelqSjMwZ0fvxQ9

[@rtberbary0101]

for some reason, it keeps failing for me. doesn't start the training eventhough i changed nothing. only uplaod my photos and trigger word same as you did. anyone else having this issue?

[@AINxtGen8]

Have you tried clicking the "see log" button in the left hand window after clicking the "start" button? Does the log show anything?

[@rtberbary0101]

@@AINxtGen8 i figured it out! apparently there is a limit on photos. you can add a maximum of 99 images for the training. anything beyond that results in an error

[@shirleywang9584]

Hi, I'm Tess from Digiarty Software. Interested in a collab?

[@zorayanuthar9289]

Great guide but poor choices relating to models... Cameltoe come-on 😂

[@sdprompts]

AI images 👍 AI voice 👎

[@AINxtGen8]

Thanks for your feedback! I totally get it about the AI voice. My English isn't good, and when I tried recording myself, it sounded pretty rough. I worried viewers might struggle to understand me. While AI voices can't match a fluent speaker's emotion, I think it's better for tutorials than my voice right now. I'm always trying to improve, though! Any suggestions on making the videos better? I'm all ears!