Fine Tune Flux Diffusion Models with Your Photos

[@reo4153]

Excellent video , well explained

[@loicbaconnier9150]

Excellent 😊

[@thanartchamnanyantarakij9950]

Great from Thailand

[@TrelisResearch]

thanks

[@loicbaconnier9150]

Hello from Paris

[@TrelisResearch]

Salut Loic

[@loicbaconnier9150]

thanks for all

[@exzxm3801]

Great content, your channel is great. In this video, you added [Trigger] to your txt files, do you think it should be the actual token word defined in your script. Else you are training on the word '[trigger]' which isn't the token you will use later to trigger the lora. I found that flux (dev) loras are strong enough without needing a token in prompt anyway. Also mostly people are saying no description needed in the txt files as flux understands the images well enough on its own with clip+t5, unlike previous SD training which neeed the guidance?

[@exzxm3801]

also be great to get your take on fine tuning this hyped reflection type prompt (like how you finetuned for functions previously)

[@TrelisResearch]

ok, that's quite a nuanced question. So, what the trigger word offers is theoretically a way to have the model use your lora ONLY when that trigger word is present. However, that possibly relies on not overfitting the model... and basically everyone (including me here) overfits the model by doing tons of epochs. If you're going to only use your lora for your own purpose (where you always want the lora's effect), then this is moot. You don't need a trigger and the model will "use" your lora for all prompts.

[@TrelisResearch]

@@exzxm3801 well there are two vids on function calling you can check out . Regarding reflection, I haven't dug too deeply. I have a video coming out on thursday about DSPy which is basically advanced prompting and the effect is VERY strong on performance. What reflection is doing is specifically putting in a and step. Anthropic already do thiking and my gut feel is that their engineers would have found any obvious tweaks to this kind of approach. Probably does help a lot. Does it help to also close the data gap from llama 70B to claude sonnet 3.5, probably not. #speculation

[@exzxm3801]

@@TrelisResearch i think i agree with your hunch on overfitting is why loras often work without the token in the prompt. But I still think you want to be putting a relevant token in the txt files.

[@exzxm3801]

@@TrelisResearch I guess the curiuous part (if there is one in any of this possibly fakery) is are training structured/ in depth complicated prompts into a finetune showing any improvement over using a simple complicated prompt up front (in the same way finetuning on functions made your function models improve). I do think there is room for improvement on prompt design, the issue in the reflection (aside from the does it work bs) is it is requiring the extra tokens to get to an eventual correct response. I wonder if training in some sort of stop and surmise tag halfway through so it can reviews its own progress might help (would need to be finetuned on specific structured synthetic data I imagine for this)

[@ravitejakonda9236]

Can we merge different trained LoRAs, such as combining a person LoRA model with another person's LoRA model to create a single model? Is it possible to combine both models into one and continue training them as a unified model? Additionally, can we train a model with two different subjects combined into one, ensuring that both subjects are learned together in the training process?

[@TrelisResearch]

You can train two loras separately and then apply both to the same model to get a combined effect. OR you can take all of the data and just train one lora. To first order, there shouldn't be all that much difference in the two approaches - although training separately gives you more flexibility as to which style to apply.

[@AICohort]

Can you please create a video on merging multiple Lora and generating frame by frame images with prompting ​@@TrelisResearch

[@sharadsisodiya3853]

please share code as well

[@TrelisResearch]

All in the description!

[@Username56291]

Hi

[@TrelisResearch]

howdy