ComfyUI Tutorial Series: Ep11 - LLM, Prompt Generation, img2txt, txt2txt Overview

[@AndreyJulpa]

A lot of useful information in one video without any fluff. Thank you!

[@DezorianGuy]

Somehow, other AI KZbinrs are prominently promoted by KZbin and Google, but you’re truly the only tutor one needs to have, imho.

[@Searge-DP]

Great video. It's always fun to see my LLM node in action :) If you want to have more control over the seed on the LLM node, you can also turn the random_seed into an input and connect either a primitive node or another seed generator to get the option to use different seeds every time you run the workflow. I've also tried the gguf versions of Phi-3.5-Mini, for example from bartowski/Phi-3.5-mini-instruct-GGUF on huggingface, and those have great results with smaller llm models and less vram use.

[@SebAnt]

Wow!!

[@pixaroma]

Thanks, i already did that on the simple workflow at the end of the video that only generate prompts by adding seed as input :) I left it without random seed on flux workflow just in case people want to generate more on the same prompt so it is faster. Thanks for the models i have to check it out. Do you know any similar to florence, that get better description from images?

[@pixaroma]

Can you also add an image input to the node? Or a new node that has that input? So i can get a description from an image. It kind of worked with a path to the image but didn't want to add extra nodes just to get the paths in the string format and then use instructions on it

[@Searge-DP]

@@pixaroma No, the node and models don't support images directly. It also can't load files from your drive. In your video it didn't actually look at the images when you pasted the file paths, it picked up the subject from the image filename. What I usually do is combine what you have here: 1. load image 2. get caption with florence2 3. refine caption with searge-llm 4. ??? 5. profit

[@ivo_tm]

Another great video! As MrBeast says, the next step after 10k subscribers is not 20k, but 100k !!! Keep it up!

[@lakislam.photoArt]

i can only say : u r unbelievable. your tutorials are out of this world 👏👏👏

[@pixaroma]

Update: The Searge node cannot provide descriptions of images; use it for prompt generation instead. Use Florence for generating descriptions from images. Therefore, use Searge with LLM for prompt generation from text instructions, and Florence 2 for generating prompts from images. Join the conversation on Discord discord.gg/gggpkVgBf3 or in our Facebook group facebook.com/groups/pixaromacommunity. You can now support the channel and unlock exclusive perks by becoming a member: kzbin.info/door/mMbwA-s3GZDKVzGZ-kPwaQjoin

[@digitalblizz9102]

I am here to say: You're the best explainer out there when it comes to ComfyUI. Keep it up!

[@pixaroma]

Thank you ☺️

[@baheth3elmy16]

Great video! Thank you for the diversity in your videos!

[@camilocortes8074]

do not have words to describe the amount of value this video provides, congrats!!

[@WasamiKirua]

thanks for your free content. you always bring a lot of value to the community for free. hope will never change

[@odrammurks1497]

Thank you so so much for these videos. You're doing an amazing job at explaining! ❤

[@НастенькаНастя-ь8ц]

Your lessons are really very detailed and useful. I'm doing well, it's fantastic, thank you for your work

[@pixaroma]

Glad you like them 🙂 and thanks for support on membership

[@kaiserinvictoria4897]

Straight to the point and useful stuff as always,keep going!

[@fernandohildebrand6319]

Thank you. The best tutorial I´ve found.

[@rmeta3391]

Excellent work, very detailed on Discord. Well organized and easy to follow.

[@GenoG]

Very well done my friend!! I've been using the Searge LLM for a few days while eagerly awaiting your video. And, as usual, I learned a BUNCH of new ideas and tech tips from this video. I love that you included the Img2Text options!! Thank you! You earned FiDolla!! 😍

[@59Marcel]

Another well explained and easy to follow tutorial. Looking forward to giving these a try. Thanks.

[@pixaroma]

Thank you ☺️

[@jgag777]

Your videos are awesome and the explanations are accessible. Thank you.

[@jorgeluismontoyasolis9800]

Thank you so much! Really useful, as always. Greetings from Perú

[@GenoG]

Thanks!

[@pixaroma]

Thanks again for continuous support ☺️

[@Ozstudiosio]

new enjoyment day with new video thats all :)

[@pixaroma]

Thanks ☺️

[@BryceHaymond]

Great tutorials so far. The use of a path to an image for a LLM model to see it is not how those work, but you probably figured that out.

[@pixaroma]

Yes I found out later :) need a vision model

[@joeshmoe4207]

Honestly baffling given the quality of the other parts of the video.

[@jennifertsang6572]

Great video!! Always learning something from you!!

[@OhItsAnthony]

22:40 On Windows 10 you can hold Shift and right click and the right click menu will have an option to 'Copy as Path', so no need to go to the File Explorer ribbon menu.

[@pixaroma]

thank you for the shortcut 🙂

[@dmitrykonovalov9366]

Very very good tutorial

[@MaxwellSmi41483]

Been absolutely loving this tutorial series and accompanying Discord channel. I have been following without any issues or setbacks until this episode. Searge node cannot be installed as it requires a maximum of Python 3.11, however ComfyUI is currently running on 3.12.7. I have tried everything including backing up the python_embeded directory and installing a 3.11 version in a new python_embeded dir. When i go to run this it tells me that there is no module called comfy, so crashes on line 1 of main which is import comfy.options. If anyone else has experienced this or can help me with this frustration, I would be very grateful indeed!! 🙂 Keep up the great work, learned so much and always grateful for the workflows you include 💯👍

[@pixaroma]

A member of our discord channel made a version that works that comes with that nodes and a few other installed, you can use that installer and install comfyui in different folder github.com/Tavris1/ComfyUI-Easy-Install

[@MarekCezaryWojtaszek]

I have the same problem with Searge for Python 3.11 vs Python 3.12.7 that is shipped with the current version of ComfyUI. I was just about to downgrade my python_embedded to tackle that, but apparently it won't work either. Did you manage to find a solution for that issue?

[@pixaroma]

​@@MarekCezaryWojtaszek you can install this comfyui in a different folder and comes with all the settings and those node installed, a member of discord ivo did this installer, I didnt find other solution than this yet. github.com/Tavris1/ComfyUI-Easy-Install

[@huongnguyen105]

thanks, very useful tutorial

[@DivineSwd]

This video is so good *-*

[@ASAS-ci3xu]

Thank you so much man. Your great work guided me to explore the ComfyUI workflow from zero to a little understanding. By the way, your opening and closing animation are great too! Is it Kling too? How could you do that? Can you do an episode about this? Menu, cursor, animated background, etc... very attractive! Thanks!

[@pixaroma]

Depending on animation i use multiple tools, if i get some free time will do a video on that. I use kling, also CapCut with keyframe animation, and also a software DPanimation maker

[@SumoBundle]

Awesome tutorial like always.

[@Redemptionz2]

best tutorial!!

[@B3D]

Thank you for the tutorial it help me so much .

[@logman121]

Well explained tutorial! Thanks a lot! Is there a way how to add my own input which would always be added to generated text from Florence? Thank you?

[@pixaroma]

You can use a text concatenate node from was node suite and can combine any text you want from different sources, add a primitive node or a positive from easy use custom node and connect to text concatenate and do the same with output from Florence and the results are those together

[@logman121]

@@pixaroma Thanks for quick reply. It works! :)

[@majdmmb3839]

Thank you.

[@erans]

Hi, thanks for the great tutorial - regarding image -> prompt, sometimes i want to use an existing image (and its prompt) - but with my own lora. is there a way to 'insert' my trigger word inside the generated prompt that was made out of an existing image?

[@pixaroma]

You can add a text concatenate node , that will combine your existing prompt with your defined text, just like i do on episode 7 where i combine my styles that are in fact multiple prompts to the actual prompt

[@YanXiao_AI]

thank you for your tutorial

[@tmkongen]

I get import failed on the Searge node install, tried multiple times. Folder created and file to put in that folder is where it should be. Any ideas?

[@pixaroma]

Some had problems on some configurations, it has something to do with dependencies, some had problems and posted on discord, only solution that found was to install comfyui with the node already installed, someone from community made an installer github.com/Tavris1/ComfyUI-Easy-Install

[@mayankgupta2937]

amazing videos as usual, if i try to add lora keywords in the prompt they gets lost while going through LLM, anyway to workaround that?

[@mayankgupta2937]

used your concatenator idea, worked out well :D

[@pixaroma]

yes text concatenate should work you can add extra things :)

[@DeeAreDee]

How would we do this with a caption generator that doesn't have it's own dedicated node? For example, Joy Caption Alpha Two? I have a Load Checkpoint node with Jay Caption Alpha Two's .safetensors file, but I don't know how to connect that to images to generate captions. Thank you!!!

[@pixaroma]

Not sure, only if you find a node for that, that was the problem for me also there are many models and hard to integrate. I did also in another episode the ollama version there only works with ollama models

[@videomark9283]

Hallo! I tried to do everything described, but when I start the queue, it says "Failed to import transformers.models.mega.configuration_mega because of the following error (look up to see its traceback): No module named 'transformers.models.mega.configuration_mega'. Where am I wrong?

[@pixaroma]

when you start the comfyui check if node got installed the searge one, most of people had that problem because it didnt install the node completely. You can try to use this installer for comfyui in different folder that install those nodes automatically, a member of discord did that installer, and worked out for people could not install it github.com/Tavris1/ComfyUI-Easy-Install

[@videomark9283]

@@pixaroma , thanks a lot! You are № 1

[@freewheelburning8834]

You have probably figured out something with creating a prompt from a path, I used Mistrel Q8 gguf model and I placed a path to a .JPG image of MT. Rushmore I also re-worded the prompt instructions you had, to "Generate a prompt from the image but don't use the path or file the name" and got a very nice image almost exactly like the picture so unless they updated the model perhaps its the wording or the image extension type Im guessing

[@logudesignstudio]

after install searge llm node missing problem, how to fix?

[@pixaroma]

Someone from discord community made this comfyui installer you can try, it install automatically all the nodes i used. I didn't find a way to fix it manually but this worked for many that had same problem like you so you can give it a try github.com/Tavris1/ComfyUI-Easy-Install

[@GigaTvn]

Are you sure mistral LLM model is able to see image by giving it a path ? I think it would need some additional programming to do that. It may recognize file name, thats why it produces subject like women portrait or architecture but image is not similar. I tested with random file name and model was not able to "see" image for me. :) Thanks for video!

[@pixaroma]

No, it seems it doesn't see, i talked with creator of the node and he said it just invented a prompt of the image info, so use Florence for that, and I will try to see if i can get another model and node that is better for image captioning. Llava from ollama seems to know how to read images, and something I saw wirh Joy in the name. I will do more research. So the searge node is better for text to text, Florence for image to text

[@GigaTvn]

@@pixaroma Great, waiting for more videos! :)

[@FabioAI_Oficial]

I have a question. which one is better? Ollama node or this Florence 2? 🤔

[@pixaroma]

well for me Florence was faster, ollama let me choose more variety of models so depends on the pc specs, but ollama took more vram to run, so I dont use ollama too much. I mostly just use chatgpt so I dont take any of my vram so i can generate faster

[@FabioAI_Oficial]

@pixaroma thank you for your help!

[@FabioAI_Oficial]

@@pixaroma After installed all, I have problems with the ComfyUI_Searge_LLM about " llama_cpp_cuda " I follow the instructions and nothing happend, other solution you have maybe? Thank you

[@pixaroma]

@@FabioAI_Oficial some people had that problem on certain pc configuration, but I dont know what triggers it. You could try to install this version a member of discord did that has already installed the searge and has the right dependencies, just instal this comfyui in a new folder, github.com/Tavris1/ComfyUI-Easy-Install

[@karamsahoo6852]

Class apart!

[@Fayrus_Fuma]

Thank you! I'm sure this feature will help a lot of people in creating interesting designs and creative solutions. (Still waiting for your Upscale videos).

[@pixaroma]

Didn't forget about upscaling, this month will be ready, i just need to do more tests

[@ESheridan]

Hello! I have a problem. When I try to download Florence-2-base Prompt I have error - DownloadAndLoadFlorence2Model The checkpoint you are trying to load has model type `florence2` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date. How can solfe this issue? Thanks in advance

[@pixaroma]

You can try to see if updating transformers work, maybe you have an old version of comfyui and your manager didn't update successfully, you can try to go to update folder and run the update and update dependencies bat file. Or you can tey to go to python_embeded folder and in address bar type cmd and press enter, then run this command: ./python.exe -m pip install --upgrade transformers

[@ESheridan]

​@@pixaroma Hello! Thanks for our help. but after update i still have the eroor - DownloadAndLoadFlorence2Model The checkpoint you are trying to load has model type `florence2` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[@pixaroma]

@@ESheridan i am not sure, i mean that node should just downloaded and use it, maybe need a different version of some dependencies but is hard to t ell. If nothing works just install a new comfyui in different folder

[@ESheridan]

​@@pixaroma I relize! Thank you for your help. I have question, what is the best florence model on your opininon?

[@pixaroma]

@@ESheridan i used a few from there but now there is a llama 3.2 so probably i will need to do some research on that. I still use chatgpt when i want something more exact

[@faycaltech5922]

cant install searge llm, i triend to install llama-ccp as they said in github but i've always an error in manager and when i start comfyui

[@pixaroma]

some still have that problem, is something with dependencies needs a certain version I think, but I have no way to test it since is different for each system. Alternative is to use Ollama like in episode 13

[@faycaltech5922]

@@pixaroma thanks, i'll check this

[@dominiquemaik3415]

This error appears for me: DownloadAndLoadFlorence2Model Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate` Plz help

[@pixaroma]

You could try I guess what they recommend and install that to see if it helps. Navigate to the folder where your python_embeded folder is located , in the address bar on top where is the path type cmd and press enter, that will open command window in the exact folder. Paste this command and press enter, then restart. python.exe -m pip install accelerate Or also you can try to fix dependencies if you go to update folder run the bat file that has dependencies in name

[@dominiquemaik3415]

@@pixaroma It worked perfectly, thank you very much!!! 🙏🏼

[@sinuva]

Nice mate, make some inpaint tutorial. Would be nice

[@pixaroma]

Yeah, I plan to do one maybe this month, right now working on upscaling

[@Valket]

Bro I am just curious, are you using ai voice over?

[@pixaroma]

Yes is AI voice from my text

[@Valket]

@@pixaroma Holy shit I will spend the entire thirty minute just listening to the way you talks rather than what you say just for how realistic it is. Its scary. What tool is that?

[@pixaroma]

@@Valket elevenlabs

[@deeceehawk]

Great explanation and workflows! There is a hot comment for both you and the algorithm :-) DEECEEHAWK

[@MaxwellSmi41483]

sorry, its not the searge llm, its the llama-cpp-python thats required wont work past python 3.11

[@pixaroma]

check my other comment, only solution was that installer I gave link for many, not sure why that cpp cause so much problems

[@MarekCezaryWojtaszek]

@@pixaroma I am getting the following error while trying to install llama_cpp_python: ERROR: llama_cpp_python-0.2.89+cpuavx2-cp311-cp311-win_amd64.whl is not a supported wheel on this platform. This is because the current version of ComfyUI is shipped with Python 3.12.7: ** Python version: 3.12.7 (tags/v3.12.7:0b05ead, Oct 1 2024, 03:06:41) [MSC v.1941 64 bit (AMD64)] So, for now I gave up. I will just use ChatGPT for help with creating good prompts and wait for Searge update to 3.12.

[@hishambarakat8544]

love your vids man, is there anyway to like support you? you got a Patreon or smth?

[@pixaroma]

Thank you ☺️ You can join membership there is join button on the channel ☺️ also under videos there are super thanks, the heat icon with dollar.

[@talismanna]

Anyone got the gguf working in "extra_model_paths.yaml" ... Nothing works for me. I must keep it in the default path.

[@pixaroma]

I will do a video tutorial about it since many seems to have that problem, will be ready this week

[@ESheridan]

@pixaroma Thank you so much for your details expalnations