Upscale from pixels to real life

This time I'm having fun upscaling old retrogame pixel art to live action characters in ComfyUI. It's basically an overview over various upscaling techniques, Tile ControlNet and a little of prompt engineering. Hope you find it useful!
The workflow: pastebin.com/jr8FEYTW
00:00 Intro
00:25 Prompting
02:40 Tile ControlNet
04:30 Latent vs Pixel Upscalers
06:36 Latent Upscale
08:07 Model based Latent Upscale
09:43 Image Upscale (pixel space)
10:59 Image Upscale with Model
12:44 Upscale with ControlNet
15:15 Wrap up with another example
20:13 Conclusions
🎵 Background Music
-- CyberPunk City by Peritune (peritune.com/blog/2020/05/22/...)
-- "Part A" by Alexander Nakarada (www.serpentsoundstudios.com)
Licensed under Creative Commons BY Attribution 4.0 License
-- Last Stop Synthwave by Karl Casey @ White Bat Audio (whitebataudio.com/)

[@TheDocPixel]

You have one of the absolute BEST channels for intermediate/advanced SD users, with real-world workflows. Your explanations, tips, speed, and dialog ist absolute top! PLEASE continue what you're doing here, and I would even try using "Advanced, Real-World Project Workflow" or something like that in your titles. Definitely NOT for beginners, and it should keep the complaints down. We NEED more of you and your tutorials.

[@latentvision]

I really appreciate the motivational comment, thanks. It works 😄

[@camilovilla8434]

For sure MagnificAI was born from this tutorial. Magnificent information.

[@MarkRiverbank]

I feel like my understanding of ComfyUI and StableDiffusion have progressed so rapidly since discovering your videos. I really appreciate the way you don’t just show a workflow, but provide an explanation and show alternatives and really make it clear why you’re making the choices you make. It’s the difference between giving a man a fish versus teaching him how to fish.

[@FamousRamos]

Your format is great! Trial and error seams to be the name of the game and you don't cherry pick. This makes your videos incredibly helpful. And this specific use case with pixel art is very interesting! I'm following the space carefully and personally I'm looking for use cases that I can implement and make a business from. But in my experience consistency is the hardest part. Say a client loves the white haired sorcerer - can we deliver more around that character or are we stuck? Idk, but I have a feeling with your videos I'm going to be able to unlock a new super power that 99% of the world is sleeping on. Thank you 🙏 so helpful, so informative 🤙

[@latentvision]

Thanks for taking the time to write. What I don't like of many youtube tutorials is that they concentrate on the latest/coolest technology ignoring to consolidate the basics. So I'm trying to do fill that gap. For example prompt engineering is very much ignored but very important. Regarding "variations". ControlNet and IPAdapter together generally grant a pretty stable image generation.

[@FamousRamos]

@latentvision yes, I have the same experience with everyone wanting to be first. But ironic you are first, cutting edge and an actual contributer to the space. Much respect!

[@AB-wf8ek]

Very good tutorial. I came to the same conclusion that pixel upscale is much better than latent and this confirms it. I haven't bothered trying an additional tile controlnet on top, even though that's what I used to do with auto1111. Now that you've mentioned it, I'll have to give it a try. I think this format is great, progressing from left to right, and your demonstration is very concise and easy to follow. I think this is a good way to cover more intermediate topics, and if someone is just learning the basics they can review step by step and do further research if they don't understand something. At least that's my opinion. Thanks for posting!

[@latentvision]

yeah you need very high denoise for latent-upscale. if fidelity to the original image is not important latent upscale can be still used, possibly together with SDXL in the second pass

[@StefanSchmidtRegensburg]

These are by far the best videos about this topic by far. Many other influencers are just users, they don't understand the concepts behind this. I learned so much. Keep the videos coming.

[@luiswebdev8292]

dude you dont need to change the style of the tutorials, this is really amazing toptier content

[@SkyOrtizCreative]

I love your tutorials I learn so much from your work!!! I know it takes a lot of time and effort to make these. I really appreciate you sharing your knowledge with us. Keep them coming, I would watch anything you would likely want to make and share. 🙌

[@youwang9156]

i am just totally shocked by the quality of your content, absolutely astonishing

[@fedoborod8890]

OH MY GOD! Again, a tons of helpfull info in your videos. A whole pipeline here) Thank you very much for sharing your techniques.

[@apacchiart3970]

Cool process, The wig in negative is a nice trick!

[@johnriperti3127]

You are really good at explaining. I'm learning so much with you.

[@Michael-gf1jn]

Best comfyui videos out there

[@alirezafarzaneh2539]

Great video! Your explanations are so clear!

[@GOLD-jb5pw]

This is one of the few great channels that explains how it works!

[@angelotsk3173]

best teacher ever 😊

[@teshtishtoshtesh3218]

I like what you're doing here, and that you make the workflow files available. I have no complaints about these videos, and I'm very glad that you're sharing how you do these things. I've learned more from your videos than most others simply because you spend that extra time to show how and why you wire things the way you do.

[@alpaykasal2902]

Amiga Forever!!! This is excellent. Thank you!

[@franciscodufey5104]

Great video! Thank you for share!

[@adrianlachlan1452]

your knowledge is amazing and great videos...I am always surprised at how endless comfyui appears to be...There is so much in some ways it is purely intimidating and I spend more time thinking where do I start....lol....its almost a university course to fully understand comfyui...lol......amazing :-)

[@gamersgabangest3179]

Ciao Matteo, ti seguo dagli Stati Uniti. Ho appena scoperto questo software e i tuoi video. Ottimo lavoro!

[@jasemali1987]

The best person to teach us everything about Ai image generation. I've dazzled of how effortless you make the process seem. Sir, you have my respect.

[@puyakhalili]

This was an amazing tutorial. Thank you for the in-depth descriptions. I had no idea that latent upscalars were so efficient.

[@Vestu]

SPEEDBALL 2! My favorite Amiga game!

[@pedxing]

your mastery of these concepts is simply mind numbing. love this.

great, thanks a lot!

[@yt-caio]

This was a master class. God bless you.

[@jaoltr]

Terrific video, thanks! The detailed walk through is great. You have a new subscriber.

[@jepperisum]

Love your work! Thanks! Learn soo much every time! Cheeers

[@AZmisc]

Excellent. Thank you very much for the technique and the explanation!

[@magneticanimalism7419]

This was awesome and very insiteful, I am still getting to grips with Comfy. Your videos are great, thanks. Also, thank you so much for the free workflow :)

[@onewiththefreaks3664]

Thank you very much, I really appreciate your work! You are really creating big stuff for the community by coding this gem and even making videos about the usage. This is fantastic and I learned tons from your videos. I really like, that you show more advanced techniques, most other channels do all the basic stuff like explaining how to install a new node in each and every video. Again, thank you so much!

[@Utopia08]

Love your tutorials! 🔥Always learn something new! But I had to learn the basics before I went in on your excellent videos. I had them saved for when I was ready for them. 📚

[@WhySoBroke]

Oohhh... the diffusion MAESTRO is here with an amazing new video!! My week is fulfilled. Many infinite thanks amigo!! ❤️🇲🇽❤️

[@weirdscix]

I've learnt so much from watching through your various videos, you're very knowledgeable. Have a happy holidays if you celebrate them :)

[@mustafamohsen]

Your content is among the best hands down. I like the deep, straight to the point approach. Maybe just raise the mic volume a little bit?

[@valter987]

your videos are always so great!

[@Drew_pew_pew_pew]

This gave me a fun afternoon recreating the golden axe characters. Awsome tutorial

[@ribbentropkuznetsov2490]

Dude. You are doing God's job here. Love your tutorials.

[@johnmcaleer6917]

Wonderful as always...

[@jepperisum]

And btw.. The format is great..!

[@maxfxgr]

Loved the video, but loved also the AMIGA

[@TheGalacticIndian]

Amiga Strikes back!

[@BernardMaltais]

Great work. I like the style of your video

[@ArielTavori]

Amazing work, thanks so much! FYI IP Adapter does great at adding skin detail, just give it a cropped example of some close up skin, even works with batch images/face model!.. 🪄✨

[@latentvision]

that's a great tip! thanks!

[@zebapy]

This is amazing! I've been trying to just use controlnet with line art and color grid models to redo pixel art but this is so much more accurate. Can't wait to try

[@zebapy]

I mean line art / canny / soft edge usually one of those.

[@UliTroyo]

If you want an easy 100k views, do this to each of the Stardew Valley pixel portraits. I’d love to see that video!

[@PaulFidika]

wow, I didn't realize how many different ways there were to upscale

[@pk.9436]

great work please keep going 💪

[@TimVerweij]

Thank you for the clear explanations. It's very useful to know why you're doing things a certain way. Little things like "the order doesn't matter" on condition concat nodes are very valuable. Edit: So in a next video you actually mentioned that it does matter. The from has lower influence.

[@hakandurgut]

Another great tutorial

[@pedxing]

also, really hoping to see a workflow like this form into something that could be used for an animation. from motion jpg, feeding each image into the pipeline and seeing what a 10 second clip would look like. =)

[@latentvision]

Stay tuned! I have great news about animations!

[@pedxing]

muahahaha! exciting times. =) @@latentvision

[@openroomxyz]

That's amazing

[@TheJAM_Sr]

Great videos. The only constructive critique is slow down a little and show where you’re connecting things to more closely. I’m glad I found your channel, seems geared for people who have an 20:42 intermediate understanding of comfy, which is severely lacking on KZbin

[@latentvision]

thanks for your feedback! It's always hard to find the right pace as I don't want the videos to be too long (you know watching a workflow is not really that much fun). I tried to increase the resolution, that should help a little. (be sure to select 1440p and enlarge the video). Most of the times there's only one place where the connection could go, but I'll try to zoom in when it's ambiguous. Also check the video description for the workflows!

[@TheJAM_Sr]

@@latentvision you Bet! I was able to follow along fine, you made a great tutorial. Seeing things laid out did give me a better insight on how upscaling works and I don’t know why I didn’t think to do other passes. These are educational videos and people who watch are here to learn and I bet they rewind and watch certain parts over and over. All the other channels out there spend so much time talking about how to install comfy and python, what feels like, on every damn video.

[@mayorc]

Non male il tuo canale Matteo. P.S. Ero un fan di Speedball 2. Iscritto!

[@aa-xn5hc]

Grazie! More such tutorials please! What about something to improve dallE images? And to have an input face?

[@hleet]

it‘s not easy but well explain. thank you

[@tonikunec]

Great stuff as always. One question, what controlnet would you use for SDXL? I've achieved great results by using the tile controller, but it doesn't work when using any SDXL checkpoint or controlnet, and as there is no tile controlnet for SDXL, as of now, the results are no so good. I've tried different controlnet types, but unfortunately haven't been able to reproduce the same results as with the tile controlnet. Anyway, I really hope they will release it for SDXL as it really makes a difference in this type of image reproduction.

[@erikdias9604]

I was on Atari, I couldn't wait to test it on Teenage Queen... uh I mean on Xenon 2!!! 😅

[@Kelticfury]

This was a great idea :) fun stuff. I didn't know VAE was lossy. Would it be a better idea generally to use a checkpoint with a baked-in VAE?

[@EH21UTB]

Another great video, thanks. One question - why do you need to change the seed from the 1st stage to the 2nd stages? I had been thinking for upscaling you'd want the same seed?

[@latentvision]

keeping the same seed you risk to burn the image (ie: you impress the same image twice). That is especially true for img2img (without upscaling)

[@EH21UTB]

@@latentvision Thank you for the reply and information. I guess then it's not such a good idea to use the Global Seed? It's such a convenient node.

[@latentvision]

@@EH21UTB SD is a game of chances. It's not about finding the best workflow but finding the workflow that have the highest chances to get what you want. Over time you discover the best practices that in your specific scenario give you the highest rate of best results.

[@dejanholmstrand3027]

Really cool - this was Stabel diffusion?

[@latentvision]

yes

[@Enricii]

Complimenti, gran bel video. Ottimo format, mi sembra di livello intermedio: nè troppo basic, nè troppo complesso. Durata giusta. Un paio di appunti al workflow: ci sono dei nodi utilizzabili per ricreare il prompt a partire dall'immagine caricata: mai provati? (simili al CLIP interrogator di A1111). Secondo, visto che parli di upscaling, come mai non hai inserito un workflow in cui utilizzi Ultimate SD Upscaler? Tra i vari metodi, di solito è questo con cui ottengo i migliori risultati. Comunque ripeto, ottimo video e spero continuerai a pubblicare video di livello intermedio-esperto. Grazie mille

[@latentvision]

i text-to-image possono essere utili in alcune circostanze, ma tendono a essere un po' generici. Ultimate SD Upscale implementa molti dei metodi che ho illustrato nel filmato con in aggiunta la possibilità di dividere l'immagine in tiles (e seam-fix), ma non fa molto di diverso da quello illustrato qui.

[@Falkov]

Inspiring! Liked and subbed. This is my first of your videos..and it sounds like you’re testing workflow tutorials (like this)..well, It’s just the sort of thing I was looking for. ,{^_^}”

[@lukefellows4319]

Love you videos! Can you do a video on how to do this process in reverse? And control for pixelisation size?

[@latentvision]

that would be pretty complicated without a trained model... but it would very interesting indeed

[@ALatentPlace]

Very nice. Mixing the good old Amiga 500 / Atari ST times with the modern AI tech. Really impressed me and made me nostalgic. :D One issue I have: After you were chosing the Speedball character, I am always hearing "ICE CREEEAM, ICE CREEEAM". That's annoying. :D

[@latentvision]

don't worry you are not alone I had "Ice cream!" in my head during the whole video 😄

[@ItsMC92]

incredible wooow, I have a silly question???????????? I was not able to use the workflow after downloading, since tries to load and it doesnt support text versions, changed to .png still not. would you please help me????

[@aldnoahz4203]

Amazing tutorial, but I would prefer a1111, comfyui for me is too complicated

[@makadi86]

do we have tile controlnet for sdxl that may work with comfy?

[@DemiGoodUA]

Great idea! How it will look in game, with animations?

[@latentvision]

that would be an interesting experiment to do :)

[@dck7048]

Hi, I don't imagine doing this video "in reverse" is something you'd be interested in, but I've found it very difficult finding resources for generating pixel art that sticks to specific styles. Do you think it's worth trying to do it with a variant of this workflow? For example, say you'd like to use your realistic human end result from this video as the reference image for a sprite head. Do you think that workflow would make sense? Sorry for all the questions, I just find that there's a significant amount of paid tools related to pixel art in the SD sphere, but very little actual information on how to generate it without specifically trained models that lead to inflexibility.

[@latentvision]

that would be really interesting. Pixel art is very difficult to do without a trained model and even so the result is generally either not great or over trained. SDXL is pretty good at pixel art if you don't need "pixel perfect" resolution. I'll make some tests to see if there's enough material for a video. Thanks for the suggestion.

[@dck7048]

I think youtube ate my comment due to having links on it, but I just wanted to add that in my research I found two good tools that help us lower the "entry bar" from the base model by not needing pixel perfection. These are ComfyUI-PixelArt-Detector by dimtoneff (palette control and pixel grid) and the 1x PixelSharpen upscaler. Anyways, I hope there's enough to make something interesting out of it! I think there's enough tools to make a good transition from illustration to pixel art happen, but putting it together with control tools feels completely out of my depth personally.

[@goor76]

very cool! How would you do the same with SDXL?

[@latentvision]

SDXL doesn't have tile controlnet, there are sone alternative techniques but they are not as effective.

[@ItsMC92]

Amaaazing but Im not abble to use the workflow, would you please help me??

[@MikevomMars]

What about the other way around - converting a real world face into a pixel character? 😊

[@latentvision]

there are some checkpoints that are pretty well trained at pixel art...

[@sherpya]

now we have the knowledge, but not the gpu 😂

[@TerrificSportsMoments]

Hi, thanks for tutorial. Could you share what hardware do you use so that it so quick? Thank you.

[@latentvision]

All the renderings are sped up in post production otherwise it would be a terribly boring video :) I have a 4090 anyway

[@goliat2606]

Is big difference with model Realistic Vision v51_v51VAE like you used and new Realistic Vision V6.0? It is possible to download v51_v51VAE?

[@FusionDeveloper]

I notice that ComfUI with IMG 2 IMG and Inpainting, it seems to degrade the entire photo each time it is used. So if you feed the output back as the input and keep editing, it will get unusable. Automatic1111 doesn't have this issue, unless it is the model or settings i'm using idk.

[@latentvision]

set the latest mask together with the inpainting. that is what a1111 does

[@sherpya]

he has green hair not blue 😅

[@latentvision]

let's agree on turquoise 😄

[@hmmrm]

the link of the workflow dosnt work

[@tazztone]

why still use sd1.5 inplace of sdxl?

[@latentvision]

Tile ControlNet is not available on SDXL unfortunately. There's a tile model for XL but it's only for anime. I haven't checked if they released a generic model... things move so fast. Apart from that when doing videos SD1.5 is much faster so it's a little easier. Anyway you can totally use SDXL as upscale model in the second pass and actually the result is generally very good.

[@Andee...]

Not working with directml :/

[@Andee...]

Can confirm that this one works: ComfyUI_IPAdapter_plus/tree/c28a04466b17d760a345aea41d6a593c0a312c95