Stable Diffusion IC-Light: Preserve details and colors with frequency separation and color match

A full pipeline and workflow for production ready product advertising solutions in Stable Diffusion.
In this episode of Stable Diffusion for Professional Creatives, we develop a full, one-click pipeline in Stable Diffusion for creating stunning advertising campaigns from a single bad product picture.
Want to support me? You can buy me a coffee here: ko-fi.com/risunobushi
Workflow: openart.ai/workflows/gCMFAhrx...
Full Assets folder (1.6GB): we.tl/t-3csFQs2saw
(If you're using Blender, enable this plugin to use 3ds files: docs.blender.org/manual/en/la...)
(install the missing nodes via comfyUI manager, or use:)
IC-Light comfyUI github: github.com/kijai/ComfyUI-IC-L...
IC-Light model (fc only, no need to use the fbc model): huggingface.co/lllyasviel/ic-...
GroundingDinoSAMSegment: github.com/storyicon/comfyui_...
SAM models: found in the same SAM github above.
Frequency Separation: they should be coming from the standard comfyUI install, otherwise you can just install them via the manager.
Model: most 1.5 models, I'm using epicRealism civitai.com/models/25694/epic...
Auxiliary controlNet nodes: github.com/Fannovel16/comfyui...
Timestamps:
00:00 - Intro
01:36 - Changes Overview
02:27 - Workflow in-depth explanation
07:07 - Color Matching!
08:01 - Advertising Pipeline
08:08 - Advertising Pipeline: Photo Prep
08:52 - Advertising Pipeline: Virtual Set in 3D
10:09 - Advertising Pipeline: Stable Diffusion Rework
11:02 - Advertising Pipeline: First Test (Ice Sculpture from Branch)
13:26 - Advertising Pipeline: Second Test (Low Denoise)
14:18 - Advertising Pipeline: Advertising Test
14:45 - Advertising Pipeline: Animation (Theoretical)
16:16 - Final thoughts
16:38 - Outro
#stablediffusion #advertising #iclight #stablediffusiontutorial #relight #ai #generativeai #generativeart #comfyui #comfyuitutorial #risunobushi_ai #sdxl #sd #risunobushi #andreabaioni

[@risunobushi_ai]

There was a tiny error in the upscaler when color matching with option 2, I updated the workflow and now it works for all three color matches options while upscaling. Sorry about that!

[@zerorusher]

This is great! What I like about these complete workflows is that I can tear it apart and learn how to use specific nodes/techniques.

[@risunobushi_ai]

Thanks! I try to keep them modular because of this, it's easier to break them down into bits.

[@carlosfernandez3565]

I like this workflow because it's usable in pro situations.

[@SouvikRg]

Amazing. There’s so much that can be done in comfyui using SD. You have been incredible in providing great resources. It has helped me learn so many things. Cant thank you enough.

[@jeremysomers2239]

This is the most comprehensive USEFUL commercial use of compfy I've ever seen. byye bye studio!

[@techniquebeats_dennis]

Well, it uses models that are not licensed for commercial use (Supir), but you can swap Supir for an ultimate upscaler, for example.

[@xxab-yg5zs]

looks amazing, can't wait to try it out. Thanks Andrea!

[@Vcr.10]

your node trees are getting x2 bigger with every new video 😁

[@risunobushi_ai]

I know right! but I'm going to blame you all for making me do this so that's fine

[@M4rt1nX]

Thank a lot Andrea. Great work.

[@MaraScottAI]

Hi Andrea, thanks for all those insights, it is very interesting.

[@yotraxx]

Your channel is the most useful to me by far. Thank you soich for sharing your knowledge.

[@risunobushi_ai]

Thank you! I’m happy people can find my contributions useful, it’s the beauty of open source

[@jahormaksimau1597]

Thank you, its a great job!

[@alexvillabon]

Good stuff! Thanks for sharing.

[@henryturner4281]

GOAT thank you!!!!

[@ImAlecPonce]

Very interesting!!! Thanks

[@ismgroov4094]

Thx sir ❤ respect sir

[@dariearminalexandru]

let's go!

[@AbsolutelyForward]

Thanks for the great revision of the workflow Andrea :) The structure for the colour adjustments is great. Good that you made it clear that there is not THE ONE setup that will work for all colour adjustments. -I think the second option (blending mode: hue + add) is missing an upstream "image resizer" node (if you want to make use of the SUPIR upscaler group) before the first "image blending mode" node at "image_a".- I still have two questions: 1) I Didn't know that is it possible to use checkpoints other than "SUPIR-v0Q" or "SUPIR-v0F" with the "SUPIR model loader" - does it work well? 2) Since the IC-Light workflow is based on SD 1.5, I assumed that you always have to work with the typical SD 1.5 resolutions (including the resolution of the input). In your current workflow v4 you work with 1920 x 1080 pixels, in the v3 workflow you worked with 1024 x 1024 pixels as an example. What of course works great is that more details of the product (fonts, logo, other elements) are preserved, but is the 1.5 model really able to deliver the best for (for example) the newly generated background? SUPIR on the other hand is based on SDXL, so I understand the higher resolution Thanks for the inspiration :)

[@risunobushi_ai]

Hey there! Thank you for getting me to think of ways to color match. Yeah I was sourcing from the original resize node for option 2 instead of the upscaler resize, I fixed it in the link in the description but I still need to fix it in the wetransfer link, I’ll do that tomorrow morning. Tbh I’ve never done a ton of testing on SUPIR, it was just one of the workflows I had laying around I had started developing for a client weeks ago and then abandoned. It might not be the best model, there’s so many different ways to approach upscaling both with SUPIR and Ultimate out there. Most of the stuff I do for clients is up till the upscaling bit, so I’m not an expert there. As for 1.5 resolution, it might just be me being a bit of a sucker for ignoring rules, but I’ve never cared for recommended resolutions. Sure, there are optimal resolutions, but as long as it works, it works, and for me it works as long as I like it. But again, the use I make of resolution is far from being the norm - I tend to bend some rules in photography as well. I like some 1.5 checkpoints at higher reses, so I stray from optimal resolutions.

[@AbsolutelyForward]

@@risunobushi_ai Hey Andrea, I just started watching your recording of the last stream where you answered my question about the higher resolutions with SD 1.5. I just wanted to clarify that I think it's really fantastic that you get great results with higher resolutions. I'm just very curious and generally always want to learn a deeper understanding of the basics :). Since I've been involved with Stable Diffusion, I've always been told to look at the relevant documentation (civitai etc.) for the specific model checkpoints (whether SD1.5 / SDXL) to know what resolutions and aspect ratios to work with to get a better result. And that sounded logical so far - image models are trained with a large set of images that all have the same resolution(s) and aspect ratio(s). Thanks for your answer + have a nice weekend

[@risunobushi_ai]

@@AbsolutelyForward oh yeah, no shade involved whatsoever with my answers - I tell people too to look at suggested resolutions and documentation. But that’s the starting point, *and* the documentation itself most of the time is there to provide guidance to the average usecase, which is very far from the ones I’m showcasing. Heavy conditioning through ControlNet for example affects suggested resolutions. Same as when I publish barebones workflows and people ask for additional groups, there’s no catch all solution to different use cases - and it’s the same for suggested settings.

[@JamalTalioui]

ty

[@houseofcontent3020]

Incredible! Once again :) Question, I know you mention the swapping with existing background and you said it's easy to implement it here. Any chance you can write what goes where for that to happen? Really appreciate your wokr!

[@risunobushi_ai]

Hey! Thanks! So there’s two ways of doing it. If the BG and the subject have the same perspective, you can either: - comp it in photoshop, and load the comped image as usual, swapping the empty latent image for the VAE encode node in the rebackground group, using a very low denoise value, or you could just bypass the rebackground group completely; or - you can load the subject and the background separately, then use a transform node that can rescale and adjust X/Y positions, connect the result to a SAM, hook up the SAM image result to a blend by mask node where image a is the BG and image b is the subject and the mask is the SAM mask, then dial in the X/Y position and the rescale factor of the subject by hand until it gets into the right spot. Plug the result into the workflow and you’re set.

[@SergioValsecchi75]

👏👏👏👏👏👏

[@user-tz4sv5nc6b]

Hi Andrea, Does this workflow allow for generating a background that integrates into the flow? I'm keen to try out this flow.

[@risunobushi_ai]

Hey there! Do you mean like the previous one? Sure, it does, you can also have the IPAdapter group influence the whole background by segmenting it and using the resulting mask as an attention mask in the SAM group under the IPAdapter group (or segment the subject, invert the mask, and use that instead, might be easier than asking SAM for the background)

[@egarywi1]

Hi Andrea, good work, your a machine putting out some incredible content in this space, I am a product photographer and feel I have to move down this path, I just need to decide on the right workflow that gives me the best results for me. I am trying to get this workflow going and I have a small issue with the Impact Gaussian Mask, half way through the process (actually 19%) it stops ConfyUI and I have to relaunch it again? I use this node in other workflows so I am a bit lost? Any clues? The last line in the log prior to stopping is listed below: Loads SAM model: /Volumes/Mac Mini Ext 1/StabilityMatrix/Packages/ComfyUI/models/sams/sam_vit_b_01ec64.pth (device:Prefer GPU) final text_encoder_type: bert-base-uncased

[@egarywi1]

Just an update on my last post, I reduced the kernel size to 41 and the now the Gaussian Blur Mask ran right through, however when it got to the KSampler in the relight section I got a runtime error which seems to indicate running out of memory, this is probably the root cause. I have reduced all the image sizes down to 1024 x 786 however made no difference? I am using a Mac M2 which has never given up on me "YET" any ideas how to reduce the load on the memory? error below: attention_pytorch out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False) RuntimeError: Invalid buffer size: 16.97 GB

[@risunobushi_ai]

Hey there, thanks for the kind words! It seems your memory is not enough to run through all of the workflow. I'm not an expert on running complex workflows on Macs, but this version of the workflow is really resource intensive. Maybe strip it down by bypassing everything you don't need, and substitute some of the more resource intensive groups, like SAM, with less resource intensive nodes like a background remover node. Unfortunately this workflow was the result of the community asking for a all in one solution, and your case is why I usually release barebones solutions - not everyone can run long and complex stuff - and not just because of monetary constraints, there's some hardware, like Macs, where money's not the issue, but the architecture and the hardware choices made by the manufacturer are.

[@Glowbox3D]

Can't wait to just prompt all this node stuff. :) Kidding. Not really.

[@risunobushi_ai]

If you take a look at the actual prompting, there’s very little of it. There’s a ton of stuff happening - selective regenerations based on IPAdapter, relighting based on selective masks, recoloring based on automated masks - that prompting is actually negligible and could even be completely taken out of the workflow.

[@mz4637]

HOLY fuck

[@risunobushi_ai]

Ahahah yeah I know

[@UnleashYourGr8ness]

Can you suggest anything that if I upload a background removed image it will automatically generate a suitable background

[@risunobushi_ai]

This workflow does exactly that, if you use the standard empty latent in the background regenerator group, and plug in the SAM output of the subject into the controlNet preprocessors. Since you want a completely empty background as a starting point, you could disable the depth controlNet entirely and just use the lineart controlnet.

[@UnleashYourGr8ness]

@@risunobushi_ai and how can know where to place all the other models

[@risunobushi_ai]

@@UnleashYourGr8ness Have you read the notes in the workflow? It's all explained, what every group does, what most of the nodes do, and how they interact with each other. The models needed are three: one SD1.5 model to start, a IC-Light model to relight (fc, not fbc, since we're relighting without injecting light from a background), and an optional SDXL Lighting model for an optional SUPIR upscale pass

[@UnleashYourGr8ness]

@@risunobushi_ai can you tell me where should I place the IC relight model

[@risunobushi_ai]

@@UnleashYourGr8ness it goes in the Unet folder

[@RenderEffect]

This works with collab?

[@risunobushi_ai]

I’m sorry, I’m not well versed in Collab, I don’t know. I run everything locally.

[@RenderEffect]

​@@risunobushi_ai okay. Thanks though! What specs do I need for local?

[@risunobushi_ai]

I have a 3080ti, and with everything turned on except for the SUPIR upscale group, my VRAM usage hits around 60-70% max.

[@andrewcampbell8938]

The final images aren't that impressive imo due to the product retaining its flat 2D appearance. Wondering how difficult it would be to create a somewhat 3D object in Blender from iphone photos of the product? But I do like the ability to change surface materials and colour/lighting.

[@risunobushi_ai]

As with all things generative AI, they may not be impressive right now. But the tech is a few years old at best (well, the current one) and a few weeks old at worst (like in the case of IC-Light). NERF and TripoSR are ways of creating a 3D object out of a 2D picture, but they are very limited in resolution with consumer hardware. Also, keep in mind that there’s only so much I can show in a 15-20 minutes tutorial that needs to explain how things are done and why things are done that way, at least before the viewers’ attention spans run out.