Silverwing Quick Tip: Octane Render Layer

This tut is made with Octane Version 2022.1 and Cinema 4D Version 2023.1
Hello everyone.
As promised this week its Render Layer time. Its a small tut that gives you and overview of the Render Layer system and how it works.
Hopefully you find everything in there you are looking for!
P.s. leave a comment if you have any good Idea for a future quicktip / tut.
0:00 Intro
0:38 Explaining Render Layers
1:20 Setting up Render Layers
3:03 Setting up the ID
5:22 Adding Shadow / Reflection AOVs
6:58 Render Layer Modes
9:32 Setting the Alpha Mode
10:54 Render Output Settings
11:47 Render Output Chart
12:44 Straight Alpha in Practice
13:53 Compositing Project Settings
14:36 Setting the Footage to Straight
15:31 How to Comp Render Layer
16:54 Dealing with the Shadows
17:45 Dealing with the Reflection
20:47 Outro

[@polygonpen]

Great tutorial, I always had a problem with the alphas, thank you!

[@SilverwingVFX]

Thank you very much for your nice comment!

[@perryharovas]

"Speaking of comments, let this commence!" LOL Love it! I don't care if they are called Quick Tips, Tutorials, Lessons, Cool Stuff, etc. I just love watching these and learning new things. Thank you for the time you always spend to help others. These are so valuable and wonderful. Cheers!

[@SilverwingVFX]

Hey Perry, thank you very much for your nice words. It´s so great to hear that you like the tuts. Comments like yours make spending so much time well worth it!

[@downeystudio]

Very clear and instructive as always, thank you very much Raphael !

[@SilverwingVFX]

Thank you! Happy to hear that.

[@10sylv10]

Amazing, love the comping examples that you do for these kinds of tutorials

[@SilverwingVFX]

Oh thanks. Nice to hear that! Yeah its definitely also a essential part of my workflow. Every render goes through Comp. At least for some color correction 😀

[@fgarciaish]

This quick tip came at the right time. Was playing with render layers last week for a current project. Didn’t know about the setting adjustments to get the shadows and reflections under the object so that’s great thing I just learned! Thank you!

[@SilverwingVFX]

That´s cool to hear. Unfortunately with the "Reflections / Shadows under the object" you would have to render twice. But sometimes that´s what you need to do.

[@ATOMVISUAL]

really cool for comp some VFX right here ! :D thanks you very much like always !

[@SilverwingVFX]

Always happy to see you comment here. Thank you very much for watching my tuts!

[@midlet]

This was super helpful. Thank you!

[@arenasvfx]

AMAZING like always. THANK YOU!

[@SilverwingVFX]

Thank you. Appreciate that you like it 🙏

[@ThomvanVliet]

Yes! Very nice as always and thanks again!

[@SilverwingVFX]

Hey there. Thank you again too 🙂 Much appreciated!

[@cellacyben101]

Thank you. You're an amazing artist and teacher.

[@SilverwingVFX]

Thanks so much. Comments like this make my Day 😊 🙌

[@FortexArtz]

Deine Tutorials sind unglaublich hilfreich! Danke Raphael! Nico :)

[@SilverwingVFX]

Hey, vielen Dank. Freut mich sehr, dass Dir die Tuts zusagen!

[@ForgingMind]

Awesome tutorial, I always learn new things from your channel, thank you. Oh and you were looking for suggestions instead of Quick Tip, how about Power Tip? 😀

[@Kuechmeister]

Mega gut, das jemand diesen Workflow mal konzentriert auf AE und Octane.

[@SilverwingVFX]

Vielen Dank. Benutze diese Octane AE combo seit Jahren. Kann man finde ich gut benutzen!

[@Kuechmeister]

@@SilverwingVFX Ich auch, aber ich musste mir das so "zusammenlügen" so konzentriert als Tutorial ist das für viele bestimmt super.

[@ArturKiraga]

just what I needed

[@SilverwingVFX]

Super nice to hear that 🙌✨

[@julianolisboa]

Very good tutorial, I managed to replicate all the steps in Blender's octane. Thanks

[@SilverwingVFX]

That´s super cool to hear. Cudos to you for doing that and managing to pull it off. Cheers and a great time to you!

[@alextorhan]

Подскажите пожалуйста как получить теневой слой без шумов?

[@awolism]

Thank you for another clear and to the point tutorial! Super appreciated. As I work predominately with stills in Photoshop I guess the workflow should translate fairly well to that program as well. I'm curious to try it out.

[@SilverwingVFX]

Thank you very much. Much appreciated! If you work within a linear workflow within Photohsop you get the exact same result. If you are working in e.G. sRGB (without linear) then you need to use different blend modes. Also I don´t know how straight alphas are handeled in Photoshop. But overall it should work!

[@awolism]

@@SilverwingVFX Yeah, I wish it was easier to achieve the same visual result in 8/16-bit sRGB as with Linear 32-bit, as I sometimes have to deliver layered PSD:s to clients, and they tend to not be fans of 32 bit. :)

[@SilverwingVFX]

@@awolism Yeah. I have that too. And I always have to work around linear workflow then. I never got it to work 100% like intended. But close enough.

[@cars103]

I wish you would do even longer tuts! 😊

[@SilverwingVFX]

Oh wow really? Thank you for the compliment! Doing long videos takes so much time to edit etc so I personally rather keep it on the 10 to 15 Min side 😇

[@Murathanaksoz]

We are very aware of the effort you have put in, as a community. However, I have a request, which I'm sure other friends would also want. There should be a way to use the shadow catcher of the Octane render engine as a reflection catcher as well, without being confined to the layer ID. If you find a way, please share it with us. It becomes essential in projects where adding a 3D object to a video is crucial.

[@SilverwingVFX]

Hey Murathan. thank you very much for your comment and also your patience. I am just an Octane. User. I have contact to Ahmet, the developer of the C4D plugin. But don´t speak with the core developers regularly. If I am asked I remember your request! May I ask why its important to you that the shadow catcher would also do reflections and should not be used the way it is shown in this linked video. Just for the understanding. kzbin.info/www/bejne/mHbKm6Gvg9yCick Cheers and a great rest of the week to you! Raphael

[@Vannellus]

The mode "Only side effects" is the only mode u should render those layers for professional compositing. Else you will get tiny edges around the masked out object.

[@SilverwingVFX]

Hey there and thank you very much for your comment. Sorry, this is wrong info! EIDT: Regarding the "Wrong Info" comment. I tested it, there is a minimal difference, so Vannellus is right. Apologies! Its barely visible in an AB comparison. So I consider the "Normal" method valid still. Now on with the text: I have comped vast amounts of animations with the "Normal" Render Layer mode without any side effects! You have to make sure you render straight alpha (Main Tab "Premult Alpha" "None"), comp in true linear space (or in fusion interpret the alpha as being linear) And last but not least interpret the alpha "Straight" If you do this, no artifacts will show up and the rendering looks like perfect. No need to render twice. Unless you want to change the opacity of your main object. Then rendering it twice once "Only Side Effects" to reveal what´s underneath makes sense.

[@Vannellus]

@@SilverwingVFX Sorry, but this is wrong info. Do a test: Render 2 layers. The object itself and the shadows on the floor. Shadows one time with "Only side effects" and one time in the standard mode. Compare the resulting composited images and u will see a a difference at the edges of your object. And use a professional compositing tool like Nuke.

[@SilverwingVFX]

@@Vannellus OK I tested it and you are right. I apologize Though the effect is so minimal that I would have never noticed it without an AB comparison. Therefore I would argue that the other method is also valid. And as I said it saves you one extra render manouvre!

[@Vannellus]

@@SilverwingVFXBut it gets very obvious, if u have a high contrast between your object and the background. U will see then very clearly that there is an outline surrounding your object and no, it's no extra render manouvre. You are just nor able to put everything on top layerwise and u have to render more pixels :D

[@ChrisPriddy]

I'd like to see how to push all this out as a transparent PNG, or some sort of output that allows clients to drop the image into other programs, like slide decks, without the baked in BG. I see you've done it in your looping example, but did not touch on output. The reflection pass would be the tricky one as it is in add mode and dependent on another layer. Great tutorial, and extremely helpful, as usual!

[@SilverwingVFX]

Thank you for your comment! I think you pointed all the important things out here. The trickiest thing is to get the reflection to work. I think it would work well with only the shadow in a GIF. The next best thing would be a photoshop file set up so the client needs to open it and set its background and save. As of now I don´t know how I would deal with the reflection in a GIF

[@sergiomarchelli]

Great tut as always! Just a question, I am using redshift and looks pretty fast on a single 1080ti, if I switch to octane, are the speeds are going to be the same. I just think Octane results are better than Redshift,,,?

[@SilverwingVFX]

Sorry I missed your reply! I hope better late then never. Speed: It really comes down to the scene and the stuff you are doing with those renderers. Octane is a spectral unbiased pathtracer while Redshift is a biased raytracer. There are scenarios where Redshift is faster and there are other scenarios where Octane will be faster. Results: If you are feeling good with Redshift and comfortable working, then stay with Redshift. There is no need to switch render engines if you are satisfied with the result. I would say Octane is better with subtle light details and energy conservation. Scenes where it is a lot about surface interactions e.g. backlit curton that transmits light energy to some sss juice in a glass ect. Redshift on the other hand is better with lots of geomtric detail. When the scene has a huge amount of polys.

[@iacdxb]

Hi, thanks Raphael. I am new to Octane and these are very helpful. Can you also help... if two three or more objects how to get separate B&W mask of the objects..? I tired but getting one mask of all object not separate...! Thank.

[@SilverwingVFX]

Hey there and thank you very much for your question. You can give object as well as materials single masks so you can differnciate them. In my channel just search "Silverwing Quick Tip: Octane 2021 AOVs" also you might want to watch the newest video. Cheers and a great time to you!

[@user-zc9eh1qn5s]

Great tut! any methods to solve the blemish/imperfection on the edge of black layer shadow/layer shadow? it's very rough,not fit well when composing... increasing samples not work.

[@SilverwingVFX]

Hey there and thank you very much for your comment and your question. Working with straight alpha I never had problems compositing the shadow. The object always covered the shadow edge perfectly. If you are talking about not sufficient sampling of the shadow. Its Adaptive Sampling. It turns off sampling automatically from portions of the render that are transparent. You can either increase the min Adaptive Amount or turn it off completely. Hope that helps you out. Happy rendering!

[@user-zc9eh1qn5s]

​@@SilverwingVFX you are absolutely right ! After re-watching it entirely, I realized that I made a mistake, putting black layer shadow upon the beauty layer.😂 thank you for sampling tips above and premultiplied alpha in video , very helpful !!

[@SilverwingVFX]

@@user-zc9eh1qn5s Thank you for your feedback. Super nice to hear that you managed to got it working. A great start into this week to you and an overall good time!

[@mirospain]

voll gut ! danke! wie verhindere ich denn das zusätzlich immer ein C4D exr gespeichert wird?? also zusätzlich zu den OctaneEXRs

[@SilverwingVFX]

Hi, vielen Dank für Dein Kommentar und Deine Frage: In C4D kannst Du einfach die Häkchen bei "Speichern" neben dem Ausgabepfaden zu "Normales Bild" und "Multi-Pass Bild" ausschalten Hey, thank you for your comment and your question. In C4D you can just untick the "Save" Checkbox next to the output Link of both the "Regular Image" and the "Multi-Pass Image"

[@michaelaforeman3393]

when you drag the EXR into after effects, it shows the beauty right away with the alpha in the comp, but I have to use Extractor to achieve such a result, what are you doing differently? For example, when you select the EXR and interpret the footage, you have the alpha options there, mine is all greyed out. I followed your video step by step

[@SilverwingVFX]

Hey there Michaela and thank you very much for your comment. Hmmm. Are you using multilayer EXRs? Having to use the extractor was giving me this Idea. If you are using the extractor all basically what AE sees from the import is a black image. So it can not interpret it. Because the data is all extracted after the fact. At lest that is how my theory goes. So I think a workflow that does not need the extractor will eventually be the solution. All the best and fingers crossed in finding a solution. If you do not find one, you can email me.

[@pommedapi3847]

Very nice tutorial, but I have a lot of noise in my shadow passes.. even if I put 2048 samples for exemple. Do you have any idea what I could search for?

[@SilverwingVFX]

Hey there and thank you very much for your comment. Yes I have a strong suspicion: Adaptive Sampling does not play well with "Layer" channels as they lie outside the beauties alpha and Adaptive Sampling is optimized to ignore anything outside the the alpha. Which is normally a good call... unless its render layers. I have requested a change to the developers already. So hopefully it will come. Until then you need to either raise your min adaptive sample amount so the shadow / reflection receives enough samples to be noise free or turn it off altogether. Hope this helps. Cheers and a great start into the weekend!

[@pommedapi3847]

@@SilverwingVFX Thank you Raphael for your kind help. Great Weekend to you to!

[@ThingaMagid]

is this the workflow for making seperate render layers for compositing? I am trying to figure out the best way to render a scene with each layer (foreground, background, smoke, debris etc), as seperate layers for comping and complete control over individual layer. I thought it would be cryptomattes

[@ThingaMagid]

following up, how would you set up rendering different layers in one go? like one render queue, do you have to render each layer seperately?

[@KexinChiang]

牛啊牛啊

[@Clemyfourtwenty]

A tutorial on the Octane Real time Mode would be cool.l but I don't think its out for C4D yet. :(

[@SilverwingVFX]

Hey, nice seeing you here again and thanks for your suggestion. You are right the real time mode is not in C4D yet as it, as far as I know, does not support the real time viewport for now. Fingers crossed it will come sooner rather then later!

[@ved.fourdimensional]

Is there any way to apply denoiser to these pass? I need reflections and shadows for an animation, but the denoiser is only applied to object ID 1

[@SilverwingVFX]

Hey hey and thank you very much for your comment. Unfortunately I do not think there is a denoised pass for the layer reflection / layer shadow. If you have really grainy results, this might be caused by the Adaptive Sampling since it cuts off the sampling from everything transparent from the main pass. This is as soon as it reaches the "Min Samples" count.

[@ved.fourdimensional]

@@SilverwingVFX wow, i didn't know that. I will try to play with adaptive sampling. as always thank you so much for the amount of knowledge you share with us :)

[@SilverwingVFX]

@@ved.fourdimensional thank you very much for your kind answer. So when working with layer shadows / reflections there are basically two options. The most easy would be turning off adaptive sampling altogether. But this then can lead to longer render times. The other would be to raise the "Minimum Samples" until the shadow / reflection looks clean but has not reached the main Max SPP count. Hope this helps 🙌

[@maksoz_mtb2924]

but when you have bg which is you need to render.i cant figured out how can i get reflection layer in the right way. ( my scene has motion blur so if i add tag octane tag there is some unwanted mb some of objects )

[@SilverwingVFX]

Hey hey, if you do not want to have motion blur, you can turn if off in the Octane Object Tag. Just go do Motion Blur Tab and set from "Transform" to "Disabled" 🙌

[@maksoz_mtb2924]

@@SilverwingVFX its not working sadly. Cuz the scene setup is. Camera mb enabled ( you will understand why ). Truck moving forward between trees. Camera is child of truck cuz we want to get everything mb but the truck. And we add oct tag to truck for not get mb. ( hope i can telling right) so we have a scene with mb but truck is not has mb. Thats what we want. if we add octane tag to all objects in the scene ( cuz we want to reflection layer same as your setup ) all scene got mb even if you disable. Btw i want the all scene's reflection not sepetare objects.i dont want to cut out bg or something Sadly i am using scatter and you know scatter not working well with mb in some of cases. i didnt understand why reflection channel includes diffuse 🤷🏻 why do we have to make complex settings. just click reflection and all done but octane not doing this

[@SilverwingVFX]

​ @Maksoz_mtb Sorry, but since I still have no idea how you have setup your scene. I can´t really help you other then (almost) blindly guessing. You need to understand the relation of camera and object motion blur! Lets say you have a normal scene, truck driving through nature. Trees and grass is static and the camera is moving next to the truck. In this scenario you want the camera as well as the truck to have motion blur! Camera motion blur is independent from object motion blur (no matter if it is a child of the truck or not). It gives blur to every movement the camera makes. So if you move your camera or turn it around it will create motion blur on the whole image no matter if objects have motion blur turned on or not. If you have a moving truck and turn motion blur off for it, you basically create a static object for every frame for the camera, so it looks like your camera is rushing by the truck in every frame. If you turn on motion blur for the truck, then the truck gets the right data to move along next to the camera and since both camera and truck moving in to the same direction there is no blur on the truck. If you have camera motion blur enabled, static objects will still get blurred if the camera moves, even if you turn off motion blur for those objects. If turned off they just don´t produce extra motion blur if they move themselves! You can write me a mail for further help. Cheers, Raphael

[@jj89677]

Would love to see this workflow with Photoshop :-).

[@SilverwingVFX]

Thank you very much for your suggestion. I personally stopped using photoshop for postwork very early on because its lack of 32bit handling capabilities. So I am not really experienced doing post in PS unfortunately. I will write it down. Maybe I can do a short chapter about bringing in Octane layers to PS at some time.

[@jj89677]

@@SilverwingVFX hey thank you so much :). I mean AAF. Isn't the best software for actual post at a static frame especially if your in a pipeline as a designer. But for my animation I will def. use this method *_* thank you very much

[@SilverwingVFX]

@@jj89677 Yeah I understand that. Especially when the client wants to have a PSD with Layers. All the Images you see from me are progressed by AE though. Even the stills 😅😇

[@user-ur5vq4lt7n]

Hi, teacher! I'm working on VFX, but I have a question. If I do PRECOMP after working like this, I get OUTLINE. How should I solve this??

[@SilverwingVFX]

Usually it has to do with Straight / Premultiplied Alphas. If you work in AfterEffects klicking on the footage, hit Interpret footage and set it to "Straight" should solve it. You can see me doing that at 14:27. I really hope this works for you too!

[@SilverwingVFX]

Oh I just have seen that you mention Precomp. If you precomp multiple layers its best to put a layer with the same alpha on top of your comp and then use the "Stancil Alpha" layer blend mode. This makes sure everything adheres to the original alpha. Hopefully this helps!

[@user-ur5vq4lt7n]

@@SilverwingVFX Thank you for your response! After copying the original on each pass, I placed stencil alpha, and if I precomp, now the pixels are cut a little inward, is it the right process?

[@SilverwingVFX]

@@user-ur5vq4lt7n The process sounds correct. If your pixels are now cut a little inward, what you can do is turn off the alpha on the layers beneath completely. This might involve loading your footage twice. Once without alpha for the layering and once with straight alpha for the stencil.

[@user-ur5vq4lt7n]

Thank you so much for your kind response, sir! Happy New Year!!!

[@user-ur9wb8qh8f]

Excuse me, what should I do if the glass outputs alpha?

[@SilverwingVFX]

In the glass material there is a chackbox "Affect Alpha" if you uncheck it, it should not contribute to the alpha anymore 👍

[@ernestomartinezart]

Octane is making the things very complicated to understand and as an artist i dont find it interesting anymore. I prefer the older ones, much more intuitive and easy to set. I could focus on the artistic part amore than the technical part

[@SilverwingVFX]

There will always be a back and forth between new useful features that add complexity and ease of use! The good thing for most of the new features you do not have to use it them if you do not care. This is but an additional feature that is really helpful for those who need it. But can be ignored by those who don´t. As far as I know It does not take anything away from the toolset of the past. If it lost your interest that is totally fine. Just work with the tools that make you do your best work! Cheers, Raphael

[@ernestomartinezart]

@@SilverwingVFX Yeah i understand the thing is that before was much easier to send the render passes (i use zdepth, matID, denoised beauty and sometimes other like occlusion or smooth normals) In the past was just a button. now i cannot even achieve them working. Octane was popular between concept artists, now is being replaced by Blender because its free, and these changes don't help in my opinion. So sad because its the best render for me. Cheers!

[@SilverwingVFX]

@@ernestomartinezart I feel like this might be more an UI / UX problem then a feature problem. I do really understand you. Though if you take 2-3h and do a small deep dive and research I am sure you will be able to figure out how to save the needed passes with the new system. And then you can save a preset that you can apply in every of your projects ✨ I am on the complete opposite side as you are. I applaude those features because I finally can have more freedom. E.g. puzzle mattes (Object IDs) AOVs from texture elements even through reflection and refraction. A lot of things I made tutorials about. And Its really really handy in production a lot of the time! Creating a smooth normal pass or a occlusion pass should be rather straight forward I think. And Object IDs rendered forever with the old system because it was a hack by the C4D developer because they did not exist in the core at all. Blender is also a great option. I also use it and cycles from time to time 🙌 I think everything is constantly in movement. If we want it or not. I have my own set of topics that I´d rather want to be static. But it´s ever evolving. That´s life 😇