[TUTORIAL] Stencil buffer in Unity URP (cutting holes, impossible geometry, magic card)

Hey game dev enjoyers!
Here we are: the mighty tutorial about the stencil buffer that I’ve been working on for weeks now. I hope you’ll enjoy it ;)
WARNING: Please make sure to follow the instructions of the “IMPORTANT” chapter (it's only 30 seconds long no worries), otherwise you would not be able to use the stencil buffer.
The shader and the 3D model of the impossible geometry : github.com/PixTrick/StencilBu...
The stencil buffer documentation : docs.unity3d.com/Manual/SL-St...
The Culling and Depth testing documentation : docs.unity3d.com/2019.1/Docum...
Chapters:
0:00 Intro
0:40 The basics
4:45 ReadMask
6:11 WriteMask
7:10 Depth testing
8:10 IMPORTANT
8:41 Setting values into the stencil buffer
10:18 Hole cutting
16:13 Impossible geometry
19:43 Magic card
22:57 Outro
I made a discord server to discuss about the videos and to build a little community of game dev enjoyers, like me :D
It's far from finished, so if some of you are motivated to help me building it, do not hesitate !
Here's the link : / discord
Also a shout out to my sister for the font i'm using in the videos !
Cheers !
#gamedev #unity3d #stencil #buffer #holes #impossible #magic #card

[@TheInSinging]

THIS IS THE BEST TUTORIAL out there for URP / Stencil / Depth mask - for AR masking as well. Don't search further. I checked other tutorials, this one is the best. Thank you for your work!

[@pixtrick]

Thanks for the kind words ! :D

[@TheKr0ckeR]

This is a high level video. Instead of showing what to do, you also explained how it works actually. Great tutorial.

[@pixtrick]

Thanks for these kind words :D

[@kushs-labs]

Thank you for this. And definitely yes regarding the Advanced Hole Cutting Techniques. Will be waiting for that video too.

[@pixtrick]

I'll add it on my list then ;)

[@Kate0lp]

It is the best tutorial on the Stencil buffer. Thank you so much for the great video!

[@pixtrick]

It's an honour ! Thank you so much !

[@-lunte.]

presentation quality off the charts! liked and subbed

[@pixtrick]

Thank you so much :D

[@armint9109]

Thank you so much for the amazing explanation!

[@pixtrick]

You're welcome :D

[@elisechevrier7996]

Trop bien ! Merci Pix !!

[@lulloxa]

Thanks a lot I really needed it ! It's very clear ! Another one ?

[@pixtrick]

For sure ! Stay tuned ;)

[@tufanaydin6340]

Thank you so much you done great job

[@Smash_ter]

11:50 you could also assign those faces as separate materials

[@pixtrick]

Definitely yeah, I didn't talk about the UV mapping either but that was not the sake of the video

[@Smash_ter]

@@pixtrick i feel the focus more on the stencil aspect was more important imo. And yet there's still a lot that i dont know about stencils like the satuates and wraps. This at least should help many get a decwnt understanding

[@RindGame]

Nice Video, Learn A Lot

[@pixtrick]

That's good news!

[@hamzzashaffi]

Thanks for sharing! :)

[@pixtrick]

It's my pleasure !

[@dpgames8165]

Man this is some good stuff! You should consider making a game someday! (:

[@pixtrick]

Perhaps someday ... Stay tuned ;)

[@watercat1248]

I think this 3D mask system i was looking for i will definitely do some tests tomorrow.

[@pixtrick]

Don't hesitate to message me on discord if you need anything

[@bovineox1111]

Disabling the ground collider is fine for a single entity but you’re quite likely to have multiple entities and those will still rely on the ground entity!!

[@pixtrick]

Absolutely, that was just an example, it's totally up to you to improve this ;) In this case what you would do instead is disabling the collision between the 2 colliders basically

[@muhammedensarsandkc4466]

Just excellent.

[@pixtrick]

Thank you so much !

[@thewightone7441]

I've been working on a feature for quite some time now, and recently came to the conclusion that shaders/stenciling was my best option. This video has brought me so close to the solution, but I fear I'm still too new to figure out the solution. Here's the context: I have a building exterior and interior. When the player enters the building, I want all exterior objects to fade into darkness, and interior objects to remain on screen (The walls move out of the way, so thats not the issue here). When the player exits, those objects fade back in. From my failed attempts and the info I got from this video, here's what I think I need to do: Separate interior objects from other objects by putting them on their own layer Have a world space blackout screen hover in front of the camera. Set the Blackout screen to its own layer "blackout" When I want to fade out all exterior objects, I call in the blackout screen Use a stencil that accomplishes the following: If an "interior" object has been rendered on this pixel, AND would be overwritten by a "blackout" pixel, use the "Interior" pixel. This seems to be very close to what you're doing in this video, so I am almost certain that this is the solution, but I can't seem to wrap my head around how this looks set up.

[@pixtrick]

I guess we already figured this out ;)

[@stylie473joker5]

Thank you for your hard work, The hole is working as intended however The wall i put the render feature on is being rendered transparent (i can see stuff behind it) any idea what am i missing ? Edit: its the Screen space ambient occlusion affecting the wall like that (disabling it fixes the issue). but that's not a good solution. Edit 2 : Ticking "After Opaque" in the Ambient Occlusion render feature fixed the issue. I hope it helps anyone having the same issue and thanks again for the awesome tutorial

[@pixtrick]

I didn't know there was such a weird interaction ! Thanks for sharing that :)

[@fleity]

I can add a bit more information about why this wasn't working and kind of still is broken. See my other comment I left :)

[@othmanmoat]

Hey I'm interested in holes that do not require a special layer for the ground ;) Also, for the hole collider, is it not possible to not have it convex? (I guess it would be possible to make a non-convex shape by making multiple convex shapes work together)

[@pixtrick]

It is indeed possible but you'll then have to procedurally generate a concave mesh that fits the hole geometry. I'll maybe make a video on that later since it's very interesting but quite advanced too

[@_TommyP]

Hi, this is so great. I was wondering if you could tell me what version of unity you are using? I'm finding it impossible to get transparent materials to render even after adding the renderobjects.

[@pixtrick]

Did you choose the right render queue for your render feature ? I'm using Unity 2021.3+ by the way

[@_TommyP]

@@pixtrick hey, yep. I ended up getting it working in a new project. No idea what I messed up. Thanks for the very quick reply!

[@pixtrick]

@@_TommyP maybe it was something related to depth priming or the render type then. Anyways, I'm glad you found a solution!

[@aureliopuertamartin3631]

Great tutorial! The card effect is really useful for Augmented Reality experiences! I have only one problem, let's say I have an object that animates from inside the card/portal in and out. Currently the object is only renderd through the portal, so in case it goes outside, and you look the portal from the side the object will disappear. How do I need to set up the materials and settings to achieve the effect?

[@pixtrick]

Hey mate, glad you found an interesting application for this tutorial :D The portal inside is indeed rendered only through the portal. I mean, yeah, that's intended. Now, if you want to render the objects even if they are in front of the portal without using the stencil buffer (basically looking the objects from the side when they are out of the portal) and only in that case, this gets a lot more tricky, because you'll need to use the depth buffer, but transparent materials (such as the one used for the portal) do not write into the depth buffer so you're a bit boned... You'll need to use a trick to make the portal material write into the depth buffer, then create another render feature to render the objects out of the portal. Actually, the hardest part will be to find that trick ... Hope that helps ! Good luck !

[@exoqqen]

did you find a solution?

[@empluna]

Hi, how did you manage to make the inverted cube invisible from the side? Like the tunnel to go through? Great tutorial

[@pixtrick]

Hi ! It's an inverted render culling, you can access this parameter in the material editor

[@kahnetop]

great explanation! is this possible in the build in render pipeline ? i struggle a lot to convert the technique correctly

[@pixtrick]

It is indeed possible, but you won't have access to the render features. Basically, the URP allows a per-layer rendering with its render features (so you don't have to worry about the materials of each object of the layer), whereas the built-in RP don't, so you'll have to write custom shaders to get the same result, it's a bit harder but 100% doable in my opinion.

[@kahnetop]

@@pixtrick thanks for the response! I finally succeeded! I think the hardest part was that i used Amplify and its stencil buffer parameters were some time buggy 😅 But you're explanation was really great and help me find the way!!

[@WalrusesAreTheOne]

Hi, one of the first portal tutorials I saw relied on a mask material instead of a stencil buffer, to make the outsides of the portal invisible. It seemed way easier to me than setting up a stencil buffer. Is there a reason this is better? Thanks!

[@pixtrick]

Hi ! I think that what you saw is also based on the stencil buffer, but perhaps the main difference here is the use of the URP which allows you to add stencil parameters on any rendered material. However, I agree it might be annoying to set up, so if you're only using hand-written shaders (shader graphs doesn't count), you can add these stencil behaviours way more easily. Does it answer your question? x)

[@WalrusesAreTheOne]

@@pixtrick Thanks! Yes, that answers it:)

[@fleity]

I will ask this here because your video is truly one of the best stencil buffer explanations I have seen over the years (especially considering read/write mask)^^ You say turn off shadow casting for the ground. That is because the shadow caster pass does not care about the stencil value. That is really not ideal. I see the workaround of setting the walls to draw shadows both sided okay but it still feels a bit broken.. the same is true for for any kind of depth pre pass unfortunately. Since the ground is disabled from the regular rendering loop it does not show up in the depth pre pass at all which will prevent us from using depth priming as optimization and the objects will also be missing in the depth texture (except depth priming is disabled, but unfortunately this only applies the the game view, scene view is always using a depth pre pass). Btw SSAO in DepthNormals mode forces a depthNormals pre pass.. so the same issue applies here as well. That is why it was not working for stylie473joker5. After Opaque mode + priming disabled copies depth from the depth written by the color pass, but that works only in very few cases. Do you have any ideas how one could fix / work around that issue? The way you show here is the way everyone seems to implement stencil buffer usage in URP but it's seems incredibly limited due to these limitation. Thanks for thinking about it :)

[@pixtrick]

Hey! Thanks for the comment and the kind words I appreciate that! Now, about your question, I don't really think there's a real workaround for that, since the shadow caster pass uses the vertex data to draw shadows. Since the fake geometry created by the stencil buffer is not an actual geometry (I mean it is not some vertex data, it's just pixels on screen), the shadow caster pass cannot render there properly. Then, I don't really see any workaround for keeping the ground shadows while not rendering them into a hole. Now, if you really need your ground shadow (I assume it would be for like a platformer game or something), I would suggest not to use the stencil buffer unless you're really forced to. Use real geometries as much as you can. I think this is not (or maybe it is but it would be such a pain regarding performance) possible to draw shadows pixel by pixel, so that the shadows are not rendered into the hole but are rendered for the ground, but that is absolutely not reasonable. May I ask you what you are trying to achieve? Maybe there's a workaround in your situation and I would be glad to help :) Have a great day !

[@fleity]

​@@pixtrickThanks so much for the reply :) I think, I figured out two possible solutions by now but either way it sucks a bit that this is so exhausting to do. Beware this will be a long text, thanks for letting me rubberduck a bit. It is in fact not a plattformer, rather an isometric adventure type of game. I am making this example up but the situation is basically like this. Imagine a hill and ground mesh with a meteor crater on top. The crater is the hole I would like to stencil out. I can not build this right into the mesh because it is procedurally placed. Inside the crater I have a plane which reads the depth texture to create some fog-ish dust. Issue 1: Shadows of the hill are drawn without the stencil cutout hence everything in the crater is shadowed. Turning of the shadows would mean no shadows in the surrounding area by the hill. Issue 2: the information in the depth texture is also wrong. This is a bit harder to explain and needed a bit of digging in the frame debugger and render pipeline. The Render objects feature only writes to the cameras color target. The color target contains a depth channel but it is only used when certain conditions are met. For example in the game view, if depth priming is disabled and nothing else forces a depth pre pass (like ssao in depthNormals mode). The scene view always uses a depth pre pass not matter what we do. When we remove the ground from the regular render loop and only render it through render objects it is completely removed from the depth pre pass because the render objects feature does not set that up and the mask definition that we have set excludes it from ALL main passes (except for shadows...). I wonder why this isn't done in the render feature but that I guess that this is the reason why the feature still says experimental. Unity didnt figure out a way to make it compatible with any possible srp renderer. This is an issue what the typical "hey draw the characters silhouette" tutorials don't have because they draw the character in the regular loop too. As for solutions, one option I can make work is not using stencils at all. Render everything in the main loop, set the queue for ground materials to 1990 and create a depth mask shader + material that overwrites the depth where it is rendered with infinitely far away. Then everything that is rendering afterwards renders as usual and even shadows are correct (because the shadow passes use the depth only passes of the shaders too). The other solution is to modify the render objects feature to also create a depth pre pass or inject the missing draw calls into the existing pass with the correct stencil renderblock set and render the stencil render objects feature extremely early (before pre passes). Second solution is more flexible / cleaner and actually allows using the stencil buffer fully but it is much more work. I am at the point where stuff renders in the depth pass but not with stencils yet. I guess my issue here is simply summarized with: Render Objects does not work with depth pre pass, stencil in render objects does not work with shadows in URP, but it could, unity just didn't finish it yet 😕.

[@pixtrick]

Hmmm your thinking process might be good, but I think there's a much simpler solution for such a problem. Basically, your problem is : how to generate a hole in a procedurally generated terrain. To me, this is much simpler than having to deal with depth pre passes and everything, plus it would be much more cleaner. As I said before, you have to use real geometries as much as you can, and only use the stencil buffer when it's needed. To me, you shouldn't use the stencil buffer at all for this problem since the crater will not move in your scene (I mean I assume it won't move). I don't know how you generate your terrain and how it should look, but I think you should use techniques such as the wave function collapse that allows you to design "tiles" that you can put together in numerous ways. That way, you can keep a procedurally generated terrain and would only have to make a tile where there is the crater. If you still really want to use the stencil buffer, well don't xD I cannot really help since I think it would feel broken ether way. Still, it was extremely interesting to read your different solutions, but yeah to me you should change your approach to this problem. Hope that helps !

[@zave1776]

Is there any chance to apply the same effect in the HD render pipeline? needed very very much

[@pixtrick]

Hey, I'm not very sure about that, you should try ! In any case, if the render features are not available, you can still use the stencil buffer through custom shaders

[@marcmantra]

The magic card example makes me think. I tested and if I let the card as Default Layer and all the content to the new layer, I can set a Render Feature with the same stencil value and using Equal to have the same effect. Without 2 render features. Did I miss something? Thanks for your tutorials

[@pixtrick]

Do you mean that the effect works without a render feature for the card itself ? Interesting ! I would be curious to see that To me, the card model would overlap all the content if you render it without a render feature, but perhaps I'm wrong, I will test that soon ;)

[@marcmantra]

It's true that using your example you created the hole and show the content, it's perfect. But we can have a hole modeled in the card and use the quad only to show the content. Not a really important thing...

[@pixtrick]

@@marcmantra oh yeah that works as well ! I just used this technique to allow you all to resize the hole in the card as you wish I agree this is not an important thing, but I didn't think about that to be honest x)

[@m1glw97]

Is this effect somehow possible with depth priming set to auto or forced?

[@pixtrick]

Not sure, but all the tests I've made so far point to the fact it is not

[@ToniehGaming]

New sub! I have a doubt, basically in the example of the letter it's as if it were a "hologram" on a cellphone using the position of your eyes (camera). Is there a possibility to obtain that texture relative to the camera position so that on the full screen I can create this hologram effect? This is not with stencils, but I have been looking for how to do this and can't find any information :c.

[@pixtrick]

Hey, thanks a lot ! I'm not sure to fully understand your request to be honest, is it about the floating text in the magic card? If that so, it's only a canvas in world space with appropriate dimensions

[@ToniehGaming]

@@pixtrick nonono, I have an eyetracker device, I can know the user position and I want that the monitor screen act like that portal to make that card effect, It's so tricky. If you atatch the game camara to the user real position works but not that good that the card effect

[@ToniehGaming]

It's hard to explain, also is similar to the 3d wallpaper efect, but that use the acelerometer, is not that real effect

[@pixtrick]

@@ToniehGaming ok i think i'm slowly getting it. Is it like a parallax effect ?

[@ToniehGaming]

@@pixtrick Not so much like a parallax, more like the monitor was a window and there were things through it. It looks the same as what the card effect looks like.

[@Tukyo]

my braiiinnn

[@pixtrick]

Imagine mine to make the video 💀

[@InstantStarWars]

Merci, maintenant c'est limpide

[@ninocoillard6775]

This is awesome however when I apply the card part to my game with a door I want to mask I can still see the shadow, do you know how to fix it ?

[@pixtrick]

Hey there ! You just have to disable shadow casting in the mesh renderer

[@ninocoillard6775]

Okay, and if I want to cast just when it is enabled and then when it is masked, disable it ?@@pixtrick

[@pixtrick]

@@ninocoillard6775 when you say enabled, do you mean when you can see through the door or do you have something that enables or disables the view through the door ?

[@RT-qi7rn]

I'd really like to learn how to do it without requiring a special layer for the ground as I'm working on a game where the layers have already been carefully setup and can't really be changed

[@pixtrick]

I'll try to make a video on that

[@helikaon00]

I can't seem to follow your instructions when it comes to NOT using a shader to solve it. You pop up the render object settings but I can't figure out how to set it up to work. I am trying to solve this without shaders because I am developing something that can't use normal shaders at all. It could use shader graph, but I don't know how to do this in shader graph either.

[@pixtrick]

Hey ! You just have to use a render feature then. Basically, instead of using a shader for the mask material, you just have to create a render feature with a Never stencil comparison operation, so that it won't show on screen. You can use any material you want for that mask then since it won't show on screen at all. EDIT : I forgot to mention that after doing that, you'll have to use the Fail operation instead of the Pass operation, because the comparison operation will always be false, so that the Pass operation will never trigger.

[@helikaon00]

Hmm, yeah, I just can't seem to figure it out. I was trying to make something for the Apple Vision Pro but you can't use normal shaders, you can only use shader graph (or other node based shading methods), because they use MaterialX which requires node based shading. Thanks for your help though! Appreciate you taking the time to respond.@@pixtrick

[@bjaksel]

Your brain is massive

[@pixtrick]

You're flattering me x)

[@Hozgen]

can we add this stencil id to shader graph? I couldnt find any method.

[@pixtrick]

Unfortunately, the stencil buffer is not supported by the shader graph ... May i ask why you want to retrieve this information ?

[@Hozgen]

@@pixtrick I want to move object/objects over other layers or on certain layers in 3d platform. This is is easy in 2D but in 3D it is not satisfactory to me. There is a solution in Unity docs but it is not working for URP. The second one is shadows, and i couldnt get rid off weird shadow renders for this technique. I also want to see proper shadows on other objects. :/

[@pixtrick]

@@Hozgen you can use the depth buffer to do that I suppose. Maybe you can contact me on discord to go more into details because I'm not sure to get your problem to be honest :/

[@hldfgjsjbd]

Can these be done in Shader Graph?

[@pixtrick]

Unfortunately no, shader graph doesn't support the stencil buffer

[@tufanaydin6340]

SSAO and DECALS not working how can we fix it ?

[@pixtrick]

Decals are not compatible with the stencil buffer, or at least I haven't find a way to make them work together yet. We discussed about it on the discord server of the community but didn't find a solution ... Regarding SSAO, I'm not entirely sure why it doesn't work as intended ... It may be either the AfterRenderingOpaques event queue or simply because objects rendered through a render feature are not considered as "real" geometries

[@Ch1psi-k]

does this work in Unity HDRP?

[@pixtrick]

I think there would be a way to use the stencil buffer in hdrp, but I don't know how honestly (i mean without having to hard code it in a shader)

[@TheGameLecturer]

This works greatly in Editor for me, but in the build it doesn't function properly (neither on Android nor on PC) and I just get plain geometry.

[@pixtrick]

What unity version are you using ?

[@TheGameLecturer]

2022.3.17f1@@pixtrick

[@pixtrick]

You have to make sure that the depth priming is off, and I'm not entirely sure but I think setting the rendering path to forward can help as well

[@TheGameLecturer]

@@pixtrick yes it is disabled and the rendering path to forward, I tried different depth texture modes as well. I'll try to isolate the problem by making a new project and make some test and I'll come back to you.

[@pixtrick]

@@TheGameLecturer all right ! By the way you can message me on discord if you prefer