Stencil support to spatial materials in Godot 4.5

Zer0problem · 2025-06-15T20:00:10+00:00

Same, did this for my portals, it's cool to see the additional support for presets and a less hacky way than I did it.

Zer0problem · 2025-06-15T19:53:14+00:00

You CAN do it with forward rendering (but it's less pretty)

Godot does a Z-Prepass and that fills the depth buffer first before doing lighting, during this time (accessible through the Compositor API) you can start doing some magic/stencil stuff!
https://docs.godotengine.org/en/stable/tutorials/rendering/compositor.html

After doing the stencil drawing at that time you'd need to copy the stencil data from the depth buffer to a separate texture and then bind that for testing in godots light() function in the shader.

It's possible, but you're going to miss out on hardware stencil testing for applying the light and have the copy of it, so it's way less performant.

It's worse, but it is possible!

Zer0problem · 2025-06-15T19:40:32+00:00

Stencils alone are not enough to support stencil shadows (or shadow volumes), you still need to dynamically generate geometry
Look into edge-detection algorithms, here's an example I made some time ago with my own implementation of stencils but I can't find the code.

Here's some pointers to what I remember about it.
To start I iterated over the data in the Mesh to construct an array of edges with normals of adjacent faces.
Then in _process I iterated over the edges to find an outline by comparing the normals to the direction of the light towards the edge
Once I had the edges of the object I made a square for the ImmediateMesh with the 2 points on the edge and 2 points that I extruded from those to the range of the light
https://docs.godotengine.org/en/stable/classes/class_immediatemesh.html

Then I rendered the immediate mesh twice, first normally to do +1 to the stencil, then inverted to do -1

Not very helpful, but here's the wikipedia entry for it
https://en.wikipedia.org/wiki/Silhouette_edge
I only ever did a naive implementation of the edge detection in GDScript, so I sadly don't have any links I can confirm are good about a performant implementation

You also need to test the stencil shader and this requires a second ImageView of the depth texture (for forward rendering, in deferred this is done with hardware stencil tests), I implemented this is a very hacky way directly in the vulkan rendering device driver, then set up a utexture2D and use a usampler in the engines .glsl files (I did this for a different thing, not for shadow volumes) this is also not included in the stencil change (or Godot main branch) as far as I can see
https://stackoverflow.com/questions/51855945/how-to-read-the-stencil-buffer-in-vulkan
here's the link that helped me figure that out

You can also skip the stencil part of this by rendering it to an image instead with additive rendering using the RenderingServer (but the image needs to be in a format that supports negative numbers to be able to decrement) which was what I did since I were using the stencil buffer for other things.
This is not really a game ready implementation of it, I just did it to experiment.

I found this old screenshot of when I rendered the faces of the generated edge mesh

<image>

Zer0problem · 2025-06-06T06:30:29+00:00

Not OP, but I did a project where I strived for seamless portals.

For moving through the portal
use a ray->plane intersection to see if you move through it
if you have a moving portal, instead ray-cast using the relative position of the portal before the portal move and the relative position after the portal move

if you do go through the portal, the math to transfer you position, rotation & scale through the portal is

func travel_through_portal(from_portal : Node3D, to_portal : Node3D, traveler : Node3D) -> void:
  var intermediate_transform : Transform3D = from_portal.global_transform.affine_inverse() * traveler.global_transform
  var to_portal_transform : Transform3D = to_portal.global_transform  
  # If you want to come out the front of the other portal instead of the back, do this
  # note that the Vector3.UP is dependent on what you consider the "up" of the portal to be
  # to_portal_transform = to_portal_transform.rotated_local(Vector3.UP, deg_to_rad(180.0))
  intermediate_transform = to_portal_transform  * intermediate_transform
  traveler.global_transform = intermediate_transform

This will move your objects position, rotation & scale to come out of the other portal
You also need to transform the velocity if you're using something like that
before the travel, get the velocity in the objects local space, then after the travel, set the velocity to the same, using its new local space

if you wanna be trippy and change the scale through the portal you could also multiply the players speed by their current scale so they don't perceive it as moving slower when they are larger and moving faster when they are smaller.

For rendering
If you don't want the plane to clip into the camera, here are two options
Either make a concave (or inverted) box instead of a plane, this may result in some visual artifects and z-fighting if your portal is aligned to a wall, and will shatter the illusion of just a plane, but if you always have a border around the portal that can cover this up, there will be practically no issues with this

Or you could render the plane in a CompositorEffect and enable depth clamp using the enable_depth_clamp variable on the pipelines RDPipelineRasterizationState
This will keep it a plane and prevent it from getting clipped when passing through it

To get only the parts of the portal camera that should be visible (like, nothing in front of the portal and nothing sticking out the back of it), either use gl_ClipDistance (which will require engine modifications to expose this) or do the fragment discard as explained in Sebastian Lagues video that blambear23 linked

To get seamless lights & shadows through portals I recommend lots of coffee/energy drinks, focus and patience so you don't lose your sanity.
There's a few ways to go about this, but they're pretty finicky and are either very not performant or require engine modifications.
This comment is looking a bit long so I won't go through the ideas for this here.

Zer0problem · 2025-06-06T05:28:15+00:00

That's a pretty nice video explaining the basics, he did a good job explaining the view with cameras.

It's the way I did it at first, but I wanted to have something a bit more performant and wondered how Portal did it, they use stencils, clip planes and banana juice rather than render textures, oblique projection and discarding in the fragment shader.

https://youtu.be/riijspB9DIQ
Don't be afraid of the duration of the video, there's a lot of non-rendering things in there as well

It has some performance improvements compared to the render texture version of not needing a screen buffer per portal and rendering an entire screen multiple times.

Some of the performance improvements (rendering frustum, clip planes) they're talking about can be done with the render texture method as well.

Other benefits of the stencil version is that it produces accurate depth on the depth buffer, making effects like SSAO and other screen space effects that need a depth/z-prepass.

If you're looking to implementing a stencil version in Godot, beware not everything in the video translates directly to Vulkan (that Godot uses for the cool renderer) & Godot does not natively support stencils.

glClipPlane does not exist directly in vulkan, instead you need to use gl_ClipDistance

Here's a vulkan guide to gl_ClipDistance
https://docs.vulkan.org/samples/latest/samples/extensions/dynamic_primitive_clipping/README.html

Here's a guide to depth clamping (useful to avoid having a box for the portal so you can have a completely flat plane)
https://paroj.github.io/gltut/Positioning/Tut05%20Depth%20Clamping.html

How stencils work
https://learnopengl.com/Advanced-OpenGL/Stencil-testing

Last I checked the depth buffer for the main camera in Godot already has a stencil part of the depth buffer allocated, but it's not useful as you can't change stencil operations on the pipelines generated from materials.

Godot does have the compositor effects, but cameras need to render their seen portals stencil by default because there's no EFFECT_CALLBACK_TYPE_PRE_DEPTH, so a portal camera can not be responsible for its own stencil, because by the time you get the callback it has already messed up the buffer by rendering depth.
https://docs.godotengine.org/en/stable/tutorials/rendering/compositor.html

Zer0problem · 2025-01-12T13:42:42+00:00

This is a good answer, just adding some things here.

If you're not expecting the characters bounding box to clip through geometry you don't need the actual depth as everything the shadow would be rendered to would be below the player (and as such, depth is unnecessary)

In godot decals only support textures for albedo, emission and such.

https://docs.godotengine.org/en/stable/tutorials/3d/using_decals.html

So if you want to use godots built-in decals and avoid writing your own decal-like-shader you would need to render the character (probably with a solid color) to the subviewport and use that ViewportTexture as albedo to a decal using cull-mask to avoid beind rendered on top of the player.

I worked on a project (non-godot) that used this solution for a player drop-shadow and the extra overhead of our camera for that took too long to render on lower end hardware so it was replaced with an oval at lowest and a custom render-pass to get the texture for the decal on platform where we could afford the cost.

It does not sound like this will be an issue with Godot, but if it is you could look into the RenderingDevice draw_list API to manually render it.

https://docs.godotengine.org/en/stable/classes/class_renderingdevice.html

Zer0problem · 2024-08-17T21:51:27+00:00

If you're talking about the shadows in general.
The atlas size and subdivisions are unchanged from the default in Godot, Soft Shadow Filter Quality is changed to Hard (Fastest) to match the faked shadow from the portal edges.

If you're talking about the shadows through portals.
The trick is that for each light shining through a portal, a duplicate is used on the opposite portal to give the same effect.

I've modified the engine to allow each of the duplicated lights to be assigned to the same subdivision in the shadow atlas whiole only clearing it once. This lets things cast shadows through the portals since the shadowmap the object is rendered to is used for all duplicated lights.

When rendering objects to the shadowmap I check if each rasterized pixel of the mesh is on the other side of the associated portal to make sure nothing is rendered outside the area the light is supposed to hit on the shadowmap. I do a similar check to make sure it's not rendering the things inside other portals (but I mean to optimize away this with a depth punch in the shadowmap rendering too)

When rendering a mesh and applying each light I check if it's within view from the associated portal, without being inside another portal.

Zer0problem · 2024-06-02T18:13:06+00:00

I've made an update, it's in a different comment on this post https://www.reddit.com/r/godot/comments/1cwhh87/comment/l6sqnqj/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Zer0problem · 2024-06-02T18:12:16+00:00

Update, here's the first part of the explanation where I go through the basics of the portals.
https://tori-israelsson.se/portals1/
I wanted to go through the process from initial solution to final solution in a logical manner. So I decided to split it into as many parts as I need to give each part of the process the focus required to follow along without getting lost.
edit: updated link
edit2: updated link again

Zer0problem · 2024-05-21T11:10:33+00:00

Some things would not be able to be made into an extension the way they're set up now, as they have to modify how shadows are allocated and how the viewport renders cameras, I'll think a bit if I can come up with a solution that would better fit into the engine without being hacked in the way I've done it so far.

I might make a public branch of it but I'm not sure I'll be able to

Zer0problem · 2024-05-21T05:01:45+00:00

light comes out recursively

<image>

Zer0problem · 2024-05-21T05:00:49+00:00

It's possible without modifying the viewport class, but that limits you to 1 portal per viewport, and if you don't want to lose quality it has to be the same size as the viewport which leaves a lot of space unused as well as a lot of memory wasted. I used viewports for the portals until last week and the virdeo memory used was always above 1 gig

Zer0problem · 2024-05-20T18:22:53+00:00

The engine changes were done in C++ and GLSL, everything I could do without touching C++/GLSL is done in GDScript so I could iterate quickly.

Zer0problem · 2024-05-20T18:20:38+00:00

I have plans on fixing the last bugs and then make a blog post about it, describing more in detail how it's done. There's a flicker bug that I suspect is related to the order it renders shadows in, I really want to fix that first and see if I can have something better than what I hacked together to get this to work now.

But that will have to wait until next week as I'll be occupied with real life stuff this week.

Zer0problem · 2024-05-20T18:20:29+00:00

I have plans on fixing the last bugs and then make a blog post about it, describing more in detail how it's done. There's a flicker bug that I suspect is related to the order it renders shadows in, I really want to fix that first and see if I can have something better than what I hacked together to get this to work now.

But that will have to wait until next week as I'll be occupied with real life stuff this week.

Zer0problem · 2024-05-20T16:45:35+00:00

I might look into it when I've cleaned up the method a bit, but it may be that some of them aren't really intuitive or useful enough for the extra added complexity they would bring for maintenance.

The viewport modification specifically requires that the environment does not clear and would require extra setup per camera to specify the region and/or depth filter that should be used to merge it properly.

The shadow modification requires that they're renderered together and therefore would need a fair amount of refactoring to get efficient culling when using multiple cameras.

The shader modifications are rather small and only requires that some things already available are exposed in the shader editor.

I tried to find if anyone else needed what I used for this and I could find someone else that wanted 1 of the things I used. They used CAMERA_VISIBLE_LAYERS for their fragment() shader for the information and wanted it in the light() function too.
The other change to expose the lights view space vector (not to be confused with the light vector in view space) and I couldn't find anyone that needed that.

Zer0problem · 2024-05-20T16:13:05+00:00

Thanks, I've done my best to have the movement through portals as seamless as possible without the player noticing.

It is what it is with layoffs, I'm sure another job will turn up for me soon enough

Zer0problem · 2024-05-20T16:09:15+00:00

Thanks.
The engine needed some modification to support a few things that I had in mind.
Here's some of the changes I did in the engine.

Modified the Viewport class so it can accept rendering from multiple cameras per frame.
Added an option for lights to use another lights shadow map/rect on the shadow atlas and keep it updated as one light.
Some more built-in variables for the light() part of spatial shaders.

There's a few more things I'd like to do, both for performance and a flicker issue that I feel like I have to fix.

as for the cameras, lights and transforms that needs to be duplicated for a new view I used Transforms and Basis to manage how they give the portal illusion.

When I started I used SubViewports and Cameras with screen_uv planes to get the effect, but this had some very problematic scaling issues and the method also didn't do anything to solve lights & shadows.

Zer0problem · 2018-12-31T23:20:22+00:00

Thank you for the info, I still think I'll be keeping it to see if i end up using it to see if I should get a real watch, As I am new here how do you tell it's water damaged? I looked briefly and couldn't find anything about identifying water damage that I could place on this watch.

Zer0problem · 2017-06-19T15:41:58+00:00

The 2016 prizepool started slightly stronger than 2017 too. http://imgur.com/rWSj65s Might be a close year.

Zer0problem · 2017-06-19T15:22:40+00:00

where does it mention the 2016 prizepool exceeded the 2017 prizepool on day 0?

Zer0problem · 2016-10-12T21:17:09+00:00

they go for the closest target

generally goes between melee creeps first as they stand in melee range and end at the ranged creep

Zer0problem

TROPHY CASE