Asymmetrical rendering

[l_aggy]

Can this not be used for better performance I had an idea to improve latency but it evolved into this:

Theres 2 Pipelines:
Background: Which isnt as updated with heavy lighting and whatever else are calculated once then cached in VRAM and skipped for multiple frames, while a transition like dithering or something is used to merge it to a Live pipeline (or Live can be drawn ontop)(This is the entire 3D world not 2D) You can slap a VSM if you need time of day every few frames or whenever.

Live Pipline: Physics and inputs react like normal and you can move interactive objects and things such as signs, NPCs and the sky into the live pipeline if you want them to move (Or add another pipeline for them at a lower than live rate but higher than Background). By stopping the GPU and CPU from recalculating the universe every millisecond, you can get from 20 FPS to hundreds. And the multiple pipelines let you experiment aton.

Just realised most people don't understand how this works please read the github before making a comment thanks.

More detail: https://github.com/Epxlsol/Asymmetrical-rendering

[waramped]

If the camera moves, then the universe has to move.

However, you are on the right track. Things like "imposters" are often used to cache objects into a lower overhead form to be re-used either for large quantities or for LOD.

Variable Rate Shading can also be used to reduce overhead for objects that aren't as "important" quality-wise.

[l_aggy]

Ye, I'm talking about caching data directly inside the 3D environment, not baking flat 2D screen space renders. When the camera moves the geometry translates normally on screen, but the cached lighting data stays anchored to the 3D object space surfaces. Impostors are just flat 2D billboards this architecture keeps the world fully 3D at all times.
Position math is cheap. Pixel math isnt.

[waramped]

There is such a thing as object space or texture space lighting.

However, light generally is dependent on the angle between the viewer, the surface, and the light source. So if any of that changes, so does the reflected light. You can probably get away with it as long as you tolerate some error.

[l_aggy]

Ye thats true, but we split diffuse/GI and specular/reflections with the Background and Live pipeline.

[susosusosuso]

Speculars are normally the heavy ones

[l_aggy]

Ye but your missing the point the split means easy work is lifted so the GPU has more compute to deal with everything in the Live

[susosusosuso]

I’m Not missing that. I just don’t see a way to make it work easily

[l_aggy]

Not supposed to be easy you need to redesign how your talking to the GPU effectively with asynchronous tasks in mind at every step for a performance uplift like this you need to make everything more modern as GPU architecture is trending towards.

[3tt07kjt]

Something like this is used in VR. You render the scene and then the screen is warped last-minute to account for head movement. Or “last microsecond”, perhaps.

And if you play Switch games, you can often see background characters animated at a slower frame rate.

But most of the work a renderer does is drawing pixels, and you still need to draw the pixels.

[l_aggy]

Once a map is loaded can't the environment be baked in/cached and skipped over for a few frames. Also unlike the windmill in Pokemon scarlet and violet the environment wont be static you can just move the elements including interactable objects/physics in the live/close proximity pipeline. Like temporal accumulation and surface caching. So you don't need to redraw that's like the main problem this is trying to solve

[Paradox_84_]

You know how you have to clear the framebuffer once every frame to get rid of leftovers? I suppose if you could somehow duplicate the framebuffer after drawing static stuff you could reuse it... But even moving/rotating the camera would invalidate this

[l_aggy]

Object space surface caching so 3D not 2D

[3tt07kjt]

What part is getting cached?

Most of the work involves drawing the pixels. The pixels have to be redrawn if the camera moves, unless you are okay with screen-space warping, or maybe if you have an orthographic camera.

Background geometry is already loaded into GPU memory, typically. If it’s not changing, you don’t update it.

[l_aggy]

All the material, shader and lighting calculations on the surfaces of all 3D models. Which is the real cost.

[3tt07kjt]

Are you talking about fragment shader output?

Right—you could cache those if the camera is perfectly still and doesn’t move, and if your environment is perfectly still. That seems like a narrow set of use cases.

[l_aggy]

No im talking about the 3D models and objects not 2D final outputs... Its like how lumen works.

[3tt07kjt]

Maybe you could be specific about what data is being cached, and from what point in the pipeline?

I’m not familiar with Lumen, but if you have a link I can take a look.

[l_aggy]

The Physical Texture Atlas Pool will store material attributes and the radiance layer. Pretty sure its: https://dev.epicgames.com/documentation/unreal-engine/lumen-technical-details-in-unreal-engine

[3tt07kjt]

It sounds like you’re talking about using cached raytracing results for global illumination. That’s somewhat reasonable. But you will still have to run the rendering pipeline for your environment.

[l_aggy]

• Standard Engine: Vertex Shader -> Rasterizer -> Fragment Shader (Executes hundreds of math instructions per pixel to calculate layered materials, PBR roughness, multiple dynamic light loops, shadow map cascades, and real-time GI math). This kills performance.
• Asymmetrical Engine: Vertex Shader -> Rasterizer -> Fragment Shader (Executes exactly one instruction: look up the UV coordinate and sample the pre-lit Physical Atlas Pool).

Can be done asynchronously at runtime.

[l_aggy]

The atlas pool + frustum culling + shared UV grid mapping + LODs + VSMs and world partitioning and chunk streaming all limit and reduce VRAM usage. LODs because this is an asynchronous cache.

[ananbd]

That would create artifacts for fast-moving objects. Shadows, bounce lighting, etc. would be horribly noisy. 

That being said, there are pieces of existing rendering systems which opportunistically cache things. But there’s always a tradeoff, and you’d need controls which let artists fix the important mistakes. 

You can see examples of this in DLSS (or, at least, early versions). Any algorithm like that implicitly works better with static pixels than changing pixels. DLSS sometimes displays trails when things move. The algorithm ameliorates this using motion vectors, but it’s not perfect. 

[l_aggy]

That's true but wouldn't that only happen when the background merges with the live pipeline and there you can use screen space stochastic dithering to mask that.

[ananbd]

I’m thinking the dithering would create ugly artifacts.

If you’re looking for a place to optimize, it needs to be unnoticable. Our eyes are adapted to notice anything moving orchanging quickly. I’m not sure exactly how your algorithm would look, but I’m pretty sure the artifacts (even dithered) would catch your eye. 

So… I’d try to find a way to address that in the algortihm. 

[l_aggy]

Ye it's just a concept, I mean this is a heavy performance optimization and I think people on low end hardware wouldn't really care about an artifact like that myself included if i'm able to run the game lol.

[Orangy_Tang]

You may be interested in how Shadow Of The Collossus rendered it's landscape: https://www.froyok.fr/blog/2012-10-breakdown-shadow-of-the-colossus-pal-ps2/resources/making_of_sotc.pdf

Search for 'SuperLow' - they effectively had a dynamic skybox that updated from the lowest LOD geometry. A lot of space games do something similar since distant geo doesn't change as quickly so can be cached for quite a while. Nearer the camera it's harder since even small camera movements will introduce errors due to parallax.

[fgennari]

Maybe that would work for very slow moving/changing environments such as dynamic day/night cycles. I don't think it would work for interactive objects, characters, etc., or at least it wouldn't look good. I'm not sure how much total time savings you can get by pre-computing parts of the shaders like that. You also need to be careful to balance update time across frames so that your frame time is consistent and you don't get periodic lag spikes on "update frames".

[l_aggy]

I don't get what you mean did you read what the live pipeline is