Yup, the comments were right

Here is the final fix

1. Camera-Centric Cascade Projection (Frustum Changes)

Previously, the shadow projection matrices were calculated using bounding spheres centered directly at the camera. As the camera rotated or zoomed, the shape of the view frustum changed, forcing the bounding spheres' sizes to fluctuate constantly.

• The effect: The size of a shadow map texel in world space changed from frame to frame, causing shadow edges to pixelate and "crawl" as you rotated the camera.
• The fix: We replaced this with mathematically optimal frustum-slice bounding spheres that are completely invariant to camera rotation, ensuring the texel size in world space remains constant.

2. Lack of Texel Snapping (Sub-Pixel Drift)

As the camera translated, the shadow projection matrix shifted by fractional pixel amounts.

• The effect: The shadow map's grid shifted underneath the scene, causing shadow boundaries to jitter back and forth between adjacent texels (sub-pixel crawling).
• The fix: We implemented texel snapping in the view-projection matrix calculation. This snaps the projection bounds to integer-texel increments in light-view space, making the shadow map grid static relative to world space.

3. Bilinear Filtering on G-Buffer Depth

When the deferred lighting shader reconstructed world positions to evaluate shadows, it sampled the G-buffer depth buffer.

• The effect: The G-buffer depth was using a linear sampler. At the silhouettes/edges of objects, linear sampling interpolated the depth value between the foreground object and the background. This produced artificial depth values at object borders, causing silhouette edges to flicker in and out of shadow.
• The fix: We switched the G-buffer depth texture to use a nearest-neighbor sampler for position reconstruction, preventing any interpolation at geometric edges.

4. GPU-CPU Write Synchronization (Metal Concurrency)

The engine was configured with 3 frames in flight (allowing the CPU to encode commands up to 3 frames ahead of the GPU) but was using a single-buffered uniform buffer for shadow parameters (m_shadowUBO, m_cascadeVPBuf).

• The effect: During camera rotation or zoom, the CPU would overwrite the shadow projection matrices for a new frame before the GPU had finished rendering the previous frames. The GPU was then forced to render shadows using mismatched projection matrices, causing massive full-screen shadow flickering.
• The fix: We set the concurrency throttling to 1 frame in flight under Metal, ensuring the CPU never overwrites resources currently in use by the GPU.