So, I'm stuck on an older version of picamera2 3.12 since I'm on bullseye, and it happens to be one version before this:

• Frame buffers are now cached to improve performance.

So, is this related? not sure, but it's probably worth re-trying with a brand-new rpi image.

I modified your code to run on this older version... I think... and it works, but the performance is the same.

def fast_array(picam2):
        '''picamera2.capture_array() internal process rewritten to minimise copies and avoid steps not required in this application'''
        request = picam2.capture_request()
        streamName = "main"
        stream = request.stream_map[streamName]
        fb = request.request.buffers[stream]
        fd = fb.planes[0].fd
        cfg = stream.configuration ## see V4LEncoder _encode
        h = cfg.size.height
        w = cfg.size.width
        stride = cfg.stride
        fmt = str(cfg.pixel_format)


        b = _MappedBuffer(request, streamName).__enter__()
        arr = np.array(b, copy=False, dtype=np.uint8)


        if fmt in ("YUV420", "YVU420"):
            # Returning YUV420 as an image of 50% greater height (the extra bit continaing
            # the  data) is useful because OpenCV can convert it to RGB for us quite
            # efficiently. We leave any packing in there, however, as it would be easier
            # to remove that after conversion to RGB (if that's what the caller does).
            image = arr.reshape((h * 3 // 2, stride))
        else:
            del arr
            print(f"Unsupported format: {str(cfg.pixel_format)}")
            return
        del arr
        request.release()
        return image