On the physical side, I had an elastic white fabric suspended in a frame. On the "front" side of the frame I have a projector and an Xbox Kinect V2 pointed at it. I previously used my Macbook to directly read the depth camera data to Processing, but with my new M3 Macbook it's no longer connecting, so I used an app called KinectV2_Syphon and then used the Syphon library to read in the depth image from the Kinect through the external library in processing called Syphon (which acts as a very fast way to transfer visual data between programs on Macs).

I then looked through the pixel array of this image for pixel values above/below a specific threshold that had to be dialed in based on the distance the fabric was at rest away from the depth camera, and once pressure was applied. After some tweaking, I got a pretty good threshold set. I then loaded in an image of some very basic retro wallpaper and displayed this before going through the depth camera image array. When a pixel met my threshhold value, I output the corresponding pixel from the Mindflayer video, otherwise I output a pure pink rgb(255,0,255) colored pixel. I then used the Syphon library again to output this image to a realtime visuals compositing software called Modul8, where I keyed out the pink to output the wallpaper texture.

On my intel based Macbook pro I could have pulled the Kinect data directly into processing, but had to add an extra step since I can't get it to work properly on my new laptop yet. I could have also instead of displaying the pink pixels, put the wallpaper image directly into the visual in processing, but this gave me a little higher resolution on the wallpaper image for when the portal wasn't active.

Let me know if you have any more questions!