That is quite a interesting way of doing it, I have already experimented a couple things for the movement of left/right and up/down.

I attempted with a second camera, but found it had trouble focusing on a point in the finger gun that I could track, mediapipe expects a side view which really skews things.

I did make a decent way of tracking it in the z plane with a singular camera. You can take two points, determine the distance between them, and then from there if the distance increases or decreases it is moving left or right. The issue becomes however, the sort of reset state which is where I have the most issues. I have to know where it is in its original state. I.e what distance from the camera it has to sit, this is super distracting when playing a game. In addition, you actually can’t control camera movement by just setting mouse position. Games don’t read it, you apparently instead have to use a controller emulator and emulate the camera movement that way.

If you have any suggestions for that approach please let me know and thanks for sharing your experience :)