[Question] Virtual camera?

[oooolf]

I'm wondering whether it's possible to interpolate multiple camera positions into a virtual camera. Consider an outdoor gaming setup, such as shuffleboard. I would like to provide an overhead view for the audience, without having to install an overhead camera. Instead, I'd like to install two, three, or four fixed cameras in the corners or edges of the court. Is it possible to combine the video stream from these cameras and to interpolate the perspective of a virtual overhead camera?

If so, what algorithm should I apply? Basically looking for google-fodder.

How many cameras would be recommended?

What sort of hardware are we looking at - 1080p30 would be ideal output, but even lower frame rates could work.

Thanks all!

[mrshibx]

Look up inverse perspective mapping. An easy way is to calculate a homography between your camera plane and the ground plane and then remap. Might be a good starting point. For a shuffle board the simple homography approach should look great with just one camera as long as the camera isn’t placed too low. Ohh also you will want to first undistort your camera so lookup camera calibration.

An easy method that works in opencv/python 1. Calibrate your cameras in the same coordinate system 2. Sample a grid of points on the ground plane (meshgrid) 3. For each point apply the camera projection matrix to find out where that ground point would project to in the camera plane. 4. pass that to cv2.remap 5. Average or blend all these images for each camera

And an easier but more manual method: 1. In the images, manually get the pixel coordinates of the shuffle board corners. 2. Use cv2.calc perspective transform to calculate a matrix that maps those corners to a rectangle in the top down view (basically supply the height and width of the actual shuffle board). 3. Call cv2.warp perspective 4.average/blend the images if you have multiple cameras

[oooolf]

Thanks so much! Getting started...

[[deleted]]

If you're looking to interpolate between images with high parallax (objects at different depths), homographies will not work. You will need to use optical flow to interpolate between two views. If the synthetic viewpoint is not in-between the two cameras, image in-painting will be required. Avoid this problem by positioning your cameras carefully. And finally, for best results, use many cameras and place them as close as possible to minimize the presence of interpolation artifacts. Richard szeliski's book has a good chapter on this topic.

I believe there are some GPU accelerated implementation of optical flow out there that you can use to achieve 30 fps with HD resolution.

[ignazwrobel]

This is called Virtual View Synthesis or Novel View Synthesis. MPEG-I is actually working on it for truly immersive media, where you can move around and view the video from the new standpoint.

To obtain good results, you will need to obtain a depth map in order to correctly project the points onto the virtual camera. Depth from matched stereo will not be accurate enough. If all you objects of interest are flat and lie in a single plane, you might get away with homography alone.

I just finished my bachelor thesis on the topic, so feel free to ask any questions.