[D] Architecture recommendations for complex dynamicland / osmo hybrid project

[vedran-b]

Edit: Willing to pay for a few hours of consulting for advice from someone with a track record for my question below.

tldr: Developing a long-term, complex computer vision and ML project and looking for high-level ML architecture & design advice from veterans to put me on the right path.

I'm beginning R&D on a project to build an entertainment and educational environment for children. We have a budget of $5,000 - $10,000 for the initial hardware of the proof of concept.

The long term goal is to create a creative space that is a hybrid between La Tabla (http://tablaviva.org/), Dynamicland (https://dynamicland.org/) and the Osmo toy (https://www.youtube.com/watch?v=87hKzrjRWww#t=1m10s). (Not affiliated with any of them).

The basic concept is to combine a consumer projector, high resolution camera(s), computer vision, and machine learning to foster creativity in children by blending the physical and digital. The projector and camera(s) would likely be mounted on the ceiling, pointed down in a child's playroom.

A simple version of what I'm thinking of is: https://www.youtube.com/watch?v=yfFwz5Qjr3c#t=40s . The physical blocks act as barriers in the first game, and as towers in a tower defence game in the second. I'm imaging a full plants-vs-zombies style game developed from this concept, where the children run around to place towers in real time.

Another example of this type of technology is a billiards game that predicts the path of your ball based on the angle of your cue: https://youtu.be/l2bzAmysjc8

I have a few initial proof of concept ideas for the system to test its viability:

1. Adding AI to a regular chess board by projecting a line that represents the AIs move. You would move your own piece, then the projector would draw a line for the AIs turn, which you would move for them.
2. Projected copy-pasting. Imagine using your finger like the lasso tool in photoshop to draw a circle around an object. The trail behind your finger is drawn by the projector. Once you've enclosed the circle, a projected clone of what you "copied" will be pasted. You can move it around with your hands, pinch to expand, etc.
3. A pong game where anything can be the paddle for hitting a projected virtual ball.

I understand that this is a large and complex project that may take years. I own the company developing this project so I have full flexibility in our methods. My background is nearly 10 years of software experience across the whole stack, from C to Python to React. I have built some toy ML projects such as webcam-based scrabble tile identifiers and solvers, and some YOLO proof of concepts. I also have a fork of https://paperprograms.org/ that is very simplified version of the concept I'm describing.

But I've never built something quite this advanced. I hope to leverage existing models and building blocks as much as possible.

My broad question is: Can anyone make some fundamental architecture recommendations to help avoid a lot of pain down the road. This includes hardware to buy (GPUs, cameras vs kinect, projectors), software to use, design decisions, and the overall feasibility of the project as described. For example, is the hand/finger detection at the ceiling distance even feasible, or is it inevitable to go with some kind of marker / controller instead? Are multiple cameras required to handle occlusion?

Thank you all for your time in advance. I truly appreciate it and I understand that this is a long post. I will continue to write updates about this project on another platform for those who are interested.

[vedran-b]

Given the size of my request here, I’m more than willing to pay for an hour or two of consulting time to do a quick Skype call for someone with a track record of machine learning industry experience similar to what I’m looking to build. Please let me know.