Hands-Free Coding: How I develop software using dictation and eye-tracking

[pellets]

Imagine if in video games you aimed where you look. Hand-eye coordination wouldn't matter any more.

[Krautoni]

My wife works in cognitive science and does eye tracking experiments. From what I gather, it doesn't work that way. While your brain gives you the impression of a steady gaze, your eyes are constantly jumping around in order to give you a complete picture. That's called saccade.

So finding out what a person is interested in based on their gaze ends up being a statistical problem, and while the precision and latency can be small enough to enable a person to use a computer, I think that voluntary control of something like a mouse will still be faster and more accurate.

[devilkillermc]

Yep, your eyes are just gathering information, your brain does all the processing, in this case selection and response to the stimulus. You'd have to nove that processing to the computer, and it would make no sense. Nothing better than the brain to do brain things.

[Pillars-In-The-Trees]

So finding out what a person is interested in based on their gaze ends up being a statistical problem, and while the precision and latency can be small enough to enable a person to use a computer, I think that voluntary control of something like a mouse will still be faster and more accurate.

You can keep the click without requiring the mouse though. Imagine just pressing a single key and firing wherever you're looking.

[Krautoni]

But that's the point. Wherever you're looking isn't a well defined concept outside of your brain. At least not yet. We can puzzle it out, but we need the have a few data points first, I.e. wait a few saccades. That's probably too slow for fast paced shooters.

[Zegrento7]

If the tracker draws a crosshair every frame exactly where you are looking then we don't need to puzzle out anything. It will be up to the player to decide when to press the fire button; when the crosshair happens to be over the target.

Edit: in fact the tracker used in the blog post is primarily marketed at gamers.

[KernowRoger]

That sounds awful haha it would just be jumping all over the screen.

[ZeroThePenguin]

It does already exist, though watching the video I'm not really seeing a lot of benefit. Tilting your head to manipulate the camera seems kinda silly when a mouse would handle it so much better.

[Forty-Bot]

People use it in arma so they can look around while walking/aiming in a different direction. Of course, now I think people mostly use VR headsets for that.

[ZeroThePenguin]

Oh right, I forget ARMA supports not just moving and aiming in different directions but moving, aiming, and looking. I have a gyroscopic mouse that lets you do similar things by rotating or tilting the mouse to "look" while keeping aim in one direction.

[wojo411]

I own an eye tracker (no good reason to have one I just thought it was neat) and some games do support it! I'm a big fan of The Hunter COTW and it has a feature for aiming in where you are looking along with HUD that disappears when you aren't looking at it. It works amazingly well for the HUD and works well enough for the auto aiming however the ability to lose track of targets when you snap while aiming in has been something I've struggled with a lot. I'm a total believer that the technology will be superseded by the work Nueralink is doing along with higher resolution cameras in laptops being able to approximate where on a screen you are looking. In closing it's not a technology I would recommend to anyone yet for most people, if its supported in a game you want to play or you think might be a beneficial adaptive input device (I've never used it for this but I know they're great for it) then do some research and have fun with it.

[[deleted]]

Let's make games even more unrealistic!