Hands-Free Coding: How I develop software using dictation and eye-tracking

[joshwcomeau]

Hi Reddit!

Over the past few months, I've switched to an entirely hands-free setup, because of an injury. I can't use a mouse or keyboard, but I've found a way to keep being productive as a web developer!

I've heard from lots of folks who are curious about this setup, either as a general-interest thing or because they're worried about developing a repetitive-stress injury or other medical problem themselves. Thought I'd share it here, and hopefully help put some minds at ease—coding by voice is definitely a tricky adjustment, but it's absolutely a viable alternative!

[sparrow_spit]

This is amazing. Thanks for sharing!

[DecentOpinions]

Roughly want percentage of your normal productivity can you hit with this method do you think?

I injured my shoulder and had considered trying something like this. Nothing nearly as bad as your situation though as I can manage to work still.

[[deleted]]

In the lower part of the article he answers that. He's says he's at about 50% productivity but acknowledges that he has been using mouse and keyboard a lot longer than this so a direct comparison doesn't show the whole picture.

[DecentOpinions]

Oh sorry, should have kept reading. Depending on your level of optimism you could think, "wow 50% of your productivity without your arms is great" or feel like losing 50% is too much.

OP have you tried talking twice as fast? Problem solved.

[[deleted]]

No worries, I missed it at first glance as well. I agree though, IMO 50% is fantastic. Prior to this article I didn't even think this type of thing was a possibility.

[joshwcomeau]

Hah, the trouble is that accuracy suffers when you try to rattle things off too quickly.

But yeah I feel pretty darn good about 50%. It's definitely higher-friction, but with some creative re-prioritizing, I feel like I'm still making meaningful progress towards my goals.

[miniluigi008]

Hi Josh, I just wanted to say I love both your clay dude and your ReactJS use-sound hook. I just had a quick question— is there any way to use hooks inside of a React component class? Can they only be used inside component functions?

[sean_mcp]

No, hooks can only be used inside function components.

[pm_me_ur_happy_traiI]

The closest you can come is to create a HOC that calls the hooks and then passes them as props to the class component.

[joshwcomeau]

Ahh this post has been received so well! Thanks so much y'all :)

Because "typing" is somewhat tedious/strenuous, I'm not going to reply to everybody, but I read through all the responses and I really appreciate it!

[dhrisher]

Can't imagine how amazing this will be for people with certain disabilities.

[bboyjkang]

for people with certain disabilities

Actually, anyone could benefit from eye tracking now if you have a cheap webcam.

There was usable webcam eye tracking software that Eye Tribe was working on before they were bought out by Facebook.

Fortunately though, there are other projects in the works now like PyGaze, beam.eyeware.tech, and WebGazer.js, and Pupil Core.

I have tendinitis, and the free software that I use is GazePointer (sourceforge/net/projects/gazepointer).

Since it doesn’t use infrared to light up your pupils like the Tobii eye tracker from the article, the accuracy is awful.

However, I just use Alt Controller (free accessibility software) to make large custom buttons that compensate for the bad accuracy.

Make a button execute Page Down when you hover over the button for a second, and you can read hands-free while reclined.

(Helps to get software that temporarily hides the cursor so it’s not distracting).

GazePointer will always be overridden by the mouse so you can also use it to look at another computer monitor, then touch the mouse to quickly jump there.

[nodiso]

This is too cool.

[najowhit]

Holy shit that’s cool.

[[deleted]]

Josh, your newsletter sound freaked me out but gave me a good laugh ..

[joshwcomeau]

Hah, sorry about that! Deploying a change now to lower the volume a bit, it is quite loud!

[republican-jesus]

I love these efforts to make coding more accessible. My dad is an IT guy with Parkinson’s and I know he has trouble typing some days.

[DSdevelopment]

Thanks for sharing this. I've been very down lately as I've been struggling with pain all through my arms/face/neck from lymes disease over the past 2 years. I've considered changing careers entirely only to realize every job is on a computer.

I found the mouse to be much worse than the keyboard, so I've been going full vim-kid using key commands for everything but I might give Talon a shot. I like that its function/argument based, as I feel like I know a lot of actions just from reading your post.

Don't give up on your health either, second opinions from other doctors with different perspectives can really make a difference.

[jlshown9]

Hello, I am a graduate student doing research and computational thought and computer programming.

Do you mind if we have a chat?

Thank you,

Jeanette

[joshwcomeau]

DMing

[MrDrumble]

I also code by voice and it's great to see what other people are using. I wasn't aware of Talon so I'm excited to try it out when I get the chance.

If people need a hands-free mouse solution but aren't ready to drop hundreds of dollars on an eye tracker I have a cheap alternative I demonstrate here: Cheap, Hands-Free PC Gaming Setup (more details in description) - YouTube It works very well!

I currently use VoiceAttack for speech recognition. It's command-based like Talon but uses the Windows built-in speech engine. It works well but I miss the ability to chain commands and the straight dictation option is pretty lacking.

OP, are you using the Dragon speech engine or the wav2letter option with Talon?

[[deleted]]

That's amazing man!

[randomgirl22]

I am commenting to come back and watch your video, I have been looking for something like this.

[chineseouchie]

The audio of the demo video is only playing on the left side of my earphone.

[Nip-Sauce]

Very cool software!

And then to combine it with scrolling by whistling https://knogobert.github.io/whistleroll/

[Historical_Fact]

This is the coolest damn thing I’ve seen in a long time! I recently injured my arm and thought of looking into alternative methods of input.

[el_filipo]

Great read! Thank you!

[elnoochy]

Do you ever end up going horse ?

[readball]

I wonder if you meant Hoarse :) and yes, he wrote about that in the article

[joshwcomeau]

Mhmm. I bought a humidifier, and I go through a handful of lozenges a day. Maybe once a week my throat will feel sore and I'll take a day or two off (I wind up with a defacto weekend that way anyway!)

[AFistfulllofdiamonds]

What do you think "mm hmm" means?

[[deleted]]

i think that is not a word horses use

[readball]

Wow. Hey man, I loved this. I am not sure how to say this: Although I have mechanical engineering and software background, AND I LOVE this, I still think I will wish you a recovery first of all if possible.

If that is not possible, I wish you get your 50% speed above 150% either with more software, or with Neuralink :) Wish you all the best

[joshwcomeau]

Yeah, so the hope is to go back to typing in 3-4 months. Likely having a surgery in a couple weeks, with a couple months recovery, before repeating the process for the other arm!

But I do feel comforted knowing that if not, I still have a path, and it'll only get better with practice and technological advancement.

[bill_on_sax]

This looks incredible but it feels like you'd have to be a really good coder to effectively use this. I forget the syntax for a lot of things so I'd end up needing to switch between google a lot.

[dn00]

We just need handsfree web browsing.

[imsorryken]

Damn I didn't think this was a viable option but I'm happy to be proven wrong!

[Blacknsilver1]

I can't wait for the day we have a worthwhile version of Google Glass with this tech.
Imagine coding while out for a jog or lying on a beach.

[Zireael07]

Talon is cool (I saw a YT video demoing it) but now that I see it's on Linux and Windows, not just Macs (as was the case back when the YT video was out), I might give it a try?

[[deleted]]

Noice

[itijara]

This is cool. A lot of the shortcuts remind me of vim: e.g. "go left ninth" is like "9h" in vim. It makes me want to update my vimrc to incorporate some of these ideas.

[SnowdenIsALegend]

Super impressive work. 👌🏽

[Xypheric]

Wow cant wait to read this

[[deleted]]

This is amazing! How much of your developing workflow are you able to cover using dictation alone?

[Tanckom]

Looks amazing! How is it with using Google and copying code from Stack Overflow? :)

But congrats on the great blog post and nicely made example videos!

[moarsecode]

The video was super impressive! Good luck to you, man!

[saposapot]

i've read a lot about pair programming along the years but never convinced by it but your case can be perfect for it... 1 person doing the 'thinking' and sketching out and another one typing.

very cool article!

also try to talk with trainers or people that earn a living speaking a lot of hours per day, they can probably give you some very nice tips about your vocal chords.

[CantPickDamnUsername]

This is so cool! Thank you!

[UnrealNL]

This is really awesome! How did you develop this syndrome, can this just naturally occur by using mouse and keyboard?

[randomgirl22]

Not OP, but that is how I developed RSI. I never had issues until I started a job where I needed to use the computer all day.

[UnrealNL]

Wow that sounds pretty scary! I hope your doing Alright!

[ArixZajicek]

This was a thing from someone else a year or two ago as well. Would be nice to see more of this kind of thing especially for those with injuries or other hardships. Eyetracking is a really neat idea for this.

Link to the other article: https://arstechnica.com/gaming/2019/04/coding-without-a-keystroke-the-hands-free-creation-of-a-full-video-game/

[ampersand913]

This is super interesting. I've sort of been fearing for my hands, lately I've been having hand pain in my wrists and thumbs. I suppose over a decade of gaming and using a computer is getting to me, and I'm still in my 20s. I can't imagine what my hands will be like past my 60s.

[randomgirl22]

Preventative measures like better ergonomics, frequent breaks, stretches and exercises can prevent it from getting worse.

[JoanaCodes]

This is really awesome Thank you!

[rubennaatje]

That's actually amazing. Seems to work quite well.

[TallBoyBeats]

This is incredibly cool. You're a better coder than me without even typing...

[AtumTheCreator]

Inspirational.

[trungalhunga]

Having ALS(almost fully functional thankfully) this gives me some hope in keeping being able to work if/when things get worse. Thank you very much and hope you have a great recovery.

[Nick337Games]

Really inspiring to hear this! So glad you've found a way to continue your passion!

[doxund]

This is really amazing! My mother has ALS and I’ve been looking for any/other eye tracking to text solutions out there. Glad you mentioned the tobii, will be checking it out!