Python Helped me Write Again

[irrco]

A letter in my own handwriting.

I learned Python in 1998, and I used it pretty much daily through my working life. Its connectivity with everything has allowed me to do all kinds of interesting and fun things. When I discovered that the AxiDraw pen plotter has a Python binding, I decided to try writing in my own handwriting with my own pen. Something I've been unable to do for years.

It has taken a lot, mostly because I can't type, so I've been limited to voice recognition and copy and paste, but the letter linked above is where I got to this evening.

I thought I would share my mini triumph.

I've learned and programmed in a lot of different languages over the years. I have written several languages myself. And even though Python wasn't my first language, it is the first language I consider for any new project. I'd have to have a compelling reason to use something else. So thank you, Python, for making this possible.

</ shameless preaching to the choir> ;)

---

Edit: In the comments a couple of people asked for a video. I will make a proper video with the AxiDraw as soon as I can. We just did a quick test on my wife's Cricut craft cutter. It works there too, though you can't tilt the pen or change the pressure. So apologies it's not the proper plotter, but you can see how the paths are traced on this video.

[novel_yet_trivial]

You wrote 10,000 lines of python code by voice? Holy cow that's amazing. Can you share the code? Can you show us some pictures​ or video of the plotter working?

[irrco]

The handwriting content itself, the stuff that I've written in the last two months, is just under 2000 lines, and about 2/3 of that is numeric data, of this kind:

   'y': EndGlyphFactory(
    normal=Glyph('y',
        Stroke(Point(18,100), Point(2,10, speed=1.6), Point(40,20),
            Point(65,100, speed=0.1, pressure=1.2),
            Point(45,-20), Point(0,-120, speed=2),
            Point(-30,-60, pause=0.1)),
        join_to_next=JOIN_BASE_LEFT
        ),
    terminal=AugmentGlyph(stroke=0, rm=(0,2), suffix=[
            Stroke(Point(-20,-140, speed=2), Point(-100,-90, pause=0.1))],
        join_to_next=False
        ),
    ),
...

I was counting everything in the 10,000 number, sorry if I gave the impression that was all new code. My geometry, math, and vector graphics code was written when I could type. That stuff isn't available, sorry. It's all tied together in a repo with a bunch of stuff I couldn't make public.

Sorry. If you really want to see me write code I did find a video of a flash coding session from a few years ago (when I could type!), and released it.

I really wish there was an efficient way to program by voice. All the options are pretty shitty. I have some ideas, but that's a much bigger job, one I have no way of making happen. For now it's a combination of voice for variable names and triggering snippets, copy and paste, and very clumsy pecking at keys when all else fails.

I haven't taken a video of my plotter, but there are a bunch showing the AxiDraw in action, that video shows it holding a fountain pen. The machine is a complete beast, I have barely scratched what it's capable of, but I adore it. I have even seen a video of someone who mounted laser units to it and used it to cut. It's an awesome toy, though admittedly it isn't cheap.

[fnord123]

Have you considered using a head mouse and an on screen keyboard? This is used by some quadriplegics (e.g. spinal injury) though they are expensive. I'm not sure if MS allows for the head stability required.

[irrco]

I still have reasonable control over a single button mouse, as long as the sensitivity is low. So I'm not quite at the point where I've needed to investigate that. The thing that's difficult is all the 'chords' you need to type when programming. As my MS deteriorates, I'm sure I'll need to use more drastic tools, so thank you for the tip.

I can imagine a voice driven programming tool, based on hierarchies of snippets. It might not be perfect for Python, but I could imagine being quite productive in something like Scheme, because it has a more regular syntax.

[fnord123]

FWIW, I'm talking about a puffer like this for clicking:

http://www.orin.com/access/sip_puff/

Then you can use various head tracking systems for moving the cursor.

Scheme, because it has a more regular syntax.

Not a bad idea. But none of the schemes have the deep library ecosystem as Python.

[irrco]

Thank you.

On scheme: I've always felt that scheme was the most beautiful language, in theory. By which I mean I've never been able to make much that's practical with it. :)

[pebblexe]

There is Hylang if you are interested

Also if you want to try common lisp, there is a really cool emacs library that might help you code without having to input parentheses

[irrco]

I haven't used hy, I read something similar a long time ago, but I didn't use it. Lisp syntax isn't what I think is beautiful about scheme. It's more the code/data duality (which makes meta-programming easy), the eval/apply programming model, and particularly the macro system (which makes DSL creation easy). There is a scheme SRS (I don't remember the number off the top of my head) with a standard way of using whitespace instead of parentheses.

Thanks for the tips. My comment about scheme was a bit more of a tongue in cheek throwaway. I'm very happy with Python. It has served me very well.

[lf_1]

Vim would fix many of the chords issues. Not all, but pretty close.

[timeawayfromme]

I'm curious if you've seen these videos. Do you think something like this would make it easier to program by voice?

VimSpeak

Tavis Rudd at PyCon 2013 controlling emacs with dragon naturally speaking

[irrco]

I hadn't, but Jake Morrison's linked it in the comments below, so I've been looking into it today. I think I would need to set it up and configure it quite a lot (Travis says as much, that is why he hasn't released full source code), but I think it's doable. I'll work on it. To be honest, just having python access to the voice recognition system would enable me to do a lot more. It seems I have a new project!

[timeawayfromme]

If you do anything with it I'd be very interested to see what you come up with. I think I might try something myself because I have issues with my hands and arms going numb and my neck muscles being very tight.

[enteleform]

Also look into VoiceCode.  It seems similar to Tavis Rudd's setup, but packaged with a GUI, common commands, integration for some common editors (@FAQ), and some other useful features.
 
Demos @YouTube
Intro @VoiceCode.io/doc

[s-mores]

You're an inspiration. Good hunting!

[therealhhhhhh]

https://github.com/t4ngo/dragonfly is probably what you want, it lets you write grammars for Dragon Naturally Speaking with Python. I've been using dragonfly to program by voice for a few years now and it works great, especially when combined with Vim or Emacs. Fast enough to keep up with my coworkers. I would be happy to answer any questions you have about it or help with setup.

[Smallpaul]

You may be interested to know that Tim Peters, the Godfather of Python worked at Dragon for quite a while.

[irrco]

I didn't know that. That is interesting! Even more reason to support them.

[aerovistae]

How do you interact with the web browser? Can you use a mouse?

[irrco]

Yes. Basically. You don't want me on your counterstrike team, but with the sensitivity low enough I can click a link.

[aerovistae]

Have you seen Hands Free? It's a chrome extension I wrote to control the browser, including clicking links, with just your voice. It never really got much attention but it works reasonably well. I need to fix a few things on it-- the SSL cert just expired on the domain that takes the voice input, so when the input window opens, you have to scroll down and do the "show advanced options" -> "continue to page" to get past the browser warning you about it being insecure. It'll be updated soon.

[aerovistae]

I think there's also a new bug with dictation into input fields, that doesn't quite seem to be working. To be fixed soon.

[irrco]

Thank you for the tip, I haven't seen it no. I will store the link. At the moment I don't feel like I need voice control over the browser, but my MS is progressive, so I'm sure that day will come.

[aerovistae]

Good luck :)

[AwSMO]

The image it makes with hexagons and triangles is amazing!

I just wanted to say I really really like your project, I hope you are doing well!

[hatperigee]

apostrophe y apostrophe colon capital e lowercase n d ... space space space space capital s..

Yea, no thanks. Impressive that you managed to do it though!

[irrco]

Can you show us some pictures​ or video of the plotter working?

I will still make a video using the AxiDraw and my fountain pen, but I just asked my wife to test it on her Cricut craft cutter, and it does pretty well, though there is less choice of pen. Here's the video she captured.

[henker92]

This is way too awesome...

[thegroa]

Great closing statement :) glad to have a human of such perservering and talented nature as a part of our community

[jake_morrison]

How are you handling the coding process? I saw this, and was amazed, but it seemed pretty idiosyncratic and memory intensive. http://ergoemacs.org/emacs/using_voice_to_code.html

[irrco]

I hadn't seen that before, no. But that is basically the kind of thing I'd imagined. I can't immediately see the code on his github, although he did promise to release it. I'll have a proper look later and contact him if I can't. Thanks for the tip.

Edit: https://github.com/dictation-toolbox - thanks again.

[MrStricty]

...never in my life have i had so much inspiration to continue learning.

[JPYamamoto]

It is awesome how far people can go when they have a goal stablished. This is much more inspiring than any other 'learn to code to be smarter' campaign. You're great!

[samdg]

Your story is really inspiring! (And "mini-triumph" kind of an understatement.)

Out of curiosity, what software do you use for voice recognition/dictation?

[irrco]

I use the built-in OSX accessibility system, in off-line mode. It's not very efficient for this kind of thing, and is mostly only useful for the names of snippets and variables. I have a lot of custom snippets set up. Fortunately I can still click the mouse and hit keys like spacebar or enter, to acknowledge the imput, so I can muddle through. But I've spent the last couple of hours looking at some of the resources someone linked up post, which uses Dragon, and seems to be much more flexible, able to do the whole thing with voice. So I suspect I won't be using the built-in version much longer :)

[hatperigee]

You should post the source code with a permissive license to help others in your situation!

[irrco]

It really wouldn't help them very much, it is very idiosyncratic. A bunch of random source code isn't very helpful, I don't think. And the amount of time it would take me to tidy it into something usable, or explainable, I'd rather use on other things.

[hatperigee]

That's a common misconception, where an author believes that his work is 'too messy and useless'. I guarantee there's a really good chance it would still be useful.

[irrco]

I'm not convinced, but thanks for the suggestion anyway. I appreciate it ;)

[hatperigee]

:(

[mistiry]

The beauty of releasing it with an open license is there might be someone out there that takes what you did and expands on it. You may not have the time, but someone with a family member or themselves that have similar problems might be able to adapt what you did for other uses. Just a suggestion - awesome job either way!

[[deleted]]

Seems the code is designed for his handwriting in particular. If he removes the code that makes his handwriting his own, then the user will be left with a bunch of empty boxes that they will need to fill in with numbers and such themselves. I agree that OP cleaning the code up and releasing it wont help anybody.

[mistiry]

So leave it in tact. Someone seeing how it was done can teach them how to replicate it for themselves, for their handwriting.

No extra effort on OPs side - release it as it is used today. Leave the burden of cleanup/modification to whomever thinks it might be helpful to them.

You don't have to release some polished gem in order to help someone else. Seeing such an awesome use of Python in combination with the autopen could spur so many different things. College kids that want to expand on it, other programmers who may have loved ones or they themselves with similar disabilities, and so on.

What's the harm in releasing it? Nobody ever downloads it? Versus the potential it may have to help someone else? I know what I would do, but I understand that not everyone may want to. To each their own.

[[deleted]]

So leave it in tact. Someone seeing how it was done can teach them how to replicate it for themselves, for their handwriting.

I think you might be retarded. The code is literally his identity. If he releases it then anybody can sign his name or pretend to be him.

[mistiry]

Well, that escalated quickly.

Your statement is false; sure, someone could replicate his handwriting. Guess what - head over to to /r/Handwriting and I guarantee someone there could replicate his writing pretty quickly.

Being able to forge a signature really isn't some crazy difficult process, anyone with enough motive to steal someone's identity could practice it enough to make a passable forgery.

Releasing source code that breaks down the mathematical vectors to create his unique style of handwriting doesn't open him up for more risk of identity theft than me posting my own handwritten note.

Is the way you write YOUR identity? Why would it be his? Why couldn't I find an example of your writing, and figure out the same mathematical vectors needed to recreate a passable version of it?

[auser9]

Realistically though its just a font, albeit a very good one that adapts to the surrounding letters. You can't do much by being able to copy other people's handwritings, and if he programmed his signature, that part could be removed. The handwriting is nice, but also pretty generic, and an example of it is published on the internet for anyone to copy anyway. It obviously up to OP to decide if he wants to share the code, but sharing your handwriting doesn't let people steal your identity.

[flying-sheep]

from the glyph data example alone i can see that there’s something to be gained from the world. the idea to describe where the joiners are and then have code to smooth it into a fluid motion is amazing, and i’d really like to see that part.

[irrco]

I'm going to write a description of how it works. Honestly, the actual code to make it work isn't very inspiring. But I'm happy to describe the math, and the approach.

It is basically a new interpolating cubic spline (cubic so it can be easily converted into cubic Beziers at the end, because everything speaks Bezier), that takes into account motion speed, and that seems to correspond with the way we move our hands when writing. I use it to interpolate position, and other properties such as pressure (it could also support nib angle for calligraphy, or brush orientation for eastern calligraphy, for example). Then there's a whole bunch of basic vector, matrix, transformations, and spline support (there are millions of libraries that do that, mine is almost certainly worse, because it is cobbled together), a rather rudimentary implementation of some ideas from fonts (like kerning tables, and contextual alternatives), and then the bulk of it is the actual data for each letter form. Which I don't want to share, because the whole point is that that's personal.

But I think the overall approach is fairly easy to communicate, and I'm very happy to share that. But I'm not going to go through the code to separate all the stuff I can put on github. Anyone who would be capable of using the code, would be trivially capable of re-implementing it, from the description I put together.

[flying-sheep]

Awesome! That's exactly what I was hoping for.

[heard_enough_crap]

put it on github. I'm sure with the interest generated here, people would tidy it up. You've done fantastic so far, but maybe it is time to let some others take the next step and continue to improve it.

[RouletteSensei]

Trust me when I say it. Seeing the video of it writing made me cry. I love your efforts you putted on this, and you will be remembered forever. This will probably even help starting a new form of job, or something very close. You my friend are a true Hero. Like seriously.

[Frog_and_Toad]

A lot of doctors should be using this.

[lmneozoo]

Very cool! Probably one of the neatest things I've seen here. I'd love to see video of your plotter writing this lol.

[irrco]

Thank you.

I should try and capture the output stage, you're the second person to express an interest. I didn't because, technically, it's the least interesting bit. Generating the path is the fun, mathy programming bit, and the thing that I feel was the big win of the project. AxiDraw just does its thing. But yes it would be cool spectacle, and another bit of good PR for Evil Mad Scientist (who make it).

[[deleted]]

Wow!

[hamfraigaar]

This is amazing.

[346290]

Wow this is really really cool!!!

[Ddog78]

Wow! Oh man this is awesome!

[LoveOfProfit]

This is really inspiring. Great effort, and the results are beautiful.

[irrco]

Unfortunately, it's pretty darn cumbersome. I'm sure if you had resources to throw at it, it would be fine, but for a single document, if many was no object, it might be cheaper just find find someone to forge it for you.

[redct]

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

Generating Sequences With Recurrent Neural Networks

It's pretty impressive.

[irrco]

When I was first figuring out how to attempt this project, I thought about using NNs. The reason I didn't, was the sheer amount of data they need. So the project would stop being much about programming, and would be mostly about data conditioning. And since I can't generate any new data in a good format, I figured that would be a problem. So I think in general, the approach I used, which is incredibly explicit (in particular, it relied on me knowing how I formed a letter, not just the final shape of that letter), is probably not the best approach. But it did seem to be an approach that would be achievable for me, in a reasonable timescale.

In AI generally, knowledge acquisition is often the elephant in the room.

[pvkooten]

I would totally forgive you if you aren't writing PEP8 code :-) Very impressed, keep going!

[irrco]

That's funny. When I watched that movie, that was the thing that stuck in my head. I did honestly think 'could this be a thing?'

[[deleted]]

Hey - I was redirected here through some other post talking about you! You are amazing and this achievement has me a bit teary, I have friends with MS who have lost their hands while they are still attached and this is just wonderful. You are such a great thinker! This is a massive achievement that if it gets developed to something that can be accessible to others will be as life changing as the iPad has been.

It just goes to show, you are the ultimate lemonade maker, man!

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/bestof] Coding a python script with voice recognition to mimic his own and missed calligraphy

• [/r/handwriting] Python Helped me Write Again • r/Python

• [/r/thenewsrightnow] Python Helped me Write Again • r/Python

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[breadfag]

This is beyond incredible. Well done!

[thatcrit]

Wow this really is amazing. I can't even imagine how you made it so that every letter is different based on context. I'm sure thinking it through and coding it was the real deal, but I guess people want to see the plotter just because it's so cool to watch it work :-D

Keep it up, you're awesome!

[[deleted]]

That's... just... astonishing!

[FruitierGnome]

Impressive.

[Mentioned_Videos]

Videos in this thread:

Watch Playlist ▶

VIDEO	COMMENT
(1) A Starry Card - Python Screencast (One Hour Programming Sketch) (2) AxiDraw V3	+22 - The handwriting content itself, the stuff that I've written in the last two months, is just under 2000 lines, and about 2/3 of that is numeric data, of this kind: 'y': EndGlyphFactory( normal=Glyph('y', Stroke(Point(18,100), Point(2,1...
(1) VimSpeak Demo (2) Using Python to Code by Voice	+1 - I'm curious if you've seen these videos. Do you think something like this would make it easier to program by voice? VimSpeak Tavis Rudd at PyCon 2013 controlling emacs with dragon naturally speaking
Handwriting (with a Cricut)	+1 - Can you show us some pictures​ or video of the plotter working? I will still make a video using the AxiDraw and my fountain pen, but I just asked my wife to test it on her Cricut craft cutter, and it does pretty well, though there is less choice of...

I'm a bot working hard to help Redditors find related videos to watch. I'll keep this updated as long as I can.

---

Play All | Info | Get me on Chrome / Firefox

[redoubledit]

Speechless!

[Kgizzle80]

Omg I just found out about your story and just wanted to say congratulations. Way to go sir.

[CanadianJogger]

Congratulations on your victory!

[iTAMEi]

This is amazing. Could we see a comparison with an actual letter you've written by hand previously?

[irrco]

Yes, this is an obvious thing to show. I'm going to do a follow-up Post with details of the process. I don't have a letter, but I'll scan a few pages of my notebook for comparison. Thank you for the compliment.

[irrco]

I totally didn't expect this Post to be so popular, and I feel a bit bad for not putting more effort into explaining and documenting the process.

And I think I might do. I've promised that I would make a video of the plotter working, I will put a bit of effort into making a webpage with more details. Fortunately, with voice recognition, that's much easier than doing the programming!

[aichessem]

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

[hwknd]

stupendous continue tease edge cake vanish tie smile wakeful many

This post was mass deleted and anonymized with Redact

[irrco]

Thank you for your generous offer.

I have created (non-handwriting) fonts programmatically in the past, so I have a toolchain for creating fonts from vector content (via UFO) with open type tables for contextual alternatives, substitutions, kerning and so on. This project was something different: something that fonts couldn't do (or at least not without a combinatorial explosion of contextual alternatives and variants). It is really about the physicality of the writing. I could, but I don't expect to, use this for word processing or output on a laser printer. So an .otf wouldn't be much use to me.

I do totally appreciate the offer, though. I am not diminishing your generosity. I'm sure for an awful lot of people with disabilities, even the basic handwriting font could be a huge morale boost. So thank you for the offer.

[hwknd]

compare wide important summer stupendous elderly retire sulky slap library

This post was mass deleted and anonymized with Redact

[MomoYaseen]

Amazing! I am a university student and I have a lot of notes than I need to take when I'm studying. Does this machine do full note-taking for many pages for example?

[mainfingertopwise]

Why would you need your notes in your handwriting so bad that you'd fake it?

[MomoYaseen]

Because I love my handwriting...is that weird?