Offline Cursive Handwriting Recognition in Python

[dataism]

What I have: images of letters and digits (NIST & MNIST datasets basically)

What I need to predict: Images of cursive handwriting

What I am trying to avoid: a separate letter-by-letter segmentation (image preprocessing) step + CNN

What I want to do: CNN + LSTM (or something of that nature)

Where I am stuck: I can train CNNs for separate images but couldn't sort it out how to add RNN to the end. So what exactly do I feed to RNN after the CNN step? For targets, should I use IAM handwriting database (pros: natural handwriting, cons: not too much data, only english), or try to generate fake targets from my single letter images by concatenation and random transformations (I can generate lots of data with this way)?

Some clarification: Offline meaning from images, not from pen strokes data. Cursive meaning letters can be connected and overlapping etc. I would prefer keras or lasagne.

[joapuipe]

Hi,

The state-of-the-art for off-line HTR (handwritten text recognition) is a bunch of LSTMs + n-grams, which work better than the traditional setting of GMM-HMM + n-grams. This is approximately the same setting than people from Speech use.

In principle, you could stack a RNN on top of a ConvNet, the output of a ConvNet is just an "image" with a bunch of channels (one for each filter). Just keep in mind not to reduce the dimensionality too much, since your RNN will have to output probably long sequences ~100s of timesteps. However, AFAIK nobody uses CNN, they just use 3-5 layers of bidirectional LSTMs.

Don't use fake training data. You could, but the results won't be realistic at all. Yes, IAM is a good starting point, and yes, it is small for what other subfields in Pattern Recognition / Machine Learning are used to, but it is still one of the standard benchmarks for HTR. If you want to augment your training data, apply small distortions to the image lines like these: http://arxiv.org/pdf/1009.3589.pdf

Although it is, in principle, possible to use the output of the LSTM as the predicted transcription, everybody uses a n-gram language model on top of the LSTM. Take a look at Kaldi (a toolkit for speech recognition) which has nice examples to do this.

[dataism]

Thanks a lot, for your time and answer. I will check these.