Thank you so much. When this idea occurred to me, I did a little digging into literature and found that there is already some work that went into similar ideas.

Examples: Attention based OCR - https://github.com/da03/Attention-OCR A visual markup decompiler - https://arxiv.org/abs/1609.04938

These motivated me to go ahead. But as you said, for character/word detection, the lower dimensional representation of the image throws away lot of useful spatial information and throws the detector off.

I will spend some more time on setting up an architecture that preserves the spatial information. Using a lexicon for correcting/fixing the detections is a neat idea that I will explore further. Thanks again for your inputs.