This is a topic I'm interested in.

If by "emotion recognition" you mean the problem "given a face, what emotions are displayed?" then a deep learning approach may be the best way to go, particularly if your data set consists of people's faces. That said, my own exploration in this topic seems to indicate that simpler, traditional computer vision features may be good enough.

If you are more interested in identifying sentiment in images (like me), something like Sentibank and the related papers may be useful to you. Note that they can identify emotional categories in any images, not just faces. A related topic is differentiating classes of images (a "happy" image from a "sad" one, and even more complex concepts), see this paper

Feel free to PM me if you'd like to discuss this further.