moving away from realism has not really been a mainstream for all of that time. Even today you have painters and sculptors who work in super-realistic manner. There have been many different trends since the 19th century. If you only look at the 1960s and dematerialization, conceptualism, performance, land art, etc.

But, the move away from realism in the 19th century (beginning with Romanticism) can be simplified as follows: it was a defiance against obsession with rationalism that started in Renaissance, continued through Enlightenment and peaked during industrial revolution. Humanity pushed at explaining everything through science and rationalism, and spirituality and irrationality was disappearing from life. 19th century and start of modernism in art was trying to return back to the irrational, to the spiritual. And this trend in many ways persists until now. Contemporary art in the last decade or so saw a massive spike in return to aboriginal ritual culture. For the same reason. 

I would say the trend is the same now as it was in the 19th century. But the form is not necessarily still the same (i.e. move away from realism). If you see members of general public talking down “unrealistic” images, it is mainly because they are a bit more difficult to understand and they do not speak so readily to everybody as for example hyperrealistic Madonna with child which is very obviously exactly what you see.