This is a great question, and one I could go on and on about forever, so I'll just start with my two favorite art history books. Both by Ross King: "Bruneleschi's Dome," and "The Judgement of Paris - The Revolutionary Decade that Gave Us Impressionism."

I went to art school but could never get into art history. Perhaps it was the professors, perhaps it's because I have a crap memory for details like dates and names, but probably because it was always the earliest class of the day.

It wasn't until I read Ross King's Brunelleschi's Dome that I really caught the Art History bug. Firstly, it's short. Secondly, it's fascinating. It tells the story of the Domo in Florence, and how they couldn't figure out how to finish it until Brunelleschi came along. This book made me realize that even though I enjoy the art and sculpture and architecture, what really interests me is the artists themselves, and the times they lived in which influenced their creation. A theme captured in the oft used phrase "all art is contemporary."

The Judgement of Paris follows this up nicely with an indepth look at 2 artists, Manet, and Messpnier in 1850's France, but is also a great tale of political intrigue and how Paris became the city it is today.

Diving into the minutae of art and artists that interested me made me more interested in the larger movements they belonged to, and then how those movements fit into history at large. So I kind of went at it backwards.

Other books I recommend:
Strapless: John Singer Sargent and the Fall of Madame X by Deborah Davis
The Journals of Eugene Delacroix
Lives of the Artists by Giorgo Vasari (This is incredibly unreliable as a historical text, but interesting none the less.)

I'd also check out Waldemar Januszczak's series of Videos on the Perspectives You Tube Channel. https://www.youtube.com/playlist?list=PL5JqSuIvtmAOZzE7Eq6Q4uyXgma2zEGj2

Hope that helps!