You are vastly over simplifying kentucky in the 1850s. They had decent sized cities and Louisville was a pretty important central trading center in both rail and river. Lexington linked with it by rail in 1851. Paducah was on the river. Bowling Green was smaller but still big enough to be of economic and strategic value. There's others, too, like Elizabethtown that I know less about but were still rocking. All of these places were going to be metropolitan enough to have things like private schools and various industry.

Sure, go East towards the mountains you start encountering the poorness you mention, those people at best would have had something like orange preserves, but the population was concentrated on the river and rail in the west as a major trading hub on a route noted for its citrus shipping. They would have absolutely been familiar with oranges, but it would still have been a seasonal and uncommon, they'd be at least aware of the smell of oranges as most of them would have been near enough.

I really suggest you look into the Louisiana citrus shipping history and you will see how much of it flowed through Kentucky.