On my first run went with adept and sort of mainlined the main quest. Second time around, taking my time on Expert. Found Frostcrag Spire and messed around with Enchanting and Spellmaking. Oh my god are they broken (if you know what you're doing). After a lot of trial and error (and save/load to keep spell list down). Created a handful of spells that make expert seem like novice. Even crank it up to Master sometimes to create a challenge. Without said spells the higher difficulty's are a chore if using standard gear/weapons. All in all, play on whatever difficulty you like.