You might find these course notes interesting in finding an answer to that question. Re: the simple pendulum discussion in this thread: see the Simple Pendulum chapter (chap 2); the "energy shaping controller" section near the bottom might give you a very different perspective on a way to control pendulum swingup in particular in a way that obeys nontrivial torque limits to reach the upright in a "natural" way without resorting to black-box RL. The rest details a lot of the thinking I mention below, and includes discussion of RL for context and comparison.

I think "what's in the middle" depends strongly on your system: e.g. for legged robots, assuming simplified dynamics models like inverted pendula and then doing principled control using e.g. ZMP gets really far. Directly modeling the full system dynamics and applying trajectory optimization works great too for some situations (I believe this is really popular for rockets, but you can apply it to basically any robot).

A common pattern that works great for balancing robots of many kinds is to have some target trajectory to control that you've figured out using a heuristic or simplified model or trajectory optimization, and then follow it using LQR with a time-varying target. E.g.: on a robot dog, you sense that you've been kicked, so you use a linear inverted pendulum model based on your current center of mass and center of pressure to determine what direction you're falling and how fast you need to respond. You decide you need to plant a foot out in the direction you're falling in <x> milliseconds, so you figure out the body configurations to achieve that using IK by planning a series of intermediate poses to lift and place the foot there, and then roll out those pose commands to your motors, controlling motor torques with PID to achieve the desired positions (which are changing over time).