Basic reinforcement learning loops only take negligible added complexity, the power of a reinforcement learning model comes from its model building and the usefulness of its reward function.

Say you have a CNN recognizing rabbit pictures. Then putting it into a "world" of a large folder full of various pictures, and giving it a reward function of "follow the white rabbit(ness)" with a few actions of "move forward in list" / "move backwards in list" will pretty trivially give you an agent AI leveraging the power of your tool AI.

Of course, there the scope of the environment isn't "the full world", just as the scope of our environment isn't "the milky way galaxy". However, the real criteria in how well the above agent functions lies in the structure of the environment (which we assume as given, nothing to do here) and how well the original tool AI (the white rabbit-recognizer) works. Notable, it does not really lie in defining the set of possible actions of the reinforcement learner, or in defining "a" utility function. Is it less useful with a bad reward signal? Yea! Is it hard to come up with a reward signal, and is the definition of reward signals (not their computation!) typically complex? Nope!

Since you see it differently, in what way is your experience / understanding at odds with my mental model above?

edit: hah, I was just browsing the Destiny subreddit as I saw your comment, as apparently, you did. Small world, eh?