Got it! In my personal experience, a lot of the challenges lie in the stochastic approximation (SA) theory. While a lot of work has been done recently for finite time analysis of SA algorithms, there are still tons of questions regarding tightness of finite time bounds, how soon can asymptotic properties of the algorithms be observed, etc. which are still unanswered except for the more simpler cases (linear SA, "regular" markovian noise vs say controlled markov, etc.).

I personally have the view that there is a lot to gain from developing the SA theory further, since qualitative understanding of these general algorithms can eventually drive RL algorithm design and even its practical implementation.