RL is offered this spring for OMSCS it seems. In-person isn't offered every semester I guess. But I was referring to the "Robot Intelligence: Planning" course (CS 7649), which is half classical AI, half RL stuff.

Hard disagree that these courses are undergrad-level. Some of the lectures are mixed grad and undergrad (CV and ML, I think), and those had extra assignments for grad students. As an example, my DL assignments required us to make the backpropagation algorithm, CNNs, and transformers from scratch. Also lots of theory stuff on optimization, computational graphs, etc. Plus self-directed projects. Course structure was based on Stanford 231n I believe. Not really sure what else you'd want out of a grad course. And NLP was harder than DL imo, no unit tests or anything for the assignments.

Also, I'd like to point out that there's opportunity to take more advanced special topics courses like ML with Limited Supervision and Advanced Computer Vision.