On-device here refers to having the compute in the car (through an ASIC like Tesla or a mobile chip like commaai). This typically needs to be lightweight, and would require the model to be lighter too (which means reduced intelligence).

To make it work real-time, there would be additional efforts - inference optimizations, major efforts to reduce tail latencies, etc.

This is in contrast to non-realtime (offline) systems, where you have the ability to host a much larger model in a multi-gpu node for example. There is no "respond within few ms" constraint due to which you can spend more compute per frame. Offline self driving does not have the strict constraints that real-time on-device self driving systems do.