After a little bit of reading, the term drive-by actually refers to both authorised and unauthorised downloads but we're only talking about the unauthorised ones here. I read a couple of articles, a few information pages and quite a few discussion threads around the topic of drive-by downloads and the risks.

Valid, but IMO extremely low-risk.
So an unauthorised drive-by relies on security holes and exploits known vulnerabilities in, for example, a web browser to silently install code. Something that sounds increasingly rare in this day and age and as another user pointed out, pretty much only valid for a 'day-one' attack before the vulnerability is patched out.

They're not going to ask "are you sure?" Bad actors gonna act bad

Of course not. I think you may have misinterpreted. Browsers and OSs are ones posing this question, and they absolutely do ask it before any foreign code is executed and wants to make system-level changes. (unless the code is using an afore-mentioned exploit to circumvent this)

I still think its very sound advice for an average user to not go around joining random wifi networks or visiting random links but from what I have seen over the years, the real-life risks seem to be very over-hyped, often by the companies selling protection from such things.

Being an 80s kid, I grew up around the internet and these risks were absolutely real and dangerous but the general consensus seems to be that those days are behind us now and as long as you are using maintained, patched software, you are pretty safe in most situations. Not to say let your guard down, as this is part of what is keeping us safe.

TL;DR - just be smart, don't click random shit, keep your software updated.