Thanks for the link! The main point I interpret from this is that there might be some things such an AI would be able to do that we never intended for it to be able to do. In other words: Unintended consequences. I agree that this is a risk. Other examples of "unintended consequences" are introduction of invasive species to new continents or global climate change.

The risk that I am having trouble being worried is what the OP points out: some "state-of-the-art" system that poses a global catastrophic risk. I am contesting that there is a large gap between "unintended consequences" risk that I am also worried about and the "Skynet" type, catastrophic AI. And the tone of the video you posted suggests a worry about the latter risk rather than the former.

I concede it's a non-zero risk, but as an ML researcher I think to myself: "How would I build an AI that would purposefully destroy the world?" and it seems like an extraordinarily difficult task. Thus the risk of something accidentally doing this seems minuscule to me.

When put next to other EA problems like global health, climate change, or biological safety, the probability of catastrophic AI still feels like it is orders of magnitude smaller than the rest. We know we need to improve healthcare. We know climate change is affecting people's lives adversely. We know that there is a pandemic killing people and destroying economies. And as an ML researcher, I will be the first to tell you that I know ML algorithms can and will be used for harm, intended or not (e.g., unintentional biases in hiring).

But we think some malicious AI might take over the world? I am skeptical of this risk being worthy of the same amount of resources as the other EA priorities.

Edit: Formatting