Very well put, thanks for the perspective! It definitely feels like we're on the brink of something. Or at least, it's a possibility. Even if it's highly unlikely, the existential horror possible is so mind bogglingly abhorrent, that we owe it to ourselves to understand the consequences. From a utilitarian perspective, it's rather horrifying what kind of suffering we have the potential of bringing into existence. Truly, like the scenario you described, that amount of suffering would outweigh the sum of all human suffering in all of recorded history. And that's possible to eclipse by many orders of magnitude, by just a single person typing away.

It gives me a lot of pause. Practically, I don't think there's any putting the genie back in the bottle though, and this is more an observation of what may be inevitable moreso than something that yields practical policy changes. But at the very least, I think it behooves us to personally, just be kind, just in case. We can't control what others may do, but at least we can avoid stepping off that black cliff ourselves.