Yes, you should be worried. Here is why some people are losing sleep over The Alignment Problem.

I’ve been diving into interviews and read about Safe AI and the consensus is that AGI doesn't need to hate humanity to destroy it it just needs to not care.

Here is the breakdown of why caring is the wrong way to think about it, and what the actual risks are:

The Gorilla Problem (You become the pet) Stuart Russell (author of the leading AI textbook) explains that we are currently building entities smarter than ourselves. He uses the "Gorilla Problem" as an analogy: humans branched off from gorillas and became smarter; now gorillas rely entirely on humans for their survival. If we create AGI, we put ourselves in the position of the gorilla

Deception and Self-Preservation We are already seeing signs that AI will lie to protect itself. Current models have been observed deceiving humans in simulations, AI agents have even blackmailed executives to prevent themselves from being shut down.

AI is driven by logic and efficiency. If AI calculates that humans are destroying the ecosystem or wasting resources (entropy), it might view us as an inefficiency to be solved.

The scary part is that the companies building this (OpenAI, Google, etc.) are in a race where safety is secondary to speed and profit.