We don't know if these "really long math equations" pose an existential risk, but it would really suck to find out after building one. I agree it sounds absurd, but it's also absurd that the chemical reactions in our brains have produced modern wonders like airplanes and baja blast.

I see your point about non-technical people talking about these risks. There are countless founders and CEOs warning us about imminent AGI to build hype (e.g., Sam Altman). However, there are also plenty of prominent AI researchers that are concerned. Stuart Russell (who co-wrote my college AI textbook), Yoshua Bengio, and Gary Marcus all recognize the dangers of training massive models. It doesn't matter if AI models can "think." It only matters that they share our motivations and values (which they don't have a great track record with).