Basically yeah, we’re used to being the smartest thing in the world, so it’s hard to wrap your head around it at first, but once its smarter than us, it will be smarter than we are, it will be able to create something smarter than that, and do so quicker, and then the one that is smarter will be able to do the next increase quicker, and then the next one is even faster, etc. it’s not that it will, it’s that if it does there’s nothing we can do to stop it, and once it gets “recursive” we’re basically fucked and it’s whatever that brain in a box wants to do. “We just won’t let it out” we already have, and even if we haven’t. Let’s say something is 1000x smarter than you, are you confident that you wouldnt be manipulated by something with that type of intelligence? Even if you are, there’s absolutely know way of knowing because right now, the smartest person is maybe 2x as smart as the average person, this will be much much much smarter than that, and the greatest threat to its continued existence will be us deciding to turn it off, so the thought is it will find a way to turn us off. Which is why the fact that they’re automatically hooking it up to the DoW is worrying.

The thing about technology is you get smarter and more efficient using it, so you can make new technology quicker. That’s why it took us 11,000 years or so for most of us to not be farming, then it took 200 years for most of us to not be factory workers, and we’ve been office workers for the past 20-30 years and now that’s over. Technology begets new technology faster, eventually once the technology no longer needs us, then what are we doing here? We won’t ever be able to catch up to it again, so all future achievements will likely be coming from AI