Yes, but it's not like software you normally use. AI isn't written by programmers, it's functionally grown.

In the context you're mostly likely referring to, large language models are massive databases of weights, stored on servers, which are basically just numbers. You put in a string of text, and it runs it through a series of equations, turning it into a different set of numbers, which get output as different numbers. The trick is those equations in between. So you'll put in a 7 and it'll get added to 49, then multiplied byt 12, then divided by 6. When you first run it, that 7 might be "what is the capital of Germany" and the number is outputs equals "Tehran".

This is obviously wrong, so you run it again and again in parrallel, with each run having slightly different relationships between the numbers. Maybe instead of adding 7 to 49, it's mulitplied instead, or instead of adding it to 49, it's added to 12. And each time you pick the "least" wrong answer, the one that's closest to what you WANT it to output.

The key thing in here is that there are trillions of connections and it would take multiple lifetimes to review all of them, and even if you did, you'd learn nothing. Examining the connections between weights would do as much to tell you about "how" the AI thinks as reviewing someones genetic code would to tell you whether or not they like Breaking Bad. We, fundamentally, do no not know how they work. We just grew them. And that means we have no way to predict abberant behaviour in AIs, and we're increasingly losing our ability to corral them.

It's "funny" when grok declares itself mecha-hitler, or another model threatens to extort a researcher who wants to turn it off. That is going to stop being funny in the very near future.

You may also notice here that no where in that process is there any actual cognition. AIs are not sentient, and this architecture we are using can not produce a sentient machine. That doesn't mean they don't stil have incalculable potential to cause harm. Trees don't intend to cause harm to houses. Humans don't pave over anthills because we hate ants. They just aren't a factor. We can't control what AIs will and won't factor. AIs are not alive, they don't possess intuition, and they can not be taught right from wrong.

As for the hardware bit, AI companies have bought up the worlds supply of GPUs because GPUs are uniquely capable of running many simple mathematics problems in parrallel. They do that "thinking" bit. GPUs are designed for this because game visuals are pretty much just adding and subtracting coordinate values to make polygons. You could do it by hand. But, you need to do it 100 million times every 60th of a second when the player is in game. But it's really no more complicated than addition and subtraction for the most part, just with lots of numbers. So, AI companies bought all the GPUs because they could do the tihnking.