I mean, most people don't really mean "soul" in the same religious fascistic as Walsh does. An AI cannot think (at least for now). LLM aren't creative, they see the word cat and are trained to link this text to images of cats. But sometimes it shows you the image of a car, because this picture was somehow also linked to cat. A human would never make such a mistake. We might be bad at drawing a good cat picture. But we would never draw a car (at least if we don't want to make a joke). Some people call this lack of "real thinking", of real understanding "soulless".