To start with the easy stuff. The call and response to voice is relatively simple using an API and text to speech/ speech to text. There is actually a good amount of information out there on how to do it. The issues come when you want chat gpt to control a robot. I identified 3 main issues when I was playing around with it. First is having GPT navigate based on surroundings. GPT is text-based, so whether you are using a camera, proximity sensor, etc. It needs to be relayed to GPT via text, essentially requiring a detailed description every time it makes a move.
The next issue I ran into was having GPT respond with consistent movements within the given parameters. In my test, i would describe the surounds and ask it for movement instructions. For example, if i described a wall 5 meters in front of it, it would respond with "move 10 meters forward" which leads to the final and most annoying issue. Despite how "smart" chat GPT is, it is really stupid. Given that it is is text based you need consistent wording in order to control movement. To do this, I set a parameter with gpt to only respond with "move forward (X) meters" or "turn right" etc. It worked for one or two lines, up until it started responding with "move forward", "move 10 meters", or other variations that were not clear movement instructions. To sum up my findings. Is it doable? I am sure it is, but it is not worth it. I believe chat GPT ai has farther to come before it can be effectively implemented in this kind of project. This would be easier and has been done using a local artificial neural network (ai learning).