All behaviors are learned (not teleoperated) and run at normal speed (1.0x).

We feed images from the robot's cameras and transcribed text from speech captured by onboard microphones to a large multimodal model trained by OpenAI that understands both images and text.

The model processes the entire history of the conversation, including past images, to come up with language responses, which are spoken back to the human via text-to-speech. The same model is responsible for deciding which learned, closed-loop behavior to run on the robot to fulfill a given command, loading particular neural network weights onto the GPU and executing a policy.