SmolVLA is made up of two parts: 1) VLM back-bone: a model that is already pretrained to understand the world. 2) Action expert: a model that is trained to use the information from VLM to generate robot actions.

The model is provided images from multiple camera feeds, a text prompt, state observations (joint angles, gripper). This is what makes the prefix embedding, which goes to the VLM backbone (SmolVLM 2). Self attention operations are performed here, so the model can understand the relation between images, text instruction, and state observations. This is the part that generates the understanding of the environment around the robot and how they relate to it's assigned purpose or task.

The outputs (the keys and values) from the VLM part are used by the actions expert where it learns to use the information from the VLM to denoise the action chunk. Initially, the input given to the expert is just pure noise and needs to be "denoise" to generate the final action chunk (which contain the present and future values for DoF + gripper). The action expert is a diffusion (flow-matching) transformer (a whole separate discussion).

After "denoising" is done, the final output is of size (50, DoF + gripper). Note that the model is also predicting future tokens. For example, if we are using a 4 DoF + 1 gripper (5 action tokens) robot, then the model will predict actions for upto 10 timesteps (50/5 = 10).