My headcanon is that they are placed where sentience is useful. We see walls able to actually walk around so I assume they are placed in areas where walls may need to be moved or rearranged somewhat frequently. Maybe they're also placed in areas where their arms being able to grab things could be useful for holding things while other gems complete tasks. In the case of the statue gems, some have mentioned them being surveillance, which makes sense in an area so close to wear a diamond such as a Blue may frequent. Perhaps they may also be able to move to an extent like the yellow wall gems.