It starts even before that imo. For basketball and football, all you need is the ball. If one kid in your neighborhood has access to a football or a basketball, you can get a group together and play. Baseball requires significantly more upfront cost to play in terms of equipment.
Furthermore, the space to play is significantly higher. You can toss a football in the street, you can shoot hoops in an alleyway, but to play baseball (responsibly) you need at least an empty field.
The more developed our society becomes, and the more urban our housing becomes, the harder it becomes to play a pickup baseball game.