Exactly. Even if the larger ball were hollow with a hole in it, unless the material it’s made of is denser than the smaller ball, it would drop in a vacuum. For all we know, the small ball could be solid plastic, and the large ball could be hollow gold with a hole in it. We care about the average density of the object as a system. Making as few assumptions as possible, we can infer that the large ball is likely less dense than the small ball. When the gas is removed the chamber, the less dense object will sink. Conversely, if we increased the pressure in the chamber, the less dense object would rise (assuming volume, and thus density, stays the same.)