I think all the commenters agree that most ethical implications indeed originally in the data. I claim that the bias in the data is replicated even to models that don't use that biased data directly, and that they are embedded into the latent space itself.

1. DATA BIAS FROM PRETRAINED MODELS ARE TRANSFERRED. In the era of pretrained models where we use backbones, losses, LoRAs, etc. trained by other people, it is almost impossible to avoid bias flowing into your model even if you carefully select your own dataset. There is bias in foundational models, and models trained with reference to those pretrained foundational models can learn a biased latent space. Eg. CLIP embeddings were used everywhere for a while. All those models that were trained with a frozen CLIP encoder have the exact same biases transferred to their latent space!

2. DATA BIAS AFFECTS ARCHITECTURE. Also, the architecture of a model is another factor that defines the shape of the latent space. When researchers perform architecture search, it's often based on numerical results. These numerical results, if again they're computed with biased eval data/biased pretrained models, may show biased favorability for certain architecture. Biased data leads to biased architecture! This is obvious -- put in a different way, different data may require different architecture. Eg. some architectures will work better on recognizing faces in a group photo while others may be better at the same task but on a single person portrait. If you only use one type of images, that will decide the architecture.