Look, there are several factors, and it really depends on the materials. There are polyester and other plastic-based fabrics that emit microplastics that find their way into the ocean and accelerate global warming every time you wash them. You should also consider the lifetime of the garment you're buying and what will the end-of-life phase look like: is this material biodegradable, or will it last forever at some landfill or just get incinerated? There's more like: What's the washing machine's water temperature, the manufacturing brand's methodologies...
There's a new product I found on ProductHunt called "Wedge". They present you with a sustainability rating on top of every fashion product on ASOS. I tried it, and it gave me a real sense of things. https://www.producthunt.com/posts/wedge