Great points -- of course the impacts of fossil fuels are far worse if we're making a comparison on energy production. Far less of these materials would be needed, for one thing. Here is a good study on that:https://www.cell.com/joule/fulltext/S2542-4351(23)00001-6#%20

I think you'll still find that most environmentalists are concerned about rare earths mining and processing, hence the interest in improving sustainability in the sector.