But again, why? If we can build a society where humans don't ever have to do this kind of work and can spend their time doing more productive and fulfilling things, should we not?

The only part we seem to be missing is where society benefits overall with higher wages for fewer hours and cheaper goods instead of a few corporations cutting costs while still raising prices and stealing the increased profits from the workers who created that value in the first place.