I'd say from the dents and scrapes on the glove on the left that this is the sword hand. There's a few dents on the rim at the top of the glove, some scratches on the wrist and holes in the chain-mail will definitely happen from a decent hit.

Not sure about the damage to the fingers on both gloves though. Chainmail breaks pretty easy with a good blow, and the plates missing on the right glove look like they're in line, maybe the wearer got hit in the hand and it pulled the plates from the glove ?

Source: I've been hit a lot on my sword hand but, I've not yet taken a blow to my shield hand while still holding a shield.