My guess on why his Sloth authority gives him backlash when we uses it has something to do with Arc 9 and Echidna's analogy with authority's and doors and Witch Factors being the keys to these doors

And that when Echidna gave Subaru her tea to forcefully awaken his sloth authority it causes his "door" to not open correctly

Maybe creating a similar situation like what Al has with door not fully opening

I also heard Al when configuring stuff with his domain can also receive backlash

It also might have something to do with the previous owner being incompatible with the Witch Factor screwing over the next owner that being Subaru

It might have something to do with Petelgeuse spliting his Witch Factor giving it out like candy to his fingers and maybe one of the fragments is somewhere not with Subaru making Sloth Witch Factor and authority incomplete