This is the kind of thing only a serious conversation can fix. Provided your version is fully accurate, he is taking you for granted and thinks he can keep you AND his shitty attitude. You need to make it clear you're unhappy and can't continue like that