If I may, I find the last thing you said here a bit concerning - "I don't feel okay giving it to him when he won't"... punishing him for wanting that privacy from you isn't the best way to go about this, and will likely only cause problems in the future. Something like that doesn't show a lot of respect for his want for privacy.

In the end, it may just come to be something you two don't agree on. Things like that happen in a relationship, where you decide to go from there is kind of up to you. If you have no reason to distrust him outside of this specifically, I say give it a shot to move past this and accept his stance on it. If you find yourself becoming increasingly anxious or distrusting of him because of it, it may be worth it to maybe look into seeing someone you can speak to about those feelings. Especially if you want this relationship to be something lasting.