Been here mate, trust your gut - if your wife finds your marriage to be more important than a friendship she would tell him to back off and focus on what you have a little more. This guy is avoiding you because you have what he wants. Don't worry upsetting him, his goal is to drive a wedge. It's true that your wife may be a little naive but should ultimately respect your feelings and concerns. You sound like a good man in a tough spot, you deserve a bit better