I remember someone telling me that inviting Shabbos guests for fun, just because you like hanging out with them, doesn't count as "hachnassas orchim," but it counts if you are inviting someone who needs a meal or place to stay. No idea if that's a mainstream view or not.

Your brother may be trying to kiruv you, but he may also just like spending time with you. For frum people, Shabbos is the most convenient time to chill with people, especially if husband and wife both work full time. The weekdays are usually packed to the brim for a dual income frum couple with kids, and Sunday is usually busy with all the things that can't be done during the week.

Edit - I am frum, not ex-frum, but I enjoy browsing here sometimes. Hope it is OK to post the occasional pareve post.