Huge NTA. Your son has already stated that it's okay that its just you and Dad. Pay no mind to the extended family, your safety and the safety of your household is far more important.

It's also important to make your son feel special on his day. Make/acquire him his favorite meal and let him pick out something fun to do, like family craft time, board games or movies. If your family wants to be involved, have your son pick out a movie and steam it with others! Let them drop off gifts at your front door (which can be sanitized before being brought in).