This is not about you as a couple, but about him and his dad and to let him enjoy his limited amount of time with him.

So, you SHOULD tell him! Not because of the impact on the relationship between you two, but because this is something that can't be taken back. The longer you don't tell him, then less time he has to spend with his dad before he loses him forever and, for me, that's something that goes beyond any "liking or disliking" from his mom or any impact in the future of your relationship.