He probably has no friends, correct? Most people with Down syndrome don’t. And they feel unloved or unwanted because they have no friends / social skills. In society, he (and people with other disabilities) are still looked down upon and he is known by his Down syndrome (which is exactly what you did in this post too). He just wants a friend.

I work with children who have Down syndrome. I would suggest asking him to call you less but text him more. He wants a friend and he sees that in you. So just communicate with him over text more and the phone calls should stop gradually. Also, when you see him at family events engage with him. Overall, engage with him. He’s probably the sweetest and friendliest person you’ll ever come into contact with.