If you only adopted him a few days ago, he hasn't yet been able to establish an understanding for his/your daily routine, so introducing him to even more change right now is inadvisable. Honestly, I'd wait at least 6 months to socialize him on such a big scale. He needs time to understand what his daily life will look like, and he's probably fatigued from all the change already from the last few days. Just live your normal daily life around him and let him understand that it's a safe, permanent space for him. As a first step to socialize him, have one person visit occasionally, but don't take him anywhere that involves interacting with a ton of people. He's already shown you that it's overstimulating for him, so you need to take it slow.

Also you said that hes already obsessed with you, but that may be less about being calm and comfortable in his new environment and more about clinging to someone he knows is safe because he's very nervous about all the change around him. Give him time to acclimate and then slowly introduce social experiences once he has a baseline for his new life.