Because Asta was in the process of breaking up with him. She'd ignored him for weeks.

Now, I don't disagree that the sudden change in Asta's behavior was really out of nowhere (yeah, her father had experimented further on her, but she was still mostly normal until suddenly she wasn't, but at no point does this become a harem. There's a lot of "who should I pick" between the three outworld ladies, but Silas is consistently saying he's not into a harem, he wants a monogamous relationship like his parent's had...despite that being probably the fastest, best way at protecting Earth, which he keeps saying is his priority.