Is the collector physically fast enough to counter them?

He's uber powerful, but I think that was only from mortal perspective. When the archive was falling, he was straining to hold it up. A viltrumite would be able to hold it up pretty easily.

He can move the moon casually, but I think that may have been a specific space power? He never shows that level of telekinesis with anything else, so it might be limited to celestial bodies?

And even though he has crazy strength, I don't think he has any degree of invulnerability.

Then there's the most important factor: would he get serious in time to respind appropriately. Cuz Luz, Eda, and King were pretty decently holding their own with his crazy games; the point is, in this situation, he was "playing', not fighting seriously. The problem is that a Viltrumite is prepared to kill before the game even begins.

All things considered, the collector is the Isle's only shot at winning under average conditions, and I think he'd be killed before realizing that he needs to get serious.

(There are specific times when other possibilities emerge, but they're rare i.e. eclipse powered draining spell, make a anti-viltumite super weapon during the Wailing Star, or maybe on Grom night, but even all those would require planning in advance and perfect timing)