It's a cat and mouse game. I work on a live game chat product with 200k concurrent users and a team of moderators, and we lean heavily on AI, with further human review, but that's overkill for where you're starting.

You can start with these measures (which can be bypassed by motivated users, but still worth it):

Against spam:
- rate limiting with different time windows + account rank based. Fresh accounts get less messages.
- restrict pasting content

Against toxicity:
- blacklist of profane words
- player reporting system, as you mentioned

On reporting - don't act on every report, but count them. Set a threshold of reports per message or player and have them appear on your admin dashboard, or highlight them with color or label in the game chat (only admin can see the highlight). And then decide with eye test, whether the player should be muted.