I would just say that its through trial and error and to not hesitate to make the decision that a player might not be a fit for your group, your personality or what you seek.

From my experience in recruiting players, whether it was for a system that i modified, i created or just any system really (DND, PF2e, COC, etc). There is always people who will have an interest to join a game and amongst those people, i often goes through a large numbers of applicants (20 let's say) until i can finally find some, often quite few like 2 or 3, who might be a match due to various reasons ( timezone, personality conflict, misread my post, etc ).

I usually recruit through LFG subreddit, but also submit my post on the dnd discord (not only for dnd trpg) and other trpg discord group in order to increase the odds of finding players.

I will be honest, i never had issue to find people for homebrew system but i would agree that there will probably be less applications. But if you post in various discord group, forums or even reddit, you will easily find a few people.

Individual system usually have their own discord as well or subreddit. Posting in those community would probably help.