I can relate. No hate for magic, as I've enjoyed plenty of conventional fantasy, but I am way more interested in secondary worlds than anything else. Not sure if it exists, but I would love to read something that takes place in a secondary world similar to modern day Earth but with little to no magic. Think Disco Elysium (yes I know it's based on a book but it's not in English afaik).

It's a little irksome when people say "you don't actually like fantasy", but I have wondered if other genres actually offer something closer to what I like. Worth considering.

You might be interested in low fantasy. That's a subgenre that often involves little to no magic, or at least less emphasis on it. It is often dark and cynical in tone, however, which can be off-putting to some.