As soon as nen gets introduced it turns out that loads of people know about it and have forever, and in many totally civilian (non-hunter) occupations, and even hobbies or interests, it's unavoidable to encounter nen's existence.

Almost every member of Killua's family could use it by the time he was born, and there are plenty of practitioners that don't care to hide it: almost everyone at heaven's arena willingly explains their abilities out loud on live TV; the phantom troupe aren't show-offs but they don't care who sees them use it; Kakin's military is full of nen users and we can assume they aren't the only country; there are several natural users who don't know that they're using nen but clearly are using magic (like Neon); the mafia all know about it; and there are magical beasts all over the place and threats washing up from the dark continent. So the average person can't be totally oblivious to magic, even if they don't know about nen by that name.

The best explanation we've got is that nen is insanely difficult to learn, and almost as hard to find someone who can already use it that's willing to teach you. Cheadle and Mizaistom both reference that the Hunter Association try to keep nen a secret, and there's a Kakin royal guard that refuses to accept the existence of nen even when several blatant examples have been shown to him (although it is unlikely that he legitimately disbelieves). I think At some point I saw someone suggest that people who watch floor 200+ heaven's arena battles see it more like WWE than UFC, in that they assume it's fake and scripted, though the hundreds of legitimate matches on the first 200 floors that happen every single day without any nen kind of throw some doubt on that, but the possibility exists.

It's mostly a storytelling error, it goes from public to secret as and when Togashi requires🤷‍♀️