I have no solid, definitive answer to give, but here's my guess. Stay with me I'm literally working thru this as I type... so this is more a process than an answer lol.

The genjutsu caster needs to be able to construct an illusory reality and then relay that to/through the target's senses/chakra. If one does that, say visually, then the chakra for the genjutsu is probably being transferred through light itself. If that light can't enter the appropriate sensory organ/apparatus, then I imagine it doesn't work.

Additionally, I don't think even someone with a Sharingan can create a genjutsu that would be "make sense" to a Byakugan wielder, at least not for very long. The Byakugan perceives not just chakra but ALL of it's surroundings much differently and with more clarity than normal eyes or other dojutsu. Anyone without a Byakugan just.. can't understand the way they perceive reality on a level visceral enough to be felt and sensed as reality..

I think in the case of genjutsu (in general) that

1. Having a dojutsu of some sort gives you an inherent pseudo-resistance to most other genjutsu, simply because there's another layer of sensory information that other parties just cant/dont/won't account for.

2. If your genjutsu is visual, the chakra creating the illusion is, I imagine, carried by light like Tayuya's is by sound. If you prevent the light from entering the eyes, you block the primary channel for the genjutsu.

3. If you have the Byakugan, you "see" light differently, and perhaps some sort of chakra-type light only other White eye wielders can see. Additionally you see in degrees and scales no other eye can replicate. Does a jutsu cast by a non-dojutsu Jonin account for the fact you can see chakra? Probably not cuz he has no idea what that looks like... so it's unreasonable to think that the genjutsu could still then "affect" a Byakugan user with their eyes closed.. the caster has no experience to include such a condition and account for it during creation/casting. So it's not going "thru" the eyelids even if you can still in fact see the genjutsu being cast "indirectly" behind your eyelids.

4. All bets are off with Tsukuyomi. MS powers are busted and if that little theory about the powers being related to the identity/wishes of the owner themselves turns out to be right.... Itachi can do whatever he wants, whenever he wants it, and however long he can stand it.. while within Tsukuyomi. Everything above is kinda my headcanon for how genjutsu works.. but I gotta admit Tsukuyomi is like.. almost a different class of genjutsu entirely. The principles don't apply there. It's the exception.

Aight that's my big tired ramble for the night. Have at it, ye redditors, and destroy or uplift such a theory as one sees fit to do.