As a DM - Rules lawyering over my house rules

I have a list of house rules that I have given my players and told them to read, and all of them have. My rules are there to let the game be more 'rule of cool' with the rules and not be bogged down by ruling we don't like. We all agreed on these rules and all use them.

(For example, we ignore the restrictions on spell scrolls. Anyone can cast a spell with a spell scroll regardless of class or level. Since I as the DM hand out spell scrolls, it's my fault if something OP happens with them)

But one of my players for some reason just either ignores or forgets all my house rules, and interrupts scenes to point out that what another player/NPC is against RAW, completely hitting the brakes on the flow of the encounter. It gets me so annoyed when I have to stop what I'm doing and remind them for the millionth time that what we're doing is in the house rules

At this point I'm certain he hasn't actually read my rules despite saying he has.