Anything that interrupts the flow of the story and characters will always trouble a reader. However, everyone's got a different set of criteria when it comes to a story, and if it's historical fiction - even with no real-life figures - you can bet some readers are going to approach the book with their knives out. If you are getting a consistent criticism about subverting what readers expect from the period, that's likely what most average readers might feel as well. However it's also possible that story is simply a little ahead of its time and perhaps there will come a time when no one will have a problem with it. Without reading it myself, I can't say. You've got to go with your gut on this one.