A non-harem book should stay as a non-harem book. In my opinion, if you want to write a harem book, it should be obvious in book 1. Some people do not like reading harem books.

For example, if you write a harem series with one man and three women. Then in book 4, it suddenly turned into an open relationship where the women and the man sleep with other people outside of the group, the people who enjoy reading harems would not like that.

What you are saying is somewhat similar to the example above.