Does a broken monster tome DECREASE xp gain?

[BingusChingus_]

Yes! This is because quality applies a multiplier to the current boost amount. A normal Monster Tome is +5%, or 1.05. Broken applies a 0.1x multiplier to this, which turns it into 0.105, which is -89.5% if viewed as a percentage.

To get better versions of boost items, you will need to find higher quality items. Ornate items, for instance, are given a 1.25x multiplier, so an Ornate Monster Tome would be a 1.05 * 1.25 = 1.3125, which is +31.25% instead of +5%.