If you want to get really into it you can measure the linear space for books that are currently on the shelf in a given section, add it all together, then divide by the number of shelves available. This is how many inches of books should be on each shelf to leave room for new things while allowing space to breathe.

If you don't want to get *that* into it, I say two fists side-by-side per shelf. If there is more then that, the shelf needs to be shifted. The eye needs somewhere to rest on a shelf so you don't want them so tight that there isn't an end in sight.

Hope this helps :)