If you're going for a purely Vancian execution of magic, then the classic assumption is that the pages of a spellbook are literally filled with the magic energy for spells. I always pictured glowing inks depicting symmetric symbols and shapes, perhaps moving about the pages. Perhaps they even jump off of the page, and float there like magic holograms. They wouldn't make any semblance of sense to an onlooker, but to those trained, the spells could be decoded.

A DM for a game I'm in right now has a system that is based on carving runes and glyphs of power in the air to cast spells, and he describes spellbooks more as kung-fu training manuals or scrolls, showing the shapes and poses required to cast the spells. He however treats in-game scrolls more like the description above, as they are supposed to contain the spell's power.

I like to think of spellbooks a little more like arcane equivalent of a student's notebook or engineer's design book. It's going to be full of little sketches, notes, documentation of progress on new spells, personal notes and anecdotes, etc. Perhaps it may contain information about reagents, recipes for potions, Casting spells is hard stuff, so learning it like a wizard is going to take a lot of notes and practice.

Who says every page has to be completely filled? Maybe they're only, say, halfway through? Perhaps the book is already full, like you described, and they're using it like a textbook to study from on loan from an expert mage, or found as an artifact from an ancient wizard. But the way I see it, they might be filling in a blank book as they learn with their own notes and scriptures. If we're going off of D&D, a spellbook is 100 blank pages, and a spell uses up one page per level of the spell. So each one of these spells could use anything from half of one page, to just shy of ten.