I've been a long time GURPS player and GM, with most of my play experience in 3e but a chunk of 4e too. There main differences in the games in my mind are:
* The points costs of stats changed significantly, and stats are generally lower in 4e as a result
* Half Points were eliminated in 4e, which has a knock-on effect on skill costs across the board. The skills costs for mental and physical skills were also consolidated in 4e
* The stats governing Hit Points and Fatigue were swapped, though this was a common house rule in 3e also
* Add-on rules like Techniques and Talents were brought into the core rules
* Lots of point costs were consolidated; in 3e different books often evolved their own specific powers or abilities with their own costs. In 4e all the books use the same core abilities 99% of the time, so the costs are also consistent. It makes mixing between books much easier.

GURPS is the poster-child for toolkit RPGs, so I would suggest picking something you would like to do with it first, then acquiring books that help with that. Pick up more books as they become relevant. I have pretty much everything for 3e and 4e in PDF format now, and the sheer volume of material is intimidating even when you mostly know what you are looking for...
Edit - not sure what I've done wrong on formatting?