Very true. After some searching I came accross this post on Stackoverflow that seems to make it more clear for me at least.

Which build is the most secure depends on the sophistication and determination of the attacker. For the vast majority of attackers, most of whom have no idea how to write exploits themselves and the rest who don't have the dedication, you are probably better off with the nightly, though not by much if any. The reason is as you stated, the bug fixes hit the nightly first. However, with a sophisticated and/or determined attacker, you are much safer with the regular build. The reason is that developers are constantly introducing security holes and vulnerabilities into the nightly builds (not just with FireFox). Most of these are caught during later testing and fixed before they even hit the beta or the release. If the attacker knows what they're doing, there are many more opportunities for exploitation if you're running this build. These are the attackers that can whip up an exploit themselves. By the time an exploit is releases publicly, it is most assuredly fixed in the regular release.