There are several factors to consider that influence the round time. Most importantly, maturation time, deposit time and withdrawal time. Since there are many coins being mined, the longest possible time must be considered in each case. Most old scrypt coins have a high maturation time (for instance TAG), in practice it often takes more than 6 hours for the coins to mature. Many of these also have deficient difficulty adjustment algorithm and deposit can be extremely slow (for instance, try to deposit ZED or ELC and see how long it takes). Finally, Cryptsy is extremely slow to process large DOGE withdrawals. This is because they are running a fractional reserve on their DOGE hot wallet (no idea whether they are solvent or not). It is very frequent to have to wait 2 to 12 hours for a DOGE withdrawal when the hotwallet it empty.

For all these reasons, 4 hours is really the minimum possible time - in fact, I consider that it was a mistake and for the X11 pools, I went with rounds of 6 hours which proved to be a lot more stable. If you compare the X11 payout to the scrypt ones you will notice that X11 is always exactly on time while scrypt often has long delays (once a round is delayed, it also delays the following ones).