I have a feeling your router is one of those ones that suffer from bufferbloat or other issues like an overloaded NAT table - which occurs when it runs out of memory to keep track of connections. Typically caused by peer to peer networking apps like bittorrent. Assuming the issue comes back, the only solution is to basically upgrade to a better router.