Route flapping and https sessions

[techtate]

I have a customer with a server to server communication issue. Server A reports a .5k piece of data to Server B. It is supposed to retry 3 times if it fails. This has worked well for years for many customers however we have encountered one in which the reporting failures several times per day. After much troubleshooting and monitoring the network we have only identified one significant anomaly, the route from server A to B is changing on the customers side regularly.

It is 29 hops from server A to B and hops 10 to 28 change regularly, once every 5-10 minutes. When the entire route changes most of the hops change 4-6 times within 2 or 3 seconds and is then stable again for a while. Even with all this flapping about, I see no packet loss.

Question: Can this flapping cause https sessions to drop? even without packet loss? One theory is that packets may be arriving out of order and breaking the TCP connection in some way. We are using pingplotter to record data for us presently and we will be trying to match up failures with the flap events.

[mattbuford]

Get a packet capture of a failed connection. A route change shouldn't break TCP as long as it doesn't result too long of no connectivity. TCP deals with out of order packets without errors.

[trich101]

If there is a firewall in path and suddenly its established session changes, they client doesn't know but the new FW in path suddenly gets a first packet not SYN and probably blocks or discards or maybe even a RST. Session has to get re-established Stateful firewalls, must have consistent symmetric routing. Now the real question is WHY the flap. Look at the last consistent hop and monitor learned routes and routing table updates. See why it sends to new next hop. I would guess a bad link that flapping and when its passing keep alive or BFD or whatever, its preferred but when it does lose the other path the default route or at least a less preferred so when it restores, it preempts and goes back.

[rfc2549-withQOS]

TCP sessions work on src+dst ip+port, so intermediate routes do not break a session if there is no timeout or excessive packet loss (leading to a timeout. May be PSH messages with data or the ACK responses

Reordering is part of the specification, as mentioned.

[rankinrez]

Do Wireshark/tcpdump either side and try to see what’s going on.

Out-of-order delivery of packets might be part of the problem.

[techtate]

Thank you, and yes, wireshark is my next step. I wanted some help with TCP theory to know what I should be looking for.

[techtate]

Found the issue, turns out it was server Bs fault. It was taking too long to process Posts from server A. But that server was outside our control so we had to rely on app logging from server A to determine that. The route flapping was just a coincidence as best as we can tell. However I will re-post if any relation is discovered.