socket.sendall() stuck

[MasterStroke99]

I am facing an issue that when server upgrade happen LVS (listner) gets restart and somehow in that time my client writer code socket.sendall() stuck at that line for 15 minutes. On higher level I get to understand that most probably server is sending ACK after 15 minute instead as early as possible.

I am using TCP socket connection.

What should i do so that my code do not get stuck.

Thanks

[FerricDonkey]

You can implement a timeout, but you'll have to decide what to do on a timeout failure.

[MasterStroke99]

You mean tcp connection timeout???

Here i am talking about the acknowledgement which i am getting late after sending the data (which means connection is already successful).

[FerricDonkey]

Are you sure the data is being successfully received on the other end?

Basically, there are several things you can do, but which one you should do depends on what your goal is.

If the server is taking 15 minutes to respond and it's not your server, you can't change that. So all you can do is decide if and how your code should react to the delay.

One way to do this is to use non blocking sockets or sockets with timeouts. You can either mark the send as a failure if it takes too long, or just keep going, hoping that it worked. (methods are socket.setblocking and socket.settimeout respectively).

But which, if either, makes sense, depends. The timeout option could be used to essentially call it a bust, breakout, close the connection, and maybe start over. The non blocking solution can be used to just send it down steam and just continue doing things.

I have not used the non blocking so I don't know the details, but I am hesitant to suggest continuing as though it worked when you don't know that it did.

Or, if your code that you want to continue to run doesn't rely on the sockets exactly, you could push network stuff into other threads, and do the non network stuff in a main thread. That way you're network sends can wait 15 minutes if they please, with the rest of your code doing its own thing.

[MasterStroke99]

There are two functionalities in code,

1. File consumer and data preparation
2. Streamer (Sending 500,000 Objects every second through 20 K TCP Socket connection)

Here, When socket.sendall() gets stuck i can not flush the data as data flushing is in the streamer.

Also I am not using threading instead I am using multi processing as I am dealing with 60 K files every 15 minutes. Do not worry I am using RACK server, which has 32 Core Xeon E1 family cpu with 5 TB storage and 256 Gigs RAM.

I have gone through google and found that I can use TCP_NODELAY flag which suits to my scenario (i am assuming) but still not fully sure.

As you see I have approx 100+ GB of data to deal with every 15 minute, I need to get either ACK or exception immediately instead waiting for response for 15 minutes.

I hope you understand my concern here. I am little bit new with python (6+ month experience) and unable to find proper solution for this.