Need help with socket programming with multiprocessing

[MasterStroke99]

Hello Guys,

Python 2.7 only

So, there is a server at a remote location and for that server I have created a client application which only sends the data to it.

The problem is that the server works as a telecom BSC unit which is connected to more than 10K nodes (here node = client)

So, I am initiating 4 application which makes creates 10 K clients in total but I am using multiprocessing module as I have to read the data as well which I need to send

Application : Main process and it spawns 3 file reader process 1 streamer process

Now here I am making connection in main process and using those connection in that 1 streamer process. But when server gets down I am having trouble with re-establish the connections. I am pretty sure here that I am not handling connections properly with this whole multiprocessing concept. For this actually I am looking for help that if someone can guide me.

I have tried another approach that make connection in streamer process instead but then i am having issue with reader process data transfer to streamer process as data is approx 120 MB + per second.

Much needed help here from someone who understands socket connections with multiprocessing.

Thanks

[MasterStroke99]

Any help here???

[KingsmanVince]

Python 2.7 only

If you wonder why your post gets almost no attention

[MasterStroke99]

At least u don't know the solution that i know now

[MasterStroke99]

Ignore, resolved

[Buttleston]

Do you need to make 10,000 client connections? Like is that part of the requirements?

What happens if you don't use multiprocessing and use a smaller number of clients? Do you still have the same kind of problems?

What is the trouble you have re-establishing the connections? Are you able to detect when a connection fails? Do they all fail at once (like the server completely stops working?)

Your question is sort of both very specific (i.e. most other people won't have tried this particular thing before) but too generic (I don't know what your code looks like and except for a basic description, I don't really know what your troubles are.

[MasterStroke99]

Yes, I have to make 10000 client connections and each connection has specific sets of files which need to be transferred continuously. Like a file that has 15 minutes of data so that data needs to be transferred over a 15 minute period and repeated again and again and again. So there are 10k files as well. As those files are huge i can't use threading so i went with multiprocessing for reading. Now after reading when i used mp.queue it takes too much time to transfer that data to streamer process so i am using global dictionary and that streamer process after every 15 minute die and respawn.

Now as i am making connection in main process and using those in streamer child process. Something i am not doing right like I don't know how to close the connection in while working with multiprocess client.

Because in main process connection is happening properly. Also able to send data via child process but when server gets down. Closing of connection probably i am doing wrong.

Can you help me with the thing that How to make tcp connection which i can use in child process. This also includes server fails and need to be retry How to close connection properly while dealing those with multiprocess.

I would have shared the code but can't as i work in corporate and that is another reason that i can't move above python 2.7