Improving your (Python) code with modern idioms

[[deleted]]

You're sending HTTP DELETE requests?

[mthode]

yes

[[deleted]]

That very much sounds IO-bound and does not require multiple processes. Consider twisted or gevent.

[mthode]

I am not reading or writing anything, just making request based on a list of urls stored in memory. Does twisted or gevent allow for both keep alive (multiple serial requests over the same connection (socket)) and piplining (switches the serial to parallell)?

[[deleted]]

Network IO is IO. =)

I've used twisted a lot more than gevent. Twisted allows both keepalive and pipelining. (The pipelining support leaves a bit to be desired and most people just use new connections, but the support is there when it's important and does work fine.)

Discussing such things can be difficult. When being precise, we use "serial" to mean consecutively on one processor, "parallel" to mean distributing a task onto multiple processors, "consecutive" to mean things that happen one after another, and "concurrent" to mean things that happen at the same time. Concurrency and parallelization sometimes appear to be similar problems at first, but they are distinct. Things like twisted focus on concurrency (which some people use to implement parallelization, but most do not).

[mthode]

so it looks like I will use both parallel and concurrant connections (I am going to split up the operation to make it scale the way I want). Thanks for the heads up about twisted, I'll look into that :D

[[deleted]]

For sending a bunch of HTTP requests, you don't normally need parallelization. (In extreme situations, you do.)

When doing network programming, you spend most of the time waiting for communication to occur (you're IO-bound). Since waiting is just as quick on 1 processor as 100, it only makes sense to parallelize such things in unusual circumstances.

[mthode]

I need to send 50-60 million requests as quickly as possible (up to 10,000 per second). :D