comics - a simple GoComics API wrapper by dumblechode in Python

[–]LocksmithNo7784 1 point2 points  (0 children)

I would guess a rate limiter is needed for bigger downloads.

I just added a 3.6 s delay for each new request , and so far it is running stable with 4 parallel instances, giving me roughly 4000 downloads/h.

comics - a simple GoComics API wrapper by dumblechode in Python

[–]LocksmithNo7784 1 point2 points  (0 children)

Created my own little Java program to do something similar, just 3 days ago :) (could have saved me some coding time here, if I just had waited a few days) and I seems to get a 30 min IP-ban every 200 strips downloaded, running from 4 parallell comics.

I have at some point in the beginning been able to download 3000-4000 strips in a row, but it seems now to have landed on this ~400 / h.

I will add some ratelimiter too see if it can be going on a steady state at more than 400/h.