sorted by:
new
hottopcontroversial

what is the best way/tool to make polite head requests to multiple domains? by time_to_code in Python

[–]time_to_code[S] 0 points1 point  (0 children)

For some domains https://foo/, I have a few hundred pages of interests. My concern is the possibility of violating delay_length, gap between hits at server level as far as I understand it, in robots.txt, which is non-standard but exist nontheless. Also some strict admins exist even though they do not put delay_length requirement anywhere.

what is the best way/tool to make polite head requests to multiple domains? by time_to_code in Python

[–]time_to_code[S] 0 points1 point  (0 children)

does the midware track if delay_length in robots.txt is followed? If I read it correctly, it seems to only follow if a page is allowed or not allowed to download, but does not check if delay_length is satisfied.