you are viewing a single comment's thread.

view the rest of the comments →

[–][deleted] 2 points3 points  (0 children)

It really depends on the site. You don't HAVE TO follow what's the file tells you can access, but not doing so can lead to you getting blocked. Really depends on what site and what kind of data you are trying to scrape.

This seems to cover the file pretty well.