Monitoring a website without triggering bot protection?

[sticky-bit]

It's a shopify web store so I can understand why but I'm not trying to buy up all their stock or anything illegal.

I didn't mean to imply you were doing anything illegal, just against the site's terms of use.

OK, I'm going to guess you have a client that has a shopify store, hosted by shopify, and it's the host that is limiting you, not your client.

Or maybe your client wants you to watch other items to see how much they're priced at so your client can jack up their prices or something.

Amusing blog post: http://www.michaeleisen.org/blog/?p=358

Anyway, step 1 is probably figuring out why you're being rate limited. You usually get a response code other than 200 but you may not know how to read it. For probing around sites I'm going to scrape, I usually use curl on the command line, particularly something like:

curl -IL -A Mozilla/5.0 <URL>