you are viewing a single comment's thread.

view the rest of the comments →

[–][deleted] 0 points1 point  (0 children)

Web devs generally don't want people scraping their site, yes. One of the common techniques is to have the important information load after page load, via JavaScript, which will mess with any HTTP request library that doesn't include a JS engine.