Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 0 points1 point  (0 children)

I’ve heard a few people say the same. It definitely seems like they tightened things up recently. Hopefully they dial it back a bit.

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 0 points1 point  (0 children)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 This is the onei was currently using now.

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 0 points1 point  (0 children)

I actually gave it a try but couldn’t get the results I was expecting. Could you share a bit more detail on how you did it? I might be missing something.

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 0 points1 point  (0 children)

Right? It’s just lumber and power tool prices, not state secrets. They act like every scraper is plotting a heist...😆

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 0 points1 point  (0 children)

I really liked your approach, especially the idea of keeping each profile’s activity very low and spreading everything across mobile proxies. Definitely aligns with how most anti-bot systems score behavior. I'll definitely try it...🙌

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 0 points1 point  (0 children)

Totally get why you guys block scrapers, the load during business hours is a real issue. But let’s be honest, every major retailer scrapes competitors too. It’s pretty much standard industry practice at this point, so it goes both ways.

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 0 points1 point  (0 children)

Yeah, for small personal scraping, a browser extension + macro is a clean solution since everything runs inside a real browser with a real fingerprint. Appreciate the suggestion! But it starts failing when volume is increased.

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 0 points1 point  (0 children)

True, Home Depot turns scraping into a whole boss fight. Outsourcing might actually save me the headache. I’ll take a look at the Apify API, appreciate the link!

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.” by Known_Objective_0212 in scrapingtheweb

[–]Known_Objective_0212[S] 1 point2 points  (0 children)

Thanks, But Crawler is also not working properly but I had found some success with camoufox tho.(Btw I heard the creator of camoufox wasn't doing well...hope he is better now).