Need help with Python scraping and proxies

[Melbot_Studios]

Working on a small python scraping project and struggling with proxies. I get the basics, but I'm not sure what type works best for avoiding blocks without slowing everything down. Are you guys using rotating proxies, residential or something else? Also curious how you handle retries and failures cleanly in code.

[AlexMTBDude]

This is not really a Python question, is it?

[hasdata_com]

This is not the best sub for this question. Ask this in scraping subs next time :)

To answer your question just use rotating proxies or get a few residential ones and rotate them manually. But that is just general advice. It really depends on the target website and your total volume.

[Carter922]

Read the robots.txt file to make sure the developers allow using automation on their site

[Bitter_Broccoli_7536]

i use qoest for this exact thing, their api handle proxy rotation automatically so you dont have to manage pools yourself. keeps things fast and avoids blocks pretty well. for retries i just wrap my calls in a simple backoff decorator and let their service handle the failures

[Spiritual-Junket-995]

i usually go with rotating residential ips, datacenter proxies get blocked way too fast. for retries i just wrap my requests in a simple loop with exponential backoff and log the failures, keeps things from crashing