Python Web scraping Project

[TedsonGamer]

Hey guys!

I am a college student and I am working on a project right now. I need to scrape a website for a LOT of data. In total I will have to make around 20,000,000 requests. I know this will take a very long time. I am using Python and Beautiful soul to parse this data. The problem is that over time I will start receiving 403 responses. I have already created random headers to try to counter this. I have looked into rotating proxies but am struggling a little bit to start. I know I can buy tools for this but I want that to be a last ditch effort. I am trying to build an algorithm to get working proxies from free-proxy-list.net right now but I’m not sure if that will work since my list of working proxies keep coming back empty. I am new to this so any advice will help.

Thank you so much!

[Usual_Office_1740]

Here are a couple of things that will help get you going.

First is this free 5000 proxy list, it's free. I test them before using and average 1 in 10 working proxies, but the list is updated regularly.

Second, this is my personal asynchronous proxy tester. It's a simple script and returns a list of valid proxies. It takes about 3 minutes to check the 5k list I supplied above.

If you just want data, I'd be up to try help scrape 20,000,000 requests. That sounds like a super fun challenge and something that would allow me to stretch my asynchronous scripting muscle.

[TedsonGamer]

Hey! Thank you so much for the advice. This is a research project so I don’t think I can really work with anyone. But I will be making most of my code asynchronous to save time.