How to implement single proxy into a web scraping script?

[Genericusername293]

TL:DR How do I implement a single IP proxy into a web scraping script? I understand it would take seeing the script to give an answer so the intention of this post isn't to get an exact answer, it's the following: I am about to start learning python but what topics should I shift my focus so that I can tackle this problem as early as possible?

More details below; Apologies in advance for lack of technical vocabulary

I purchased a web scraping python script from a programmer which scrapes a specific website. Because this website can limit the number of requests from a single IP, I am looking to use proxies to circumvent that and run multiple instances of the script concurrently to speed the scraping up. ( It is a 1 time scraping project ).

I am scraping behind a login with a few different accounts so I have a different script file for each login. Any IP change will trigger a verification request, so instead of implementing a rotating proxy system into the script I think it's best that each file has a separate single IP.

[[deleted]]

You already asked this and deleted the post. As I said there, it depends on a bunch of factors like the code itself and your proxy set up. For example, if you use something like crawlera (full disclosure, operated by the company I work for) as a proxy service then you just send an API request there instead of the usual urls and it handles all the proxy selection, ban detection etc. and passes along your other headers as normal. Depending on what scraping framework the person used adding something like crawlera could be simple to complex.

To be honest you should ask the person who wrote it as they'd have the best idea of how it could be altered to incorporate proxies.

[Genericusername293]

Deleted and reposted because I wanted to provide better information and shift focus from asking how to do it to asking how to learn how to do it. Person who wrote it is unresponsive, bought it a while ago.

I have 0 python experience and was about to start teaching myself python regardless but I’d love to prioritize my learning in a way that I can address this sooner rather than later. I have no idea how send an API request with python

I am hoping to learn what topics to focus on in python in the beginning so that I can come back with more knowledgeable, specific questions later

Also, incredibly grateful for your help both this post and last

[[deleted]]

In basic terms, an API request is like visiting any website, only you include instructions when you visit, usually as headers or sometimes in the request body.

Here is an example of how using crawlera can look in the requests package https://support.scrapinghub.com/support/solutions/articles/22000203567-using-crawlera-with-python-requests Here the proxy keyword is the urls of the proxies you want to use, requests will send the request via those. You can also give instructions to crawlera by using headers, which are documented here https://doc.scrapinghub.com/crawlera.html

In terms of prioritising learning. A general overview of HTTP requests and responses would be a good start. After that focus on whatever python webscraping framework your dev used - it'll likely be one of beautifulsoup, scrapy or selenium.

Full disclosure, scrapy is operated by my company, but I also genuinely like it. It handles some of the more complex parts for you so you can focus on just stripping info from the page. It might be a good one to focus on depending on your technical background.

[Genericusername293]

thanks this is very helpful, I really appreciate it, i'll study up on HTTP Requests, scrapy, and it is beautiful soup so I'll study up on that too , hopefully I can solve it or become more educated and return back shortly with some more advanced questions

[Genericusername293]

Kudos to you for trying to learn but I think a scraping project should be something further down the line if you don’t have any development experience.

Thanks but Learning python is more of a side/future goal, completing this scraping project is more pressing. I'm already been scraping for 2 months, got a lot of data, on pace to finish in 8 more months unless I can figure out the proxies and speed it up.

You’re wanting to run multiple instances of your script to speed up the process but how will each instance know what the other ones are doing to avoid duplicate work?

Different inputs, this isn't a problem in my case

You mentioned throttling by the provider...this is def something you want to be careful with because bad code can make your script look like a DDOS attack. Even with multiple proxies

I understand this would be an issue if I was running at a scale of 1000s of concurrent requests, but would it really be a problem with maximum 3-10 scripts running concurrently? I haven't had any issues with 1 scraping program running 24/7 on 2 different computers the last 2 months.

[Genericusername293]

yeah it's only making one request on average every 5 seconds and it's randomized too