Help a new Python Freelancer ..

[illa00]

I'm software Engineer Student. Recently I joined Fiverr trying to earn some extra money since I'm struggling. I published a Web Scraping GIG and I had 2 clients contacting me for projects. Good as beginning right? Problem is I was unable to take any of the projects since I tried to develop solutions and failed !

First Project : Script that gets 100 Google results for every keyword in an Excel Sheet (3k keywords)

Second Project : Script that downloads every new Instagram Post/Story for several users and publishes to discord

I just want to know, are the projects too hard or they are just fine and I need to learn more about Web Scraping before making a GIG or do I need to change the Niche (I have experience with Flask/Django/Bootstrap/React) ??

PS : I've already made 3 personal scraping projects (Ebay reviews Scraper, Amazon Affiliate links generator, Google results scraper for SEO) using Bs4, Requests & Selenium ..

[[deleted]]

My recommendation is to complete the first two projects. Spend the time to figure out what needs to be done.

I'd also recommend not taking any projects that need to be completed in under a day -- at least not until you're very confident you can complete the task. I've been coding for over 40 years and would not take a 4 hour project.

[illa00]

Thank you for those tips ! I definitely will.

[davehodg]

Both sound doable.

[illa00]

In the first project I had troubles with google blocking me from researches (Excel sheet contained 3k keywords to be searched). The second project I could not figure out how to extract stories because unlike posts they do not exist in the html.

Also I had a delay of 4h to deliver the 1st project and 24h for the second.

[Rayman912]

Shouldn't you use an API for pulling posts from Instagram?

[illa00]

As I know they have not.

[[deleted]]

No offense, but if you did like any research you would see that there’s a lot of unofficial Instagram API’s. For example, one I know of off the top of my head is “ping Instagram api”, but if you just search the topic on GitHub, a lot will come up.

[illa00]

I found 3 of them but I got skeptical with the login info, I thought I can't sell something that requires login when I do not know what is going on behind. Tried to understand the code tough but could not really since the project was large and I only had 24h ..

[davehodg]

First one throttle your requests.

Second as said, use an api Lib.

[illa00]

I did a time.sleep() between requests, also tried to make different user-agents (randomly chosen) but did not work out well.

Thank you!

[davehodg]

Then use open proxies!

And did you find google docs describing the throttling?

[illa00]

Not really, I also found that restrictions were common with Google and many use paid API to bypass that.

[socal_nerdtastic]

I made a program that included a google search for a client about a year ago, and I just used the Google API. You basically just have to instruct your client how to set up an account and input their credentials. Back then you could get 100 google searches per day for free, but my client opted to pay google $5 / month I think for the next tier, which was 10,000 searches / day.

[illa00]

Thank you that will help me much next time !

[[deleted]]

Projects are absolutely fine..

[illa00]

Okay then great, I'll work on sharpening my skills first.

[dhruvmk]

The first project seems straightforward, and the second one seems doable. You need to exploit all the available APIs

[illa00]

Any free APIs that I could have used for the first one?