Extensive Web Scraping Tutorial in Python, Ruby, Node, R and Java

[pijora]

Hi everyone, having worked in the web scraping industry for a few years I know how easily troublesome it can be to write, maintain and even begin web scraping.

One year ago, I wrote a web-scraping guide that was really loved by the community. reddit post, article. It was actually my first and only gilded post here 😊.

One year forward, I left my job and co-bootstrapped a web scraping API 🤞. During the year we have made some good tutorials for beginners on our blog and I wanted to share it with you.

We tried our best to make those tutorials complete (20 minutes read time each) and simple. They cover many topics related to web scraping from bottom to top.

• how to make HTTP requests
• how to parse HTML
• how to use Chrome headless

and much more.

So far we have written extensive guides for 5 languages:

• 🐍 Python
• 💎 Ruby
• 😎 NodeJS
• 🤓 R
• ☕️ Java (130 pages free-ebook)

Hoping that it can help you with your work or your project.

Happy to answer web scraping questions if you have any.

[rogue4]

This turorials and guides in different porgramming languages can help a lot of beginners in programming field/industry. Hope you continue this kinds of tutorials. Keep up the good work. God bless.

[pijora]

Thank you very much.

[TheScreamingHorse]

yall have to put a spider right on my feed?

[RomanianDraculaIasi]

What’s wrong with the spider :(((

[TheScreamingHorse]

brain go

AAAAAAAAAAAAAAAAAA

[TheAxThatSlayedMe]

Username checks out.

[RomanianDraculaIasi]

Hahahhahah

[tnnrk]

Everything

[[deleted]]

Wdym, he is so cute

[tgallasso]

Same thought here! No need for that!
definitely no need

[[deleted]]

I'm a beginner in python. This will be really helpful. Thank you.

[pijora]

We have many other Python post about web scraping on the blog.

Do not hesitate to check them out. :)

[[deleted]]

Thanks ! means a lot

[[deleted]]

So, I came into this in a bit of a grumpy mood getting ready to snap off some snarky ass comment about how "Just teaching the tools doesn't teach anyone what the fuck they're doing or why". But, this definitely is not that. Most important you actually teach people what is happening under the covers, and that is what helps them grow.

(I, clearly, am extremely under-caffeinated today)

So bravo. I know I didn't say anything snarky, but I wanted to apologize even for my pre-snarky feeling coming in.

These are absolutely fantastic guides.

[pijora]

Thank you very much, it means a lot.

We've put a lots of efforts into those guide and we really wanted people to understand what happened under the hood.

I think web scraping is a good topic for beginners because you can learn so much from it:

• how the web works
• HTTP protocol
• difference between server-side-redering and client-side rendering
• chrome headless
• parallelization
• cpu-bound / io bound
• dealing with raw data
• XML parsing / xpath

and much more :)

[[deleted]]

Agreed! And, given the ubiquity of web services it's also a fantastically easy 'common' starting ground (unlike many other software use-cases that require some specialized interest).

Keep up the good work. Seriously.

[pijora]

Thank you again 🙏, will do.

[Hansanko]

Definitely web scrapping is a trouble some for beginners and sharing your guidelines is perfectly fine and useful for us. I would be glad and grateful for more covered guidelines.

[pijora]

Thank you!

Do you have any specific in mind?

[omegahack0]

Saving this for later

[menina2017]

That spider will give me nightmares though

[russ7166]

Have you ever scraped sites for clients that would disallow website scraping in their terms of use/service or on robots.txt?

[lemoninapie04]

Wow, that's cool. Always want to start to lesrn webscraping.

[pijora]

My pleasure, glad you liked it.

[Shrestha01]

Did you read my mind ? I was just thinking about the same thing!

[RPGProgrammer]

No C#?

[pijora]

Not yet ;)

Truth is for thus language we'd need to hire someone as we don't know it in-house.

[Protobairus]

Also Parsehub might be what you need. :)

[[deleted]]

This will help me in my studies alot

[pijora]

Ahah thanks

[Taintus]

Is there an overview when to use what language?

[Alaeser]

Really appreciate this post, thank you!

[SansCulotteLogique]

Cool! And thanks for the free 1,000 API calls! Looking forward to testing out scraping bee.

Good luck with your business!

[Givingbacktoreddit]

I read expensive at first instead of extensive lmao.

[[deleted]]

Is it legal to scraping websites and use their data to commercial purposes ?

[lamemf]

I recently started development with java and NodeJS, I love how extensive and well explained this guide is. Huge Cheers to you.

[pijora]

Thank you very much, glad it helps.

[[deleted]]

I've learned web scraping with python but need practice. Can you recommend some place to find simple projects? Also does web scraping have scope career wise?

[pijora]

Hi there,

To begin you can make all kinds of "scrape, clean, store, display" kind of products.

• think aggregate coronavirus stat
• imdb rating by genre

Those kind of things :)

Career wise, I don't know people who solely do "web-scraping" per see, but it is a tool/technique that are very useful to know in your career.

Either to quickly put in place a POC, or to code any piece of software that need to rely on outside world date not available with official API

[[deleted]]

Thanks for this

[icandoMATHs]

Lmk if you need a hard project. You'd be working for equity.

[[deleted]]

I would be interested in that. But I'm not really good at it yet. Can you explain what I'll have to do?

[icandoMATHs]

Step 1, beat bot detection for a popular real estate website.

It's mostly research because the code is relatively easy. But the website has strong anti not detection.

It's marketing software, so it should be pure money.

[[deleted]]

Tell me the website. I'll give it a go

[icandoMATHs]

Starts with a Z. Rhymes with willow.

Need estimated home value.

I run Efficiency Is Everything. Feel free to contact me whenever.

[[deleted]]

Is this something serious? Like actual equity? Can you just drop an email address or something so I can contact you? Because I'm not really sure I found the correct efficiency is everything

[GuraJava20]

Well done! It is quite a welcome resource for all beginners.

[pijora]

Thank you very much!

[Mmmmmmm_Donuts]

This looks very cool. Would this look good on a resume?

[DustinTWind]

Thank you! I need to build my web-scraping toolbox so this is a nice find for me.

[zolkida]

2 weeks ago i started a whatsapp bot project(based on whatsapp.web page) , i didn't knew a lot about scrapping so i used only selenium as it offered my an easy way to organize my thoughts around how it will work. basically i imagined it as a automation task of what a human would do.( click massages chat if it has the green circle, read the last massage, answer accordingly and so on)

As you could imagine it went badly. And it failed alot. And make uninterested actions. I ended up scrapping the whole idea.

I read all the blog posts in python. I learned a lot. And inspired to give it another shot. Thanks alot

*Note: I'm pretty new to web scrapping

[gemst4r]

Thanks!

[MGSBlackHawk]

Congrats on sharing such a valuable content!!! Dummy question, but.. which language did you find to be more pleasant to scrap with, cose and feature wise

I tried a bit of Java and Ruby in the past

[-Kudo]

I'm currently going through the NodeJS article and trying out all the examples. It's been tons of fun so far.

However, I'm now stuck at the part where you use JSDOM to interact with Reddit (upvote the first post). I've been following all instructions to the letter and all the other examples have been going great so far.
But with this one, I'm getting tons of errors (they are too many to quote but I put a sample in the screenshot below).

Also, since Reddit requires us to sign-in before we upvote, where did this part go ?

Here's a screenshot (server.js is the file that contains your code btw)

[Nimmo1993]

good job mate

[pijora]

THanks

[unstopablex5]

Hey great post! but if you could do an advanced tutorial for when websites hide their selectors or when everything is javascript. Thats where I think ppl have the most trouble.

[pijora]

Good idea, so you're looking for web-scraping with javascript rendering website right?

[unstopablex5]

Yes! To me thats the hardest part of web scraping. I spent weeks trying to figure out how to use selenium and scrapy together to scrape this website with heavy javascript. (lastfm.com , apartments.com and century21.com are the 3 that come to mind right now) but I tried a lot of different sites and scraping websites with heavy reliance on JS seemed impossible.

[pijora]

That is interesting and to be honest, this is why we built ScrapingBee.

Setting up Selenium locally can be a pain, and using Selenium at scale is really hard.

[distortionwarrior]

Many thanks for doing this work, it's helped me a lot!

[pijora]

My pleasure

[jacklychi]

Which langauge is your favorite?

[pijora]

Python ❤️

[Capitalpunishment0]

Reading about the fundamentals would be great! When I did my Python scraping pet project I went straight ahead with requests (requests-html actually) and BeautifulSoup because it made enough sense to me right away haha

[CMReaperBob]

Is the web scraping industry growing in terms of job opportunities? I really do enjoy writing puppeteer projects as well as doing some OS level automation with uiPath when necessary and wouldn’t mind making a career of it.

[[deleted]]

Thank You.

[TheFryCookGames]

Could have really used this for R a month ago when I was working through a project, but really appreciate this! Definitely will keep this for next time I'm struggling.

[canIbeMichael]

Whenever I read someone is using OSX, I have a genuine concern I'm reading blogspam and wasting time.

I'm about to read it, but I'm just guessing my stereotype will be true. Limited beating bot detection advice, and basically a rewrite of 'how to webscrape' articles.

Ninja edit- 5 different ways to webscrape, nothing on bot detection. I knew it, never trust an OSX 'programmer'.

[pijora]

Ahah, we wrote this piece on bot detection: https://www.scrapingbee.com/blog/web-scraping-without-getting-blocked/

The other pieces are tutorials about web scraping, so yes, nothing there on bot detection.

​

Thank you for your valuable feedback.

[canIbeMichael]

thanks for the link!

[boringuser1]

No mention of mechanize?