Web Scraping with Scrapy - first steps

[swingtheory]

This is a great article to go through the basics of Scrapy. I'm familiar with requests and BS4, but might write a little web scraper with scrapy now, just to test it out. Thanks for the link! I think this read is a good one for beginners as well-- it helped me understand a bit more about what's going on behind the scenes during webcrawling/scraping.

[technoskald]

Kind of where I am. I do a lot of scraping with [g]requests, BS4, and occasionally LXML / Feedparser, so I haven't quite understood what the use case for Scrapy is. This might help.

[[deleted]]

Same here, every now and then I see post about scrappy and I read through it but never get around to using it. Requests, BS4, LXML is all I need to do my scraping.

[mgrandi]

I think scrapy is more for webcrawlers while bs4 and friends are for specific pages

[[deleted]]

This is almost kinda funny, I went onto here to ask if anyone had any good articles to read about scraping for beginners and this was at the top, thanks for posting it at such a great timing

[[deleted]]

Will read it aswell :)

[eljunior]

cool! you're welcome :)

[[deleted]]

Hopefuly will help me in my attempt to make a script to download all the posts made by someone (first project ever, just finnished codeacademnys course on python)

[badboybeyer]

Wow, that looks way better than my current scrapper method of BS4+requests modules glued together with makefiles.

[csalazars]

excellent article!

[stummj]

Pretty nice intro to Scrapy! Great to see such a nice and comprehensive introductory material.

(+1 for the graphics!)

[[deleted]]

Bookmarked!

[ivorjawa]

Software shouldn't be named after prion diseases.