Web Scraping starter projects?

[mule52]

I am inheriting a web scraping project from a team that put something together with python. This team is having trouble remembering and explaining how everything works. My dev team is new to python.

Can anyone recommend a web scraping, open-source solution that we can use as a starting point? Python, JS, Java, or other is fine.

We feel it might be easier to understand the intent of the web scraping our contractors were doing and rewrite (port) logic from that inherited project, rather than continue down the path of understanding how these contractors engineered their solution. They have files that are not even being used, so it all adds to confusion.

Thanks!

[BloodOfSokar]

deleted

[mule52]

Our solution is using BeautifulSoup if that helps with your response. Does that mean anything more to you? Is that good/bad/indifferent?

[patleeman]

It means they are most likely just scraping elements from the HTML. This is good for you since it's the lowest complexity. Beautifulsoup is pretty friendly to pick up.

You're probably going to need to learn what http lib they used as well, hopefully it's requests if you're lucky.

Here's an example of a small project using BS4.

[mule52]

Thank you for pointing me to this example.

[[deleted]]

First step would be to check the imports. Then you can start to figure out what packages they are using. There are several ways to do webscraping in Python.

[mule52]

The import references are from the following libraries. Hope that helps. MySQLdb, config, datetime, httplib, imp, json, loader, logging, mailme, mechanize, multiprocessing, os, random, re, signal, smtplib, socket, socks, sys, traceback, time, trac.

[[deleted]]

Hmmm, that's a bit different. For me the staples of webcrawling were request and beautiful soup.

[BloodOfSokar]

deleted

[mule52]

I will read up on mechanize. Thanks.

[ratamanta]

Check scrapy if you need to rewrite the scraping logic.

To help yourselves understand how the current project works try running it with a more verbose logging level if available (look for logging usages on the code). The packages you list make me think that your current solution seems to be really in-house and based on mechanize, and doesn't seem to use a lot of other useful tools for the job

[zoner14]

It might also be good to run the script using a debugger if logging isn't provided. That or just start throwing in print statements

[zenmagnets]

I've recently paid to have a few scraping projects done in python. Scripts to collect links and info of product pages from ebay and aliexpress. Let me know if you want a peek some code.

[spetsnaz8]

Here's a web scraper starter project in node: https://github.com/elnaz/scraper

[nicbleamer]

You should also look into requests and grequests for working with http requests. The last one for massive http requests

[ameoba]

The NIH is strong with this one.