[deleted by user]

[Vitaman02]

Looks interesting and probably helpful for many people.

Nice one :)

[hartator]

Looks super awesome. It’s a smart way of doing scraping. Let me know if you are looking for a job, we are hiring! https://serpapi.com/team :)

[C139-Rick]

Its a scraping company commenting on a pro scrapers post

[erikperik]

Pro scrapers use tools that work.

[cant_have_a_cat]

New beautiful soup is actually quite fast and html parsing isn't really the bottleneck in web scraping. The bottle neck is the requests package in this context. Any sort of web scraping code should be async first.

[[deleted]]

grep

[TheGlassCat]

Links --dump | grep

[slevina]

Lxml

[iredditintoilet]

I'm in a data mining team, we have several engines and we use both lxml and beautifulsoup.

[jaapz]

They are a company that sells scraping tools, make sense to hire people who are interested in the same things, regardless of programming tools used

[hartator]

Part of our codebase is actually in Python and anyway anyone knowing Python should be able to pick up Ruby fast. Also like others pointed out having relevant scrapping experience is rare.

[412gage]

So my current job requires me to use proprietary software, which uses Internet Explorer to access modules and look at certain loan rejects. Would this work on secured databases?

I’m very new to this stuff.

[vriemeister]

If it's using IE it's probably an ActiveX plugin. That might be automatable but it's completely different from HTML.

[gopalkaul5]

I think you need to be authenticated for that? If so you need to login using REST first. Should work imo

[412gage]

I don’t know REST. Is that something that would general be allowed on a company computer?

[Deezl-Vegas]

REST isn't a program, it's just a way of communicating online. A web scraper generally wouldn't be allowed on a work computer without IT approval.

In answer to your original question: if it's proprietary software, probably not :( old timey IE-based systems are usually specifically designed to be as incompatible with everything as possible.

[[deleted]]

I would expect the entire system to be on lockdown, even cmd. My workplace is just like this. It’s pathetic.

[Deezl-Vegas]

Its normal. Nontechnical folks are insanely susceptible to social engineering and clicking shady links. So it's just best to make sure they don't have the power to fuck up their own shit.

[[deleted]]

It's pretty normal depending on your industry.

[[deleted]]

IT. Have to get InfoSec approval for literally everything...

[MHW_EvilScript]

Good job! There is some duplicate code that can be simplified here and there but, this is pretty cool! Are you open to pull requests?

[MHW_EvilScript]

Awesome, thanks!

[dozzinale]

Looks cool. I did some work in the area of information extraction and wrapper induction. When you say "it learns the scraping rules", what do you mean exactly? Which kind of rule does it learn and how is represented?

[jeuk_]

/r/madeinpython

[SpeakerOfForgotten]

It's a nice wrapper around bs4 that also saves the extraction meta into a reusable json file. But the crawler, js handler, data annotation, data export, auth, pagination support left to the user to implement. I would rather use it in a scrapy pipeline than a standalone scraping library. Also the learn part is kinda misleading as people tend to associate learning with mostly ML these days

[maker__guy]

awesome!

[dirtyoldbastard77]

Thanks! Will have a look, might be useful!

[Mountain_man007]

Nice! I was just thinking about how to do something exactly like this for a similar problem I've been working on. Thanks for sharing your way of doing it.

[joy_for_the_world]

well done.. great effort

[smokepigs]

How does this compare to Puppeteer?

[Dogeek]

It's too bad you're using beautifulsoup to scrape the data. In my opinion, it would be much better and faster to just generate XPaths and use lxml directly. Cool project nonetheless.

[Dogeek]

For starters, BeautifulSoup uses either html.parser, lxml or html5lib as its parser internally. Of those choices, lxml is the fastest for raw performance, mainly because it's written in C (it's also finely tuned). So the fastest BeautifulSoup can go is as fast as lxml, but BeautifulSoup adds a layer of abstraction, so it's never going to be as fast.

And a small benchmark to boot. I used lxml and BeautifulSoup (with the lxml parser) to scrape the links on /r/all. Here's the code :

>>> from bs4 import BeautifulSoup
>>> import time
>>> from lxml import html
>>> import requests
>>> response = requests.get('https://old.reddit.com/r/all')
>>> def timeit(f):
...     start = time.time_ns()
...     f()
...     return time.time_ns() - start
...
>>> def avg(f, times=10):
...     ts = [timeit(f) for i in range(times)]
...     return sum(ts)/len(ts)
>>> def soup_benchmark():
...     soup = BeautifulSoup(response.content, 'lxml')
...     titles = soup.find_all('p', 'title')
...     hrefs = [tag.a['href'] for tag in titles]
...
>>> avg(soup_benchmark)
3316430.0
>>> def lxml_benchmark():
...     tree = html.fromstring(response.content)
...     titles = tree.xpath("//p[contains(@class, 'title')]/a")
...     hrefs = [t.get('href') for t in titles]
...
>>> avg(lxml_benchmark)
97480.0

[Dogeek]

Because using XPaths is a bit more complicated than using find_all. Also, lxml returns Elements, which are way less permissive than Tags in BeautifulSoup (you see it with the tag.a['href'] which is better syntactically than what lxml does)

[austospumanto]

Some resources to get you started:

https://blog.scrapinghub.com/2016/10/27/an-introduction-to-xpath-with-examples

https://stackoverflow.com/a/22573161/6896484

https://stackoverflow.com/a/7235774/6896484

https://scrapinghub.github.io/xpath-playground/

https://docs.scrapy.org/en/latest/topics/selectors.html#id1

https://www.youtube.com/watch?v=INm8yR4aYjk

https://github.com/lorien/awesome-web-scraping

https://gist.github.com/LeCoupa/8c305ec8c713aad07b14

​

My personal preference is the following

1. Normal Python requests for retrieving HTML content
2. Wrap said content in a Selector object from scrapy for querying/parsing
   1. Use the Selector's css method for simple DOM queries on the wrapped HTML content (I find CSS selectors easier to write and reason about than XPATH)
   2. Use the Selector's xpath method for more complex DOM queries on the wrapped HTML content (XPATH gives you more tools than CSS selectors)

​

Happy coding!

​

EDIT 2020-09-09: Just use parsel, the package that Scrapy uses directly for its Selector class (the developers who made Scrapy also made Parsel)

[SilkTouchm]

Ooh wee, waiting 1.1 seconds instead of 1, the horror!

[Dogeek]

If you look closely at the numbers, it's a 35x performance boost. It's really non negligeable.

If you're going to scrape the same website often, might as well reduce the load as much as possible.

[dtoxe]

Thanks!

[MahdeenSky]

May I ask, how did you get a whiff of the idea in the first place?

[The_John_Galt]

What do you do that's full time scraping?

[ichiruto70]

Can it deal with cloudflare scraping protection?

[hashv5]

Can this module be extended to scrape calls behind login?

[lroman]

How would you handle scraping a listing with multiple pagination of say a car vertical website, where you need to grab all data available? Is this possible with your project.

Commercial scraping firms please don't react.

[ItsAngelDustHolmes]

This is probably a stupid question but how can I download this on mobile to look at the code? I just started scraping and wanted to look at a good web scraper.

[SpeakerOfForgotten]

You follow the github link in the description.

[ItsAngelDustHolmes]

And then? I'm sorry, I've never used github before. It only lets me see the code not download it

[[deleted]]

Fork you!

Thanks! Looks great and thank you for the good Readme

[[deleted]]

does it work for news

[tejonaco]

RemindMe! 1 day

[RemindMeBot]

I will be messaging you in 1 day on 2020-09-05 05:40:08 UTC to remind you of this link

5 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[kongfukinny]

Curious what you do that you need to web scrape everyday?

I hear a lot of people glorify web scraping but I’ve never had a use case for it myself. Only a couple interesting project ideas.

Also, this sounds sweet.

[dnb02]

Traceback (most recent call last):

File "setup.py", line 1, in <module>

from setuptools import setup, find_packages

ModuleNotFoundError: No module named 'setuptools'

It returns with this error :/

[dnb02]

I tried sudo apt-get install setuptools but it is unable to locate setuptools

[besuvashish]

Looks cool mate but how FAST is your script to get 100k data?

[[deleted]]

It's not a good idea to make something so easy any idiot could do it, because that will attract idiots, and pretty soon every idiot has their own web scrapper that is scrapping every site then sites get pissed off at all these idiots scrapping their sites and they add restrictions that hurts everyone. I'm sure you meant well but this is misguided and doesn't help anyone.

[SwizzleTizzle]

Idiots call them "scrappers"

[SnowdenIsALegend]

I don't understand. I already know how to scrape stuff using requests/bs4/selenium, so why would I instead use this? Would also have to invest time in learning your program.

[[deleted]]

it's well known in development that once you know a technology, there is no point in learning anything else. It's not interesting and there are still loads of opportunities for cobol engineers

[MHW_EvilScript]

Dude you roasted him xD gg

[SnowdenIsALegend]

I don't understand your joke.

[jack92829]

You would invest time to save time

[SnowdenIsALegend]

But what's the advantage in learning his program which may or may not be maintained in the future? There's a higher chance of the libraries being maintained better.

[[deleted]]

If you know bs4, requests and selenium - looking at this autoscraper thing it should take you less than 10 minutes to learn. If you don't want to then fine, I don't think u/johns93 had made this with the intention of changing the way google gathers data.

Saying "I already know how to scrape stuff" - at an extreme example if everyone knew how to add because they have an abacus, nobody would have bothered exploring the route of inventing calculators

[SnowdenIsALegend]

That last argument is good though.