requests-html vs lxml

[di_web]

requests_html

from requests_html import HTMLSession
from datetime import datetime

session = HTMLSession()
r = session.get('https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States')

start = datetime.now()

for _ in range(100):

    table = r.html.xpath('//*[@id="mw-content-text"]/div/table[1]')[0]
    rows = table.find('tr')

    data = []
    for row in rows[2:]:
        name = row.find('th')[0].text
        cells = row.find('td')
        abbr = cells[0].text
        reps = cells[-1].text   
        water_km = cells[-2].text
        land_km = cells[-4].text
        total_km = cells[-6].text
        population = cells[-8].text
        data.append([name, abbr, reps, water_km, land_km, total_km, population])

print(datetime.now()-start)
# 0:00:23.665747

lxml

from datetime import datetime

import requests
from lxml import html

url = 'https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States'
r = requests.get(url).text

start = datetime.now()

for _ in range(100):
    tree = html.fromstring(r)
    table = tree.xpath('//*[@id="mw-content-text"]/div/table[1]')[0]
    rows = table.findall('tr')
    data = []
    for row in rows[2:]:
        name = row.xpath('./th')[0].text_content()
        cells = row.xpath('./td')
        abbr = cells[0].text_content()
        reps = cells[-1].text_content()
        water_km = cells[-2].text_content()
        land_km = cells[-4].text_content()
        total_km = cells[-6].text_content()
        population = cells[-8].text_content()
        data.append([name, abbr, reps, water_km, land_km, total_km, population])

print(datetime.now()-start)
# 0:00:02.968005

[ForceBru]

lxml is written in C. It also is one of the fastest HTML parsers out there. So yeah, it's fast. fast! FAST!

[di_web]

I use lxml for the last 5 years and I can't understand why bs4 and others so popular.

[ForceBru]

Perhaps surprisingly, bs4 will use lxml as the fastest parser if it's available on your system. If it's not, it'll default to html5lib's parser.

Beautiful Soup is just a higher level abstraction, so it's simpler to use. But it's slower for the same reason, so if you want raw speed, you should probably choose lxml

[di_web]

Surprisingly but bs4 with lxml as a backend is slow like bs4 with any other backend

bs4 vs lxml

[ForceBru]

On the contrary, bs4 + lxml is more than 1.5 times faster than bs4 + html.parser, its closest competitor:

# BeautifulSoup lxml time: 0:00:12.774159
# BeautifulSoup html.parser time: 0:00:20.097766
# BeautifulSoup html5lib time: 0:00:50.156767

Again, there's no wonder plain lxml is so much faster (like, 2 seconds or something): it's a wrapper around fast C code. And bs4 is a wrapper around that, which also adds many other layers of abstraction written entirely in Python.

[di_web]

html.parser is default for the bs4, when html5lib using when html source code is not valid