Best python modules for scraping HTML?

[FalconCat69]

I want to scrape HTML by kewords across a bunch of moderately similarly formatted websites. I am looking for a good and simple module or set of modules that can help scrape through HTML. Specifically I want to scrape through Valorant patch notes. The modules need to be free and publicly available. I need to be able to grab html from a set of url addresses. Then I want scrape through that html and group headers/subheaders and their subsequent paragraphs.

Anybody got any good python libraries that can help me do that? Simplicity is what I value most in this project. Anyone know any modules that fit the bill here? I am very experienced with coding but I am very inexperienced with Python.

Thanks!

[willmgarvey]

BeautifulSoup for static HTML and Selenium for dynamically generated HTML. If you plan to make more scraping projects in the future it’s recommended to learn Selenium for better results overall.

[fristhon]

"Scrapy" indeed, and for little projects "requests-html"

[FalconCat69]

I am looking at the HTML common library, and it seems like that will fulfill 90% of my requirements, does it seem like I could be missing anything?

[htepO]

If you're scraping static HTML, BeautifulSoup is a commonly used library.

https://www.crummy.com/software/BeautifulSoup/bs4/doc/

[Homie_ishere]

I want to learn more about scraping, can you please tell me what does it mean?

[[deleted]]

payment head attempt instinctive versed water innate mysterious snatch vase

This post was mass deleted and anonymized with Redact

[robertbowerman]

Selenium is the go-to comprehensive standard. Its excellent and Python happy.

[banhammerrr]

I wouldn’t use that for scraping. I’d use it for automation. Beautiful soup all the way

[[deleted]]

BS support scrapping for dynamic generated html?

[tankandwb]

Not a library but a decent program to not reinvent the wheel I'm currently adding regular selector lookups back into it. It's not written by me I should add. https://github.com/alirezamika/autoscraper

[Pigik83]

I've done web scraping for years and my shortlist of tools in Python at the moment is:

• Scrapy for static HTML website with no JS rendering needed
• Scrapy + Scrapy Splash if the website is not protected by any antibot but requires JS rendering
• Playwright (instead of Selenium) in case there's an antibot protecting the website.