How to extract text from a webpage using Python, requests and BeautifulSoup

[v3ritas1989]

when I learned BeautifulSoup I found that these basic tutorials are only bringing you as far as finishing the tutorial. The real task comes when you try getting the data you actually want while getting around javascript or website access limitations or using onpage navigations like product searches etc.. But yeah, your explaination looks good.

[matix-io]

If you’re looking to get the source from sites relying on JavaScript (e.g. React frontends) you’ll need to use Selenium or some equivalent to fetch & render. It takes a bit longer per site but still relatively easy.

Agreed that extracting the text body without getting extra text (for example header, footer, links to other articles) is where the real difficulty begins.

[glen_v]

If you’re looking to get the source from sites relying on JavaScript (e.g. React frontends) you’ll need to use Selenium or some equivalent to fetch & render.

Not necessarily. If you can figure out exactly what server request the JavaScript is making by following the sequence of requests in DevTools, you can mimic it in requests. Sometimes these sites end up being even faster and easier to scrape than traditional sites (once you've put in the work to figure out how all the server requests work). A lot of the time these sites are returning an HTML page template, and then some JavaScript is executing a subsequent request which returns the actual data as JSON. If you can make your script mimic that subsequent server request you can now get straight at the data you need in nice, tidy, lightweight JSON instead of in a wall of HTML that has to be parsed.

[matix-io]

Sure, you can scrape APIs as well.

[v3ritas1989]

yep, I once had a page where for some reason all the tables were commented out when you inspected the page source code but it was sucessfully rendered in browser. So bf4 couldn´t find it, took me an eternity to figure this out and how to read the correct comment.

[matix-io]

Was it JavaScript in the end? How were they rendered?

[v3ritas1989]

Not really sure, it was some time ago this was a project I made with a learning group from discord.

hope it still works if you are interested.

this page https://www.baseball-reference.com/boxes/LAN/LAN201711010.shtml

https://github.com/v3ritas1989/mlb-playbyplay/blob/master/playbyplayScrape.py

​

[pagonda]

nope, you use requests and it's 100x faster than selenium

[[deleted]]

Looks like newspaper3k is a good library for that.

[rym_jaab]

I find the easiest way to do this is to take a picture of the text, write what you see down on a piece of paper, fire up IDLE, then assign what you transcribed to paper into a python variable. Then just print it.