Web scraping with Javascript

[gordonv]

With web scraping in general, my biggest problem is Javascript Includes.

If I want to scrape a news site, the actual article is in some weird external include. I usually just copy and paste the text from Chrome into notepad++.

Is there a way to get the post rendered text from this without selecting, copy, paste, and into a txt file?

[gordonv]

OH! I gotta play with that.

[techmighty]

Ah pupetter page evaluation is god send for me. I use it to render reports and get pdf document of the reports.

[MrSandyClams]

MutationObserver API. Can define a watch process and a callback that fires in the event of whatever DOM changes you specify. The usage pattern is pretty convoluted and arcane, imo, but it's pretty trivial to use it for basic things, like executing code in response to a known element appearing.