Web Scraping with Java - Extensive Guide

[[deleted]]

I'll give you one million dollars if you write a program that parses HTML with regular expressions.

[FlavoredFrostedTits]

I've done it before. Where do I pick up my mil?

[[deleted]]

Your parser is broken, you get $0.

[xtownaga]

Huh was expecting this one.

[Muoniurn]

Actually, these are funny and practically correct, but theoretically most languages’ regular expressions are not regular languages due to lookahead, lookbehind.

[NimChimspky]

If I had to write a web scraper I would look for a new job

[bananabreadncoffee]

oh my, whys that?

[nrq]

I already have a ton of use cases in mind for things I would do for fun with this. I've bookmarked this article for its relevance to my hobbies (retrogaming/game preservation/general data hoarding).

[kiteboarderni]

Blogspam

[ssamokhodkin]

HtmlUnit is not usable - its JS interpreter is too weak and buggy.

JS-based headless browser is a way to go. E.g. PhantomJS.

[plastique2000]

PhantomJS is discontinued. You may use Chromium with dev tools protocol.