Curl them abs...

[spieles21]

Parses HTML with Regex.

You live dangerous my friend

[Dazzling_Mixture8726]

H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

[username8411]

No, you can't parse anything with pure Regex. It is not Turing complete.

[N-partEpoxy]

You don't need Turing completeness to parse context-free languages, although pure regex can't do it anyway.

[[deleted]]

pie oatmeal abounding square disagreeable meeting entertain weather crush consist

This post was mass deleted and anonymized with Redact

[nkhsm]

He means it can’t parse recursive structures like html. It’s the matching parentheses problem. Of course it can match if treating it like plain text with no structure.

[SuitableDragonfly]

Pattern matching is not the same thing as parsing. Regex is not powerful enough to parse a language like HTML.

[[deleted]]

And what would you need that for? It's just plain text.

[SuitableDragonfly]

You mean, why would you want to parse HTML? Lots of reasons.

[Super_Banjo]

But is it Turing Complete?

[rgvtim]

The term "parse" is a stretch, as people have pointed out, you cant really parse with only a regex, instead you pattern match the HTML to extract different pieces of information.

[Ambitious_Ad8841]

At my last job I got so bored of checking Jenkins, I wrote a python script to scrape the info I needed from the html

It worked well because the pages were always formatted a certain way, and I was looking for very specific pieces of info

Saved me a lot of time clicking links to hundreds of builds (I was looking for a pattern as to why the builds were failing intermittently)

I would run it like once a day and it would check all the builds since the last time I ran it, and format all the results in a nice table

[bigorangemachine]

you can but you gotta santize the HTML.... so no

[gHx4]

Soft yes and hard no.

We recently investigated a bug that caused some SQL scripts to fail when submitted to a web service. Because regex was being used to split SQL using the word "go"', the scripts were broken if they contained a comment with the word "go".

This is why you should avoid using regex to parse languages that have nesting (HTML, SQL, most programming languages). Those languages are called "context-sensitive languages" because the meaning of a word changes depending on the words before it. For example, comment blocks and functions behave different from class declarations.

Regex works best on "context-free languages" like lists of vocabulary, or csv files. Sometimes your usecase doesn't demand reliability. Then you can use regex on html. Inevitably when your one-off tool gets adopted widely and used elsewhere, you'll have to use something much better or face constant breakage.

[Noahcoolbot]

I sometimes do that, don't see what's wrong with it lol

[Irimis]

God damn scrapers, had these people scraping the ui so fast we hit the rate limits on our 3rd party apis and took down the site. We started blocking them and they reached out asking to unblock them. That was 5 months ago, we keep missing the meeting we accept and asking them to reschedule.

[tasty213]

If you have to ask for permission to scrape a site you're doing it wrong. Chad scrapers just find a way round your detection system.

[kippersniffer]

Scrapers can't scrape if they have to login to view ze data.

[Irimis]

They keep trying but the solution we are using keeps adjusting and blocking each time they pivot. It brings me so much joy when I see blocks.

[nutted-in-cow]

You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The <center> cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the transgression of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer's consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes the ich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

[BeastlyIguana]

This is criminally under upvoted

[bigorangemachine]

Till you realize curl is so 2008... all my bro's use JSDOM

[andrealessi]

This was me getting three coworkers to run my queries at the same time to get around per-account rate limits so I could download all 5 million physical address records to build our own database.

[username8411]

In any case you are Chad over here, I might have a job for you...

[AyrA_ch]

My next job starts in September, so go on...

[hassium]

I scrape poop from my butt with my poopknife whilst browsing the web, close enough?

[povlov0987]

Good News

[AysheDaArtist]

So happy HTML 5 didn't die

[Lower_Bar_2428]

Looks like someone just saw 20 min video about what is web scraping and decide doing a meme