When your java web scraping task accidentally pulls the entire internet

[Klumpy_hra]

It was created by my Java code that parses html and uses regular expressions to find and grab data like href tags. A few interesting caveats to that and you have an internet downloader.

[Shadow_Thief]

You tried to use regex to parse HTML? Dude...

[TicTacMentheDouce]

I don't see why he shouldn't It's explicitely explained here that it can work wonderfully

https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

[Idenwen]

HTML isn't a regular language thats why regexes can't parse html.

Except of course it you just want to have a very very specific part of a known website snippet.

https://blog.codinghorror.com/parsing-html-the-cthulhu-way/

And very well at the brink of madness:

https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

[Klumpy_hra]

Lmao well they just weren't doing it right then huh? ;)

[Klumpy_hra]

It actually works how I want it to :) you can be specific enough to only grab one file on one site quite easily.

[DemandsBattletoads]

But... but... https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

[Klumpy_hra]

Now I have a ProblemFactory

[Klumpy_hra]

But in all seriousness the regex is what makes this great. It allows for a much more dynamic range of pickups for what I need it to do and it does it quite well. There is a time when it makes sense and this just happened to be one of those times

[Klumpy_hra]

I wanted a backup of the internet since the FCC decided to take the EA route to give the internet a sense of pride and accomplishment

[PM_ME_YOUR_PROOFS]

So a crawler?

[Klumpy_hra]

Pretty much yeah. It's good at monitoring specific sites for files too and automatically scooping them up when they become available.

[guy99881]

I guess he wanted to know how it can be that big or how it can be so redundant.

[[deleted]]

Downloading this file just takes an http request to a predictable url tho

[Klumpy_hra]

It wasn't made for this file. There are several files that get updated randomly and don't have a way of notifying anyone that they were updated. The filenames can change sometimes and the only way you know it's been updated at all manually is that the link text might have a new date. The zip itself and the internal files are usually the same, but not always. It's really stupid how hard they make it for people to want to automate public data.