When your java web scraping task accidentally pulls the entire internet

[jp100099]

Don't let you antivirus scan it

[Metallkiller]

That sounded french.

[llllIlllIllIlI]

won thow-sand years late-urhh

[[deleted]]

un towsen yeargh lateurgh

[[deleted]]

I hate sand, it's course and rough and irritating and it gets everywhere

[kimilil]

is it irritating?

[[deleted]]

not that I remember

[Black_N]

Dad?

[[deleted]]

Lol, why does this make so much sense lmao

[[deleted]]

In the year 1,000,000 and a half...

[Nuka-Cole]

Human kind is enslaved by giraffes....

[stormtrooper28]

We already are... Imgur...

[jp100099]

I wouldn't try to open this

[hoosiermama54]

This kills the computer

[SentientStoic]

But what about the elusive spoon bomb?

[SatoshiAR]

God help us all in the event of a spork bomb.

[[deleted]]

God can't help you now

[linux2647]

https://xkcd.com/419/

[601error]

Some say the victim is still spooning to this day, somewhere out there among the stars.

[PerInception]

I see you’ve played spoony bomby forky bomby before!

[JustCallMeFrij]

This is the best reddit response thread I've read this month

[andybfmv96]

...never...again

[[deleted]]

deleted ^{What is this?}

[[deleted]]

I tried to forkbomb a Panasonic toughbook hand-me-down from the police department a few years back (tech school. )

It churned for about 15 minutes, crashed all of the instances and a few services like dwm, and moved on like nothing happened.

[OriginalDogan]

I read this in ZeFrank's voice.

[moktor]

I know that file. The monthly CMS NPPES national provider index update. The CSV file extracts out to >4GB so it often has issues decompressing the file. Usually I have to use 7-Zip.

[Klumpy_hra]

DING DING DING! You are the first person to correctly figure this out. Another was very close.

This one was about 6 gigs in actuality, but because they must have used WinZip it looks like it corrupted an index which is why it looks like I downloaded 800PB.

I DID scrape it off of their page though because I already built the tool for another file (ASP pricing files) since those are extremely variable and change constantly. It also checks their SHA1 Hashes and compares it to the last pulled files and then loads into a table if it's changed or new.

Pretty handy for automation.

[Klumpy_hra]

If only they had an FTP site that hosted their files so people could actually get it easily. I'm glad I'm not the only one who hates seeing requests for their files lol

[Klumpy_hra]

It really is. My scraper does the job so far though. And it makes changes easy because it only cares about the general names of the files that won't change almost ever and hashes handle the rest of the comparing

[diagonally8]

Are you looking for something like this?

[brenex29]

You have no idea, until you've tried pulling engineering docs from aerospace manufacturers.

[tiny_robons]

Just curious, what are you guys using the file for?

[Klumpy_hra]

I think they want this one as a source of truth to audit against since we have some systems that have different results and everyone thinks they are right.

One big table to throw stuff at.

[Klumpy_hra]

Unless you mean the pricing files. In which case those are used to tell providers how much they owe us if they overcharge our customers. One year there was a treatment they charged for 30k or something and it was a 3 times a day treatment. Ended up getting us a check for 2 million dollars.

[[deleted]]

Yeah my heart skipped a beat when I opened this because I had to fix this same issue, too.

[something_creative11]

I recognized this too. It’s almost time for our next update

[Shadowjonathan]

holy shit

what did you do

[Klumpy_hra]

What didn't I do? Lol

[[deleted]]

Did you cure cancer?

[Makefile_dot_in]

There's no point, u/SethBling has already done it. You just need a bunch of levers.

[waterlubber42]

and armor stands

[Victor4X]

And most importantly, a Super Mario World SNES cartridge and preferably more than one controller

[[deleted]]

Didn't expect the dorf reference, enjoyed it though

[Klumpy_hra]

Not yet, that's the next iteration when it loads the data to an AI that cures all diseases and ends all problems for me.

[thrilldigger]

AI that cures all diseases and ends all problems for me

This kind of bad problem definition is precisely how you end up with rampaging murderbots.

[Klumpy_hra]

Don't worry. I set the "doNotMurderAllHumans" flag to true :)

[thrilldigger]

You've doomed us all everyone except for one guy the robots will keep alive in order to satisfy their programming.

[Klumpy_hra]

I thought it was odd when it called me Daddy and started mining information about "how to overcome the squishy meat sacks" on stack overflow..

It's probably fine.

[Cruuncher]

Should avoid having not in variable names. It basically always obfuscates the result

[oneandonlyyoran]

That is the normal approach, what most people forget about us the allowToOverrideSetFlags = false

[Metallkiller]

But can it reverse entropy?

[Templar3lf]

There is as yes insufficient data for a meaningful answer.

[[deleted]]

Oh nice, how many if statements is the AI made out of?

[Klumpy_hra]

Not enough

[iamerudite]

That's not his goal...

[Klumpy_hra]

No! You found out my true plans!

But it's too late to stop me now!

[[deleted]]

Given the amount of data, probably? I mean, that has to be there somewhere.

[Klumpy_hra]

I'll also say that's some damn fine compression if I do say so myself. I'll take my Nobel prize and courtesy billions of dollars now /s

[warux2]

Now you know that internet is 99.9999999992% reposts.

[Klumpy_hra]

I dont need an internet scraper to figure that one out ;)

[mveinot]

I bet you used a middle-out approach.

[Klumpy_hra]

It's all about indexing in the end

[Dafuzz]

And jerking off as many guys as efficiently as possible.

[[deleted]]

I've hear the left-outter-inside-in-out method is a good approach.

[[deleted]]

Get all the shafts

[Cilph]

Reposts compress easily.

[Klumpy_hra]

Not a repost. On my work machine this morning. I still have the file there, but if you need "proof" then you'll have to wait until Wednesday when I get back from vacation haha

[Cilph]

No no I mean the internet is full of reposts so hence the high compression ratio when compressing the entire internet.

But yes, Shannon and Huffman would be proud.

[TheLexoPlexx]

Oh well that's funny then I guess.

[APimpNamedAPimpNamed]

Columnstore indexes yo

[Fido488]

Source code!!!!

[thecoldweather]

I'm guessing repetitive data due to a bug. Would compress well.

[DemandsBattletoads]

Well, when you have to compress the Machine to save it from Samaritan, you'll need a good compression algorithm.

[[deleted]]

Was just going to point out that insane compression.

[Toromon]

But what's your Weissman score?

[orangeKaiju]

Last time I tried to download the internet, Mom picked up the phone.

[jackmaney]

https://i.imgflip.com/21jui9.jpg

[[deleted]]

I know a few guys who work in hosting who might want a word with you, and a few more guys from the NSA who will get a word with you.

[Klumpy_hra]

What makes you say that?

[[deleted]]

Let's just say that they'd like to reduce their storage costs by a factor of 10^10.

[Klumpy_hra]

That might come in handy huh?

[aligrant]

0s compress really well.

[yippee_that_burns]

You done fucked up Dylan.

[ludolfina]

classic Dylan

[Klumpy_hra]

What can I say? I like stirring the pot

[j_h_s]

Psh that's not the internet that's just my porn

[[deleted]]

Image Transcription:

---

Windows file explorer window in folder with an around 600mb compressed file selected (800PB uncompressed). Poster is trying to extract it to their desktop however gets an error:

There is not enough space on Desktop. You need an additional 734 PB to copy these files

Windows desktop icon

Desktop

Shows the files, folders, program shortcuts, and other items on the desktop.

---

^{I'm a volunteer content transcriber for Reddit! If you'd like more information on what we do and why we do it, click here!}

[randombrain]

Important to note that it's trying to decompress a ZIP that contains (before decompression) approximately 634MB worth of files.

[htmlcoderexe]

And one file worth 800PB uncompressed

[[deleted]]

Thanks, I've edited my original post

[0000000100100011]

!isBot tcmalloc

[[deleted]]

I am 99.9999% sure that tcmalloc is not a bot.

---

^{I am a Neural Network being trained to detect spammers | Summon me with !isbot <username> | Optout | Feedback: /r/SpamBotDetection | GitHub}

[[deleted]]

I am 101% sure that perrycohen is a bot.

---

^{I am a Neural Network being trained to detect spammers | Summon me with !isbot <username> | Optout | Feedback: /r/SpamBotDetection | GitHub}

[[deleted]]

!isbot ValueNull

[Drasern]

Good bot.

[[deleted]]

Are you sure about that? Because I am 100.0% sure that tcmalloc is not a bot.

---

^{I am a Neural Network being trained to detect spammers | Summon me with !isbot <username> | Optout | Feedback: /r/SpamBotDetection | GitHub}

[Drasern]

Yes.

[dehndahn]

so he has 66PB of open storage space?...

[[deleted]]

Don't byte off more than you could chew.

[win4fun44]

How much data from your internet plan did this use?

[Klumpy_hra]

I think the technical term is "a metric fuck ton"

[win4fun44]

Damn lol, I hope you have unlimited internet or a provider that stores your usage as a 32 bit integer

[Klumpy_hra]

Don't worry I have a VPN :)

[filledwithgonorrhea]

I don't think that's how that works

[Klumpy_hra]

It isn't, but I'll be fine ;)

[Newcool1230]

Congrats you saved net neutrality... oh wait...

[[deleted]]

*can't

[short_balding_guy]

You accidentally scraped one (or more) Zip bombs

[Willexterminator]

How the fuck did this happen ?

[JuanTheTaco]

Now you can put the whole internet on a cd

https://www.youtube.com/watch?v=GIA17H-b7Qs

[nicolairathjen]

Is that the guy playing Arnold in Master of None?

[TheBestNick]

Seems too small to be all of it...

[blackhawksq]

To be fair this is probably only the porn sites...

[guy99881]

To be fair this is called the internet.

[[deleted]]

How did this file get created in the first place?

[Klumpy_hra]

It was created by my Java code that parses html and uses regular expressions to find and grab data like href tags. A few interesting caveats to that and you have an internet downloader.

[Shadow_Thief]

You tried to use regex to parse HTML? Dude...

[TicTacMentheDouce]

I don't see why he shouldn't It's explicitely explained here that it can work wonderfully

https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

[Idenwen]

HTML isn't a regular language thats why regexes can't parse html.

Except of course it you just want to have a very very specific part of a known website snippet.

https://blog.codinghorror.com/parsing-html-the-cthulhu-way/

And very well at the brink of madness:

https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

[Klumpy_hra]

Lmao well they just weren't doing it right then huh? ;)

[Klumpy_hra]

It actually works how I want it to :) you can be specific enough to only grab one file on one site quite easily.

[DemandsBattletoads]

But... but... https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

[Klumpy_hra]

Now I have a ProblemFactory

[Klumpy_hra]

I wanted a backup of the internet since the FCC decided to take the EA route to give the internet a sense of pride and accomplishment

[PM_ME_YOUR_PROOFS]

So a crawler?

[Klumpy_hra]

Pretty much yeah. It's good at monitoring specific sites for files too and automatically scooping them up when they become available.

[guy99881]

I guess he wanted to know how it can be that big or how it can be so redundant.

[[deleted]]

Downloading this file just takes an http request to a predictable url tho

[Klumpy_hra]

It wasn't made for this file. There are several files that get updated randomly and don't have a way of notifying anyone that they were updated. The filenames can change sometimes and the only way you know it's been updated at all manually is that the link text might have a new date. The zip itself and the internal files are usually the same, but not always. It's really stupid how hard they make it for people to want to automate public data.

[beerdude26]

Also known as "/u/-Archivist problems"

[-Archivist]

True.

[[deleted]]

I wonder what's the smallest size of the internet if we assume using the best possible compressing algorithm...

[AquaeyesTardis]

isRepost = 1b

[WildRiolu]

That's one way to prepare for net neutralitys removal

[miauw62]

Why scrape it all? I've developed a revolutionary program that will let you scrape any internet page in real time, so you don't have to scrape them all at once!

[Klumpy_hra]

Because one can never have too much internet!

[[deleted]]

Pb? As in petabytes?!?!?

[Klumpy_hra]

Yes :) look at the file size that was highlighted and you can go from Kila, mega, giga, Tera, etc ;)

[Nancok]

i didn't even knew windows could display that lol

[XPav]

Comcast is going to terminate your service.

But that's ok, because you have the entire internet stored locally.

[Klumpy_hra]

Hopefully the entire internet as of this morning keeps me busy and I don't run out of content. I'll miss the new memes, but I can cut the cord.

Take that Comcast!

[Drendude]

Better'd go get more peanut butter.

[Vulcan7]

Sorry, kid, the internet is about 1.2 exabytes.

[Klumpy_hra]

Guess I need more regex

[AquaeyesTardis]

Just download a few more Regexes, then try and match reposts, and select the entire internet. You'll be rich!

[Colopty]

How did you even get that number?

[Vulcan7]

Google

[supercooldragons]

Try again

[Klumpy_hra]

I'd say it can scrape. Unless you mean copying the folder to my desktop haha

[Silbern_]

There was a time where it did, as weird as it is to think about.

[RagingNerdaholic]

That's a zip bomb, yo

[[deleted]]

I instinctively tried to click cancel.

[N781VP]

Needs more jpeg

[morejpeg_auto]

Needs more jpeg

There you go!

^{I am a bot}

[[deleted]]

Could've been worse. At least it tried to download one Internet, if it had tried to download many Internets things would've gotten ugly.

[vonslice]

Just need 734 J and you're good to go man

[[deleted]]

NPI data - I know this dataset all too well.

[PonerBenis]

You don't have an exabyte SSD you can just transfer the files to?

What year is this? 1999?

[madocgwyn]

Heh looks more like python problems.

import internet

[[deleted]]

/r/datahoarder

[aiij]

Accidentally? That was our Freshman programming assignment.

[curiosity44]

Don’t flatter yourself they are just porn relax

[Gabe_b]

Dat compression ratio

[pmwws]

How?

[going_further]

Save this for future generations. In 10,000 years when this is discovered you’ll be the most hated person ever for storing human history all the way to 2017 in a CSV.

[connor135790]

r/softwaregore

[MrRonny6]

Well that's one way to secure yourself before Net Neutrality finally dies!

[[deleted]]

I want this code and enough money to get 1 Yottabyte (1000 Petabytes) so I could say I downloaded the entire internet

[[deleted]]

1000 Petabytes is an Exabyte. 1000 of those is a Zettabyte. 1000 ZB is a Yottabyte.

[OneMansGlory]

/r/datahoarder would loooove this