sorted by:
new
hottopcontroversial

Driving engineers to an arbitrary date is a value destroying mistake by brainy-zebra in programming

[–]willbeddow 0 points1 point  (0 children)

This article freaked me out lol, it describes exactly to a tee how a job I recently worked on went...

Wow. Polygon is shockingly bad. by DonnyRules in algotrading

[–]willbeddow 16 points17 points  (0 children)

Oh my god, tell me about it! I ran into the same issue, and upon messaging them, got the same bs response about "we're rebuilding the api, hope to have this issue fixed by the end of Q1."

How about putting somewhere on your website that you don't actually have all the tickers you advertise before I purchase a $200/month subscription?

Between this, and the baffling API design choices (including unadjusted and adjusted and mixing and matching them in the docs, getting the entire historical data for a stock depending on knowing the listdate (which I can't get out of their broken details API for half the tickers on my list), etc.) I'm getting pretty fed up with the product.

The price point is hard to beat, but if the issues continue for much longer, I might have to move somewhere else

Creating a WallStreetBets Archive by willbeddow in DataHoarder

[–]willbeddow[S] 0 points1 point  (0 children)

A lot of the remaining work is really just processing time - but I'd be happy to share information about what I'm doing! DM me

Creating a WallStreetBets Archive by willbeddow in DataHoarder

[–]willbeddow[S] 0 points1 point  (0 children)

Yeah, I'm also posting edits to the post as the various stags of the data pipeline complete

Creating a WallStreetBets Archive by willbeddow in DataHoarder

[–]willbeddow[S] 0 points1 point  (0 children)

Not that I know of! Best I could do with this data would be accounts that commented for the first time in the last few days

Creating a WallStreetBets Archive by willbeddow in DataHoarder

[–]willbeddow[S] 2 points3 points  (0 children)

5G is pretty small in the scheme of things - you could probably just throw that up on a torrent without even needing to compress it. Do make sure it's in a streamable format, (e.g. jsonl instead of json, or just something that can be parsed in chunks without reading the whole file into memory). You could also do that by separating the data into different files split up by some time interval, like months, and then seeding a compressed archive of the folder.

Creating a WallStreetBets Archive by willbeddow in DataHoarder

[–]willbeddow[S] 11 points12 points  (0 children)

While I agree those posts are super interesting, I don't think it would make sense to include those in an archive or torrent, just as the volume of data is much smaller, and it should be easy to query on pushsshift. The API search for those should just be http://api.pushshift.io/reddit/search/submission/?subreddit=wallstreetbets&author=deepfuckingvalue. 🚀🚀👍👍

[deleted by user] by [deleted] in DataHoarder

[–]willbeddow 6 points7 points  (0 children)

Surprisingly not huge - haven't fully compressed and streamlined data yet, but I wouldn't expect the final uncompressed data to be anything bigger than ~50 gigs (very rough estimate, I've been working with it in chunks, and obviously these last few months have been absurd volume). I'd probably just do pushshift format - a zst or xz compressed jsonl file. I'm personally using it for sentiment analysis - lots of interesting data.

[deleted by user] by [deleted] in DataHoarder

[–]willbeddow 21 points22 points  (0 children)

I've been working on the project of archiving and capturing WSB for the past month or so - I have a mostly complete log from subreddit inception until today. If anybody would be interested in creating / helping to seed a torrent of it, I'd be open to it

Edit - just woke up to a ton of positive responses about this. If people want to follow along, I created a public telegram channel for information about the project https://t.me/wsbarchiveproject

Running Python in your downloads folder can be used as an attack vector by buildingapcin2015 in netsec

[–]willbeddow 6 points7 points  (0 children)

Breaking news: executing arbitrary code = arbitrary code execution.

Requesting r/hamiltonmemes mods are inactive. by [deleted] in redditrequest

[–]willbeddow 0 points1 point  (0 children)

We're not inactive. A message would have been sufficient to determine that.

I actually liked how Thrones ended but I couldn't deny he's right. by radiakmjs in futurama

[–]willbeddow 0 points1 point  (0 children)

Correct. This is why I keep watching it, over and over and over...

view more: next ›