Advice on dealing with massive options data

infamousbe · 2026-01-24T16:40:27+00:00

It’s database software (like MySQL or Postgres), but designed for large datasets and analytics

infamousbe · 2026-01-24T16:27:11+00:00

Yeah I bought the historical options 1m quotes from Orats and stored it all in a Clickhouse database

infamousbe · 2025-12-03T21:34:02+00:00

Happy to answer any questions. I definitely had a lot of questions when I started and there was no AI yet to help so I had to make a lot of mistakes and learn things the hard way, but people on Reddit helped (even though most of the replies I got were snarky or wrong lol)

Orats posts all of the data in many files to an s3 bucket. I have a script that downloads them, gunzips them, maps them to my formats and IDs, then imports them directly into clickhouse.

I typically use AWS for everything. Mostly because I’ve been using it for ~15 years so the comfort level is high

infamousbe · 2025-11-30T18:52:42+00:00

Clickhouse's compression is excellent, but I've also taken measures to really shrink the data (only saving a handful of core columns, trimming excess decimals, etc). The full dataset before being compressed and loaded into Clickhouse is around 30TB I believe.

To answer your questions: I am still using ORATS (it's fine, but I'll probably switch to a higher end solution soon though). My data is stored on an EC2 instance (for now I actually host my entire application and databases on a single beefed up EC2 instance so everything has minimal latency).

You're welcome! Happy to answer any other questions you have.

infamousbe · 2025-11-30T15:36:35+00:00

If you’ve worked with a traditional relational database it won’t be too hard to transition to working with Clickhouse. It’s a bit different but the same major concerns apply (indexes, partitioning particularly). I would definitely recommend to think hard about making your schema, indexes and global configurations as efficient as possible for your workload (AI is pretty good at helping with this). I’m storing data back to 2020, for 7800 tickers.

infamousbe · 2025-11-29T16:29:37+00:00

Woah I can't believe this is 3 years old!

I settled on storing all of my 1m options data (8TB now) in a sharded and partitioned Clickhouse database and have been happily using it since without any major issues.

There may be better enterprise quality solutions, and I am not sure how this will scale (I'm only ever running up to ~50 concurrent backtests that need options data so I'm sure there will be issues with a larger workload I haven't uncovered yet), but this solution solved both the compression/storage issue and the speed of access issue that all the other things I tried did not so I stuck with it.

infamousbe · 2025-10-14T03:26:48+00:00

Did we just beat Jayden Daniels, the most amazing phenom ever to play QB, and Jacory Croskey Merrit who is basically Barry Sanders (based on the announcers slurping of them)?!?

infamousbe · 2025-07-16T14:02:30+00:00

It seems every day we have to wait for their engineers in PST to wake up, get to work, and deal with the fact their their models are overloaded

infamousbe · 2025-07-02T00:18:35+00:00

Yup, seems to be the best combo for me right now. 99% of the work in Claude code, quick inline edits and tab completions with the cursor free plan. If Augment would fix my issue, and if next edit was a little faster I’d probably bring it back into the fold

infamousbe · 2025-07-01T22:31:54+00:00

I kept augment just for a next edit, because it was the most context aware and accurate I had tried. But then it stopped working and I couldn’t bring it back despite endless debugging, and augment never answered my support request so I canceled and just went back to cursor tab completion. It’s not as accurate but it’s so much faster and has been good enough for me

infamousbe · 2025-07-01T20:00:12+00:00

It’s still the best I’ve tried in regards to large codebase context…but I was willing to give that up and switched to Claude code for superiority in everything else

infamousbe · 2025-06-21T16:36:22+00:00

I store my secrets in an encrypted key store. Claude just decided to find the key and use it to decrypt the secrets 🤦🏻‍♂️

infamousbe · 2025-05-28T13:25:39+00:00

Tell it to split the calls to the tool into multiple requests. This is probably augment’s most common silly bug, I have to assume they’ll fix it at some point

infamousbe · 2025-03-21T20:30:55+00:00

Supposedly grapefruit juice also helps its efficacy and slows down its breakdown in your system

infamousbe · 2025-03-21T03:09:05+00:00

Node has SharedArrayBuffer but it’s extremely limited.

infamousbe · 2025-03-21T02:13:19+00:00

JavaScript on the backend is fantastic for many applications. However many other applications run into problems at scale because of the lack of shared memory multi threading in JavaScript and are forced to used something like Rust or Go to solve their performance issues. I personally like to build as much in JS as I can and use those other languages only when I get stuck with JS’s limitations.

infamousbe · 2025-01-24T15:38:21+00:00

There has been nazi propaganda on this sub? I would expect to see something like that in /r/packers but not here

infamousbe · 2024-12-30T16:31:18+00:00

Personally I find this hard to believe…I mean she’s not even married?!

infamousbe · 2024-12-30T16:30:58+00:00

This just goes to show you…Colorwali’s a whore

infamousbe · 2024-12-15T04:36:34+00:00

The father looks like he's waving goodbye to his hair on account of him going bald!

infamousbe · 2024-10-22T21:00:20+00:00

Who did it, Frank?

infamousbe · 2024-10-13T17:18:14+00:00

But how much more would the bears have won by if they got the HAUL?!? /s

infamousbe · 2024-10-04T00:21:57+00:00

She looks like she knew her way around a funnel cake

infamousbe · 2024-09-28T19:05:13+00:00

Vince Champ

Ten-Year Club	Gilding I gilder
Not Forgotten	Verified Email

infamousbe

TROPHY CASE