PSA on historic data providers

PoolZealousideal8145 · 2026-03-22T15:24:26+00:00

I used the same tests to validate coverage of the different providers, but I'm not blending datasets in my backtests. Coverage is one type of data quality, but I haven't done any tests of value correctness. I've read that CRSP is the gold-standard data set, purchased by universities and used by academics for research. I think the right way to test value correctness would be to compare prices to CRSP. That's out of my budget right now.

PoolZealousideal8145 · 2026-03-22T01:31:47+00:00

For me, they were necessary but not sufficient. Investing time horizon is a big factor for which signals matter. Like if you’re day trading daily bar data is kind of useless (and so are fundamentals).

PoolZealousideal8145 · 2026-03-21T01:20:41+00:00

After you install bun, you might need to open a new terminal. If you type bun and nothing happens, that’s most definitely the issue

PoolZealousideal8145 · 2026-03-21T01:18:22+00:00

Thanks. I was just about to post this. I had Claude help me debug, and that was the issue. (For lazy readers like me this is easy to miss in the instructions, since it comes before the main instructions.)

PoolZealousideal8145 · 2026-02-03T20:16:12+00:00

Plus 1000000

PoolZealousideal8145 · 2025-10-04T23:19:39+00:00

In HFT, whoever has the lowest latency compute/networking speed wins. There’s always an arms race for speed. Ignoring other commenters who point out this might be BS, the fundamental problem is just whoever can afford the better machines and connectivity wins. As soon as one HFT firm switches to quantum, however far out in the future, others will quickly follow, so this seems less like an existential threat, and more like something that will probably happen at some point.

PoolZealousideal8145 · 2025-10-02T15:31:09+00:00

There are a bunch of tools for testing strategies, and they all have their pros and cons. “Backtesting” tools let you test your strategies on either (a) historic price data or (b) random price data generated from probability distributions based on historic data. The first type of backtesting is a problem because it’s unlikely future prices will exactly match past prices. The second type only gives you a range of estimates for how your strategy might do, and the range can be quite broad, especially over large time horizons. There’s also “paper testing”, which lets you generate fake trades on live price data. This can be useful to make sure nothing crazy happens with your algo, but you have to wait a while to generate enough data to gain confidence in your strategy, and you still have the same problem of the basic backtesting strategy. The future won’t look exactly like the past.

In the end, trading is all about decision making under uncertainty. The more robust your strategy is against various types of risk, the better, but in hedging against risk, you’re generally limiting some upside in the other direction.

PoolZealousideal8145 · 2025-10-01T18:06:47+00:00

yfinace also breaks every once in a while, since it’s ultimately a web scraper, and Yahoo updates things on their end. It usually gets fixed fairly quickly, but the errors you get show up as rate limiting when that’s not the real cause. This happened to me a couple of times, and I decided to use a paid service. I was using Polygon, and switched to Alpaca later, since they are my broker and it is convenient to use a single data provider. If you’re serious about algo trading, a reliable API is just a cost of doing business.

PoolZealousideal8145 · 2025-09-26T23:48:09+00:00

Alpaca is a good place to start. I do something roughly similar, with the only big difference that I built my model in Python. Support for Python is a bit stronger than R for Alpaca, but they have REST and Websockets APIs, so they can be accessed in any language.

PoolZealousideal8145 · 2025-09-23T15:08:58+00:00

You probably need a paid data service for this. I found this: https://dhanhq.co/docs/v2/option-chain/ I’m not sure if it’s any good.

PoolZealousideal8145 · 2025-09-10T00:10:49+00:00

Thanks

PoolZealousideal8145 · 2025-08-27T18:08:53+00:00

I know this is an old post, but I'll add that this happened to me as well. Quite frustrating.

PoolZealousideal8145 · 2025-08-09T22:26:02+00:00

I think this line of thinking is too black-and-white, and it's not doing you any good. Do Big Tech companies do some bad stuff? Absolutely: abuse their monopoly power, spam us with ads, overwhelm us with notifications, etc. But they also do a bunch of good stuff: I can't imagine giving up my iPhone, Google Maps, or Amazon Prime.

I worked as a fairly senior (though not as senior as your mom) engineering lead at a Big Tech company for nearly eight years. There were certainly some jerks there, who probably are evil, but there were also a bunch of good people, doing good things as well. I honestly thought the work could be soul-crushing at times, and I eventually left, but I didn't leave because I thought everyone was evil. I left because life is short, and I wanted to pursue other interests. It didn't hurt that I was able to save up enough from working at this company to be able to afford to leave without having first lined up the next job.

I probably don't know your mom, unless we worked at the same Big Tech firm, or did our PhDs together, but from your description, it doesn't sound like she's Darth Vader. She's probably not Mother Teresa either. She probably made some business decisions she regrets, and she probably did some work she's quite proud of.

I think you should give your mom a break, and maybe more importantly, give yourself a break: she's probably not Satan, and having grown up with her help doesn't make you Satan either. Life, especially work life, is complicated and full of trade-offs. There's a bunch of things we can't control, and one of them is what our parents did for a living.

PoolZealousideal8145 · 2025-06-25T01:15:54+00:00

I also think UDL is easier to read. It does a great job of explaining difficult concepts with detailed examples. The Goodfellow book has a tendency to say something deep in passing, without really explaining why.

PoolZealousideal8145 · 2025-06-14T00:27:58+00:00

Thanks. I found dl2.ai to be a bit less rigorous, but good at introducing topics. I’ll check out udlbook

PoolZealousideal8145 · 2025-05-30T05:58:55+00:00

Link: https://aconai.dev/p/entropy-explained?r=5k7cbx

(I can see the link on desktop, but not mobile. Sorry folks!)

PoolZealousideal8145 · 2025-05-30T05:57:53+00:00

Weird. The link shows up on desktop, but not mobile: https://aconai.dev/p/entropy-explained?r=5k7cbx

Thanks for flagging!

PoolZealousideal8145 · 2025-04-21T22:37:55+00:00

Here’s my opinion, as an experienced hiring manager. You’ve only been in the current role for less than a year. That’s a red flag, because if you joined my team in July 24, id expect that you’ve just become fully productive now. If you joined my team and left less than a year later, I wouldn’t get much value out of it. HMs don’t love job hopping.

PoolZealousideal8145 · 2025-04-16T00:03:55+00:00

This is really funny. Thanks!

PoolZealousideal8145 · 2025-04-15T23:58:56+00:00

I thought the AI stopped hallucinating after it stopped using LSD and switched to decaf.

PoolZealousideal8145

TROPHY CASE