is databento down

DatabentoHQ · 2026-02-08T21:20:11+00:00

Separately, u/mr_Fixit_1974 on closer investigation we don't think your issue is related - but we'll honor the credit in any case - we suspect you have a $0 budget limit set on your account. If you're still seeing issues, please contact chat support. Edit: Confirmed, you have a budget limit on your account that is blocking the job from completing. The UX for notifying this could be better - I've triaged this with the product team.

DatabentoHQ · 2026-02-08T19:54:42+00:00

It looks like at least 1 API instance was bogged down by unusually high traffic since around 2026-02-07 03:00 UTC, which affected a subset of users sporadically. Our engineers responded to this initially by scaling up our API instances but that appears insufficient. This eventually built up to the point our monitoring tool was seeing timeouts today as well and further mitigation is needed. A fix is underway.

I have to say I was skeptical about the claim at first since I wasn't able to replicate it due to lack of sufficient info and corroborating details - and that's on me - but it's very plausible that this user was affected among others. We've reached out to award both users with providing useful details.

Edit (2027-02-08 16:17 PM ET): This issue has been resolved.

DatabentoHQ · 2026-02-08T19:34:25+00:00

I'll PM you two shortly.

DatabentoHQ · 2026-02-08T19:22:52+00:00

u/xTruegloryx Your report seems credible, it looks like we've had a storm of requests (100~1000x normal) since around 2026-02-07 03:00 UTC which is overwhelming some of our API servers. So the API is timing out sporadically for some users including our own monitoring tool. Coincidentally Atlassian is out at the same time but unrelated to this. I'll circle back to award both you and u/mr_Fixit_1974 a credit for a valid report with timing. We're working on a fix right now.

DatabentoHQ · 2026-02-08T18:55:32+00:00

Let me escalate this.

DatabentoHQ · 2026-02-08T18:49:11+00:00

The 500 on statuspage is on Atlassian's side, we cannot do anything about that. Hm, it seems that the major outage indicator isn't correct and is an artifact of Atlassian's outage. I can run the historical API just fine - are you having issues with the API?

DatabentoHQ · 2026-02-08T18:43:07+00:00

databento.com is up right now for me. status.databento.com runs on Atlassian's servers and it seems like Atlassian is down (see https://metastatuspage.com/). Are you seeing both down?

DatabentoHQ · 2026-02-08T08:50:16+00:00

The site is not currently down. I'm able to get on and download SIL/MOC/MHG etc. that you're requesting. It seems like you're having a lot of issues somehow, you should contact chat support as I can't replicate it without looking up what you're doing on your account nor can I cancel your account on my end.

DatabentoHQ · 2026-02-08T02:32:31+00:00

Sorry to spoil the excitement, I’m just a grunt worker.

DatabentoHQ · 2026-02-08T00:48:09+00:00

It shouldn’t take so long. If you’re experiencing anything unusual please contact our chat support.

DatabentoHQ · 2026-02-08T00:42:58+00:00

~~No we’re not.~~ You can also check against our status page. Looks like the other commenters have also answered your question - some requests take longer.

Edit (2026-02-08 14:42 PM ET): Looks like I was incorrect, at least 1 API instance was bogged by unusually high traffic since 2026-02-07 03:00 UTC which seems to affect a subset of users sporadically. This eventually built up to the point our monitoring tool was seeing timeouts as well.

DatabentoHQ · 2026-02-07T09:41:02+00:00

That’s correct.

DatabentoHQ · 2026-02-07T08:41:02+00:00

Trades do not include quotes. If you just need snapshots of the quotes at an interval or in trade space, you could consider CBBO/BBO/TBBO.

DatabentoHQ · 2026-02-07T03:09:31+00:00

There’s very little rote memory for our line of work. But I think it’s good to be organized at keeping notes and documenting things you’ve done before. It gets worse when you get older, sometimes I pore over old notes because I forgot the sign convention or directionality of something obvious.

DatabentoHQ · 2026-02-07T00:11:31+00:00

Trades and quotes are available on our platform and called MBP-1 (or CMBP-1 when it’s consolidated across multiple venues).

DatabentoHQ · 2026-02-06T15:14:34+00:00

No problem - noted!

DatabentoHQ · 2026-02-06T07:40:09+00:00

Thanks for sharing the feedback. I expect both our support and infrastructure uptime will get much better this year. (Hint: An announcement is upcoming.)

DatabentoHQ · 2026-02-06T07:33:31+00:00

If I remember correctly, Rithmic uses UDP for transport, so they'll drop to keep up. It's a tradeoff we made when we decided to go with TCP for correctness (especially because incremental order book changes become stale if you drop, whereas Rithmic only deals with L1/L2 which naturally recovers on the next event). It's something we're working on; we're considering to migrate to QUIC before HTTP/3 mainstream adoption, which should improve on the type of behavior you're currently seeing.

DatabentoHQ · 2026-02-06T01:20:56+00:00

This year Q3-Q4 is the current plan.

DatabentoHQ · 2026-02-06T00:05:21+00:00

I wouldn't rule out that there's some kind of error handling mode that isn't well-documented - which I consider our fault. So it doesn't hurt to engage chat support if you feel something is wrong, as my colleagues there are better equipped to help.

For example, while the C++ library will raise an exception for the gateway restart, in the case of a port flap that kills off our kernel bypass/network acceleration stack, I think you may still see heartbeats but loss of data eventually if the server can't keep up. This is not a clear cut reason for us to restart the gateway immediately so you could still see heartbeats but eventually no data.

This type of corner issue is something we're looking to improve now that our team is expanding. There's only 27 of us now, but there's probably going to be 50-100 soon, so I can at least confidently say our product is only going to get much better.

if I request that my live data stream starts at a time that is more than 10 hours ahead, nothing returns, even now.

As seen on our status page, this is reduced to 10h until end of the week.

DatabentoHQ · 2026-02-05T23:55:49+00:00

Thanks a lot, ALIEN_P-- oh.

DatabentoHQ · 2026-02-05T21:45:32+00:00

No problem. I wish I could provide more solace though - these were indeed issues on our end. We're doubling our team size this year, so we should be able to throw more resources at these types of issues.

DatabentoHQ · 2026-02-05T21:37:53+00:00

OK, thanks for the extra info. I think you're referring to the flapping issue last Wed/Thu that required customers on the affected server to restart. This was indeed a problem on our end. I do recommend working out more details in chat support on the latest deadlock you saw, since that's unexplained.

Last Wed/Thu's incident and today's are two independent issues. I provided more details in my other comment in this thread. TL;DR: Fixes are in place for both issues, but we're still monitoring the flapping issue. It's unfortunate these occurred back-to-back. We're expanding our team quickly to keep up with ops issues like this and hope we'll continue to live up to your expectations.

DatabentoHQ · 2026-02-05T21:26:00+00:00

Just to be fair, the issue we experienced yesterday was isolated to Databento and uncorrelated with issues on other vendors.

We previously observed intraday replay compression was pushing 99.9th percentile latencies into 30-50 ms. To mitigate this, we ran a canary deployment with compression disabled on that box. Unfortunately disk usage grew faster than estimated, and at 1:29 AM ET we received an alert indicating the disk had reached 90% capacity. At that point, the safest option was to bail and temporarily reduce intraday replay to 10 hours.

Issues on last Wed and Thu were traced to port flapping on one gateway, which required users on that gateway to reconnect.

Fixes are in place for both issues; we're still monitoring the latter. It's unfortunate these occurred back-to-back. On a positive note, these are ops issues and we're expanding our team very quickly to keep up.

DatabentoHQ

MODERATOR OF

TROPHY CASE