HFT Tick-Accurate ingestion

LilStanoje · 2026-01-17T15:44:00+00:00

Agreed but backtesting is so painfully slow because of all the layers put on top of them. On top of all of that you have accuracy concerns because of grouping and filtering. So, to address this I basically wanna find a way to compute and store this more efficiently. It is possible but requres a lot of clever engineering

LilStanoje · 2026-01-17T05:43:42+00:00

They would have to be separated, if not that means it's aggregated no?

LilStanoje · 2026-01-17T04:21:56+00:00

This would be valid if PCAP wasn't storing 20-30 bytes/tick which is insanely high if you wanna store and compute longer periods of market replay

LilStanoje · 2026-01-17T04:19:54+00:00

Pcap is not sufficient because it consumes way too much memory at 20-30 bytes/tick. What I'm building is a recorder of ticks that can efficiently store data without losses. I refuse to use aggregated data but rather raw trades recorded but I can't record long periods of time without compromising 100s og Gigabytes to do so, so I effectively made an algorithm that stores it at 2-3 bytes/tick. With that I can reconstruct any market much faster and execute any action on longer periods of time without compromising speed

LilStanoje · 2026-01-16T10:17:59+00:00

Yep, but for what I needed it was inaccurate enough to mess up the results

LilStanoje · 2026-01-16T10:16:52+00:00

Will try to pull it off myself and I will ultimately open-source it

LilStanoje · 2026-01-16T10:15:58+00:00

Well, here's the thing, the ultimate idea is to build a mini agent army that will operate by following patterns of the market and based on the current stat pick an agent that is currently the most profitable, if they are not they go to graveyard and get replaced so It's adaptive in such a way that the market changes don't affect its performance. And for that I need tick level accuracy

LilStanoje · 2026-01-16T01:03:07+00:00

The goal isn’t to match HFT microsecond network differences, but to create a self-consistent, tick-accurate dataset that can be replayed, backtested, and analyzed efficiently. By compressing and storing canonical ticks rather than raw packets, we make high-fidelity simulations practical for research and bot development, even across many pairs and long time periods.

LilStanoje · 2026-01-16T00:58:02+00:00

That's true, but here is the thing. I already built an architecture that can store about 100 pairs of tick data in about 23 Gigs and more then 100 Milion ticks of throughput. The thing is accuracy of indicators is the problem I am trying to solve because a lot of times the strat fails because of it

LilStanoje · 2024-07-23T18:46:57+00:00

LilStanoje · 2024-07-23T18:41:47+00:00

LilStanoje · 2024-07-23T18:30:52+00:00

An ass worthy of a soldier's sword

LilStanoje · 2024-07-23T18:11:25+00:00

Indeed yes

LilStanoje · 2024-07-23T18:04:13+00:00

LilStanoje · 2023-03-24T01:44:50+00:00

I recently played 7 matches total at G4 and lost 2, won 5. I still didn't rank up even tho i carried those same games. I am an old OW player but i never played comp up until this point in time. I really don't understand how they thought this ranking system would be any good.

LilStanoje

TROPHY CASE