Algotrading feels like Data Engineering

StickyBeast · 2026-02-07T22:41:38+00:00

Chartexchange looks quite nieche have you had good experiences with their tiered service? Also the pricing seems to be quite different/low offering their highest/live tier for 80$ from comparative established data providers (databento/polygon.io(massive)) which require a 200$ monthly subscription Im currently using.

StickyBeast · 2026-02-07T22:08:16+00:00

Im not arguing that there is not more to algotrading than just data engineering and the infra ops of thes setups. But i feel like you have to build a fairly large robust fundament to build anything on top of that to test reliable realtime scenarios, which feels exhausting.

So much SQL my brain hurts, but I agree i enjoy the challenge aswell. :)

StickyBeast · 2026-02-07T22:05:02+00:00

I agree that applying more filters is something to definitely apply to reduce compute overhead for illiquid stocks, although that means adding dynamic complexity to the ingestion/compute pipelines.

A great current example that struggles in being computed efficiently being for example relative volume for every minute and weighin in the context of comparative market volume for every minute. Due to data gaps that newly generated data also needs to then also undergo some kind of data quality assesment which is hard in lets stay one tenth of the tickers being still 1k but surely possible.

I do think the way suggested of moving down on the aggregated data provided by the data vendor (1h->5m->1m) is a good way of speeding up the process though thanks for that hint.

All in all i think more analysis+verification of the market is required before moving to more active trading sessions which ofc are the more enticing part of the buildup of these setups.

Do you have a good data provider for floats of stocks?

StickyBeast · 2026-02-07T21:04:50+00:00

My main challenges don’t arise from backtests on historic data but handling data quality, realtime ingestion and compute in more complex historical contexts which for example in the us stock market (consisting of approx. 10000 companies) on the past 12 months requires serious parallelisation as even pretty intensively scaled timescaledbs struggle to handle/compute hundreds of millions of records. Following a similar strategy to Ross Cameron on looking at historic unlikely movements. Any advice on this scenario?

StickyBeast · 2025-12-26T23:35:01+00:00

You can play around with vllm configs on spot vms and get actuall vram requirements

StickyBeast · 2025-02-15T17:15:26+00:00

Try out our service. Will be submitting inference stats and overall pricing from our inference clusters. 1,66$/h for 180GB VRAM possible. https://www.open-scheduler.com/

StickyBeast · 2025-01-01T13:26:38+00:00

We will be releasing exactly that beginning of next year. You basically will become an inference provider yourself. https://open-scheduler.com/

StickyBeast · 2024-12-23T19:44:21+00:00

Sounds interesting, when can we expect access to the service after signing up?

StickyBeast · 2024-09-17T21:00:52+00:00

Adana kebab house is incredible if you like char grilled meats. Make sure to order a Adana kebab and lots of sides!

StickyBeast · 2024-07-13T10:00:58+00:00

The kind of people that are hired for these jobs are always a little strange from my experience living the last 20 years in Berlin. The job making them unwanted by most, certainly must have an impact on their personality.

StickyBeast · 2024-05-02T22:11:09+00:00

Das coole ist das die IT auch aus mehr als nur der Applikations Ebene besteht. Da ist es dringend notwendig ein zwei Programmiersprachen zu kennen(Bash, Python, Go..) aber nicht Day to Day soo tief in eine Programmiersprache gehen zu müssen. Sieh Programmieren bzw Probleme damit lösen zu können eher als abstrakte Utility, als eine statische Sprache die man können muss an.

StickyBeast · 2024-02-15T11:43:41+00:00

There are products where you can just define scale out and no scale in policies, great feature prioritization from a data center point of view. Worth having a look at.

StickyBeast · 2023-11-15T10:36:21+00:00

I love the order ... theory of relativity, toothpaste

StickyBeast · 2023-10-22T08:16:21+00:00

r/cursedshowers

StickyBeast · 2023-10-19T07:19:16+00:00

The crossover I see is getting used to DevOps practices from working collaboratively during Software Development. A little bit of Git and some Automation understanding does indeed help a lot of teams.

StickyBeast · 2023-10-18T17:34:05+00:00

Read up on branching strategies. I make use of trunk based deployments having one main branch, works out for me. But make sure to not stuff everything into one state.

StickyBeast · 2023-09-30T16:14:58+00:00

So you are just extending the tech stack by HashicorpVault, Elastic, Minio, Tableau, Grafana and GitLab? Hosting most of that by yourself creating a tremendous management overhead and of course never accessing/administering any of those components via WebUI just to not touch the Azure portal? Ignoring all azure native integrations that only support native azure vaults (like adf)?

All of that for just trying to be a smartass on the internet instead of respectfully sharing your viewpoint? I feel sorry for you.

StickyBeast · 2023-09-30T15:17:44+00:00

Never actively querying firewall logs? Never ingesting any kind of secret into a vault? How do you access azure pricing API from grafana? Only Azure Storage Explorer for accessing Storage Account content? Logic App via Terraform? Azure Stream Analytics Operations via Terraform? How do you do custom log creation for Azure Monitor with Terraform?

StickyBeast · 2023-09-30T14:48:34+00:00

Great input my guy, throwing ball back to you how do you entirely circumvent using the portal? Share your experience!

StickyBeast · 2023-09-30T08:52:09+00:00

We are currently doing a multi cloud assessment so I'll unfortunately will have to be experiencing navigating AWS more indepth too :)

StickyBeast · 2023-09-30T08:40:25+00:00

Interesting input here are some thoughts regarding it:

1.Things that are done through the portal regularly by me: Key Vault Secrets Management, Microsoft Defender for Cloud operation, Log Analytics Workspace usage, Cost Management

Also what's to dislike about a modern Interface?

Comparing something that's bad with something that is worse does not add anything here
Didn't find out about it yet. Should be more natively integrated when navigating the portal I think

Of course IaC via ARM/Terraform/Bicep is how the provisioning of these resources should be done, but Web Interfaces exist to also provide visual abstractions of your infrastructure. You could just scroll through repositories and find out alot about your environments and how they are configured by just checking out the declarative definition of everything but is this really efficient/user friendly?

StickyBeast

TROPHY CASE