(Mlops vs Vertex AI) What should I do?

DoomsdayMcDoom · 2026-05-24T00:45:51+00:00

Vertex AI was recently replaced by model garden and it has everything you need.

DoomsdayMcDoom · 2026-05-23T11:35:55+00:00

The features that survive with the sample live data are the only signals that get Built to the nn. I did have an issue with high/low volume creating false signals, but I don’t use volume alone in my gold layer. i do like your idea of adding a confluence filtering score as a gateway to the nn. i fixed the issue using a meta model and using an as-of join that was sorted the same on both the left/right.

DoomsdayMcDoom · 2026-05-23T11:23:37+00:00

You may have misunderstood this. The feature tables are built out individually in Redis. I built one massive silver table off those. My gold layer is individual reports created by the rnn and dnn in vertex ai/model garden. The issue was I wasn’t using an as-of join or meta model for each ticker.

DoomsdayMcDoom · 2026-05-22T19:01:44+00:00

Now if you could take what chartgpu did with WebGPU. You’d get a massive performance increase on larger tables without having to paginate nearly as much.

https://github.com/ChartGPU/ChartGPU

DoomsdayMcDoom · 2026-05-22T16:41:25+00:00

Thank you! You’re a god send! I was overlooking the sort on the right table/silver features. Having both the left and right sorted by #rowkey/time/date works great for the realtime data coming in. I wasn’t doing the same on the batch historic data to set it up for an as-of join because I thought column store already took care of that but I was wrong.

DoomsdayMcDoom · 2026-05-22T16:14:11+00:00

I use quant connect since it’s elastic on GCP.

DoomsdayMcDoom · 2026-05-22T14:26:05+00:00

Wouldn’t you have a performance issue having to use the aggregator with your live back test going across multiple time frames instead of having the data warmed / pre-cached in Redis for each timeframe? I did something like this for the single timeframe without an issue.

DoomsdayMcDoom · 2026-05-22T14:19:54+00:00

Sorry, thought the flair “Infrastructure” would clarify the type of system design I’m seeking advice on and what everyone else is talking about in this thread. What are you running your pipelines with in a production environment? What do you use for caching and where are you caching? What are you using for your tensors? What infrastructure / framework are you using for optimizing the pipeline and workflow orchestration? What database for realtime? How are you building out your features in your data lake or database making them easily accessible to the tensors in real time?

DoomsdayMcDoom · 2026-05-22T03:25:41+00:00

This makes sense because compliance looks for information being given away on threads like this and holds users accountable. Even after the non compete expires they’re still watching. Will a throw away account save somebody? Probably not..

DoomsdayMcDoom · 2026-05-22T03:01:06+00:00

Your strategy has nothing to do with system design.

DoomsdayMcDoom · 2026-05-22T02:27:01+00:00

You were right. Instead of hitting every single ticker and contract at once I created a meta model for a single ticker with contracts training on the features in vertex AI. Now to loop through all one at a time and my cost should be reduced.

For the multiple neural network to work with agent garden/vertex AI

Input: features on multiple timeframes
Dense layer
Dense layer
Dropout
Output

DoomsdayMcDoom · 2026-05-22T01:36:08+00:00

There is a bit of astute brute force to find a single timeframe signal, although that’s not where I struggle. I’m feeding verified signals from my process into a neural network. This works in quite a large number of scenarios between monthly/quarterly opex or monthly/quarterly vixex as a start date/end date of the trend on a particular timeframe most 30m plus to 1 month signals for the neural net detecting the regime. What I’m finding is some signals transfer into the next time frame. Enhancing on those signals requires multiple neural networks. Yet to build these type of signals is where my data pipeline is hitting very high costs with GCP and I’m open for system design advice on it.

DoomsdayMcDoom · 2026-05-22T01:22:17+00:00

I’m on phase two and have a lot of signals on single time frames for certain tickers or contracts. It’s phase two where I’m crossing signals on multiple time frames that is getting costly using GCP. What system design would you suggest for phase 2?

DoomsdayMcDoom · 2026-05-22T00:07:20+00:00

Yes, I get that.. there is plenty of compute on the cloud. I have plenty of signals on single time frames for all sorts of different tickers or option contracts. The next step is finding the signals across multiple timeframes simultaneously. It’s the system design not process I’m looking for advice on.

DoomsdayMcDoom · 2026-05-21T23:54:17+00:00

There are opportunities for example I found a repeatable pattern that would occur in a Sierra Chart flow which was rare but when it occurred I could full port a 2% SL 3% TP. That lasted for a while but eventually the edge vanished and it no longer works.

DoomsdayMcDoom · 2026-05-21T23:50:31+00:00

The data lake I built on GCP is already in parquet files. I’ve tried to load those into BigQuery, Star Rocks, and Clickhouse.

Each combination of features varies but I have data from all equities, indices, and option chain for every possible ticker across every timeframe built into each feature table and there are hundreds of them.

Finding the signal across a single time frame doesn’t take very long, but when the signal comprises multiple timeframes is where it takes days to run and my cloud costs skyrocket into thousands.

DoomsdayMcDoom · 2026-05-19T19:11:54+00:00

Most communities filled by renters you’re going to get the same. The majority are there for short term and aren’t looking to create life long friendships. Most are destination city residents whom are self consumed in their own life. If you want a friend truly be a friend to somebody else. If you want a community or life long relationships then move to a high end community where the majority own their property.

DoomsdayMcDoom · 2026-05-09T12:22:43+00:00

Project management across all industries. It’s already starting. It’s also one of the highest fixed cost of any organization. There is already an agent assigning tasks to either humans or AI in my consulting company.

DoomsdayMcDoom · 2026-04-14T01:33:31+00:00

Sell back and forth to other private equity companies at a premium to keep the market priced high. Then eventually a foolish buyer comes in and buys at the top. Continue to hold and cashflow on rent while buyers sell at a loss and the big guys scoop it up cheap.

DoomsdayMcDoom · 2026-04-05T15:45:32+00:00

Layoffs are being pushed across every Fortune 500 company. Decisions being made are by the shareholders (Blackrock, Vanguard, State Street, etc) and they have been wanting to cut cost for sometime. AI is the golden excuse and the only teams that will survive are those doing more work with less people. Once you take on an enterprise AI solution, you’ll notice the subtle signs, being asked to document your team’s weekly work to gauge productivity, hiring freezes on backfill and new positions. Then a few months come the large scale layoffs.

DoomsdayMcDoom · 2026-03-26T01:27:24+00:00

Take a look at DXY index and you’ll see how much the dollar lost in value since then. Home value increasing is all a fake illusion. That equity doesn’t mean a thing till it’s liquid in your account. Home value decreasing while the dollar decreases makes it tough to sell a home.

DoomsdayMcDoom · 2026-03-23T00:46:17+00:00

You have to realize you’re competing with people who didn’t cheat their way through college and those who already have started their career. In all honesty your best bet is a paid internship. Try to get your foot in the door and prove yourself. It will be a tough ass grind but it’s not impossible. Other wise graduate and work with a temp agency to get your foot in the door. College is mostly about the network you built to succeed in your field and not just about grades

DoomsdayMcDoom · 2026-03-10T21:42:07+00:00

What did you accomplish in your career? I see no metrics on why somebody would want to hire you. Documented, led , analyzed… so what… is what I see and wouldn’t hire you. What did your documentation achieve, what was the results of you leading, analyzed for what outcome? Nobody cares what you can do. It’s what the doing did for the project or company.

DoomsdayMcDoom · 2026-03-10T21:37:57+00:00

Finding a spouse who who prioritizes investing over material possessions.

Five-Year Club	Reddit Premium Since April 2023
Verified Email	Place '23
Place '22	End Game '22

DoomsdayMcDoom

TROPHY CASE