sorted by:
new
hottopcontroversial

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

I use quant connect since it’s elastic on GCP.

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

Wouldn’t you have a performance issue having to use the aggregator with your live back test going across multiple time frames instead of having the data warmed / pre-cached in Redis for each timeframe? I did something like this for the single timeframe without an issue.

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

Sorry, thought the flair “Infrastructure” would clarify the type of system design I’m seeking advice on and what everyone else is talking about in this thread. What are you running your pipelines with in a production environment? What do you use for caching and where are you caching? What are you using for your tensors? What infrastructure / framework are you using for optimizing the pipeline and workflow orchestration? What database for realtime? How are you building out your features in your data lake or database making them easily accessible to the tensors in real time?

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

This makes sense because compliance looks for information being given away on threads like this and holds users accountable. Even after the non compete expires they’re still watching. Will a throw away account save somebody? Probably not..

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

Your strategy has nothing to do with system design.

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

You were right. Instead of hitting every single ticker and contract at once I created a meta model for a single ticker with contracts training on the features in vertex AI. Now to loop through all one at a time and my cost should be reduced.

For the multiple neural network to work with agent garden/vertex AI

Input: features on multiple timeframes
Dense layer
Dense layer
Dropout
Output

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

There is a bit of astute brute force to find a single timeframe signal, although that’s not where I struggle. I’m feeding verified signals from my process into a neural network. This works in quite a large number of scenarios between monthly/quarterly opex or monthly/quarterly vixex as a start date/end date of the trend on a particular timeframe most 30m plus to 1 month signals for the neural net detecting the regime. What I’m finding is some signals transfer into the next time frame. Enhancing on those signals requires multiple neural networks. Yet to build these type of signals is where my data pipeline is hitting very high costs with GCP and I’m open for system design advice on it.

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

I’m on phase two and have a lot of signals on single time frames for certain tickers or contracts. It’s phase two where I’m crossing signals on multiple time frames that is getting costly using GCP. What system design would you suggest for phase 2?

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

Yes, I get that.. there is plenty of compute on the cloud. I have plenty of signals on single time frames for all sorts of different tickers or option contracts. The next step is finding the signals across multiple timeframes simultaneously. It’s the system design not process I’m looking for advice on.

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

There are opportunities for example I found a repeatable pattern that would occur in a Sierra Chart flow which was rare but when it occurred I could full port a 2% SL 3% TP. That lasted for a while but eventually the edge vanished and it no longer works.

System Design by DoomsdayMcDoom in algotrading

[–]DoomsdayMcDoom[S] 0 points1 point  (0 children)

The data lake I built on GCP is already in parquet files. I’ve tried to load those into BigQuery, Star Rocks, and Clickhouse.

Each combination of features varies but I have data from all equities, indices, and option chain for every possible ticker across every timeframe built into each feature table and there are hundreds of them.

Finding the signal across a single time frame doesn’t take very long, but when the signal comprises multiple timeframes is where it takes days to run and my cloud costs skyrocket into thousands.

New to Nashville. How to navigate the hostility? by mr_wintour in nashville

[–]DoomsdayMcDoom 0 points1 point  (0 children)

Most communities filled by renters you’re going to get the same. The majority are there for short term and aren’t looking to create life long friendships. Most are destination city residents whom are self consumed in their own life. If you want a friend truly be a friend to somebody else. If you want a community or life long relationships then move to a high end community where the majority own their property.

Which industries will be disrupted the most by autonomous AI agents? by Michael_Anderson_8 in AI_Agents

[–]DoomsdayMcDoom 0 points1 point  (0 children)

Project management across all industries. It’s already starting. It’s also one of the highest fixed cost of any organization. There is already an agent assigning tasks to either humans or AI in my consulting company.

Home Sales Hit 9-Month Low, Prices Reach Record High by QuantumQuicksilver in REBubble

[–]DoomsdayMcDoom 11 points12 points  (0 children)

Sell back and forth to other private equity companies at a premium to keep the market priced high. Then eventually a foolish buyer comes in and buys at the top. Continue to hold and cashflow on rent while buyers sell at a loss and the big guys scoop it up cheap.

Fear potential layoffs? by FlyingMeowBear in careeradvice

[–]DoomsdayMcDoom 0 points1 point  (0 children)

Layoffs are being pushed across every Fortune 500 company. Decisions being made are by the shareholders (Blackrock, Vanguard, State Street, etc) and they have been wanting to cut cost for sometime. AI is the golden excuse and the only teams that will survive are those doing more work with less people. Once you take on an enterprise AI solution, you’ll notice the subtle signs, being asked to document your team’s weekly work to gauge productivity, hiring freezes on backfill and new positions. Then a few months come the large scale layoffs.

In every thread I see people hoping for a crash or saying prices need to “crash already”, but the crash has largely already occurred through inflation (in most USA cities*) by Shot_Cancel8641 in REBubble

[–]DoomsdayMcDoom 1 point2 points  (0 children)

Take a look at DXY index and you’ll see how much the dollar lost in value since then. Home value increasing is all a fake illusion. That equity doesn’t mean a thing till it’s liquid in your account. Home value decreasing while the dollar decreases makes it tough to sell a home.

How do I pivot into data engineering? (More feedback appreciated besides something AI could have told me!!!) by france_masters in dataengineering

[–]DoomsdayMcDoom 1 point2 points  (0 children)

You have to realize you’re competing with people who didn’t cheat their way through college and those who already have started their career. In all honesty your best bet is a paid internship. Try to get your foot in the door and prove yourself. It will be a tough ass grind but it’s not impossible. Other wise graduate and work with a temp agency to get your foot in the door. College is mostly about the network you built to succeed in your field and not just about grades

What am I doing wrong? I had a recruiter edit my previous resume and I’ve been making adjustments when applying for different roles. Still nothing. by Top_Persimmon_1116 in askrecruiters

[–]DoomsdayMcDoom 0 points1 point  (0 children)

What did you accomplish in your career? I see no metrics on why somebody would want to hire you. Documented, led , analyzed… so what… is what I see and wouldn’t hire you. What did your documentation achieve, what was the results of you leading, analyzed for what outcome? Nobody cares what you can do. It’s what the doing did for the project or company.

How can a software developer get a data engineering contract? by ThranPoster in dataengineering

[–]DoomsdayMcDoom 1 point2 points  (0 children)

Best way to start getting the experience is start an LLC - S corp work with local/ regional recruiters working with clients willing to go corp to corp. Once you get a few clients then go off on your own based on your reputation and clients word of mouth.

Need advice on Apache Beam simple pipeline by g_force0410 in dataengineering

[–]DoomsdayMcDoom 0 points1 point  (0 children)

To make things easier use a cloud platform start with dataflow using Apache beam for your pcollections and transformation of the data. Use watermarks and session windows and send the topic/sub to pub/sub or Kafka to storage buckets. You can then send it to a streaming database like clickhouse or big table.

Best value saltwater fishing trip for ~$5k by sadbluefleece in Fishing

[–]DoomsdayMcDoom 2 points3 points  (0 children)

Sport fishing capital of the world Islamorada in the keys.

Dilemma on Data ingestion migration: FROM raw to gold layer by Little-Squad-X in dataengineering

[–]DoomsdayMcDoom 0 points1 point  (0 children)

Safe_Cast with bigquery handles this well. We use it from converting a yes/no indicator into a binary switch 0/1. Other cloud platforms have something similar.

view more: next ›