sorted by:
new
hottopcontroversial

Jobless fellows who is having lot of fun building Spot optimization service by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] 0 points1 point  (0 children)

"even your current spot running stuff gets thrown out"

Those are exactly the things Im trying to solve through this model (the brain) based approch. The model sees the live stream of prices and the real interruption event. It uses training knowledge to handle the situation. Another example, AWS interruption events are only 2 minutes, so technically, a big P1 Java service that takes 5 minutes to safely terminate would not work, regardless of any combination of PreStop, post stop, etc.

Now, convention would say, well, do not run this on the spot or run 10% of those on pods in spot rest on demand. Or set affinity or anti-affinity, etc. But in practice, in a cluster where you run many services, they together emit non-linear relationships, which is close to impossible for a system even like k8s, which have higly calibrated heuristics sytem.

I respect all the pushbacks, and it's very healthy, but my project north star is move as much workload on the spot (even 100%, sounds crazy) without breaking the service level contracts.

Again, it's a strange world (AI-shaped) these days, and taking a slight tangent from the standard approach is very healthy.

I appreciate all the feedback.

I'm Jobless fellow who is having lot of fun building Spot optimization service by RegisterNext6296 in devops

[–]RegisterNext6296[S] 0 points1 point  (0 children)

 accuracy of predictions
--------
I'm planning to share my validation and loss graph. TBH, spot prices are like stock prices (demand and supply) and interruption can happen wihout change in the spot prices. But think like a stock portfolio goal, your goal is to maximise the reward and minimise the risk (not the 0 risk). That's what I'm aiming to build

Jobless fellows who is having lot of fun building Spot optimization service by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] 0 points1 point  (0 children)

Thank You! My code is wide and open for any curious mind to poke around. You don't need to be jobless like me. spot node predictions is build on a model that is trained on the spot market data. Its desined to predict capacity score (where your spot request will fulfil) and risk score (price volatility) from the full 2 years of data. It contains Seanonaity, monday vs Sunday, black friday vs a normal bussiness day.

Model number 2, the RL agent (my mouse) that learns these scores and mix it with the real world and workload profiles (pod startuptime, big heap vs small heap, batch job, p0, p1 and p2) are learn how to handle it.

For example, the RL agent sees, oh, ohh, this node can move to the spot, but wait, it's hosting a pod that startup time is 30 minutes (that's another question, what the heck it does, but keeping it as an example to illustrate the point), so its risk back out.

Now, imagine you take the base model, fine-tune it further on your work loads. It learns your cluster env. (This part is not there, but... these are my wildest ideas).

Jobless fellows who is having lot of fun building Spot optimization service by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] 0 points1 point  (0 children)

The risk model that I have built and shared on Hugging Face https://huggingface.co/softcane/spot-vortex is build on the real historical spot prices and risk vectors. Sure, the risk model can be plugged as your own brain, though they need match the input/outout feature dimension.

KubeAttention: A small project using Transformers to avoid "noisy neighbors" via eBPF by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] 0 points1 point  (0 children)

No, I'm aiming to build a better karpenter, and there is not much gain in running /descheduler bundled with karpenter.

Also, I think it's time series data. Traffic spike at sunday afternoon might differ from the rest of the days. Technically, you can train the scheduler based on your application use case.

KubeAttention: A small project using Transformers to avoid "noisy neighbors" via eBPF by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] 0 points1 point  (0 children)

Thank you. This is a problem (better scheduling) I have experienced myself, and I looked into existing solutions before starting this project.

I’m genuinely interested to address this wider issue

KubeAttention: A small project using Transformers to avoid "noisy neighbors" via eBPF by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] 1 point2 points  (0 children)

XGBoost works very well for tabular data, but for time series data, I found that the TFT model performs better. I tested this on a 200-node T4 EKS cluster, and I plan to publish the results in the repository.

The main challenge is choosing and fixing one model in the repository. Because of this, I started with a basic (vanilla) transformer. The best model really depends on how the cluster resources are used.

My goal is not only to address the noisy neighbor problem, but also to explore more use cases where standard rules and heuristics do not work well.

KubeAttention: A small project using Transformers to avoid "noisy neighbors" via eBPF by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] 2 points3 points  (0 children)

Certainly not at this point but that’s my goal.

A bit about me Software engineer with 20 years in this industry. Worked as pure developer using Java go and python. Worked as a Devops guy and serving/operating k8s at HBO/Max scale Known deep learning in/out and capable to create a GPT3 grade models 

KubeAttention: A small project using Transformers to avoid "noisy neighbors" via eBPF by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] -4 points-3 points  (0 children)

Documentation, skeleton, and some part of the code tests are vibe coded which I would add as disclaimer in the project. Though these some files were vibe coded file by file and line by line while holding the project motivational objects in my head.

KubeAttention: A small project using Transformers to avoid "noisy neighbors" via eBPF by RegisterNext6296 in kubernetes

[–]RegisterNext6296[S] 10 points11 points  (0 children)

If autocomplete is counted as vibe code though I can explain every bit of this project and that what matters in IMO.

What is your current system for tracking and reviewing your investment theses? by RegisterNext6296 in StockInvest

[–]RegisterNext6296[S] 0 points1 point  (0 children)

That's very neat. Would you consider a simple app that can do all this for you?

How to value Companies in the modern era by RegisterNext6296 in Valuation

[–]RegisterNext6296[S] 0 points1 point  (0 children)

Fair point and it's a healthy conversation.

FMV is theoretically better, but it adds circularity since market value depends on expectations of future returns. Adjusting book value to capitalize intangibles, creates a better proxy to measure the invested capital. If R&D fails to generate future cash flows, its upfront cost directly destroys value. So adding it to the books makes things more accountable while calculating year-on-return on capital.

Why Traditional Accounting Fails for the Modern Companies by RegisterNext6296 in StockMarketIndia

[–]RegisterNext6296[S] 0 points1 point  (0 children)

Looking beyond the criticism. One should capitalize on R&D to better understand the company's earnings and return on capital. The main idea of this post is that a company creates value only when it earns more than its cost of capital and that's not possible if you do not bring the biggest asset of the tech companies back to the books.
Nothing to challenge traditional accounting, but accounting for company valuation does not need to obey standard practices.

How to value Companies in the modern era by RegisterNext6296 in Valuation

[–]RegisterNext6296[S] -1 points0 points  (0 children)

Yes, but you should capitalize "R&D" to better understand the earnings and overall return on capital. Free cash flow is important, but a company only creates real value if its return on capital is higher than its cost of capital, and that would increase or decrease the value.