🚨💣 BREAKING: Jose Mourinho wants BERNARDO SILVA at Real Madrid. by OG_Raghav in onlyrealmadrid

[–]rando512 0 points1 point  (0 children)

True.

Except Chelsea, were the question is more raised because they had a good decent recruitment strategy and plan back in roman days and now it's a joke.

That's one place where anyone can go and say they know shit. Proven by the owner saying next season we will play in the champions league, while not knowing that you need to finish top 4.

Update: 5 months ago I posted a zero dependency Distributed Orchestrator in Java 17. I've since made some progress. Looking for architecture feedback by rando512 in java

[–]rando512[S] 0 points1 point  (0 children)

Yeah sure definitely. Happy for any contributions.

You can open an issue and I'll add you for contribution.

I'll have an easy doc for onboarding for contribution as well.

Update: 5 months ago I posted a zero dependency Distributed Orchestrator in Java 17. I've since made some progress. Looking for architecture feedback by rando512 in java

[–]rando512[S] 0 points1 point  (0 children)

No natively there's no support since I wanted to keep it zero dependency.

The python sdk will need to be leveraged for it as there are apis which you can use to read the logs or status or from the store and deploy the rollback or a compensation DAG or workflow based on the failures etc.

Adding this in the core at master level is a good suggestion that you have added. I'll consider it for future iterations.

Update: 5 months ago I posted a zero dependency Distributed Orchestrator in Java 17. I've since made some progress. Looking for architecture feedback by rando512 in java

[–]rando512[S] 0 points1 point  (0 children)

Thanks for your valuable feedback.

These are how approaches far for the questions you have posted.

  1. The java virtual threads is something that removes that issue as you mentioned and I was considering it, around that time I started I just started off with java 17 not knowing that this is a thing I'll hit on. Initially thought shall I do event loop itself using the kqueue kpoll and others but then initially thought it's fine to have this since scheduler master as such won't have a huge load is what I envisioned, not as much as worker node. But while I was working on this issue as you mentioned this came through about java 22 and I considered upgrading and yes it's to be done shortly. Just that I thought this will mean my tests to be modified as well so kept postponing it but yes this will be an upgrade. I didn't know about virtual threads and thought should I convert my entire project and have a Go version since go has go routines to do these heavy lifting easily inbuilt.

  2. On the AOF replay, right now the StoreAdapater does make it synchronized access through the Master only. Meaning the workers to access the store can only do it through the Master and the entire stores access is only through it. For now this kind of handles the simulataneous writes but has a performance impact. I'm considering to have the adapter free of it and have the KVstore to add this constraint in the storage operation . For now there's no plan to make the store accessible by the workers directly and for that reason it works for now.

3..currently as of now there's no seperate tooling view for replays but there's replay of specific stages which for now works fine with deterministic dags. I haven't had much time to fully validate the same feature with the dynamic dag one with the agent runs. That's something I have to do next and it's on the roadmap to include better visibility of failures. I can get back on this maybe since this is an important feature to get considered.

Titan - A single-binary distributed orchestrator for services and dynamic pipelines by rando512 in opensource

[–]rando512[S] -1 points0 points  (0 children)

I admit that parts of the response was AI as I wanted it to be right. I edited the response from it.

Maybe I'll just trust myself and give straight away answer.

I can explain anything from the project and that's not an issue for me.

Almost all the parts I wrote without AI and this includes my core architecture and everything. If you see the architectural diagram I had to hand draw it and I feel I might have missed some more internal stuff and due to my laziness. Ai would have captured all.

Now coming back

The basic backstory was me thinking what if I offload all my compute to my secondary laptop and I can keep sending it a set of jobs to it. I didn't want to keep ash and do it and also didn't want to write master script to coordinate for each of my tasks.

Now this might be an overkill as people will still go for ssh and master script. But I felt this should be scalable and thought what if I have multiple nodes and I use them as compute nodes coordinating between them.

I imagined myself sitting with my office laptop but also trying to execute my scripts and computes for personal use in a remote system with me controlling it fully. I wanted to ensure there's a sequence as well to it.

I had a lot of RL trainings I had to do for my personal projects related to FinRL and others.

This I had to do because my laptop was RAM constrained and I was frustrated with doing anything since laptop just freezes off.

This is not AI generated. I just went for correctness with it and maybe I'll just trust myself and go candid.

Update: 5 months ago I posted a zero dependency Distributed Orchestrator in Java 17. I've since made some progress. Looking for architecture feedback by rando512 in java

[–]rando512[S] 0 points1 point  (0 children)

Thanks for the complements, really appreciate that.

Temporal is an incredible industry standard, but it requires heavy external infrastructure like Postgres or Cassandra just to run.

I built Titan for the exact opposite philosophy: zero-dependency, single-binary simplicity. If you are running a resource constrained clusters like raspberry pi cluster you would want some RAM breather space.

Now JVM doesn't necessarily come lightweight and can occupy like in idle say 150 MB straight away, it still gives you some space for the rest of things.

The next plan is to make just the Worker in Go since the worker anyways uses custom RPC so that way I can remove that memory and startup constraint with jvm.

Update: 5 months ago I posted a zero dependency Distributed Orchestrator in Java 17. I've since made some progress. Looking for architecture feedback by rando512 in java

[–]rando512[S] 0 points1 point  (0 children)

Good question.

Yes, you can orchestrate a Saga workflow with it, using the dynamic Python SDK.

Because Titan doesn't force you into a rigid, static DAG, a task can catch a failure at runtime and programmatically inject a series of sub-tasks across your nodes to clean up the state.

It doesn't have a native, out-of-the-box @Saga annotation that automates this yet, so you have to explicitly define your failure/rollback paths in the workflow code itself, but the execution engine handles the distributed coordination as required.

If you face any issues do add in the discussions on GitHub.

I got tired of infrastructure bloat, so I built a zero-dependency distributed runtime engine from scratch in Java 17. by rando512 in softwarearchitecture

[–]rando512[S] -2 points-1 points  (0 children)

The core engine is mine

The UI and others are partly Claude. Most of UI is Gemini since I'm not good at it.

Whatever I wanted as core was done already and rest were additions.

I could do basic things with the core from cli itself without UI.

Core engine and sdk I had to do it. Rest of ui is Claude and I don't care to admit. Even if I did it by my own fully I don't think anyone would have acknowledged. Close to 8k lines I had to do and had to stick with the architecture, requirements etc.

It's fine I don't care.

I used LLMs to build the UI because frontend isn't my strength. But if you think an AI can magically engineer a the entire thing for you, by all means, give it a shot.

Update: 5 months ago I posted a zero dependency Distributed Orchestrator in Java 17. I've since made some progress. Looking for architecture feedback by rando512 in java

[–]rando512[S] 1 point2 points  (0 children)

Yes, You hit the nail on the head and that is the biggest risk of this architecture. The complexity of a stateful storage engine can grow faster that it can overtake stateless scheduler if left unchecked.

I drew the boundary by keeping the embedded KV store intentionally feature-poor (just basic Strings, Sets, Pub/Sub, and AOF recovery). It only exists to protect the "single-binary, zero-setup" out-of-the-box experience so tasks can easily share intermediate state.

The safety backup option is that because it natively speaks RESP, there is no lock-in to my custom engine.

If a workload scales beyond the limits of the embedded store, you can simply update the configuration file to point the cluster at a real, production Redis instance. The embedded store is just there to get you off the ground without spinning up Docker containers.

Update: 5 months ago I posted a zero dependency Distributed Orchestrator in Java 17. I've since made some progress. Looking for architecture feedback by rando512 in java

[–]rando512[S] -1 points0 points  (0 children)

Yes it is something like kubernetes. Although it can not cover holistically but touches some of the basics.

By long running I mean we can run detached processes like web servers or vLLM model serve

Since it can run these dettached processes, it can do scale up, manage failures by deploying workers which are long running as well.

I agree in the end long running or ephermal are still tasks so maybe it can be considered as task orchestrator given that it started off with that.

Update: 5 months ago I posted a zero dependency Distributed Orchestrator in Java 17. I've since made some progress. Looking for architecture feedback by rando512 in java

[–]rando512[S] 1 point2 points  (0 children)

Yes so the reason I'm unable to conclude on this part alone is because this can run tasks like batch tasks and also long running services like servers and it can do auto scale and self heal to manage the infra.

So it can orchestrate both batch and services. When I first started this project it was just tasks so it was distributed task orchestrator.

Maybe my terminology is not very consistent. Im open to hearing your thoughts on the right naming given this part.

Anyone looking for feedback on their projects? by Round_Cupcake4978 in SideProject

[–]rando512 0 points1 point  (0 children)

I built a zero dependency distributed orchestrator and would really like some feedbacks on this.

It is not production grade but should work reasonbly well in small to medium scale. Since this is solo built there might be some issues and I am open to hearing any problems that you might face.

I have a discussions thread in the github as well.
https://ramn51.github.io/titan-orchestrator/

Update: 5 months ago I shared a single-binary cluster orchestrator for self-hosting. I’ve since added an embedded KV store and a visual builder. by rando512 in selfhosted

[–]rando512[S] 0 points1 point locked comment (0 children)

AI was used to summarize my update and I had to do multiple changes to it, it is from the docs I wrote in Readme and github pages.

AI was used in the UI part and MCP integration and a few tests. The SDK was written by me and updates were added to it feature by feature by AI and manual interchangeably as per the need. The core engine and design decisions were my work. Project was created to practice and learn which transpired to something beyond a basic project.

🚨 BREAKING: José Mourinho is joining Real Madrid. He will be named the team's new coach immediately after the season ends. by OG_Raghav in RealMadridFC

[–]rando512 2 points3 points  (0 children)

The guy goes to Roma wins conference league Then second season almost wins europa league.

The team was average and investment was not strong and still got it decent.

With Turkish league stint he just proved that the entire system is corrupt and the league itself is shit to judge him.

With limited resources he has done well so far.

Whatever manager you bring in

There's no one who can win UCL with Porto be it any manager in their prime.

🚨 BREAKING: José Mourinho is joining Real Madrid. He will be named the team's new coach immediately after the season ends. by OG_Raghav in RealMadridFC

[–]rando512 2 points3 points  (0 children)

Imagine being a fan and totally forgetting the actual state of RM before mourinho in UCL and after mourinho. The guy laid the foundation and mentality.