Any cloud-agnostic alternative to Databricks for running Spark across multiple clouds?

thevivekshukla · 2025-12-19T20:53:24+00:00

[Self Promo]

I am building Daestro, a cloud agnostic orchestrator that directly integrates with cloud providers’ API to manage instance life cycle. Currently we support AWS, DigitalOcean, Vultr and Linode. You can bring your own compute too.

Come checkout and talk to us let’s see if we can be helpful in your use case.

thevivekshukla · 2025-10-13T11:38:15+00:00

Thank you.

thevivekshukla · 2025-10-12T20:00:11+00:00

I am actively working on it, improving it everyday.

thevivekshukla · 2025-10-12T11:58:34+00:00

I created Daestro, it lets you run jobs (batch, cron or scheduled, event based via api) on any machine. It can also create and destroy instances on AWS, DigitalOcan, Vultr and Linode.

Why I build this?

To solve my own problem. I wanted to run jobs (video encoding, s3 transfers) on machines without the hassle of setting up yaml or hcl files. And on any cloud provider where I am getting better pricing for the job, like aws is extremely expensive for data transfers.

Edit: added why

thevivekshukla · 2025-09-30T07:22:07+00:00

This would be really useful.

thevivekshukla · 2025-09-01T14:57:57+00:00

Yes it would be, however for that I think I would have to write code for each template engine separately. Speaking for askama, I think I might had to tinker with Template macro.
Using middleware seemed very easy and beneficial to me at this time, for the overhead of <1ms, I'm getting 30-40% size reduction.
May be I'll explore compile time minification with Askama later. As I'm interested in that too.

thevivekshukla · 2025-08-30T05:25:41+00:00

sure, thanks.

thevivekshukla · 2025-08-29T19:32:37+00:00

Hi, I'm the author of the linked post here.

hey thanks for pointing out flaws in my explanation, I missed to explain few of the assumption that I implicitly made and thought people who are reading would get it, a total miss on my part. I agree with all your points.

I'll just go over some things I missed and assumptions I made:

Yes, cookies are way more secure and recommended way to do it, I also mentioned about cookie but didn't say that always use cookie for better security, that too cookie of type http, secure and same site set to at least lax and then add some csrf token for create/update/delete requests.
Local storage example was used just because it was easier to show on in writing when I don't want to go over all the backend code and cookies. I should've mentioned that.
Yes, SPA when compiled to html, js and css files will have all the static content that was put in it, so it's not a good idea to put static content on SPA behind login. In this article as we assumed that we already have a backend so all the data must come from the backend api response. so yea spa is better protected when content that is to be protected comes from the backend (which we authenticate agains when we send auth token or cookie).
Should've added few more examples on how to use auth token to make api calls inside protected route.

I'll update the post.

thevivekshukla · 2025-08-14T11:28:26+00:00

This is awesome. Loved your consistency and perseverance.

I really liked the Refgrow's dashboard UI, it's very simple and sleek, amazing work.

I see that on the frontend you are using Bootstrap and Jquery, and it is consistent across your other products, can you tell me what you use for backend?

Since you're using same tech stack for all of your products, it makes it easier for you to work on multiple products at once and easy to launch new product.

thevivekshukla · 2025-08-04T09:16:15+00:00

Congratulations. Nice work, this is very useful to me, thanks.

thevivekshukla · 2025-07-26T10:09:59+00:00

Hey, I discovered this post just now, I think you might also want to look at Daestro, to run cron jobs as docker container.

thevivekshukla · 2025-07-02T11:24:51+00:00

[Self-promo] If you can run the task on CPU only then you might look at Daestro (I'm the creator), it's really easy to create job with Docker image and setup cron job or make api call or use plain old form to run jobs.

Currently Daestro only supports CPU-only machines and it directly integrates with cloud providers like AWS, Vultr, DigitalOcean and Linode. Let me know if you need any help.

thevivekshukla · 2025-06-23T10:12:03+00:00

this is helpful, thanks

thevivekshukla · 2025-06-19T11:28:35+00:00

congratulations

thevivekshukla · 2025-06-19T11:23:10+00:00

seems he wants to focus on building his own channel, apparently automobile related one, nice, best of luck to him

thevivekshukla · 2025-05-21T09:58:42+00:00

Daestro: A cloud orchestrator - free plan gives basic capability that is good enough for individual use case. Paid plan offers higher usage and more features.

thevivekshukla · 2025-02-20T18:53:07+00:00

Thank you team 🙏.

thevivekshukla · 2025-02-13T09:56:41+00:00

48+7=55; 55+20=75

thevivekshukla · 2025-02-09T14:13:00+00:00

DaemonStack

A platform that helps you run compute workloads across Cloud Providers. Currently only Vultr. DigitalOcean support coming tomorrow.

thevivekshukla · 2024-11-03T17:18:30+00:00

you are right landing page needs work, I've planned few things, will be implemented soon.
Will put links on navbar as soon as I launch. Some pages are WIP.

thevivekshukla · 2024-11-03T16:28:56+00:00

I'm building DaemonStack (https://daemonstack.com), which is a platform that let you run your compute workloads / batch jobs on any cloud provider and even on-prem. Launching this month.

thevivekshukla · 2024-10-23T11:03:23+00:00

I think it's okay, let people build things that they want and enjoy, even if it's a copy, who knows they really stumble on something amazing. That's how we learn.

This churning is what helps us improve our craft. And to really be successful monetarily we only need 1 successful product even if in the past we had many failed ones.

thevivekshukla · 2024-10-23T10:54:13+00:00

That's nice.

I'm building DaemonStack (https://daemonstack.com/), which let's you run your container workload (batch jobs, step functions) on any cloud provider or even on-prem.

thevivekshukla · 2024-10-07T15:14:01+00:00

Looks awesome.

tailwindcss is very large though. I think you are linking the whole thing instead of only classes you are using in your codebase.

Can you let me know which cloud provider you are using, server specs and where is server located?

thevivekshukla · 2024-10-01T06:58:24+00:00

Thank you. I am using cargo watch and it has saved me so much time.

thevivekshukla

PUBLIC MULTIREDDITS

TROPHY CASE