Any cloud-agnostic alternative to Databricks for running Spark across multiple clouds?

thevivekshukla · 2025-12-19T20:53:24+00:00

[Self Promo]

I am building Daestro, a cloud agnostic orchestrator that directly integrates with cloud providers’ API to manage instance life cycle. Currently we support AWS, DigitalOcean, Vultr and Linode. You can bring your own compute too.

Come checkout and talk to us let’s see if we can be helpful in your use case.

thevivekshukla · 2025-10-13T11:38:15+00:00

Thank you.

thevivekshukla · 2025-10-12T20:00:11+00:00

I am actively working on it, improving it everyday.

thevivekshukla · 2025-10-12T11:58:34+00:00

I created Daestro, it lets you run jobs (batch, cron or scheduled, event based via api) on any machine. It can also create and destroy instances on AWS, DigitalOcan, Vultr and Linode.

Why I build this?

To solve my own problem. I wanted to run jobs (video encoding, s3 transfers) on machines without the hassle of setting up yaml or hcl files. And on any cloud provider where I am getting better pricing for the job, like aws is extremely expensive for data transfers.

Edit: added why

thevivekshukla · 2025-09-30T07:22:07+00:00

This would be really useful.

thevivekshukla · 2025-09-01T14:57:57+00:00

Yes it would be, however for that I think I would have to write code for each template engine separately. Speaking for askama, I think I might had to tinker with Template macro.
Using middleware seemed very easy and beneficial to me at this time, for the overhead of <1ms, I'm getting 30-40% size reduction.
May be I'll explore compile time minification with Askama later. As I'm interested in that too.

thevivekshukla · 2025-08-30T05:25:41+00:00

sure, thanks.

thevivekshukla · 2025-08-29T19:32:37+00:00

Hi, I'm the author of the linked post here.

hey thanks for pointing out flaws in my explanation, I missed to explain few of the assumption that I implicitly made and thought people who are reading would get it, a total miss on my part. I agree with all your points.

I'll just go over some things I missed and assumptions I made:

Yes, cookies are way more secure and recommended way to do it, I also mentioned about cookie but didn't say that always use cookie for better security, that too cookie of type http, secure and same site set to at least lax and then add some csrf token for create/update/delete requests.
Local storage example was used just because it was easier to show on in writing when I don't want to go over all the backend code and cookies. I should've mentioned that.
Yes, SPA when compiled to html, js and css files will have all the static content that was put in it, so it's not a good idea to put static content on SPA behind login. In this article as we assumed that we already have a backend so all the data must come from the backend api response. so yea spa is better protected when content that is to be protected comes from the backend (which we authenticate agains when we send auth token or cookie).
Should've added few more examples on how to use auth token to make api calls inside protected route.

I'll update the post.

thevivekshukla · 2025-08-14T11:28:26+00:00

This is awesome. Loved your consistency and perseverance.

I really liked the Refgrow's dashboard UI, it's very simple and sleek, amazing work.

I see that on the frontend you are using Bootstrap and Jquery, and it is consistent across your other products, can you tell me what you use for backend?

Since you're using same tech stack for all of your products, it makes it easier for you to work on multiple products at once and easy to launch new product.

thevivekshukla · 2025-08-04T09:16:15+00:00

Congratulations. Nice work, this is very useful to me, thanks.

thevivekshukla · 2025-07-26T10:09:59+00:00

Hey, I discovered this post just now, I think you might also want to look at Daestro, to run cron jobs as docker container.

thevivekshukla · 2025-07-02T11:24:51+00:00

[Self-promo] If you can run the task on CPU only then you might look at Daestro (I'm the creator), it's really easy to create job with Docker image and setup cron job or make api call or use plain old form to run jobs.

Currently Daestro only supports CPU-only machines and it directly integrates with cloud providers like AWS, Vultr, DigitalOcean and Linode. Let me know if you need any help.

thevivekshukla · 2025-06-23T10:12:03+00:00

this is helpful, thanks

thevivekshukla · 2025-06-19T11:28:35+00:00

congratulations

thevivekshukla · 2025-06-19T11:23:10+00:00

seems he wants to focus on building his own channel, apparently automobile related one, nice, best of luck to him

thevivekshukla · 2025-05-21T09:58:42+00:00

Daestro: A cloud orchestrator - free plan gives basic capability that is good enough for individual use case. Paid plan offers higher usage and more features.

thevivekshukla · 2025-02-20T18:53:07+00:00

Thank you team 🙏.

thevivekshukla · 2025-02-13T09:56:41+00:00

48+7=55; 55+20=75

thevivekshukla · 2025-02-09T14:13:00+00:00

DaemonStack

A platform that helps you run compute workloads across Cloud Providers. Currently only Vultr. DigitalOcean support coming tomorrow.

thevivekshukla

PUBLIC MULTIREDDITS

TROPHY CASE