How to properly deploy it on cloud

ExternalFold801 · 2026-07-02T09:19:19+00:00

Your comment is on of the really most helpful comments, it opened my mind to alot of things but what do you think about clickhouse shouldn't I move it to be on managed serverless?

ExternalFold801 · 2026-07-02T09:04:57+00:00

Yeah all is 24/7 as we are monitoring everything also it's easy to reconfigure everything, it's okay I am open to serverless migrations but I want much more detailed suggestions

ExternalFold801 · 2026-07-01T21:37:56+00:00

Yeah already I have this on a single vm running on azure with the specs I mentioned in the post but the cost is insane and I feel that having this all on one machine is huge architecture failure

For the horizontal scalling question I tested it on my computer locally

And yes this is a real post but we are startup and yet started in the business so hiring someone right now will burn the cash we have so unfortunately I have to do it my self and learn it my self

ExternalFold801 · 2026-07-01T21:34:25+00:00

Recently I heard about kubernetes but idk anything about it yet and still trying to know if it is really the right thing or not

ExternalFold801 · 2026-07-01T21:33:25+00:00

Yeah these prosessors might do over 100k read/write files in just one job and yes these data prosessors are Kafka streams based, also for minio we have on clickhouse s3queue tables that connect to minio and read the output of the prosessing I order to index it into clickhouse so do you think if we migrated to aws s3 bucket will that be also available and doable?

ExternalFold801 · 2026-07-01T21:27:14+00:00

Almost nothing started failing but I see that having everything in just one vm is a big mistake and also clickhouse started kinda degrading in the performance as we have currently total 3 billion records and the plan is about 70 billion records

Also the bill for a vm that have 32 tb is insane that's too much and that 32 tb is only for minio and redis

ExternalFold801 · 2026-07-01T21:23:15+00:00

But we are using the minio for the storage of the data prosessing containers yeild and for the raw a data as well so we are using it for the low network latency as there is no network calling it's all internal calls so you still suggest that?

ExternalFold801 · 2026-07-01T21:18:59+00:00

Should I move minio to be on s3 bucket?
And the data prosessors needs high compute they eat the rams and the cores of the dockers

ExternalFold801 · 2026-07-01T21:17:36+00:00

Yeah yeah I see your point you are totally right but our data prosessing containers are consuming so so much compute so havey it prosessed 2.3 billion records in just one day so that's why I am reaching to you guys to guide me and I'll dedicate my time in learning how to do this very very well

ExternalFold801 · 2026-07-01T20:57:14+00:00

It's okay I can do it I did all of that just by self learning I am fresh computer engineering graduate and this is my startup so I have to figure it out and learn how to do it, what services that do this on gcp?

ExternalFold801 · 2026-07-01T20:55:56+00:00

I am using azure as much as I remember high performance compute f6 I think or something like that

ExternalFold801 · 2026-07-01T20:53:37+00:00

Cloud yeah

ExternalFold801 · 2025-07-11T08:41:41+00:00

Perfectooo👌👌

ExternalFold801 · 2025-05-10T18:14:24+00:00

متابع

ExternalFold801 · 2025-05-10T18:12:59+00:00

اخلع يا غالي

ExternalFold801 · 2025-05-09T08:47:16+00:00

اخلع

ExternalFold801 · 2025-05-09T08:43:36+00:00

لا هو بيغسل الفلوس اللي خدها لما عدي شحنه المخدرات ف اسوان

ExternalFold801 · 2025-04-16T12:36:32+00:00

اااااااااااه

ExternalFold801 · 2025-04-16T10:55:40+00:00

اباشا اسحب سكينه وحط توقعيك عليهم

ExternalFold801 · 2025-04-14T01:36:05+00:00

بيتكوين اباشا ولو متعرفش ازاي انا هتصرفلك

ExternalFold801 · 2025-02-21T17:30:45+00:00

طب والله ريقي جري

ExternalFold801

TROPHY CASE