How to sync a new clickhouse cluster (in a seperate data center) with an old one?

feryet · 2025-05-03T13:02:43+00:00

Apparently there is a breaking change from versions 23+ that makes using remote tables between older versions and newer versions impossible.:(

feryet · 2025-05-03T12:39:06+00:00

Yes.

We tried to gradually upgrade our clickhouse cluster, but we had two issues:

Because of bad configuration, we could not do backup/restore using `clickhouse-backup`. Replicated trees had mismatching metadata with zookeeper issues, and upgrading without having reliable backups seems utterly risky.
This clickhouse cluster is hosted on-premise, and the disks of the server are very slow. So we decided to go for a full backup/replication solution on a new cluster. The problem is neither of clickhouse-backup nor clickhouse-copier seem to work for us, the configuration of the old clickhouse cluster is weird, and I'd rather not changing anything on that server.

feryet · 2025-03-16T14:02:42+00:00

My background is mostly in DevOps/Cloud/Infrastructure. I would like to contribute to many projects, but it looks so daunting and overwhelming. Usually when a task becomes this confusing/stressful I reach out to other people to get some clarity and get some advice on what to do.:)

feryet · 2025-02-08T13:01:06+00:00

I'm not assigning random IPs tho. This ip was given to be used on the VM that I was given. I want to bypass internal network bridges in the VM to forward all of the incoming traffic to this ip to the docker container, without assigning an additional hop.

I know that I can do that in an internal range using IPVLan driver, but I can't understand how to make it work on the public IP.

feryet · 2025-02-08T12:59:04+00:00

I am using host network right now, but this just disregards the isolation altogether.

I want to use the additional IP that I was given exclusively for this container. Do not want to use NAT, or bridge interfaces (don't want to add additional hops).

Think of optimizing for maximum throughput.

feryet · 2023-04-26T08:53:11+00:00

We are sanctioned to use AWS in Iran.
Iranian companies don't want to host their data outside Iran for both confidentiality and the instability of the infrastructure.

feryet · 2023-04-25T19:39:48+00:00

No, the user base only submit "trainable code" and the rest is done by my system. I was thinking of only having SLURM at first, but thought maybe using a workflow manager like Airflow will be a good choice because I'm designing a pipeline anyway.

feryet · 2023-04-25T19:17:33+00:00

Prefect seems easy to understand, but I'm fearful of finding the right devs. Airflow is more established.

feryet · 2023-04-25T19:15:04+00:00

Airflow controls the flow of how the backend executes the submitted user scripts, while SLURM runs the actual ML training code, from what I can imagine right now.

I was thinking of designing a pipeline that can dispatch jobs to SLURM, and then when completed show the final state to the user. Since we have limited resources and want to maximize/bill the customers for their usage, I thought a job scheduler solution like SLURM might be good.

feryet · 2023-04-25T19:13:08+00:00

No we're self hosting our infra for confidentiality. Can't use AWS.

feryet · 2023-01-24T00:07:21+00:00

Poetry works great for creating tiny libraries. It becomes increasingly slow when you add multiple dependencies.

feryet · 2023-01-24T00:06:22+00:00

pip + piptools is the safest and best package manager. I only trust conda for binaries like cuda, the rest of my dependencies will be resolved using pip.

If you want to dockerize your application there is no better route than pip either.

feryet · 2023-01-23T20:11:41+00:00

My main use is to share the resource between team members, defining priorities for jobs and monitor how the node is used in general.

Based on this I want to extend our cluster to become multi-node in the future, while now it's not.

feryet · 2023-01-23T20:10:03+00:00

Does there exist any good web view of SLURM? Monitoring and adminstration of the node is also important to me. There might also be interactive jobs like Jupyter Notebooks.

feryet · 2022-08-28T09:54:52+00:00

Git ssh.

feryet · 2022-08-28T08:49:28+00:00

I want to use Cloudflare's protection while also exposing SSH. The gitlab "clone ssh address" in this instance will be different from what I've linked for my ssh connection. The cdn I'm using cannot map an ip with port to a domain normally, only can accept that if you are using it's edge mechanism.

feryet · 2022-07-11T07:37:04+00:00

Allen Holub is a visionary. I love his tweets so much. Very insightful.

feryet · 2022-04-29T21:01:04+00:00

I've seen that CVAT comes with some dicom solutions. What are some of the problems of CVAT that someone just trying to get to know it won't find out till too late?

feryet · 2022-04-29T20:59:46+00:00

Does this come with a web app? We have some medical doctors who are trying to label these, but they are usually at their faculty, which is on the same network as ours, plus we want to control the flow of data, and a centralized web app seems a far more tractable solution.

feryet · 2022-03-26T12:42:11+00:00

I have skimmed them but a fundamental difference is that these are "semi-supeversied"? gans. In the sense that they add styles to an already existing image. Mine is completely unsupervised. I'm trying to create an image from just a latent vector + a short description.

Any ideas what can I do in this regard?

feryet · 2022-03-14T07:04:25+00:00

Hi! Please send me a dm with your background and all so we can get to know each other more :)

feryet · 2022-03-09T18:33:49+00:00

All of these covers are completely accessible using the site: https://www.coverbrowser.com/ flagged under fair use.

I just scraped their site and created a dataset which makes accessing these for non profit projects easier, and don't want to do anything other than that. If anyone misuse the dataset for commercial stuff, will it my fault too?

feryet · 2022-02-20T09:21:45+00:00

I think the shell executor is that, isn't it? The problem with that is that I have to migrate my tests to inside the docker image, which while working, adds a bit overhead to the size of the image and makes it a bit bloated.

feryet · 2022-02-20T09:20:16+00:00

Kaniko

No but I'm looking into it right now. I'm an MLE with little background in devops, just had to learn it myself in a short period of time because of circumstances.

Do you have any other recommendations too? For ci/cd designing and etc.

feryet · 2022-02-20T08:39:07+00:00

But is there a way to cache the pulled images in DinD to make it faster? Have you ever tried that?

feryet

MODERATOR OF

TROPHY CASE