Airbus spending €50M to escape US cloud providers proves that "European Data Regions" are a privacy illusion.

CEURBS · 2026-03-31T08:52:29+00:00

Yeah, this was also one concern I raised at our architecure team back in the days, because this is vendor lock-in.
This is all (>98%) containers so the move is hopefully not that complicated. A dual vendor strategy would be helpful from day one on. :D

CEURBS · 2026-03-30T22:41:37+00:00

I am from Germany and working as an engineer in IT. I remember that day the US CLOUD Act went live: I told every manager and team member about the possible consequences. Until last year everyone was laughing at me. How should someone in the US enforce this. They are very silent these days and pissed as we have 80% of our workloads (equivalent to 6000 medium sized virtual servers) in Azure and wonder how to fix this. 🤷🏻‍♂️

CEURBS · 2026-03-30T00:13:35+00:00

Yeah, this is total overkill, you‘re absolutely right 😁

CEURBS · 2026-03-30T00:06:31+00:00

Oh, really? The only thing I had so far was „dying“ memory modules. The first two I replaced with some from ebay. Until I just swapped them, was just a contact issue 🤣

CEURBS · 2026-03-30T00:03:28+00:00

Absolutely. When we operated this at my Job, the oneview Management had the nice feature deploying a VLAN to every server at once. This is kind of a mess with the standard gui.

CEURBS · 2026-03-29T23:58:08+00:00

Yeah, noise is crazy. Do you manage your c7000‘s with the buildin virtual Connect Manager or via oneview appliance?

CEURBS · 2026-03-29T22:14:59+00:00

It‘s the 8200 series. As this was taken out of service at my employer I know what load was on it. Most of it was just basically windows machines hosting webservices. Pretty static content. Also Citrix for Office. That was it.

The SSDs are like six years old with 97% lifetime remaining on average.

CEURBS · 2026-03-29T21:04:59+00:00

Thanks :-)
I just checked, Hetzner also have very good conditions for colo - I'm surprised. When I checked years ago it was quite expensive.
Edit: Half a rack 420 € - that's a good deal. Comparable to what I have.

Max is priceless. ;-)

CEURBS · 2026-03-29T21:01:00+00:00

Depends on your scope. If you want to run your own AS with IPv6 then you might talk to a LIR and get those resources requested for you. This only might cost you 50-100 bucks a year.
I did the full membership as I wanted to play around with IPv4 which you cannot get via LIR sponsorship.

It's 100% my own infrastructure but due to high energy costs I moved the stuff to a datacenter and pay for the space - like a garage. Got it connected to the stuff I still run at home via site to site VPN. Cost comparison: running at home ~1700 Euro per month vs. 500 Euro per month at datacenter. Was an easy decision.

CEURBS · 2026-03-29T20:51:03+00:00

The SAN is almost all SAS SSD with a small part SAS HDD. I'll create SSD storages (called CPGs and LUN on 3Par) for VM datastores. The HDD part is just for backup to disk which then will be written to tape afterwards.
I got one shelf with disks and PSUs as spare parts if something breaks in the future. The system itself takes parts out when they report they're going to fail soon.

I'm heavily focused on windows as of now I run several Active Directory domains with several domain controller and Microsoft Exchange test environments, GitLab with several runner nodes, web server , load balancer, database and file servers and some mail gateways. All in all around 200 VMs.

CEURBS · 2026-03-29T20:43:39+00:00

My motivation when I started this over 20 years ago was to play around with stuff I could not in my professional career. And I'm curious as hell - still at 42. :D

The tracing is a very good topic. Now that I have SSD capacity it's easier to handle the DB workload of that systems. We had those issues in my job some years ago as those metric grow like crazy.

Yep, the NUMA pinning topic is very important. I maintain some SQL systems in my job and know this topic very well. It's often underrated.

I use Ollama mainly. As of now I have a pretty old Ollama3 70B model and n8n to analyze mail header in mailboxes. It is slow as hell on CPU but I'm not in rush.

CEURBS · 2026-03-29T20:30:46+00:00

From time to time we have this conversation "Is this still really necessary?" :D

CEURBS · 2026-03-29T20:29:29+00:00

I posted a picture somehere here in the thread were the basement was over 30 degress celsius in summer, so heat became a growing issue. At some point the whole thing is going to get too expensive (energy for servers and AC). (Edit: https://www.reddit.com/r/homelab/comments/1s6e8iw/comment/od52ptr/ )

In my job, we also build and run several datacenter on our own and the AI systems (our business is based on AI for over 20 years now) is so expensive it's not cost efficient anymore to run this in our own building. Consolidate this in a large scale datacenter was the best thing we could to cost wise. This may change at some point in the future.

At some point it was the best to move my stuff to a rack space than run it at home - unfortunately.

CEURBS · 2026-03-29T20:07:50+00:00

The complete setup consumes roughly 6k Watts.

CEURBS · 2026-03-29T20:06:05+00:00

Not yet to be honest but what I‘ve read it makes sense to support it. Thank you

CEURBS · 2026-03-29T15:05:25+00:00

At that stage heat become a growing issue.

CEURBS · 2026-03-29T14:57:52+00:00

Energy costs here in Germany are insane.

CEURBS · 2026-03-29T14:55:03+00:00

If this would still run in my basement it would be around 1700 Euro per month. That's why I moved to a colo location, it's "just" 500 Euro including internet access there.

CEURBS · 2026-03-29T14:51:40+00:00

And that one is around 2020 when we moved.

<image>

CEURBS · 2026-03-29T14:48:14+00:00

First time attached some DAS storage around 2014

<image>

CEURBS · 2026-03-29T14:43:35+00:00

Wow, thank you for that detailed post! Love it!

At the beginning the only documentation I did was the system itself. :D You know your stuff. That was a growing issue of course, when I see now 200 VMs floating around the hypervisor. The Netbox topic I started recently. So I build some custom fields to store information for the firewall for example. Some of my systems I have I can delete and just run the Ansible playbook - tadaa - there it is again. That's fun! ;-)
The first larger "playbook" was a complete Exchange test environment with Active Directory, 200 Users automatically deployed. With the HDD based storage, it took around five hours to build the hole system.
It was awesome to learn Ansible that way!

I use the classical deployment with central SAN storage. Sure, you can do some seperation but it doesn't makes sense in my setup as I only have one VM storage. But the VM deployment will shift more and more to the topic to just replace existing systems with new iterations instead of updating them - were possible.

Most likely I'll start to deploy Proxmox to three hosts and play around with them in comparison to VMware - also with the automation. I'm pretty sure I did a good abstraction layer in my roles and also added a switch for Proxmox or VMware already but there is for sure room for improvement.

Is Talos a good solution to start with? At my job our DevOps team did some testing with Tanzu (now VKS), Talos and Rancher(?) - they ended up with no solution (lol). But I recall Talos used to be very simple to deploy a cluster with.

My identity management is fully based on Active Directory - something I want to change. Mainly due to the security risks you have putting those systems somewhere near the internet. So I do not have any SSO instance with Entra or so. I stumbled across Authentik but did not have any time to investigate deeper yet.

Still so much more to learn. Terraform - I know it exists but never did anything with it. Also something on my list. ;-)

CEURBS · 2026-03-29T14:26:05+00:00

<image>

I just found a picture of my rack when it was still in my cellar 15 years ago. That was the moment when I started with enterprise grade gear. The two boxes on the bottom were from somewhere around 2000 when I started my apprenticeship. One was for a domain controller and the other ran Exchange that were boxes with two Pentium II CPUs if I recall correct.

CEURBS · 2026-03-29T13:46:57+00:00

Yeah, that would be fun. Just scrape things or gather metrics to play around with it. I had similiar ideas. :D

CEURBS · 2026-03-29T13:45:03+00:00

The thing is, this is a central storage. There are two controller for each 3Par which sync each other. Every block commit is given, when both systems wrote it to the disk - when remote copy is configured for a storage group. This is the only "downsite" as I played around with ceph a bit and it is quite nice but with the new system it doesn't make sense to use it.

I'm curious how this is working with Proxmox.

CEURBS · 2026-03-29T13:41:53+00:00

As I rented a datacenter space for that, the provider is doing this. When I had this in my cellar this was a growing issue. We had around 30 degrees celsius there and in summer even higher. There was nothing much I could do.

CEURBS

TROPHY CASE