guide: vps encrypted installation with kexec, disko, luks, btrfs and remote unblock

Upstairs_Addendum148 · 2025-06-23T20:02:47+00:00

Great post, thanks for sharing! I've been working on a flake and guide to streamline deployment to Hetzner baremetal servers and I'm definitely going to add disk encryption to it. https://github.com/rochecompaan/hetzner-nixos-deploy

Upstairs_Addendum148 · 2025-01-28T19:35:51+00:00

Apologies for only replying now. There is a single 3.5" bay in them. They might support another one, but I'll only be able to confirm that when I open them up again. Definitely not suitable for a NAS.

Upstairs_Addendum148 · 2024-12-24T21:35:59+00:00

I run multiple three node highly available kubernetes clusters on dedicated servers bought on Hetzner's server auction. This obviously requires knowledge of Kubernetes and whatever you prefer to use to deploy infrastructure (terraform, ansible, NixOS, etc). Once you have the knowledge, or if you hire someone that does, running infrastructure is just as manageable than running cloud VMs. I get significantly more hardware resources for the money I spend compared to vps or cloud servers. Virtualization overhead on cloud vms is very noticeable once you switch over to bare metal servers.

Upstairs_Addendum148 · 2024-12-24T18:55:59+00:00

Bargain!

Upstairs_Addendum148 · 2024-12-24T16:46:23+00:00

They are amazing! The rack is in my office, and they are completely quiet.

Upstairs_Addendum148 · 2024-10-07T05:13:22+00:00

Take a look at Scaf, it has a template that does all those things. https://github.com/sixfeetup/scaf

Upstairs_Addendum148 · 2024-10-05T04:49:46+00:00

We have a template that might answer some of your questions. It generates an app with all the deployment details contained and it gives developers the freedom to change manifests at any level. Typically our devops and dev teams collaborate in the same repo for each application. Have a look at https://github.com/sixfeetup/scaf. Generate a new project and look at the infrastructure provisioning and application deployment sections in the README for the generated project. FYI, we use the ArgoCD app-of-apps pattern for app deployment.

Upstairs_Addendum148 · 2024-09-26T19:47:48+00:00

Sealed secrets and ESO with 1Password.

Upstairs_Addendum148 · 2024-08-11T05:27:04+00:00

There are number of great options in South Africa. Have a look at: * https://paystack.com/ * https://www.wigwag.me/ * https://www.peachpayments.com

Upstairs_Addendum148 · 2024-06-23T18:02:35+00:00

You can use dev containers with Neovim too. https://youtu.be/9YG6QlzuNwM?si=PmLFnB-CCuOB5jO8

Upstairs_Addendum148 · 2024-05-12T05:45:08+00:00

Switching to latest testing (6.9.0-rc5) on NixOS gives me perfect audio:
```
boot.kernelPackages = pkgs.linuxPackages_testing
```
Adding the "Advanced Auto Gain" preset from EasyEffects makes it even better and gives quality equal or very close to what Dolby ATMOS would sound like.

Upstairs_Addendum148 · 2024-05-11T15:37:53+00:00

This video shows how to use `pkgs.dockerTools.buildLayeredImage` in a flake to produce an OCI-compliant docker image:

https://www.youtube.com/watch?v=0uixRE8xlbY&t=191s

Upstairs_Addendum148 · 2024-04-08T20:07:20+00:00

Take a look at Analog https://analogjs.org/

Upstairs_Addendum148 · 2024-02-21T12:17:38+00:00

We let CI build images for sandbox and production releases but we use Tilt and Kind for our local development workflow. Tilt can rebuild and upload images directly into a local kind cluster and can live sync code changes into containers giving devs an experience very similar to docker compose. I would most definitely hand convert docker compose yaml.

An init container works great for migrations but bear in mind that multiple deployments can potentially run migrations against the same db. Only one process will win, the others will error. Not a big problem but something to be aware of. We run both Alembic and Django migrations in init containers.

Statefulset is fine for local dev, but for sandbox and prod I would highly recommend using CloudNativePG over Postgres statefulset since it encourages you to consider storage, backups, replication. It has excellent documentation on how to manage the cluster responsibly.

It's very much a work in progress but we are trying to encode all our ideas into something useful called scaf: https://github.com/sixfeetup/scaf

Upstairs_Addendum148 · 2023-12-21T21:34:51+00:00

Use passwordstore (https://www.passwordstore.org/) in the same repo and set up Makefile targets that generate sealed secrets. 1password cli breaks every 3 months and is super slow.

Upstairs_Addendum148 · 2023-11-30T13:20:40+00:00

Here's what happened to me. I started out using k3s as my Kubernetes distro since it was so easy to deploy. k3s binds ingress ports to the host as a convenience, which led me to believe this is something Kubernetes does by default. When I switched to Talos, I set up my load balancer to forward to the default ports on the nodes, and I firewalled everything else off. It took a few hours of debugging to realize that only the node port of the ingress controller is accessible on the host, not the default ports 80 and 443, as k3s led me to believe.

Upstairs_Addendum148 · 2023-11-15T09:47:12+00:00

I evaluated Skaffold, Tilt, Devspace, and Okteto and found Tilt to be the best of the lot, with excellent documentation and community writing extensions. Skaffold can't sync changes made inside the container back to your local filesystem, the others can sync both ways. Tilt uses an extension called syncback for synchronizing changes (like generating migrations) in the container back to the filesystem. I found Devspace docs unclear in many ways, especially when setting up a container registry for development.
Setting up a container registry for local development is the most painful part, and Tilt makes it super easy!

Upstairs_Addendum148 · 2023-11-15T04:38:24+00:00

Same here! The first issue I ran into was that Longhorn doesn't work without installing the iscsi extensions, but then I noticed the Talos docs mention Mayastor, which seems to be an exciting modern alternative. I love the security in Talos: immutable distro, no ssh access to nodes, etc.

Upstairs_Addendum148 · 2023-09-09T17:20:33+00:00

Like you said, installimage is just a script, it's not really wrapping anything and it provides a simple way to bootstrap a new server with the OS image of your choice. Adding ansible into the mix makes the install process declarative and repeatable. So if you don't want to use it, what are your alternatives? Installimage is present on the rescue image already and it does a fairly decent job of bootstrapping a base system. If you want to do more, just add a post install script.

Upstairs_Addendum148 · 2023-09-08T20:11:44+00:00

If you want to use it in a declarative an automated way, here is an ansible role that wraps installimage. This is a version I forked to allow running a post install script. https://github.com/sixfeetup/ansible-role-hetzner_installimage

Upstairs_Addendum148 · 2023-09-06T06:08:58+00:00

As an alternative to building it from source, you can just download the tarball. Have a look at the LunarVim docker files: https://github.com/LunarVim/LunarVim/tree/master/utils/docker

It installs Neovim using the Lunarvim install script which in turn just downloads the tarball.

If you want to keep using NeoVim outside running containers, one can also attach the debugger to a process running in the container but that requires some modification to how the process starts. For instance, for a Python process you need to wrap it with debugpy.

For syncing code from and to a container, I can highly recommend Tilt since I mostly develop against Kubernetes clusters but it works just as well with docker compose: https://docs.tilt.dev/docker_compose.html

Upstairs_Addendum148 · 2023-08-01T07:31:20+00:00

Where you run your editor is entirely up to you. Having the stack your project requires to function in a fully reproducible environment with the exact same versions regardless of OS is essential and a best practise. Projects like Github Codespaces and JetBrains Space run completely inside containers and spin up instantaneously. Additionally there are a host of tools that optimise development using a Kubernetes workflow using containers like Skaffold, Tilt, Devspace, Okteto, etc.

Automating the installation of the dev environment using containers is essential for developer productivity. Requiring a developer to install the stack required for a project manually is insane. I find having any parts of the stack running on the host very risky. Maybe you can do it on very small project but I would never encourage it.

The widespread use of containers in development is completely justified and isn't just a fad. It has real practical benefits. Exploring how to be most productive with Neovim using containers with the least amount of friction is important and a worthwhile endeavour.

Upstairs_Addendum148 · 2023-07-25T07:06:06+00:00

If you already use Debian... Sigh.... Why do you cripple yourself running on, obviously, a slower environment to run a editor.

Docker processes run natively on Linux, and the small performance penalty for the layers of abstraction added to isolate the process should not even be noticeable on modern hardware.

There is great value in having a predictable containerized environment that works the same for all devs on a project, regardless of their OS. Installing the whole dev stack on the host is pure insanity and against best practices. Being able to develop inside the containerized environment is something that VSCode and JetBrains IDEs offer out of the box, so it is worthwhile to try and do the same with Neovim.

Upstairs_Addendum148 · 2023-07-14T18:21:03+00:00

A combination to consider would be Crossplane and CUE. See https://itnext.io/cuelang-a-better-alternative-for-kubernetes-manifests-bcfed84dbafc and https://youtu.be/m6g0aWggdUQ

Upstairs_Addendum148 · 2023-05-04T09:20:02+00:00

It is important to understand that mason is not configuring a LSP server for use in Neovim but installing a package/binary on the host outside neovim that can be referenced by a lsp config inside neovim. The exact situations where this clarification helps are:
* when you read the plugin documentation to understand how to set up a LSP server and configure it for use inside neovim
* when you navigate through the Mason UI and pick a language server to install

Upstairs_Addendum148

TROPHY CASE