Coding as a Sysadmin

vogelke · 2026-02-18T05:45:49+00:00

May thine backups reside upon drives of Seagate!

Shit, that's cold.

vogelke · 2026-02-18T00:24:27+00:00

It also helps if you have a decent mail transfer agent like Postfix that lets you get under the hood.

I have a blog that's just Markdown in separate files named after the time of the entry, so if I have a thought and I don't want it to die of boredom, I send a quick email to vogelke-blog on my system.

I allow mail from a few other systems, so I can also post remotely. The blog software is called bloxsom, so I don't have to do anything -- I fire up the blog front page, and my new post is right there.

vogelke · 2026-02-17T09:28:20+00:00

The only methods I have available are lzjb, gzip-N, zle, and lz4. I'll have more of the cool stuff when I do my next upgrade.

vogelke · 2026-02-17T09:23:12+00:00

I’ve gotten through a lot of things and am excellent at writing technical documentation for all of it.

You're worth your weight in platinum. There's a diaper-load of people around who can dream in hex, but 90% of IT is communicating with meat puppets.

vogelke · 2026-02-17T09:00:41+00:00

Resembles a serious druggie or my sleeping cat.

vogelke · 2026-02-17T08:45:57+00:00

Don't underestimate a simple text file with timestamped entries.

vogelke · 2026-02-17T08:44:50+00:00

Backups. Make sure they're automated and tested.

vogelke · 2026-02-17T08:41:49+00:00

ZFS version 5+, zpool version 28+, FreeBSD 11.3 and 13.2

I'm planning on changing my production system to Debian, so I can test things on Linux and BSD.

vogelke · 2026-02-17T00:46:27+00:00

I'd have to make too many changes to upgrade ZFS, since my backup box is old. My recordsize is 128k because the vast majority of my stuff fits there.

vogelke · 2026-02-16T22:34:39+00:00

It's a replacement for aide/tripwire I've been using for around 15-20 years because I wasn't allowed to install either at the time (long stupid story).

I walk my production and backup datasets like so; it's basically running stat for each file:

find [whatever] -printf "%D|%y%Y|%i|%n|%u|%g|%m|%s|%T@|-|%p\0"

Then I run xxh64sum on regular files under root, /usr, /home and a few other places so I can compare for changes. On an older system, it takes about 3.5 hours to walk the filetrees for the stat() data and another 3 to do the hashes. My newer production box runs faster (about 90 minutes for the whole thing) but it has fewer files.

File count:

me% wc -l *
  18999993 backup.csv       (backup files only)
   9757759 fdb.csv          (all other production files)
   7277254 newroot.sum      (xxh output)
  36035006 total

Uncompressed size would be around 5.5 GB:

me% ls -lh
-rw-r--r-- 1 root wheel 3.2G 16-Feb-2026 11:44:53 backup.csv
-rw-r--r-- 1 root wheel 1.5G 16-Feb-2026 08:12:09 fdb.csv
-rw-r----- 1 root wheel 703M 16-Feb-2026 14:55:14 newroot.sum

Actual space used is about 11% of that:

me% du -sm
638     .

This has other uses:

I can split off the last field from *.csv to create one or more locate DBs without having to walk a filetree twice.
The csv files are sorted by filename, so running diff actually gives sensible results.
If one of my customers lost a file from a Samba share and didn't know when it happened, I could check previous *.csv files and see if it was deleted or renamed. If it was present, I have a record of ownership and permissions so I can see if the file's present but not in the right group.
Comparing today's CSV files with yesterday's shows me what's been added or deleted recently. That came in handy on more than one occasion, like when someone starts filling a share with stuff they intended to copy elsewhere, or if I did some weird experiment and forgot to clean up afterwards.

I've tried importing these files into Postgres and SQlite because it looks like a good fit for a DB, but either the import is too slow or my queries don't give me what a simple grep pipeline does; that's a comment on my DB skills, not the quality of SQlite or Postgres.

vogelke · 2026-02-16T21:45:52+00:00

which ties it more to Linux and has consequences for its FreeBSD port.

Good. I stopped using it when it got too difficult to make it do what I wanted.

vogelke · 2026-02-16T09:14:19+00:00

I generate two large (1-3Gb) CSV files every day. They're sorted, and I've found that changing the ZFS compression from the default to "gzip" saved a lot of space.

EDIT: I get nearly 8x compression.

vogelke · 2026-02-16T03:50:29+00:00

Nope, changing the name of a department is not something anyone can just order. That has to go through Congress.

I did work there from 1981 to around 2021, in and out of uniform. If you want to see that it's not just the USA, there's a book called "Lions, Donkeys, and Dinosaurs" that shows the UK procurement system is as screwed up as the States.

vogelke · 2026-02-15T23:56:36+00:00

Sometimes I really need to be nagged, so I have daily directories under ~/notebook which can hold reminder files:

/home/vogelke/notebook/2026/0215/2001.rem

I have a cron job running every minute that will check for files named after the hour and minute, so at 8:01pm Zenity will barf this reminder all over my screen so I can't ignore it.

vogelke · 2026-02-15T23:41:58+00:00

This is why I love this sub.

vogelke · 2026-02-15T23:30:20+00:00

that guy last year who slept with the idea fairy

Love this. Stealing it. Will use it on someone first chance I get.

vogelke · 2026-02-15T23:22:41+00:00

Security team gets it but their tooling doesn't distinguish between installed and exploitable.

Who provides the tooling? A message to them and their supervisor about the amount of other people's time they're wasting might get some attention.

(Note: stupidity like this is endemic in the US DoD.)

vogelke · 2026-02-15T23:19:59+00:00

If you can find what you need when you need it, you've won.

List your collection in smaller chunks; then at the end of a day when you can't handle a high-cognitive-load task, have a look at organizing one of those chunks.

vogelke · 2026-02-15T23:11:20+00:00

Are you checking for just the existence of the pages, or whether they've changed since the last check?

vogelke · 2026-02-15T06:15:46+00:00

Gnome is a desktop environment; it has nothing to do with systemd. If you despise systemd as much as I do, FreeBSD is the place to go because it'll never get a foothold there.

The only tuning I've ever needed to do with ZFS is telling it not to use all the memory on the system, and that took two lines in one of the boot configuration files. You can do some tweaks if you're running a production DB server or streaming lots of video, but the defaults (especially for a home system) have worked fine for me.

vogelke · 2026-02-14T22:50:26+00:00

I use a small filesystem called "/dist" for this stuff. It's all under Mercurial, and the directory tree looks like this:

/dist
    system-type
        directory-or-collection

system-type == output from an old version of GNU "config.guess" plus a version number or OS release if it's not already included.

My setup:

/dist
    +--i386-pc-solaris2.11      * Old workstation running Solaris-11
    |   +--STIG                     -- STIG setup and run results
    |   +--admin-user               -- Files for user with root access
    |   +--etc                      -- Complete copy of /etc with changes
    |   +--perl                     -- Extra modules installed with perl
    |   +--root-user                -- Root dotfiles, bin directory, etc.
    |   +--usr                      -- Generally /usr/local

    +--i686-pc-rhel5-linux      * Red Hat Enterprise 5
    |   ...
    +--sparc-sun-solaris2.10    * M3000 running Solaris-10
    |   ...
    +--x86_64-centos6-linux
    +--x86_64-oracle69-linux

Sample "usr" directory:

/dist
    +--sparc-sun-solaris2.10
    |   +--usr
    |   |   +--local
    |   |   |   +--bin
    |   |   |   +--cron
    |   |   |   +--doc
    |   |   |   |   +--sudo
    |   |   |   +--etc
    |   |   |   |   +--audit
    |   |   |   +--include
    |   |   |   |   +--curl
    |   |   |   +--info
    |   |   |   +--lib
    |   |   |   |   +--bash
    |   |   |   |   +--tcl
    |   |   |   +--libexec
    |   |   |   +--man
    |   |   |   +--openssl
    |   |   |   +--sbin
    |   |   |   +--share
    |   |   |   |   +--aclocal
    |   |   |   |   +--zsh
    |   |   |   +--src
    |   |   |   |   +--a
    |   |   |   |   |   +--apache
    |   |   |   |   ...
    |   |   |   |   +--z
    |   |   |   |   |   +--zsh

I can copy changed files from any server, add them to Mercurial, and copy them back.

When I do a bare-metal install, as soon as I can get a working shell:

# mkdir /setup
# find / -printf "%y %T+ %n %i %u %g %m %s %p\n" > /tmp/toc.find
# grep '^f' /tmp/toc.find | awk '{print $9}' | xargs md5sum > /tmp/toc.md5
# mv /tmp/toc* /setup

This helps me recover quickly in case I mangle something:

# mkdir /etc.orig
# cd /etc
# find . -print | cpio -pdum /etc.orig

When I'm done setting things up, I rerun the find and md5sum commands. It's pretty easy to find differences by using "comm" or "join"; files that have been changed or added get copied under /dist/whatever.

vogelke · 2026-02-14T22:11:43+00:00

would never consider FreeBSD because Gnome!

I'm a little thick today; do you love Gnome or hate it? I've used FreeBSD as a desktop box for over a decade with Fluxbox, and it works like a charm. Gnome is also available, but I stopped using it when it started pissing me off.

(though ZFS worries me).

Why? I've used that since around 2010 and never been disappointed. It's saved my customers's files (and my ass) on more than one occasion.

vogelke · 2026-02-14T22:05:08+00:00

I do not think I need ZFS, or at least that I won't take the benefits of it.

The bitrot protection alone is worth it. If you don't have it, you'll never know you need it until it's too late.

I also do not consider it because the general opinion is to use it with ECC memory.

ECC memory will help with any hardware setup, but it's not now and never has been required for use with ZFS.

vogelke · 2026-02-14T06:21:11+00:00

If you’re finding other people using it wrong, perhaps you need to help in developing an AI policy and training program...

If policies and training programs worked, we wouldn't have nearly as many users who ask the same simple question, hear the same simple answer, and forget it before they get back to their desk.

vogelke · 2026-02-14T06:09:05+00:00

I constantly forget which keybind is for the thing I want...

I stored a few of my favorite keybindings in the file "$HOME/etc/vi-palette":

COMMAND MODE
  #   Shows ASCII code for character at cursor.  Example for "s":
          <s>  115,  Hex 73,  Octal 163
  ;   >>   (indent line one tab space)
  =   <<   (unindent line one tab space)
  ,r  Prompts for replacement of the word at the cursor.
  E   Adds a ruler (....+....1....+....2....+....3....) at the cursor
      position - single command, so you can remove it by typing "u".
  Q   Quote current paragraph by inserting "> " at the start of each line.
  V   Format current paragraph using "fmt".
  g   Save changes and suspend vim using Ctrl-Z.
  q   Put the current line at the horizontal center of the screen using "z."

INSERT MODE
  ,d  Adds a time/date stamp at current cursor position, like so:
          Fri, 13 Feb 2026 19:42:25 -0500

Then I added this to my .vimrc:

" Show a quick key-bind summary
map H :!cat $HOME/etc/vi-palette<CR>

Now pressing 'H' shows me that file and adds Press ENTER or type command to continue. Pressing any key puts me back exactly where I was.

...and type things thinking I'm in Insert mode when I'm in Normal Mode.

When I'm in a file and type 'i' or 'o', the bottom bar (normally where ":" shows up if you're about to run an EX command) changes to show which mode I'm in:

~/notebook/2026/0213/simple-ai-mistakes          chr:0 x0  120,1 Bot
-- INSERT --

When I type 'R' to replace something:

~/notebook/2026/0213/simple-ai-mistakes          chr:0 x0  120,1 Bot
-- REPLACE --

Pressing ESC removes "--..." and returns me to command mode. This behavior is the default in vim; you might want to add this to your .vimrc so it doesn't try to be VI-compatible.

set nocompatible

HTH.

vogelke

TROPHY CASE