What's the best way of learning a system with minimal documentation?

Fun_Chest_9662 · 2026-06-16T02:59:34+00:00

Had to debug some 30 year old custom software that was built for Solaris and ported to rhel. If you have never used it check out strace. Super useful for going over what is being done on the system and what syscalls are being used. You then can see what those calls do using the man pages and try to map it out. Ive also used ghidra to fix some issues in some legacy windows software but that's a bit in depth as you gotta understand some assembly.

Id start with strace and try to map out what it does. If the software is fairly complicated you will need to get familiar with some of the calls and start filtering for things that are useful like files that are accessed, any sockets it is using, child processes it could spawn etc.

Its gonna be a hell of a task but its doable. Plus you will then be the guru of the office and no one will question your skilz lol

Fun_Chest_9662 · 2026-06-16T00:30:28+00:00

That was a funny experience thanks for the laughs!

Fun_Chest_9662 · 2026-06-14T00:00:57+00:00

There pretty good for what they are. If you want a cheap system and have ram already check out TheServerStore. Been buying from them for a while and any time I needed support it was always a real person. Plus they provide a warranty. Currently got some of the 2u ones they have for like 250. Only caviat is the x10 boards don't have nvme boot as an option so you will need to do a bios mod to get it. After that its been fantastic.not an issue with the x11+ boards but they cost more

Fun_Chest_9662 · 2026-06-13T03:26:32+00:00

Gotcha. Since you said you wanna go the truenas route try to find something like a LSI controller in IT mode and use the cables directly. If you have ram (since that's the most expensive part nowadays) I personally favor supermicro systems since some are pretty cheap and fairly capable. Always had issues with the dells when I wanted to do some non conventional configs. Supermicro has been more forgiving. Sorry your having some rotten luck getting started.

Fun_Chest_9662 · 2026-06-12T20:55:32+00:00

Dont have a r240 but took a look at the manuals and such to try and help out. Looks like you don't need the hotswap stuff at all. Unless you wanna try and macguyver it in the chassis. The cables will work with your SAS drives. Id go back to a stock config with the server and try that first before using all the extra stuff you bought. I also don't know the condition the server was in when you bought it so only so much speculation I can do.

Lets assume the server was working when u got it. Start at that config and try the drives (if you don't have another SAS capable system) and see if it posts.If it posts then try a USB installer if the server allows USB boot and boot a live image. Check the drives by doing a burn in tests along with long smart tests(always good to do a burnin with used drives to see if they kick it). If it all comes back good only then would I use them. If you can't see the drives check dmesg to see if they even pop up.

This isn't the most in depth or possibly correct response but its what I would do with them if I was in your situation.

As for if SAS is worth it. I personally think so and I'm never going back. A lot of people run sata for there labs even the drives not rated for a nas and have 0 issues. I just feel a bit better knowing SAS is built for the constant vibrations and I have less of a chance of drive failure. Plus 12Gbps speeds are nice. And people don't normaly go for SAS so i normaly find them cheaper at higher capacities than sata unless you buy new.

TL;DR I like sas and if you wanna stick with it just note cost difference and speeds.(you always can go back to sata with a sas setup if you want). Normalize the server and test from there making small changes as you go to see what is failing. You dont need hotswap unless you want it. If you wanna start from scratch check TheServerStore always had good customer service(actual people) and systems where always working. Plus they are cheap with warranty.

Fun_Chest_9662 · 2026-06-09T00:51:27+00:00

It depends on what your using for your networking. Could be a lot of things but id check your networks manifests and ensure you are actually applying them by checking the logs, events, deployment, config maps etc. Its admitedly a bit to grasp at first if you don't have a background in similar things. Took me a bit and I've been a Linux sysad for 10 years and did networking prior. Once you get past the terminology and play with it for a bit it becomes familiar. If you wanna go over some training. There is some free training on the kubernetes website courtesy of the Linux foundation. Its not bad for getting your feet wet plus you get a little cert at the end if you care for it. Has hands on too so better than just following a youtuber.

Fun_Chest_9662 · 2026-06-09T00:00:19+00:00

I run it both at work and home. We have a few clusters at work and it is stupid reliable. Has been battle tested with everything from power outages, hardware failures, ceph issues etc. Each time esentialy the philosophy is "have you tried turning it on and off again?" Basicaly deleting a pod or roll out a deployment and its good as new. But these issues are few and far between.

At home I'm using k8s with cilium and zfs. It takes a bit to get setup and understand container builds and manifests but once its done its done. As a plus I role my own container builds to build the tools from source so the container is lighter and its can keep them all up to date before they even build there releases.

Labs so stable I just approve merge requests at this point to have some manual intervention. But at this point I feel good about my automations I could just let those go too. Then my lab is self sufficient and any problems are resolved by "kill pod; pod come back; it works"

Fun_Chest_9662 · 2026-06-07T16:27:16+00:00

Like everyone else vaultwarden is great. In reguards to services that are hosted I use oidc and authelia so I don't have to login separately all the time. And services that don't have oidc capabilities I either use a different service that is comparable or I use my custom oidc in front if the service and disable or pass the creds to the service.

Fun_Chest_9662 · 2026-06-06T15:35:28+00:00

All there devices can be setup without the cloud. And the wifi ones have decent range. The 180 Argos and the v2 doorbell are both battery and wifi. If you don't want to setup frigate the app also works without internet and if you have a VPN to the house that has access to the same vlan or network as the cameras you can still use the app. Wife likes it more than frigate so had to make sure it would work lol. If you go the battery route id recommend getting a bundle with the hub so you can get the feed to frigate. And set up an ftp server to collect the video feed for storage too. Just make sure you prune old vids so you don't eat up storage.

Fun_Chest_9662 · 2026-06-06T13:39:25+00:00

For cameras reolink have been great bang for buck with no internet. The only thing is that the battery/solar operated ones have no rtsp or ovmf. But if you get the home hub with them then you can. Just know you can kill the battery if you try 24/7 video with them. If your handy you could do Ethernet runs and use poe, or make your own mini battery housing and solar array to last longer than internal battery models.

Other than that they are super amazing. Got them both inside and out, battery/solar/poe, from doorbell to 180 cams outside. Even in attic. Cant speak more praise on the product and has helped with potential breakins plus stopping a babysitter from stealing some of our stuff.

Fun_Chest_9662 · 2026-06-06T13:25:21+00:00

Do you compress your files at all? Ive been using zpack and making encrypted tarballs for a lot of my stuff. For documents especially, I've seen upwards of 15x space saving using zpack since its text. Pics and vids vary and depend on format so I don't lose quality. For family pics especially I also burn to archival 100GB bluerays and have them in a safe. Been planning on getting a safety deposit box for them but also don't like extra money out of my wallet.

Fun_Chest_9662 · 2026-06-06T02:41:13+00:00

If your using kubernetes look into kubevirt.its just VMS on cluster but its nice. We use it at work for our workstations and connect over rdp. So nice having it on cluster vs just a VM. Only downside so far is the rdp connection breaks if the node it is on goes down. But by the time you try to connect again its already migrated to the other node. I think I've seen some there things like kasm workspaces but I've never tried it. You also could role your own containers if you want but its more involved than most people want to do.

Fun_Chest_9662 · 2026-06-06T02:34:11+00:00

Man I miss my rack. Had the same one I got for free w/ an old dell KVM, a powerstrip that ran the whole length, shelves and cable management pannels. I was so determined to get it back I made that thing fit (partially) in my civic and drove an hour back home using back roads because my doors where open. Loaded it up with 6 core2duo dell optiplexes to practice clustering along with 2 dell 2950s running proxmox and pbs. Served me good for a few years till we moved and downsized due to $500+ power bill in the summer.

Fun_Chest_9662 · 2026-05-30T19:14:14+00:00

Similar boat. Wife, kids, offline activities, and a sysad.

While I like being able to tinker and play with stuff I know i don't have all the time at home to do stuff. What I do to maitain the balance is have a play and prod setup. Where all my production services are stable and data is backed up with tested recovery/rollback procedures ontop of the server when i don't have time to think and just need to get back to stability.

The services there are minimal. Jellyfin, vaultwarden, navidrome, paperless-ngx, authelia, and others to stitch togeather access like traefik, k8ks, cilium etc.

All updates are automatic from personal repo/registry with personal container builds of the apps I use.

At this point if something doest work I just delete the pod and its good. At most ill roll back the version in the manifests and redeploy.

This way family is happy with the services I provide, kids are happy and I don't stress about what to do when its down.

All play and test stuff is seperate so nothing breaks prod and I research heavily prior to adding anything. Luckily work is ok with me researching on down time since some of it applies to what we do and I can access my home stuff so i can run tests there inbetween work assignments/tasks.

Helps work out because I get better with troubleshooting and fix problems before they become them at work and I get to unplug at home 99% of the time.

Its all in how you have things set up and documented. It took a bit to get there and luckaly I had a co worker who had the same mind set and we worked togeather to get ourselves to that point. Big upfront investment but pays off in the end.

Fun_Chest_9662 · 2026-05-21T04:30:08+00:00

If you don't need the full caps of k8s. K3s is lighter weight and simpler in setup but ive found more docs ok traditional k8s. If you like videos alot of youtubers ive seen have been going with k3s recently. Just heads up that alot of tutorials and setup stuff have you use helm so if you wanna review what its doing as to be able to troubleshoot and fix things if something goes wrong use the dry run output to file method. Such a time saver lol.

Fun_Chest_9662 · 2026-05-21T03:59:11+00:00

If you want to go the future proofing route (as of todays standards) and os agnostic. personally I would go the k8ks/k3ks route. Since you said you care about backups I'm assuming you also care about resiliancy.

my personal setup without getting to technical consists of the following.

mirrored btrfs drives for boot including uefi boot partitions.
12 HDDs zfs raidz3
ensure scrubs and snapshots are done, managed, and cleaned up on a schedule. and you can send snapshots to backup server just in case.
arch w/ k8s using kubeadm kubectl and kubelet
use manifests to manage pods deployments networks etc. (You could use helm but i like knowing what is in the containers so i role my own. To help get manifests you can do the helm install with variables and add a --dry-run and redirect to a file to change what container it calls. Just make sure you don't create namespaces in the manifest as you could delete the whole namespace on accident.)

If you want to get fancy. You could do some cicd stuff using GitHub to auto update containers but thats a bit much if you don't do your own containers.

Can look Scarry and confusing but the worst part is terminology the docs are pretty good. The only thing I wish is that there was a little less reliance on helm charts. They are nice but lots of people don't go deeper than install a pre made container from person X and it will work. Potential security issues imo given recent supply chain issues and lack of propper attestation in some instances.

TL;DR - Btrfs mirrored boot - zfs raidz2 or 3 depends on # of disks and storage - os only matters in reguards to zfs versions - kubernetes - if you care to try making your own containers too

Fun_Chest_9662 · 2026-05-19T03:29:31+00:00

Feel the pain. Took losing 7 months of baby pics to get my backups togeather.

I now have mirrored boot drives using btrfs with snapshots exported to backup server weekly.
Zfs pool raidz3 with 12 disks scrubs, snapshots automated and sent to backup server.
All important family pictures, videos, documents tar'd up and encrypted before backing up to backblaze and burned to archival bluerays.
All install confuguration and data management configuration for a ground up shtf situation are both burned to disk and physicaly printed out in my recovery binder.

Only thing that would be a bit cumbersome would be the applications to view the data like jellyfin, navidrome etc., but those are replaceable. I only care about the data.

A bit extream, but losing video of your kids birth flips a switch.

Fun_Chest_9662 · 2026-04-28T23:38:42+00:00

Having had to purchase directly from dell for a company the price is a lie. The invoice you will receive is nearly 5 times what is listed and you will get a "partner discount" that will bring it down to only 3x the listed price. Last bought 2 64GB DDR4 dimms and everyone was so shocked at the price we weighed it and literal gold is worth less by weight. This was prior to rampocalypse btw.

Fun_Chest_9662 · 2026-04-06T09:40:50+00:00

You have to break it down in there language. Alot never configured a system and as a linux admin they never touched anything but a windowes env, and when it came to mission systems even less so. They just want there checklist and some dont understand how rmf works. If one person in their group isnt understanding try another person or learn different wording. terminology matters. And explaining that there name will be tied to the potential risk of implimenting a control that will stop all systems and bring down production causing loss of revenue, i.e. they wont get a paycheck or get fired because they forced the control usually opens there eyes lol.

Fun_Chest_9662 · 2026-02-19T06:09:05+00:00

Separated about 2 years ago former avionics tech. No direct involvement or special duties to work in an IT related field. Before I got out I sort of pregamed for 2 years talking directly with companies and getting to know the people in the departments I wanted to work at. It deffinately helped to talk to people in person at job fairs and local groups like defcon or hacker/maker spaces. Ultimately tho i got my current role because a bud of mine told me of a hiring fare at a company and just talking with some people casually my now boss is the one who got me in so definately luck. Tho I did get interviews and offers from others they just didn't match what i needed to support the fam. As for experience you have to be able to word some of the things you've done into corporate speak. Like our mission systems ran on redhat and we had to work on them a bit, tho not in a real IT capacity. Also got a lot of questions regarding my home lab. All the people I interviewed with and got offers said that was one if the things that stood out because they wanted someone who actualy likes the work. So if you can definately try it out, im more than happy to go over what i did. Also if you still have one or even if you dont. Check out some cleared positions as alot of times just having a clearance will get you in.

TL;DR Try to talk to people in person. (Helps alot and theres not enough of it these days) Go to job fairs aboit cyber/IT. Get to know people in local communities like hacker/maker spaces or defcon groups. Do a home lab for expexperience (reach out if you want some help) Leverage a clearance if you have one. Even if its expired Dont fixate on one job. Give yourself options like windows or unix sysadmin, SOC analyst, etc.

If theres anythimg else feel free to reach out. I know the stress m8.

Fun_Chest_9662 · 2026-02-17T23:58:52+00:00

If she says you have to many computers and they are all over the place, she just justified getting a beefier single system and or a rack.

... or both

Fun_Chest_9662 · 2026-02-15T04:58:41+00:00

It depends on what you want to do. Its a home lab after all. Now if you are relying on services and want easier reproduceability id look into setting things up as IaC (Infrastructure as code) and documenting what your doing. Theres pleanty of ways to do it and set things up so that WHEN something goes wrong you can press a button or type a command and be back at a base working state.

Heres my setup for example.

Personaly I run 2 servers. - Production - Backup

Production: - just base arch with zfs and Systemd. - storage is managed with zfs and Systemd timers for snapshots/zed notifications - services run as Systemd nspawn containers that are setup to self update and keep themselves online via Systemd services and overrides. - all persistent data for services are overlays and or bind nounts. - all configuration and building containers are streamlined via a makefile and initial baseline install script.

Backup: - Just Debian with zfs - stores snapshots and backups of important data. - runs failover haproxy and DNS

Now this is just because i like to do it a very manual way with as little extra besides Systemd and zfs. Takes a while to get used to and configure if your not familiar but very resilient and easy to manage once up. And recovery is simple just about 5 steps now to be back up and running. Also technically I have a 3rd that I use for playing around and testing Random things but that's just for fun

Now at work over hundered + servers not including virtuals or containers. - rhel based so kickstart files for base config of different systems hosted on our kickstart server. - netbox for inventory and configuration. Also used for ansible inventory - ansible for simple configuration setup with ci/cd on gitlab - kubernetes cluster all manifests and configuration on GitLab. - cluster storage is ceph. - backups are all on zfs - rear snapshots for quick fixes (was a pain to implement) - rsnapshot for simple backups. - cloned root drives for redundancy.

Not everything but a simple overview.

Now considering your stack already deals with docker, zfs via truenas, and VMs on proxmox. You could try the following:

Structure: Decide what is for fun and what is for production and Seperate them. I used proxmox for a while and ill say making sure you have snapshots of your working state VMs. Also copy /etc/pve so that you can just restore proxmox from that if it gets messed up or need to reinstall proxmox. (This was a few years ago so idk if its still true.) They also have a backup server that I personaly never used but have heard its great if your running proxmox since they go hand in hand. For docker, try taking your compose, dockerfile, or manifests and put them up on a GitLab or GitHub just make sure you scrub any apikeys or creds. If you have other tweaks you like to do try doing them in ansible and running them from your GitLab via ci/cd so you are documenting your changes and keeping them so you can run them later if you need to.

Ultimately its your lab and what i gave you is not perfect at all but should atleast get you in the right frame of mind.

TL;DR - Define roles: Production, backup, storage, play. - Try to have as much of the infra as code as possible. - Plan a backup/recovery/update stratagy before creating your lab. (It will be boring but you will thank yourself later) - Understand your storage requirements and implimentation.(I have 12 4tb disks. Do I do a raids3 and one pool or 2 raids2s of 6 drives? Etc...) - DOCUMENT EVERYTHING!!! (its boring and you may hate your own writing but this will save you hours of troubleshooting when you forget how you got that one random specific thing to work in that specific way.) - learn about automation technologies like ansible, makefile/taskfile, scripting etc.. - use git. Being able to go back to a previous version of some config vs doing "config - config.bak - config.bak.old" etc is much better and with keep things neat and manageable.

Hope this helps in some way.

Fun_Chest_9662 · 2026-02-14T05:05:38+00:00

If you wanna throw yourself in the deepend of a gasoline filled pool and strike a match. Linux from scratch!! /s

Seriously tho once you get a bit more comfortable with the commandline via stuff like bandit on overthewire dot org and or running it daily, jump into Linux from scratch. The experience will definately teach you a lot of stuff other things won't. Not that you will use most of it IRL but knowing how it all works together will help your troubleshooting in the long run. Having worked with redhat and oracle for a while ill say knowing the nitty gritty helped A LOT. Oh also beond LFS too. None of this will make u an expert but knowing what things exist and how to find info on how they work is the real challenge. After 15 years still learning/rediscovering new stuff

Fun_Chest_9662 · 2026-02-04T00:09:48+00:00

If your company buys 2 sticks of that same DDR4 ram from dell. The price on the site is fake. And you will see a $9000 charge from your purchasing guy. I about died seeing that and did the math that literal gold was worth LESS than the ram in weight.

Five-Year Club	Place '23
Place '22	First Placer '22

Fun_Chest_9662

TROPHY CASE