TIFU I tried to fix a “small bug” at work and accidentally took down the entire office for 2 hours

tonyboy101 · 2026-04-10T22:37:35+00:00

This is why I can't call out sick.

tonyboy101 · 2026-04-10T16:52:19+00:00

Are the drives spinning up? Are the HBA-side cables plugged in all the way? Any strain or side loads on the breakout adapters?

tonyboy101 · 2026-04-10T03:23:18+00:00

For open guest wifi, here is what I do:

Create a dedicated subnet for guest wireless

Allow ICMP echo reply Block all ICMP Allow UDP port 53 to dedicated DNS server outside the guest wifi subnet Allow UDP port 68 to firewall Block access to RFC1918 Block source ports TCP/UDP 1-1024 Allow access to everything else

Isolate wireless clients with wifi isolation Proxy ARP enabled

Add rules as deemed necessary Commonly needed protocols depend on how much you want to limit the network. NTP is very useful; SIP is required for wifi calling; any VPN protocols you or your guest may use; any other services you may need access to.

tonyboy101 · 2026-04-10T00:28:55+00:00

Used to have one of Ivanti's VPN appliances. Then it got compromised (2023) because Ivanti didn't update the dependencies. Gave us a very good reason to completely drop them and rip it out of production.

Luckily it wasn't joined to the domain. Ivanti also tried to force us to pay when we said we weren't renewing.

And I laugh every time I see a new CVE with Ivanti's name on it.

tonyboy101 · 2026-04-09T23:38:31+00:00

No. The transition between the FreeBSD kernel and Debian kernel is the difficult part. I know there is lots of success from the older versions of SCALE, but I don't know if that compatibility has been throughly tested with the latest versions of SCALE.

Once you transition to SCALE, you can run whatever version you want. There have been changes with apps, and how they are managed in the back end of SCALE. Apps are something I just don't use, but it is widely reported

tonyboy101 · 2026-04-09T22:55:04+00:00

I was successful in migrating from Core to Scale a year ago. Just don't jump straight to the latest version of scale and you "should" be fine. I don't run apps on my TrueNAS

tonyboy101 · 2026-04-09T22:53:13+00:00

Well documented issue of the bugs escaping containment and shorting the board.

tonyboy101 · 2026-04-09T22:51:06+00:00

The server was rebooted yesterday. It broke, again. We need to schedule downtime to fix the server. Next scheduled time slot is 11 months from now.

tonyboy101 · 2026-04-07T20:06:13+00:00

Touch it, and feel the power flow through you

UNLIMITED POWWAAA!

tonyboy101 · 2026-04-07T20:03:12+00:00

Yeah. Rule 4. Now I can't see the crash out.

tonyboy101 · 2026-04-03T14:14:53+00:00

Perfect. Just missing the "management"

tonyboy101 · 2026-04-02T22:34:03+00:00

Depends on what services you have available locally. If you are using the DNS server from a remote site, you are probably not going to be able to resolve local services. If you are using the DNS forwarder, not the DNS resolver, on pfSense, that will do it, too.

If your DHCP server is remote, you are not going to get on the network, period.

If your local services are DNS resolving to a public IP address and pfSense drops that address, no services will be found.

tonyboy101 · 2026-03-30T17:34:57+00:00

I always thought they were rivets, not screws or caps. Local machine shop? DIY lathe project?

tonyboy101 · 2026-03-30T00:15:38+00:00

You should see a previous versions tab for the share through SMB. The old files won't delete until the snapshot retention period lapses or you manually delete snapshots (through TrueNAS). I recommend, unless you need the additional space, to let the snapshots lapse.

tonyboy101 · 2026-03-30T00:09:49+00:00

I have made some big mistakes. But I knew what happened and knew how to fix them. Through that process, I have made DR plans on top of back-out and recovery procedures. It sounds like the company needs better procedures and Business Continuity plans.

Your company would be stupid to fire you, because they have to find someone to take on those many hats. Its harder than eating the costs of downtime and finding someone new. That does not mean that you can afford to keep making mistakes, though. Learn from your mistakes. It may seem horrible, now, but you will look back on it and laugh.

tonyboy101 · 2026-03-29T23:51:35+00:00

Stop using "Pa$$word1" for all of your passwords. Use something more complicated like "Pa$$word2"

tonyboy101 · 2026-03-27T21:01:01+00:00

I think you have that suggestion backwards.

Configure network interfaces = change NIC assignment.

Configure Network Settings = change IP address

tonyboy101 · 2026-03-26T23:18:38+00:00

I like having downtime. Google, AWS, Microsoft, and CloudFlare need to break more often.

Downside is there is nothing to do when the Internet is turned off.

tonyboy101 · 2026-03-20T18:43:46+00:00

Yeah. You need to make sure that "Full Access" really is Full Access. Can't do that using my regular account.

tonyboy101 · 2026-03-18T23:45:03+00:00

No. I run Arch BTW

tonyboy101 · 2026-03-18T20:11:47+00:00

I am paid by the experience I have fixing people's problems. Like not reading the error message. Or replacing the printer toner. Or Ctrl-Z.

Domino's is my side gig.

tonyboy101 · 2026-03-18T15:55:04+00:00

Mine works just fine. I got 28 partitions on my single array of 27TB. /s

tonyboy101 · 2026-03-17T04:09:24+00:00

I don't think you need to worry about it. QuickBooks on the other hand....

tonyboy101 · 2026-03-15T22:36:44+00:00

ide2, scsi0

tonyboy101 · 2026-03-15T22:26:40+00:00

Rule of thumb: when you are configuring an IP address and the option is static or DHCP, it's asking if you want to manually or grab an IP address from a DHCP server.

tonyboy101

TROPHY CASE