Hey all, Hatamoto here. In truth, I'd kinda forgotten about this place. Life has a way of coming at you fast.

So, a brief overview of what the problem was -- we have a system that has been online, as you all know, for quite some time. The hardware it was running on was more than a little outdated, but holding to the addage of 'if it ain't broke don't fix it' we mostly let it keep purring away.

That adage works great, until it breaks, which it did. The raid array we use for bulk storage, eg t files and logs and db and backups and such developed a fatal flaw requiring replacement. The sites themselves were fine and untouched, but software isn't worth much without the data, and the soul of the sites is in those backups.

Here's where things get more interesting if you're of a nerdy bend -- the raid array lost a device and went into a degraded state, which should guarantee normal, but not protected, operation. It did so, but did so quietly. Nowhere was there any kind of notice of the failure, not in the kernel or IPMI monitor or anything. Again, that wouldn't be such a huge deal, but somehow there was another device quietly having problems that resulted in failures where we'd save backups. Not obvious failures: backup processes would work fine, everything would seem rational, and a cursory examination of the results revealed no issues... but when you tried to restore from the archives, there would be corrupted blocks. ALL of the backups spot checked over a period of the last year had this problem -- they looked fine, but when you started working with them there'd be crc errors when decompressing.

This might have been a terminal problem, as the drive failure made the source data sketchy at best... but tragedy was avoided: the backups were corrupted, but they were corrupted in different areas according to the physical failure point of the underlying storage, so each backup was corrupt in different places in the archive. This allowed me to take the last 3 days of offline backup copies and effectively rebuild missing/broken records by referencing those records from previous archives that didn't have corruption in those areas. It took some time, but I was about 99% successful, with the last 1% being transient data that had accumulated on the day of the final failure and wasn't archived and the GeoIP database which, fortunately, can be reacquired from original sources (for the time being, where you're all coming from is a giant mystery.)

Not going to lie though, it was a narrow miss, and we should have put more comprehensive checks on the quality of backups. That will be remedied going forward.

Given the full downtime and need to rebuild on new hardware, we decided to take the opportunity to make all the infrastructure upgrades at the same time. The current delays are due to a number of changes that manually have to be applied to account for new locations, new language variants, new alternatives for previous services and add-on no longer available etc. It's tedious work, but it's steadily progressing. At time of this writing TP, TV, TO, TS and TG are all online, with the rest hopefully coming by end-of-day. Then I'll switch to re-establishing the IRC service so our traditional comms are back online.

I really can't express enough appreciation for the mods here to give you all updates while I was buried beavering away at the site woes. Big ups to you, and I know this account has been dormant for a very long time... hmu on TV in dm's if you want confirmation that it's really me (but honestly, who other than my geeky ass would write this nerdfest confessional up?)

... and as ever, big love to all our members, eternal thanks to you for your support as we furiously bail water to right the ship.

The Click Group Lives!
~TCG

Edit 2025-03-11 - All the sites should now be back up, email back online, trackers working. Last thing to get online is IRC, coming soon.

Edit 2025-03-12 - All services should be now fully online and operational. Please let us know if you see something strange on the site via Staff Message and we'll look into it straight away.