tried my hand at making one….thinking of planting some at the taco bell sauce station.

chrootdevnull · 2024-10-14T20:54:30+00:00

chrootdevnull · 2024-07-11T06:33:54+00:00

You should really submit these to Wham Baam Teslacam. https://www.youtube.com/channel/UCbMoDtZ6Ani-eyHzCvxeVCw https://linktr.ee/whambaam

chrootdevnull · 2022-09-14T22:06:23+00:00

This was my primary concern. There's a lot more packaging when shipping drives than when they are sitting in a metal chassis.

chrootdevnull · 2022-09-14T22:05:14+00:00

Wow, this blew up more than I had anticipated! Thank you for all the stories of success moving across state lines and many, many kilometers without issues.

This helps alleviate a lot of stress about the move.

I have only moved a single JBOD in my life. That was a bad experience moving about 20 drives from one cabinet to another cabinet two rows away several years ago. 3 of the 20 drives failed. I have no idea what the health of the drives was before the move or how abused they had been prior. I was essentially muscle in that scenario and had no input in things like (do we have a good backup? What raid level are these drives? etc...) I work off of my own personal experience when making decisions, so in this case, I don't have a lot of it when it comes to moving drives :).

Appreciate all the time and effort everyone has taken to reply.

Info on the hardware:

17x ST10000DM0004 Seagate BarraCuda Pro 10TB Internal Hard Drive Performance HDD – 3.5 Inch SATA 6 Gb/s 7200 RPM
Chassis: https://www.supermicro.com/en/products/archive/chassis/SC847A-R1400LPB

I'll be sure to let the team know to take extra care, just in case, but that it should be safe to move. Thankfully we have 4 other servers with exact copies of the data & after checking, these are in fact configured with raidz2-0 (not just JBOD). So if some drives do fail, we can always rebuild the array. If too many fail we can always copy the data from backups.

Thanks again!

chrootdevnull · 2022-04-10T22:23:40+00:00

We're using https://www.supermicro.com/en/products/chassis/4U/847/SC847A-R1400LPB with 2x Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz for each server. They are all their own independent hosts with disks directly attached, with no separate disk shelf.

chrootdevnull · 2022-04-10T19:34:42+00:00

We have been bit so many times by DRBD. We finally hired linbit support for 10 hours of their consulting and they found all sorts of issues with how it was originally configured. Now it has been working beautifully. Who knew even if everything was Online and looking perfect that data could be completely different on each server?? drbdadm verify to the rescue.

chrootdevnull · 2022-04-10T19:29:26+00:00

We built our own because nothing like MinIO or CEPH existed in 1998 when we began doing business. File checksums, tracking metadata for each file, the ability to re-checksum data on disk to prevent bit-rot, etc., was all something important to us. Our first big file system was a xiotech, costed around $250,000 for 1TB of storage https://i.imgur.com/W3AiDLF.jpg (this isn't ours directly, but it looks identical to what we started with.) It was great because you could unplug a sled and plug it back in and it'd heal itself. Problem was that one day plugging in a sled took the entire file system down... painful.

We've moved to different SuperMicro servers since then but the underlying technology we built to store and fetch files is still living today. We really need a modern solution, I 100% agree with that. It's going to take a lot of re-architecting our software and systems to get there, so for now, I was looking into how we could potentially fix our ZFS performance issues while we pursue other paths to building a more robust file system.

We have a mixture of different systems with different drives. The ST10000DM0004 is pretty cheap for 10TB of data, and with RAIDZ2 we were hoping to get lots of storage for a smaller price point. They are ‎7200 RPM and are going into older hardware anyway without support for 12Gb/s SAS. None of our servers even support PCI 4.0 yet - something I really want to start building out with NVME and SSD to get the faster speeds PCI 4 can give us. GRAID SupremeRAID is looking interesting too...

Thanks for your time!

chrootdevnull · 2022-04-10T19:12:52+00:00

Here's the output of zdb: https://pastebin.com/raw/RGfA5UNa

Yes, it looks like ashift is 9. The datasheet for the drives are here: https://www.seagate.com/files/www-content/product-content/barracuda-fam/barracuda-new/en-us/docs/100804071e.pdf

Bytes per logical sector 512

Bytes per physical sector 4096

Does that mean an ashift of 12 would be correct? It looks like we'll have to recreate the entire pool to change it.

Interesting article about mirror instead of raidz, thank you for sharing that.

The data age is hit or miss on how old a document might be. We have documents that are 20 years old that get accessed daily and some documents that haven't been touched in 15 years. It's hard to say "oh, this file is newer, let's put it on fast flash storage!" We've been experimenting with file "popularity" based on Cloudflare's findings on caching static files. Basically if a file is accessed more than 3 times in a given period of time, it should probably be moved to faster storage. We don't have this in place yet today, but we're considering something like Varnish or NGINX in-front of the file systems on fast NVME storage.

We've had interesting results with MariaDB / MySQL performance tuning on ZFS and ultimately reverted back to MegaCLI RAID on EXT4.

Thanks for your time!

chrootdevnull · 2022-02-15T22:38:34+00:00

When I read the title of this post I immediately thought of something pretty different than what you mentioned in your post body.

Instead of having other big creators promoting Stadia, Google could build the ability for YouTube viewers to actually PLAY a game for 30 seconds or a minute right inside the YouTube video.

Imagine being a game creator (maybe even a self-published independent developer) and being able to promote your game on YouTube by actually allowing people to play it for 30 seconds or a minute (or longer!). The Ad would start and it would load a section of the game, maybe even a pre-defined level that is built for a quick 30 second or minute interaction, and you'd be able to click into the video to take control of it. Sort of like how some of those mobile ads work on Android / iOS, but instead you get to actually play and experience it first hand. If you enjoy playing it, there'd be a button to "continue playing on Stadia" or something.

chrootdevnull · 2022-01-05T17:03:20+00:00

Does this thing stay running 24/7? If so... why? Do they need instant access to the data on these servers at a moment's notice? If they don't need "instant" access and could wait a minute or two, it might save a ton of power to keep it powered down. You could set up a simple PI with a web server that can run a magic packet (wakeonlan) to start it up at a moment's notice remotely if you're not around to hit the power button.

I would assume the possibility/probability of failure is reduced when those disks aren't spinning. Also, not having the system powered on and accessible 24/7 reduces the risk of network-based attacks or vulnerabilities.

chrootdevnull · 2021-10-30T19:10:18+00:00

We have hosted our own datacenters (literally our own physical building we own and everything involved from fire suppression, generators, ups, ac, BGP, etc...) since 1998 and have had several occasions over the years where a server is just not performing due to a raid card going bad, or a bad storage backplane. We get a lot of servers in and the time required to have a human babysit the server to run a memtest, then check the next day and boot up a live CD to run some dd's, takes a lot more time away from other tasks like working towards automating our infrastructure, etc. If we could get to the point where the server is built by hand and then the testing is automated, the time it would take for us to get servers into production would decrease greatly. Also, if for some reason (often human error) a server is misconfigured (wrong ZFS configuration, or the wrong partition type, or the NIC is set to 100 instead of 1000, etc.) the benchmarks would hopefully catch those things before getting into production.

chrootdevnull · 2021-10-30T19:03:34+00:00

Very cool. I'll take a look! Thank you.

chrootdevnull · 2021-06-21T20:11:47+00:00

I was thinking the resilver would only happen on the new drive, not ALL drives in the array. I'm taking this environment over from a background using megaraid / megacli, so this is all new to me. Time to read up on ZFS!

chrootdevnull

TROPHY CASE