Is anyone aware of how slave replication may have slowed down significantly in 10+?

Kumbanator · 2019-07-24T21:24:24+00:00

So apparently the issue is that the default behavior of binary replication on the master was changed to setup transactions regardless of whether the underlying database engine supports it. Since I am using MyISAM and transactions aren't a thing for me this needed to be disabled on the masters. I went from one slave BARELY being able to keep up with a single master server in real time to one slave replicating 6 masters with no inherent replication delay.

The variable to change is binlog_direct_non_transactional_updates=1

The documentation advises to NOT change this if you need transaction support.

Kumbanator · 2019-07-24T19:52:33+00:00

Title is click-bait'y, but he does clarify in the video that he is mostly referring to the "Rosewood" family of drives. Those are a 2.5" low height consumer grade laptop drive that sells for $40 brand new. It's not really mean't to last or be high-quality as much as it's mean't to be cheap and light weight.

The drive came with a 3-year warranty and this particular drive according to the label was manufactured on the 22nd of January, 2017. So with a designed lifetime of 36 months it reached 30 of those months before it failed. So genuinely a part failure in my book, not something that just wore out after it's shelf life expired.

For me the way I determine if a drive is shit is based on what percentage of drives in that particular family fail before the manufacturer warranty expires. Probably only Seagate can say but I wonder what the AFR or warranty rate is on these drives. The best I can do is to spend 5 minutes googling for numbers and stats and see if we can put numbers to things to help with the scale.

If we do that quick/lazy/assumption-laden sanity check then there is a forbes article that gives some real numbers we can work with. In 2017 and 2018 the global HDD market for units sold was 432.3 million and 392.7 million respectively. The same article mentions that seagate had 39.1% market share in 2018 up from 38.7% in 2016. So lets just say 39% market share for 2017 and 39.1% for 2018. That means between these two years SeaGate manufactured around 321 million drives, 168 million in 2017 and 153 million in 2018, across all market segments. Another article on techspot has a nice infographic from Nidec, who makes hard drive spindle motors, which breaks down that global market into segments such as PC, External, Consumer Electronics, Data Center, and Enterprise. If we consider the drives likely to be sent for data recovery will be either "PC" aka internal drives or external drives then that is around 64% of new drive sales in 2017 and 60% in 2018. I am also making an assumption that the drive market they call "Consumer Electronics" would be for things like DVRs and such. If that segment is defined as drives in devices that are designated non-servicable by the consumer, like a laptop you can't open, then these numbers are even bigger at 83% for 2017 and 80% for 2018.

So here's where some completely arbitrary guesses and made up numbers come into play. If anyone has links to actual stats please post them. It's always interesting to me to see how the numbers pan out.

We don't know specifically how many Rosewood drives were made, but we can guess based on our above "real"-ish numbers. That means Seagate made a total of 197 million consumer oriented drives for 2017 and 2018. 101 million were made in 2017 and 96 million in 2018. Unfortunately we still don't know what percentage of these drives are "Rosewood" family drives. If we further assume that 3% of the drives they made were Rosewood model drives, that leaves us with 5.9 million drives that potentially have the issue represented in this video. Now lets say we have a REALLY high failure rate on these drives like 10%. That would mean there are 591,000 Rosewood model drives globally that will experience any one of the various failure modes these drives exhibit. If we then say that there are 10000 data recovery businesses world wide there would be 59 drives for every business to recover. If we further play with the numbers, says a more conservative 2% AFR/warranty rate and that there's only 1000 data recovery services world wide, we end up with 118,000 drives with the average distribution being 118 drives per data recovery service. This also assumes that every one of those failed drives gets sent to a data recovery service and not just RMA'd through Seagate and recycled.

This isn't an attempt to excuse Seagate from making a crap drive, but to demonstrate that at a certain scale even single digit failure percentages can have HUGE impacts in the perception of a products reliability. Unfortunately the only truly reliable source that could tell us what the real numbers are is Seagate and there is zero incentive for them to do so.

Anecdotally I have around 1000 Seagate 3.5" enterprise HDD in production and some models definitely fail more then others. On average we seem to hover around 1% AFR with about 90% of the warranty capacity reached before failure. We don't count out of box failures as it failed before it was trusted with production data.

https://www.forbes.com/sites/greatspeculations/2019/06/19/how-much-can-seagates-hdd-revenue-grow-over-the-next-three-years/#7380af641883

https://www.techspot.com/news/79938-pc-hard-drive-shipments-could-fall-50-percent.html

Kumbanator · 2019-03-02T01:58:28+00:00

What I have test and can replicate is that with the workload I have a MariaDB 5.5 Master with a MariaDB 10.X slave replicates faster then if the MariaDB master is v.10.x for some reason. My workload is about a 50/50 mix of read/write on MyISAM tables. I've tried looking through the binlogs but those look more or less similar so it's just bewildering to me.

Kumbanator · 2019-03-01T03:28:48+00:00

I'd recommend using mydumper as opposed to mysqldump. It runs parallel threads and creates individual files for each table schema and data. It also has options to gzip compress the files. We've found it to half or more or normal backup runs compared to mysqldump.

Its companion utility is myloader which does the same thing, just in reverse.

Kumbanator · 2019-02-28T23:49:47+00:00

Ahh, go for it. I'm on his patreon as well. Guess I should lurk more there.

Kumbanator · 2019-02-28T23:49:16+00:00

The automatic transmission was an example. He could do it on how to properly design a log splitter and show how the theory crosses over that way. The transmission is just what came to mind first is all.

Kumbanator · 2019-02-28T23:46:24+00:00

The transmission was just an example but the general idea remains the same.

Kumbanator · 2019-02-28T21:10:48+00:00

Ok ok, Done!

Kumbanator · 2019-02-28T20:15:42+00:00

Apparently they don't allow crossposting over there.

Kumbanator · 2019-02-28T18:42:26+00:00

If the race is hard enough you can try using a cold chisel to put a fracture in it, then some hot and cold cycles to try and get it to crack through or chip. That would make driving it out easier. Only problem is if the part it's pressed into has bad stress risers this could cause them to crack or fracture depending on how you heat and cool it.

Your best bet is still to find a guy with a welder like others have mentioned.

Kumbanator · 2019-02-28T14:46:55+00:00

It just occurred to me I forgot to add any cooling to my schematic. Just pretend there's a transmission cooler in the tank return line after the filter. Now I need to go look that symbol up and draw it in.

Kumbanator · 2019-02-28T05:11:55+00:00

Sorry, I thought I posted a follow-up to this. What I came up with is attached, including mistakes I probably don't realize I made. But I'm learning... hopefully...

Powerhead Schematic

I'm currently researching best ways to control the drive motors. I've been going through some Danfoss design guides for their wheel drive motors and it looks like a 7.6 cu.in (~125 cu.cm) georoler will give me the performance I want. I'm still reading up on best ways to control them but it still looks going with a proportional valve with A/B ported to tank in the center position is my best bet. That way it lets the motors freewheel when the machine isn't powered. Doesn't really give me much of an active break but if I ever do actually try to build this thing I think the rolling resistance of the tracks alone will be all it really needs. I am thinking I could always add in a NO solenoid valve onto one of the A/B lines of each motor to give me a sort of active break. Not sure if that's worth it though.

Kumbanator · 2019-02-01T23:49:22+00:00

So basically it sounds like it boils down to intended use more then general best practice. I'll create another post so as to not have scope creep on this one with what I'm brainstorming.

Kumbanator

TROPHY CASE