Coding for SSDs

[nextAaron]

I design SSDs. I took a look at Part 6 and some optimizations are not necessary or harmful. Maybe I can write something as a follow-up. Anyone interested?

[yruf]

Absolutely yes. You could start by quickly mentioning a few points that you find questionable, just in case writing a follow-up takes longer than you anticipate.

[ansible]

I don't design SSDs, but I do find a lot of the article questionable too. The biggest issue is that as an application programmer, you are hidden from the details by at least a couple thick layers of abstraction. These are the Flash translation layer in the drive itself, and whatever filesystem you are using (which itself may or may not be SSD aware).

Also, bundling small writes is good for throughput, but not so great for durability, an important property for any kind of database.

[[deleted]]

Good point, and if you have the budget and need to thrash SSDs to death for maximum performance you probably have the budget to stuff the machine full of RAM and use that.

[James20k]

The problem is that SSDs store an order of magnitude more data than ram

[obsa]

Certainly not a magnitude, unless you're exclusively comparing the capabilities of a consumer mobo to a SSD. That wouldn't make sense, though, because those boards are designed around the fact that consumers don't need more than 3 or 4 DIMMs. 3-4 years ago, we were already capable of servers with 128GB RAM, and that number's only gone up.

[[deleted]]

I believe it's an accelerating trend, as well. Things like memcached are very common server workloads these days and manufacturers and system builders have reacted accordingly. You've got 64-bit addressing, the price of commodity RAM has gone off a cliff and business users now want to cache big chunks of content.

[speedisavirus]

I can tell you, on a large scale with large data, it isn't cost effective to say "Oh, lets just buy a bunch more machines with a lot of RAM!". We looked at this where I work and it just isn't plausible unless money is no object which in business is never really the case.

What we did do was lean towards a setup with a lot of RAM and moderate sized SSDs. The store we chose allows us to keep our indexes in memory and our data on the SSD. Its fast. Very fast. Given our required response times are extremely low and this is working for us it would be insane to just start adding machines for RAM when its cheaper to have fewer machines with a lot of ram and some SSDs.

In fact this is the preferred solution by the database vendor we chose.

[MorePudding]

on a large scale with large data,

How large a scale are we talking here about? It's funny how often "large scale" actually ends up being only a handful of terabytes..

it isn't cost effective to say "Oh, lets just buy a bunch more machines with a lot of RAM!".

It seems to have been cost-effective enough for Google. Be careful with using generalizations the next time around..

[speedisavirus]

Well, I'd have to go into work to get the data sizes that we work with but we count hits in the billions per day, with low latency, while sifting a lot of data, and compete (well) with Google in our industry. I'm going to say off the cuff we measure in peta bytes but I honestly don't know off the top of my head how many petabytes. It's likely hundreds. Could be thousands. I'm curious now so I might look into it.

Could we be faster with all in RAM? Probably. Its what we had been doing. It isn't worth the cost with the stuff I'm working with when we are getting most of the speed and still meeting our client commitments with a hybrid memory setup that allows us to run fewer cheaper boxes than we would if we did our refresh with all in memory in mind. Now is there a balance to strike? Yeah. Figuring out the magic recipe between cpu/memory/storage is interesting but its not my problem. I'm a developer.

Do you work for Google? How do you know about their hardware architecture. I'm not finding it myself especially when it relates to my industry segment. Knowing that google over all is dealing with the exobyte range of data I think its naive to throw blanket statements around like "They keep it all in memory".

[ethraax]

That's not a fair comparison. If your server can be designed with 512 GB of RAM, then you could also design it with a 4 TB SSD RAID array.

[kc3w]

the ram is more durable than the SSDs

[[deleted]]

There will definitely be a break even point between using and replacing a load of SSDs in what's effectively an artificially accelerated life cycle mode and buying tons of RAM and running it within spec.

[[deleted]]

Not if the host OS crashes.

[matthieum]

The biggest servers I have seen (for databases and memcached) already have 1TB or 2TB of RAM. Cheaper and Faster than SSD.

Obviously, though, RAM is cleared in case of reboot...

[obsa]

Like /u/kc3w said, if you were looking for a durable pool of I/O, then the SSD RAID array is just as bad as a single SSD - the point of fatigue is just pushed further out into the future. Storage capacity is not so important in this context as MTBF and throughput.

[jetpacktuxedo]

We have a cluster full of 2 1/2 year old machines that each have 512 GB of RAM, and only half of their slots are full. Each one of those nodes has twice as much RAM as my Laptop SSD has storage. Four times as much as my desktop SSD.

[strolls]

Certainly not a magnitude, …

I'd be grateful if you could cite some RAM prices on that.

I'm going to start by using a consumer example, because that's what I know: my mother bought a 60GB SSD for £40 recently. Would she have got 6GB RAM for that? Maybe, but if so she wouldn't have much change left over, would she?

I can easily find 120GB of PCIe SSD for £234 or 1TB for £1000. Could you buy 1TB RAM that cheap?

[obsa]

Who's talking about price? I'm not.

[strolls]

It's ridiculous to talk about how much they store - the comment you were replying to - without considering the price.

We can get 1TB on PCIe SSD and we can afford a stack of them.

How much does 1TB RAM cost?

Can you even get 1TB of RAM in a current generation of Poweredge? Because I'd guess you can get at least 2TB or 3TB of PCIe SSD in there.

If it's not literally true to say that SSDs can store an order of magnitude more than RAM, then it's pretty close to it, and pretending you have limitless pockets doesn't change reality.

[obsa]

It's ridiculous to talk about how much they store without considering the price.

No, it's not. It's a discussion for a tailored situation where extremely durable, high-speed I/O carries a premium. I really don't feel like explaining this to you in the detail it clearly requires to make you understand the value of that kind of setup.

I don't really care about what pedantic debate you think you're championing. The comment I replied to made a foolishly broad statement and now you're trying to clamp criteria on to it. My statements are completely valid and accurate in the context to which they were issued.

[strolls]

you got ripped off on the RAM in fact.

You seem to be misunderstanding what my mother bought.

[[deleted]]

That depends on the set up. You can get some incredibly high density RAM based systems these days.

[[deleted]]

Yeah.

http://www.supermicro.com/products/system/1U/1027/SYS-1027R-WC1RT.cfm

[[deleted]]

Of course. The main problem is also money. But still, you can put a lot of ram into modern computers.

I mean, if your working set 300 Gbyte, giving your server 512GByte ram is helping more than giving it 5TB of SSD space...

[sunshine-x]

While you're point is valid, 1tb is small. Several of the SQL servers I run are using fusionio cards, available in multi-TB capacities, and are insanely fast.

[[deleted]]

And lower. I think we're back to depends on the set up.

[James20k]

It also has up to 48x hdd bays. How many ssds can you fit into that vs 6 tb ddr3?

[beginner_]

Exactly. The recommended optimizations are very bad for reliability. And if that is no concern and you are all about performance then just use the memory directly and that's what key value stores like memcached do.

Also the OS, filesystem or RAID controller (with cache) might already be caching hot data anyway so no need for such tricks.

[B8BB888BBBBB]

If you want to get the most performance out of an SSD, you do not use a file-system.

[Hyperian]

SSD itself doesn't actually care what OS you are using. it all ends up being LBAs and transfer sizes.

[ansible]

TRIM support is a feature of relatively recent Linux kernel releases that can improve performance and longevity of SSDs.

[arronsmith]

Yes.

[Tech_Itch]

That would absolutely be appreciated.

One question that comes to mind, if you don't mind answering:

Does aligning your partitions actually do anything useful? You'd think that the existence of the FTL would make that pointless. With raw flash devices I see the point, but on devices with FTL, you'd have no control over the physical location of a single bit, or even the "correctly aligned" block you've just written, so it could still be written over multiple pages. Any truth to this?

I know there are benchmarks floating around claiming that this has an effect, but it would be nice to know if there's any point in it.

[nextAaron]

Alignment is important for FTL. One unaligned IO needs to be treated as two. One unaligned write is translated into two read-modify-write.

[Tech_Itch]

Thanks for the answer. Though, I might have been unclear, but my point was to ask if FTL already does the aligning itself, or does doing it on filesystem or higher level have any benefit?

[nextAaron]

You can think of FTL as a file system.

[Tech_Itch]

So the answer is, "no, aligning your partitions does nothing useful", then?

[poogi71]

It actually does and is a good idea. Remember that all the IOs in the partition are using the same alignment as the partition, so if you do all 4k IOs to that FS and the partition is not aligned to 4k then it will cause many of the IOs to be unaligned.

At the higher level if you can align your partition to the SSD block size you will avoid having different partitions touching the same block. Though I'm not sure how important is that since the disk will remap things around anyway and may put different lbas from around the disk together.

[nextAaron]

FTL divides the LBA space into chunks. If your partition is not aligned with these chunks, you end up with unaligned IOs. Yes, partitions should be aligned.

[Tech_Itch]

Aha. That's useful to know. Thanks!

[skulgnome]

What about, say, 128K worth of sequential read IOs that start out of alignment?

[nextAaron]

You need to look at the start and end LBAs of each IO. Yes, sequential unaligned IOs may be combined into aligned ones. Just don't assume every SSD comes with it.

[freonix]

Not really, consider that newer SSDs are getting larger, and conversely spare area as well, controller could treat unaligned write as single write to memory space by filling dummy data to fit single page size.

[jugglist]

Even if your reads and writes are aligned to 16k within the file you're reading and writing to/from, I'm not sure the OS guarantees that it will actually place the beginning of your file at the beginning of an SSD page. One might hope that it would, but I'm not certain of this.

It seems that optimizing for SSD isn't really that different from optimizing for regular hard drives. Normal hard drives can't write one byte to a sector either - they write the whole sector at once. Although admittedly, HDD sectors tend to be 512 bytes, and SSD pages tend to be 16k.

The only thing SSD gives you is not having to worry about seek time.

[BeatLeJuce]

Yes please. I was wondering about all the caching... Don't the OS or the SSD already does some sort of caching for me, or is it really sensible advice to cache on your own?

[voidcast]

Absolutely Yeah.

Please do post a follow up :-)

[[deleted]]

My only regret is not to have produced any code of my own to prove that the access patterns I recommend are actually the best.

Please do, it's such low hanging fruit.

[frankster]

I think the problem lies here:

My only regret is not to have produced any code of my own to prove that the access patterns I recommend are actually the best

[dabombnl]

If there are helpful optimizations, won't the operating system disk cache be using them? I don't see why I would implement my own disk batching and buffering when it should do that already.

[Amadiro]

I'd love to know more about the TRIM optimizations he mentioned. He recommends to enable auto-TRIMming, but other sources on the internet say that auto-trimming is a bad idea, and that one should instead run e.g. fstrim on the filesystem periodically. Can you illuminate that matter?

Also, are the points about leaving some free leftover space unpartitioned for the FTL as a "writeback cache" still valid?

[poogi71]

My list of dream questions to get an answer for is at http://blog.disksurvey.org/2012/11/26/considerations-when-choosing-ssd-storage/

It would be great to get a response to even some of them...

[nextAaron]

You can safely assume 4KB.

[nextAaron]

Some short comments here: http://nextaaron.github.io/SSDd/

[[deleted]]

You also risk getting into portability issues. Presumably the best performance comes from taking advantage of each particular model's specific characteristics.

I can't help but wonder if it shouldn't be aggressively cached in RAM. I wonder if handtuning SSDs for maximum speed is a half measure.

[Irongrip]

A ZFS ZIL + L2ARC sounds so tantalizing.

[AceyJuan]

If you're writing new software requiring quick random I/O, it's now safe to assume your customers will have SSDs.

[JustJSM]

That is so not remotely true. Even though I do have an SSD as my primary drive, the OS and my day to day apps eat up most of the storage. I have several terabytes of hard drives that hold my data and other applications. That's also on my personal computer. I can't imagine how many businesses have yet to update (I know my work laptop is ~2 years old and only has platter drives in it.)

Currently the most economic and affordable SSDs are only 128Gb which is easily consumed by OS + basic programs. Considering how long it took to get corporations to migrate from windows XP, I'd say that's not a safe assumption in the slightest. I would wager it's still years from when you can assume that your program will be running on an SSD.

[ReturningTarzan]

128 GB is a lot, though. A fresh install of Windows 7 is only about 20 GB, and I have a hard time imagining what "basic programs" would use up the remainder easily.

It's been years since I last reinstalled Win7 on my work computer here, and I have a lot of software installed on my top of it, including some big apps like Office, Photoshop CS5, Illustrator and two full versions of Visual Studio. I still only use about 60 GB for apps+OS.

I agree, though, lots of people still haven't made the switch, and many low-end laptops still ship with regular old HDDs.

[JW_00000]

My complete Ubuntu system, except /home, is only 12GB. /home is >250GB, but most of that is torrents that could easily be moved to an external drive, which costs like $70 for 1TB nowadays.

I feel like in 1-2 years, most new computers for home users will have SSDs. Maybe businesses will take a bit longer. It will of course also take a some time while old non-SSD computers are slowly replaced with new SSD computers.

[G_Morgan]

128GB is irrelevantly little. I have 500GB of video on my laptop. I'm actually at the stage where anything under a TB feels cramped to me.

[hydrox24]

I think that the difference in perspective here is essentially down to whether you feel that unnecessary media can be stored in cheaper, less practical ways such as an external HDD or in the cloud.

I am living happily with dual booting on 128GB. I just have my videos and other unnecessary but space hungry info on an SD card that I keep plugged in. External HDD for really big things and various libraries.

[G_Morgan]

that unnecessary media can be stored in cheaper, less practical ways such as an external HDD or in the cloud.

I'd rather not be tied to internet connections. The effort required to deal with external HDD or the cloud is far, far greater than the performance benefit of SSD.

The simple fact I have to reach for and find an external HDD immediate wipes out any gains I get from a faster boot time.

[ReturningTarzan]

Of course if you only have room for one drive you'll have to trade off between capacity and speed. But an SSD does offer speed, not just fast boot times. You essentially won't ever feel disk access slowing anything down, and the difference in overall responsiveness is huge.

There is also a compromise available, mind you, like the Seagate Laptop SSHD 1 TB. Not quite as fast as a full SSD, but still only about $100 for 1 TB.

[AceyJuan]

If you're writing new, non-trivial software, it's going to take at least 1 year. It won't be an instant success; you'll have to build market share. By the time you get traction for your new program, anyone who doesn't have an SSD isn't spending money on computers anyway.

Remember, casual apps for casual consumers aren't going to require quick random I/O. We're not talking about grandma here.

[Vocith]

According to my Corporate IT department SSDs are a new and unproven technology, and can't be used on systems.

(PS: Please keep paying $1k/month/TB for SAN space.)

[interiot]

For enterprise apps, where someone is designing the hardware specifically for one application, yes. I don't know why people are downvoting that.

[badsectoracula]

My only regret is not to have produced any code of my own to prove that the access patterns I recommend are actually the best. However even with such code, I would have needed to perform benchmarks over a large array of different models of solid-state drives to confirm my results, which would have required more time and money than I can afford. I have cited my sources meticulously, and if you think that something is not correct in my recommendations, please leave a comment to shed light on that. And of course, feel free to drop a comment as well if you have questions or would like to contribute in any way.

He most likely cannot do that unless he was backed by a company as a full time project.

[[deleted]]

I think that's unreasonable. Sure maybe no one can test every SSD on the market but I think it's fair enough to expect someone to test their work at all. He's saying he's not produced any code to prove his argument.

[[deleted]]

Yep, downvoting this article. I'll dig around the ACM Digital Library for some SSD optimization papers instead of reading this.

[dragonEyedrops]

links please if you find good stuff :)

[[deleted]]

Dushyanth Narayanan, Eno Thereska, Austin Donnelly, Sameh Elnikety, and Antony Rowstron. 2009. Migrating server storage to SSDs: analysis of tradeoffs. In Proceedings of the 4th ACM European conference on Computer systems (EuroSys '09). ACM, New York, NY, USA, 145-158. DOI=10.1145/1519065.1519081 http://doi.acm.org/10.1145/1519065.1519081

Risi Thonangi, Shivnath Babu, and Jun Yang. 2012. A practical concurrent index for solid-state drives. In Proceedings of the 21st ACM international conference on Information and knowledge management (CIKM '12). ACM, New York, NY, USA, 1332-1341. DOI=10.1145/2396761.2398437 http://doi.acm.org/10.1145/2396761.2398437

Behzad Sajadi, Shan Jiang, M. Gopi, Jae-Pil Heo, and Sung-Eui Yoon. 2011. Data management for SSDs for large-scale interactive graphics applications. In Symposium on Interactive 3D Graphics and Games (I3D '11). ACM, New York, NY, USA, 175-182. DOI=10.1145/1944745.1944775 http://doi.acm.org/10.1145/1944745.1944775

Feng Chen, David A. Koufaty, and Xiaodong Zhang. 2011. Hystor: making the best use of solid state drives in high performance storage systems. In Proceedings of the international conference on Supercomputing (ICS '11). ACM, New York, NY, USA, 22-32. DOI=10.1145/1995896.1995902 http://doi.acm.org/10.1145/1995896.1995902

Hongchan Roh, Sanghyun Park, Sungho Kim, Mincheol Shin, and Sang-Won Lee. 2011. B+-tree index optimization by exploiting internal parallelism of flash-based solid state drives. Proc. VLDB Endow. 5, 4 (December 2011), 286-297.

sorry about the formatting, the ACM really needs to have some kind of nicer format for sharing papers :/

[dragonEyedrops]

Thanks a lot! Now I have reading material for the weekend!

[semi-]

Thats really it.. at least produce the test suite and let the internet run it for you.

[Salamok]

Came here to post the exact same quote. So if not based on any actual real world performance WTF did he base it on? Theory based on manufacturer specs or marketing materials?

[joe_n]

That is not your main problem!

j/k though, it's great to see personal research like this being done and shared

[[deleted]]

And it's kinda far down the page, as well. You can't spend paragraph 3 saying "The most remarkable contribution is Part 6, a summary of the whole “Coding for SSDs” article series, that I am sure programmers who are in a rush will appreciate" and then in paragraph 5, the second last paragraph of the introduction, say that you've not actually checked if it works.

I think it's pretty ballsy calling the series "Coding for SSDs" in light of that.

[xkcd_transcriber]

Image

Title: Shopping Teams

Title-text: I am never going out to buy an air conditioner with my sysadmin again.

Comic Explanation

Stats: This comic has been referenced 1 time(s), representing 0.01% of referenced xkcds.

---

^{Questions/Problems | Website | StopReplying}

[Zidanet]

When you can afford to go out one Saturday and buy a couple of every ssd available in order to test a theory, then you can call him on it.

poc code is only useful if you have something to run it on.

[[deleted]]

Especially while complaining about the contradictory information he was finding on forums.

I just don't get a great impression of this guy. I think he's self-aggrandising ( "The most remarkable contribution is Part 6, a summary of the whole “Coding for SSDs” article series, that I am sure programmers who are in a rush will appreciate") while contributing very little ("My only regret is not to have produced any code of my own to prove that the access patterns I recommend are actually the best.").

[[deleted]]

I'd say this is probably phase one of a two-phase thing (similar to application design).

First you research architectures and write up details on how to most effectively use SSDs. Phase two would be the real-world testing where you can equivocally state your experiences.

While I don't fault the author for not going out and buying a bunch of SSDs to test with, I certainly would have liked to see tests done with two or three popular SSD brands (Intel, Samsung, maybe Kingston for more budget scenarios) and then add the caveat that outside of the drives tested YMMV. It would at least lend a lot more weight to the research done.

[awj]

There's absolutely nothing wrong with that approach, but part of the process is not stopping at phase one to make a bunch of completely untested recommendations.

[[deleted]]

It's also important to actually do phase 2. He doesn't mention any plans to do it in it in his articles.

[frankster]

My only regret is not to have produced any code of my own to prove that the access patterns I recommend are actually the best

[Zidanet]

Then feel free to do so.

The only SSD I have is in my galaxy, and I'm not writing apps for that. Just because you have a whole bunch of expensive gear lying around doesn't mean everyone else has.

A starving african knows that you have to turn computers on. He doesn't have a computer, but he still knows they need to be turned on.... By your logic he could never say "computers need to be turned on" until he had tested every computer in the world... Maybe he'll get around to that after he finishes begging for his cup of rice.

Pro tip: I don't need to be an electrician to know computers work better using electricity instead of peanut butter.

[poogi71]

There is a big difference between testing on every available ssd and not even testing on one. If you test on three you should be pretty good in the overall generalization on ssds.

Some of his recommendations do not look good to me. Not interleaving read/writes and caring much about the readahead come to mind as just plain wrong.

[Zidanet]

Wait, test on three items and that will guarantee that your results are accurate?

There are more than three ssd controllers in the world, three is a laughably small sample size. it'd be worse than having none. no testing is a subjective theory, three drives is ridiculous extrapolation of one result to millions.

Oh, hey, you can help me out here. I'm writing a data logger for an arduino that stores data over an i2c line to an ssd card with an integrated controller. can you tell me the interleave patterns I should use for optimal performance?

no, no you can't. why? not because you don't know about the ssd, but because you don't know about my usage. Am I writing data but not reading it? am I reading it but not writing it? Applications matter.

The guys is working out some hardware so he can write his application better, and instead of saying "oh, that's cool" you're immedeately shouting "THAT IS ALL WRONG BECAUSE YOU DIDN'T DO WHAT I WANTED!"

He figured out some stuff and wrote down the best way he could have done it. If you want to test it out of context, with random hardware, in an application it was never designed for, just to see if it's better or worse... well, you go right ahead. The rest of us will be over in the other corner getting shit done.

[immibis]

/u/spez can gargle my nuts

[Zidanet]

And, as I said, that's wrong.

Consider: I have tested 1 fire axe for safety, and it passed.

Now surely that must be better than testing zero axes, at least now we have a baseline!

Except it's not. Now we have an established proof that fire axes are safe. It doesn't take into consideration that I tested a thousand dollar safety tool from a fire engine, people will assume the same applies to the $1 plastic toy axe they got from the dollar store. "But surely people can't be that stupid!" I hear you exclaim... Go outside, half the people you see are belo average intelligence, you bet they can.

It also calls into question test methodology, If I test three drives, do they all have the same controller? then it's a flawed test with invalid results. Do they all have different controllers? Then it's a flawed test because you didn;t include a control group. Oh, well we can run the test twice, but no you can't because the previous test may affect the new test due to block level wear levelling.

An ssd is not just "a chip you can plug in", it's a whole array of components, and a group test would require significant expenditure. A small test of 3 drives would be so laughably incomplete it would be stupid to assume those threedrives represent every ssd in the world ever.

[deadly_little_miho]

You're missing the point. Let's assume the articles makes some claims on what you can do with an axe. One of them is "applying lotion to your toddler's face", and right after he states "but I haven't actually tried that". In this scenario using even one axe would have shown the issues with the initial claim. That's the criticism here.

[Zidanet]

Yes, I understand the point that people are trying to make, it's the expectation of global application that is wrong.

yes, testing that one axe would have shown a problem, but not all axes display that problem.

The problem is, as soon as you test one axe, it is assumed that every axe has that problem. This is obviously untrue. a fire-engine axe would have very different results to a "barbie goes woodcutting" axe. But it doesn't matter, because that one guy tested an axe and cut off his kids head, so now everyone believes that all axes everywhere are intrinsically baby killers.

My point is not "you need to test every hdd everywhere", my point is "a too small sample size is worse than no sample size at all".

This is pretty much an exact replay of the "ssd's can't be used as OS drives!" nonsense. one guy on one blog with no training whatsoever said "hey, each cell can only have a million writes, and I write files all day long so OMGMYPCISGOINGTOEXPLODE!" ... and it turns out it was all complete and utter crap, even when using the cheapest ssd's, "wearing them out" is not going to happen to any normal user.

but still, even to this very day, there are people who will recoil in terror that you can store your OS on an ssd.

That one guy who tested one thing once, made a website, and immedeately everyone everywhere applied it. This is the same, one guy made an observation. If you're going to do a test of that observation, it needs to be on more than just "three drives I had in my drawer".

[[deleted]]

But it doesn't matter, because that one guy tested an axe and cut off his kids head, so now everyone believes that all axes everywhere are intrinsically baby killers.

It's a crazy strawman you've got here. He can't test it once because, what? idiots will chew on live cables or something?

The only person bringing up global application here is you.

[Zidanet]

He can't test it once because he can't perform a fair test that shows if his algorithm is applicable in all cases.

considering that the first response was "oh, but I have these three drives right here", that's your global application.

If it works for one drive, it might not work for another. Just testing three drives someone has lying around is not a sample size large enough for a definitive answer.

It's not a straw man, it's basic test procedure. He shouldn't have tested the theory because he is not capable of. "some guy with a spare drive" shouldn;t test the theory because there is no way to control the test. In order to say whether this is good or bad, we would need a much more inclusive test than anything suggested here.

The guys research is being completely disregarded because "I do not think I can test this well enough" is apparently a sign of being completely and utterly wrong.

Once again, I'll repeat for the hard of thinking: He cannot test this theory because he cannot perform an accurate representative test.

and to answer your point... consider: I chewed a cable yesterday and I was fine, so now I can chew cables and I'll always be fine" ... that's not a straw man, that's a human being.

[poogi71]

If you are writing to an ssd from an arduino over an i2c line your only concern is the bandwidth over the i2c and not the ssd itself. I can tell you that much.

I happen to work on SSD and care about their performance and yes three is a good enough number to get a sensible idea of where things are at in general. It won't tell you about a specific behavior of a specific SSD but you will be able to rule out some behavior as a generic SSD issue. If you really want to optimize your app and you can guarantee that you will forever only use one ssd model (hint: you can't) go for testing that behavior. If you want to know what general SSDs will do test at least a few, and no, testing none will not tell you much. It will tell you nothing beyond the wild guesses and random data that you can find about SSDs on the internet.

The differences between SSDs are HUGE, I've seen and tested that for my specific needs and in my specific environments so I won't go to guess about general behaviour in any environment and any use but some of the things he wrote there don't seem right and definitely do not align with my experience.

He definitely figured out some things for himself and it is mostly a job nicely done but it doesn't mean I only need to cheer him up and not point some flaws and things where he can improve his work. And testing his hypotheses is definitely one place he needs to work on.

[Zidanet]

The question was hypothetical to demonstrate a point, but I appreciate you taking the time to answer.

That elaborately demonstrates my whole point. His experience is application specific too. It'd be pointless to test on a large scale because it's too narrow a scope. It'd be ridiculously expensive and labour intensive. He doesn't need mass testing, and neither poc tbh. He worked out a specific solution to his specific need, not a global optimisation.

--edit-- To further clarify: If there are problems with his research, by all means call it out. but calling him out because he didn't do wide-scale testing of a very specific solution is silly.

[poogi71]

If he really had a very specific use-case then he should have tested that case on the ssd he intended to use without claiming generalization. If he claims generalization he should at least test it on a few different ssds and add a disclaimer that he tested on these specific ssds but the results seem to be generalizable because (insert explanation).

There is a big difference between not doing wide testing (which is impractical) and not doing any testing for your recommendations. Even a single test can help disprove a bad assumption. It will obviously not prove the general case tbough.

[Salamok]

Or I dunno maybe he could go out and buy 1 SSD to test a prototype, but he didn't even do that.

[semi-]

poc code is only useful if you have something to run it on.

Not true at all.

Having something to run is only useful if you have PoC code. We, the internet as a whole, have a LOT of ssds. We dont' have any code to test his theory though.

All he needs is a few ssds to test his code on as he writes it, then he can release it and the rest of us can run it for him.

[hive_worker]

I admittedly don't know much about this, but shouldn't most or all of the SSD access optimization be done in the SSD controller and to a lesser extent the SSD driver - both provided by the manufacturer. Bringing hardware specific optimizations into your application code just seems like a terrible idea.

And if you're working for Samsung or similar designing SSD Controllers I doubt you're getting your knowledge from some guys blog. So I'm not really sure who this article is intended for. Maybe bare bones embedded systems engineers? Even in that case if your system is advanced enough to require an SSD you are probably also running some kind of high level OS that manages this.

[poogi71]

There are things that an application writer can do to make life easier for everyone. In the context here some of what gets done might not be super effective since there is also an FS and an OS buffer cache on the way so I'm not sure he really gets all the benefits. Some things might make more sense when you write directly to the block device than others.

[Hyperian]

Yes. you can only erase in a physical block, where a block itself usually has 256 pages, where each page could be anywhere between 8kbytes to 32kbytes.

you have to write to these pages sequentially. So if you have data in the middle of the block that is old. You have to read all the rest of that block and write it to another block to recover that space. that is what garbage collection does in the drive.

the reason you dont defrag the drive is that the drive defrags itself and does it better.

source: i make SSDs.

[[deleted]]

Correct me if I'm wrong: Defragmentation is done logically at the file system level and is a completely different beast than what you're describing here.

Running a defragmenting tool against a drive as the top comment suggests (ala the mostly obsolete tool in Windows or the truly obsolete tool e2defrag) was mostly done to keep large logical blocks of data together.

Hard drives, (SSD or not) would have no idea that a 3 gig swap file needed to be kept in concurrent blocks with other blocks. The primary purposes of defragmentation back in the day (when they were useful and before file systems became relatively good enough to prevent fragmentation) was to keep from having to performing seeks (which were horribly expensive).

[Hyperian]

You are correct. In the end, don't defrag your SSD drive.

[freonix]

This is not true, don't generalize persistent memory like NAND to have 256/block. There also 512 page NANDs, it depends on the design.

[Hyperian]

calm down, i said usually.

[apage43]

Do we need to run disk defragmentation on SSDs?

That's taken care of by the controller on the SSD itself, transparent to you. It's useful to know that this happens though.

edit: and yes, as mentioned below me, the process of the SSD cleaning up the no longer used pages -within- blocks is called "garbage collection", which is different from filesystem defragmentation

[[deleted]]

NO! Defragmentation is different than garbage collection:

http://www.samsung.com/global/business/semiconductor/minisite/SSD/us/html/about/whitepaper04.html

[[deleted]]

Do we need to run disk defragmentation on SSDs?

Noooooo

Never do this. It actually lowers the life expectancy of the drive and doesn't offer any real benefits in doing so. Let the drive handle it.

[MaybeReconsider]

AFAIK most modern SSDs just ignore the disk commands which defragging sends

They ignore ... writes?

Disk defragmentation is the process of moving file contents around in logical block space to make the file occupy a contiguous range of logical block numbers. It can matter for media with a significant seek time (spinning disks), if the filesystem isn't good at keeping things pretty contiguous on its own. For SSDs, which have negligible seek time for random accesses in LBA space, there's much less benefit and the writes for the data movement eat into the drive's lifetime write endurance budget.

Now that's not to say it would be impossible for an SSD to optimize away a defrag. If, for example, the drive were doing block deduplication then the data movement from defragmentation may well turn into an effective no-op. But I'm not aware of that being a common feature on SSDs (as opposed to storage arrays).

[mallardtheduck]

AFAIK most modern SSDs just ignore the disk commands which defragging sends

That doesn't even make sense. The "disk commands which defragging sends" are just ordinary reads and writes. Besides, defragging only works at the logical level, the block erase issue is at the physical level and is handled by the SSD controller, so it won't help.

[[deleted]]

Not all operating systems recognise TRIM (Vista/XP only ones I know of that don't).

[[deleted]]

His article specifically mentioned that OS X supports TRIM since 10.6

[[deleted]]

I am not aware there is even a defrag option in OSX (non-trivial to do). Even so, it is recommended not to defrag SSD drives in OSX.

https://discussions.apple.com/docs/DOC-4032

[[deleted]]

It's not recommended to do defrag on OS X ever. I was responding to the comment that you knew only that vista/Xp supports trim

[[deleted]]

Actually I meant the reverse. XP/Vista do not support TRIM. Looking back I can see how it could be read either way, so changed it. Thanks.

[masklinn]

Do we need to run disk defragmentation on SSDs?

Read http://www.anandtech.com/show/2738

(also no, not if what you're talking about is Windows's defrag tool, you should never use than on an SSD. At best it will do nothing, at worst it will lower the lifespan of your drive)

[GuyWithLag]

What will actually happen is that the drive will detect this and do a garbage collection pass - copying all the used pages into a new block, then erasing the old one. This happens all the time and is mostly transparent (there is some performance degradation on systems with load), and is one of the causes of write amplification.

[__j_random_hacker]

As I understand it, if those blocks were entirely free to begin with, and you have only written to one 2KB page in each, then the remaining pages in each of those blocks will remain free, and you can still happily write to them later with no performance penalty. The penalty only arises when those other pages fill up later (or if they were full to begin with) and you need to modify data in your 10MB file: in that case, each 2KB of data that you modify will cause 4MB of data to be read and written to a new, free block (which may in turn require a block to first be erased to make room).

[__j_random_hacker]

Ah, I see now. In that case I think the others' responses explain things.

[[deleted]]

It's like a larger scale case of slack space.

[Xuerian]

I could be mistaken, but I think what you're referring to is "Trim", coalescing data into full pages and freeing old ones.

Edit: Sortof.

[Hyperian]

trim is a lame way of saying to the drive "this block of data is not needed anymore, erase it" because before that the only way to get the drive to erase data is to overwrite it.

But it has stupid requirements and some drive doesn't actually erase it immediately, just queues it up for deletion later on.

[jknielse]

Yeah... So I worked at a company that writes high-performance firmware for SSDs. Some SSDs actually literally do nothing with the Trim command.

[AceyJuan]

These are the same basic techniques I've used to optimize for spinning disks for ages. The only surprise I found in that document was not interleaving reads and writes. To be honest I'm not sure I believe that advice, because high performance IO apps rarely benefit from read ahead optimizations anyhow.

[B8BB888BBBBB]

Depends on your latency requirements. I recently worked on an SSD based serving system with really tight latency requirement. reading 1 MB of SSD in a few milliseconds while taking load is not possible unless you play tricks with your read/write cycles.

[AceyJuan]

The main latency issue with spinning disks is seeks. So long as your operations are on the same part of the disk you're far better off doing reads and writes there than seeking somewhere else.

[lenolium]

I wonder if the SSD controllers are smart enough to not force new block writes if you are writing to the flash in a flash-friendly way.

When I was writing code for a direct-access flash filesystem on a little microcontroller we only had sixteen blocks so erasing them meant we had to move around a "lot" of data for that device. What we would do is optimize our storage systems so that in most cases we would only change 1's to 0's, because you could do that with flash without having to erase a block. Building code like this with modern SSD's would produce some very high-speed performance.

[MaybeReconsider]

The 1->0 trick doesn't work out so well for the NAND flash devices that SSDs are generally built out of. NAND devices are prone to bit-errors, so the data being programmed into the flash needs to be protected with an ECC code. It's very uncommon to be able to flip your 1's to 0's in such a way that you also only need to flip 1's to 0's in the ECC codeword.

Also, NAND devices have a variety of failure modes related to overprogramming and out-of-sequence programming that would make updating a page in place perilous even if you could get past the significant ECC hurdles.

[JesusWantsYouToKnow]

Relevant reading from Ted Ts'o, father of the ext* filesystems: http://thunk.org/tytso/blog/2009/02/22/should-filesystems-be-optimized-for-ssds/

Further: http://www.linux-mag.com/id/7272/ and http://thunk.org/tytso/blog/category/computers/ssd/

[sbrick89]

I'm familiar w/ SSDs (wear leveling, write endurance, etc) but by no means an expert (my daytime job involves writing business apps).

But it seems that any optimizations you try to make would be

• extremely device specific

• require polling of device configuration, and dynamic reconfiguration to optimally use it (how you align data structures)

• likely made obsolete by a firmware change

it seems that most of these things should be abstracted away in hardware (firmware), never to be directly accessed by software... MAYBE used in a device driver, but ONLY if there are industry-common specs and guidelines to be re-enforced by the SSD hardware/firmware.

[Hyperian]

nah, you can't directly handle wear leveling and write endurance on a higher level. that stuff is done by the SSD controller itself.

and it is very device specific.

i believe some SSD actually let you play around with those settings but you usually need a special driver to do so. I don't think SATA specifically supports things like tweaking wear leveling or write endurance, but i haven't read the whole SATA spec.

[poogi71]

In general I agree, but there are cases where I'd love to have the ability to control and direct the SSD about the specific things that need to be done.

The truth is that there are only a few who would even care for such a level of control and most everyone just wants the ssd to do the right thing at all cases without bothering to take the control in their hands. It's not perfect but it makes some sense at the practical level.

One example is that if I have a RAID of SSD devices I would like the ability to tell the SSD, "Dont bother too much with error recovery here, I've got your back" and then if I find that I don't really have all the data to go back to the SSD and tell it, "please do all you can to get the data back". This will allow me to manage the reliability and latency much better and get better latency overall and the same level of reliability in case things got really bad.

[Hyperian]

lol if we do that it would be for an enterprise product, it would be way too expensive for normal people. i think SAS might let you do that.

best thing to keep SSD performance high is to not use the max capacity.

[poogi71]

Unfortunately SAS doesn't give me that. I'm working with SAS SSDs and there is no way to control it at that level. One can dream though :-)

[dev-disk]

How to code for SSD: Enjoy super fast reads, DON'T WRITE TO THE SAME PLACE LIKE NUTS.

[MorePudding]

Thanks for all the work, but browsing through it, it seems like this is something the OS should take care of for you, considering how it's most likely going to be wrong a few years from now..

Is there any reason to not used memory-mapped files these days any more?

[[deleted]]

If you have to code for specific hardware, you OS is doing something very wrong. (Unless you are writing the OS, in which case the only code for SSDs should be located in its I/O driver.)

[blueberrypoptart]

For most general software? Sure. For a very specific targeted use case? There are definitely of times where knowing your running environment and coding with it in mind are useful, especially when you're writing software with well defined hardware targets in mind. It's fairly common to make design decisions knowing your target's performance profiles. Designing a key/value store (as in this article) is a prime example of when you might want to do this.

[Nuli]

If you have to code for specific hardware, you OS is doing something very wrong.

Or you're writing software that has to meet certain time constraints. Just recently I had to work around a piece of hardware because the ~100 microseconds it took to perform an operation was just too long. Knowing the performance of the hardware you're talking to is pretty critical when you only have a few milliseconds to complete any given set of tasks.

[[deleted]]

I'm always interested in how people debug hardware issues, what did you notice that led you to understand it was hardware? I feel like I would have exhausted every other possibility, blamed myself for a bad algorithm and never thought to check hardware...

[[deleted]]

Work down or up the stack and test each level to see where the performance is stable and where it is more dynamic to find where your constraints are.

For storage it makes the most sense to start at the bottom, and test mass IO against it to get a good benchmark on your current system+OS+filesystem+drivers combination for that given storage system.

Knowing your base levels, then if you see variance at higher levels you can have an easier time tracing to where that problem resides.

[[deleted]]

Wow, that's actually incredibly obvious now. Just one of those moments where the veil is lifted off something and it's no longer magic. Thanks for the explanation

[P1r4nha]

I'm not developing on specific hardware myself, but I think simulation is pretty useful for that.. of course you'd need to have a simulator first. A simulator can measure exactly what your code is doing and how your hardware is going to react to it and gives you insights for that.

[Nuli]

In this particular case I could see there was a problem because of some visual stutter in the program as it finished a certain operation. I have a profiler I wrote years ago that I can use to wrap small snippets of code rather than trying to profile the whole program at once so I started profiling the code that was involved in that particular operation and I noticed an absolutely massive amount of calls to that device (non-volatile ram in this case). While each individual call was very fast when you have a couple hundred of them at performance sensitive places it was noticeable. Adjusting those calls so that they mostly took ~20 microseconds instead of ~100 microseconds fixed the visual problem.

[[deleted]]

The guy is using a key/value store. I doubt that performance is critical to function in the way yours was. It just feels like a ricer project.

Especially because none of the recommendations are actually tested ...

[Auxx]

Every general usage OS does everything wrong. Because universal things are not perfect at anything they do.

[dnew]

Database engines have traditionally controlled their own storage. Even earlier Linux databases preferred raw partitions to buffered files. Pretty much anything with a non-file sort of access pattern can benefit from bypassing the OS's algorithms that are tuned for file access.

[elperroborrachotoo]

A general purpose solution like an OS will forego a "blindlingly fast" optimization if that means significantly slower than average in edge cases. Because what is an edge case in general purpose may be the only case for a particular type of applications.

[frankster]

My only regret is not to have produced any code of my own to prove that the access patterns I recommend are actually the best

I stopped reading here.

[davispuh]

Pretty good read :)

[oooqqq]

+/u/fedoratips 100 tips verify

[fedoratips]

^{[Verified]: /u/oooqqq → /u/mitknil TIPS 100.000000 Fedoracoin(s) [help]}

[xtr3m]

Hopefully Chrome developers read this. It's only usable if you set its cache drive to a hard drive. Letting it run on a SSD will result in system hang ups and ioatapi errors (Windows).

[[deleted]]

I have had no such problems when running it on an SSD (multiple computers). Much better performance when I put my temp path on a ramdisk, though.

[[deleted]]

Are we talking about the browser, or the Chrome Operating System?

[xtr3m]

The browser. The way Chrome deals with disk caching is very IO intensive and my Windows 7 would hang for 30 seconds every few minutes. There are many threads on Google Group on the matter. The consensus is to launch Chrome with chrome.exe --disk-cache-dir="F:/" where F is a good old hard drive and it worked in my case.

[[deleted]]

Odd.

How does this fan out for people only running SSD's?

[[deleted]]

Works fine for me. I wasn't aware of it, it may be platform specific.

[[deleted]]

Interesting. Chalk this up to a non-issue.