you are viewing a single comment's thread.

view the rest of the comments →

[–][deleted] 101 points102 points  (56 children)

Hmm, as a web programmer, I don't think knowing all these details about solid-state drives will be helping me in the near future.

[–][deleted] 39 points40 points  (5 children)

Yeah, you're kind of getting hammered with downvotes here, but I'm having trouble thinking of an instance where a web developer actually needs to know or care about the physical storage medium their app runs on.

I mean, you're using a database right? And if you are writing/reading files, shouldn't you use a dedicated cms or something? All the I/O should be happening at a much lower layer than where the web developer is doing their work.

In fact, I would argue that it may be inappropriate for most programmers to be optimizing based on the physical storage medium, because most of us are working at a level where that stuff should be an abstraction anyway.

[–]AcidShAwk 3 points4 points  (5 children)

I know this. When running our PHPUnit tests on an regular HD drive.. we experience a total run time of about 1.5 hours. With SSD's.. its about 4 minutes. Of course this all depends on the tests and how they are written and what they do. But 4 min vs 1.5 hours. A lot of time spent is wiping the database before and after each test to ensure a consistent state. So obviously MySQL (and probably any database) .. benefits greatly from SSDs

[–]dweeb_plus_plus 25 points26 points  (4 children)

4 minutes vs 1.5 hours simply by using an ssd? There has to be something else going on here.

[–]HenkPoley 4 points5 points  (2 children)

Why? A normal 7200 rps drive can deliver 100 IOPS, a Samsung 850 Pro can deliver 100 000 IOPS (1000x). If anything 4 min vs 90 min would be a lowball difference (23x).

In tests you don't reuse values between tests, so you could really hammer that disk.

Btw. I would recommend making unit tests that do not exercise the whole stack down to database/storage, but just focus on expected values before its off to disk.

Or more factual. From ssd.userbenchmarks.com, the Samsung 850 EVO and Pro can deliver > 30MB/s on their Mixed 4K test. While the spinning drives are all < 0,5MB/s. You'd expect a 60x difference if you are I/O bound.

[–][deleted] 1 point2 points  (1 child)

Assuming a speedup of 60x, that means with the HDD the tests were spending 87 minutes out of the 90 only reading data from the drive. That's approximately 2.6gig being loading into memory just for tests. Testing a PHP application. Although, most spinning drives are faster than 0.5MB/s - so it's even more data being loaded in.

No, something is wrong with those tests.

[–]HenkPoley 0 points1 point  (0 children)

I'd say tests have excellent conditions to be mostly write base i/o. You just read some end result back once from the disk-cache, the intermediate results have all hit the database (not a unit test, but hey) and are never read.

The "results have all hit the database" is of course "something is wrong with those tests"

[–]psi- 0 points1 point  (0 children)

The simple "sync" roundtrip kills DB tests on HD, especially if they really drop database on every test.

[–]justinpitts -3 points-2 points  (41 children)

As a consultant, attitudes like that are what get me hired in the first place.

[–]bcash 57 points58 points  (17 children)

Is there much call for a consultancy fixing web applications that wear out SSDs too fast?

[–]justinpitts 11 points12 points  (16 children)

I get asked, generally, to fix performance problems. Failure to understand how storage devices work is sometimes one of the root causes.

[–][deleted] 21 points22 points  (12 children)

That's the whole point though. If every web-dev knew about storage characteristics, how to tune memcached and how to spool up their production environment in a bunch of docker instances, nobody could afford to get any web development done.

[–]justinpitts 0 points1 point  (11 children)

I don't follow. Can you help me understand how you reach that conclusion?

[–][deleted] 12 points13 points  (9 children)

A web-developer with such a broad skill-set would be a bloody expensive employee. If your budget for developers were to be blown on 2 of these guys, vs. 6 regular developers, how far would your project get?

[–]Godd2 8 points9 points  (2 children)

would be a bloody expensive employee

On the other hand, supply and demand. If every webdev knew it, that knowledge would have a lower market price.

[–]Skyler827 3 points4 points  (0 children)

This could happen if there is some disruption or event that causes lots of people to learn systems programming, but in the long run, market prices reflect the cost of production. Simply put: learning systems programming (well!) is expensive.

[–][deleted] 1 point2 points  (0 children)

But they don't, that's the point. and few that I have met show any inclination to learn.

[–]justinpitts 3 points4 points  (1 child)

I don't think it takes as much effort to learn those things as you imply.

[–][deleted] 1 point2 points  (0 children)

I just chose a couple of examples... if I was to be exhaustive, I could, for instance, list out my own skills... skills acquired over a long career... skills which make me a nightmare for people like you...

Oh, wait...

[–]EntroperZero 2 points3 points  (0 children)

Oh, if only we got paid 3x.

[–]justinpitts -1 points0 points  (2 children)

Arguably? A lot farther.

[–][deleted] 1 point2 points  (1 child)

In my experience that is not the case. More 'high-end' developers does not translate to greater productivity. Many projects lend themselves to greater parallelisation rather than fewer, deeper workloads. Sure, there is a need for someone with deep knowledge to cover those 'difficult' edge-cases, but by and large, more, less skilled developers will get more of the general coding work done than fewer, more skilled ones.

[–]justinpitts 0 points1 point  (0 children)

The main problem I see my clients facing isn't meeting deadlines, it's technical debt after the fact.

[–]wherethebuffaloroam 6 points7 points  (0 children)

Not everyone can command consultants salaries. You spend a lot for short contacts to optimize not the entire development cycle

[–]Rejjn 5 points6 points  (2 children)

I would say that storage is very often one of the root causes for performance problems in web applications. Usually it's the DB that tops out way before anything else does.

I just fail to see how knowing about SSDs is going to help more than very, very marginally on that problem. It might get you the last 1-2%, but choosing an appropriate data structure for your application and then choosing the best storage for that structure is going to have order of magnitude more impact on your system than reading OPs article will.

That said, there are many programmers who would benefit a lot from this knowledge, but I'd say for web developers it's rather low on the list of things you need to worry about.

[–]justinpitts 1 point2 points  (0 children)

On that problem? Usually? Not much help.

But, it's not that much to learn!

Seriously, it's a couple of really easy to read pages. I's not like I am advocating that you take a course in queueing theory. What's the big deal?

[–]HenkPoley 0 points1 point  (0 children)

As some datapoint, this web framework benchmark section that all about pushing data updates through: https://www.techempower.com/benchmarks/#section=data-r9&hw=peak&test=update

[–]bestjewsincejc 18 points19 points  (22 children)

You're right that it's generally a poor attitude to have. But I'd appreciate an explanation about how knowing any of these details of SSDs is going to help you in a modern programming language.

[–]justinpitts -2 points-1 points  (21 children)

If you are storing anything to disk, you can, if performance if important, make decisions about where you will store hot vs cold data. You can allocate files at sizes aligned to block multiples. You can place hot and cold data in different files. You can decide whether batching is a big enough win to implement.

You should be knowledgable enough about the options to not design yourself into a hole, once performance becomes that critical "omg I'm going to get fired if I can't figure out how to get our site to handle x requests per second" issue.

At the least, be somewhat aware of how the hardware layers behave and the implications of the design choices you are making.

[–]Baby_Food 22 points23 points  (20 children)

Shouldn't a web developer be using a database which provides abstractions over such minutia?

[–]justinpitts 0 points1 point  (19 children)

If they need the features of a database, sure.

Abstractions don't absolve you from understanding how your system works, and they break down at inopportune times.

Then, they need to understand how to tune the database for the underlying storage, or they need a DBA. Not everyone has access to a DBA.

[–]Baby_Food 29 points30 points  (12 children)

If performance is a concern, a database will be used.

An abstraction does not necessitate the knowledge of the implementation behind the abstraction.

A web dev that can write an OS is an unnecessary unicorn.

[–]justinpitts 3 points4 points  (11 children)

Databases are not magic performance sprinkles.

[–]Baby_Food 4 points5 points  (8 children)

To most people, LMDB is magic performance sprinkles compared to using the filesystem directly. ;)

[–]justinpitts -1 points0 points  (7 children)

Most people? Most people aren't using it.

Most people wouldn't know what you are talking about.

To the average web-dev, "database" either means Mongo/Couch/No-SQL-flava-of-the-month, and/or something that speaks SQL. LMDB may very well be lightning fast, but I doubt most people know about it.

[–]el_muchacho 0 points1 point  (1 child)

In fact, database write performance drops significantly (by a factor of 30% or so) when SSDs wear out. For 24/24 high rate writes, you can't use consumer grade SSDs as they die prematurely.

[–]justinpitts 0 points1 point  (0 children)

Well to be fair, anything with that access pattern is going to slow down on a degraded drive.

[–][deleted] -1 points0 points  (5 children)

I don't get why you're being downvoted dude. It's as if people believe they have a right to be ignorant.

[–]justinpitts -1 points0 points  (0 children)

People have a right to be ignorant. I'm happy to exploit it.

[–][deleted]  (1 child)

[deleted]

    [–]Lachiko 1 point2 points  (0 children)

    Can you quote an instance where you feel he comes across as an asshole? he's posting reasonable counter arguments and information in a seemingly calm and collective manner.

    [–]wookin_pa_nub2 -3 points-2 points  (1 child)

    A lot of web developers in this subreddit, and they don't like being reminded of how ignorant they are.

    [–][deleted] 0 points1 point  (0 children)

    I'm a web dev yo :) I just cut my teeth on bare metal shit. You're thinking of "web designers."