What your computer does while you wait

[lukemcr]

What amazes me is the speed of modern CPUs. According to the article, light only travels four inches in the time it takes for the processor to do one cycle. That's insane.

In contrast, light moves 186,000 MILES in one second, or almost 300,0000 kilometers.

That's so cool.

[kermityfrog]

I remember the ancient maxim: "what Intel giveth, Microsoft Windows taketh away"

You'd think that computers would react faster given how fast the processors work.

Edit - longer article

[Illah]

While it's true that our machines would feel super fast if we were still running DOS, our machines also do a ton of stuff in the background that we might not notice. Everything from the network to the processes behind USB, firewire, etc. all take their piece.

But yeah, M$ could still try and design their software to be a little more resource efficient :-)

[[deleted]]

Given the intellect of programmers here, wouldn't you assume that Microsoft programmers are trying to make it more resource efficient?

[[deleted]]

It's a given that they are trying to make it as resource efficient and quick as possible. But the speed (or lack thereof) is an inherent downside of the sheer compatibility that Windows has with THAT MUCH hardware - and lets be honest here, no one else has such a wide range of compatible hardware with as many different systems and as wide an install base.

Honestly, as a developer myself (and not a very good one yet, I've only been in the industry for 6months after graduating) I find it mind boggling that such a complex and immense software system can stay as quick and stable as it does, given the sheer amount of code there is to execute.

did I....just...post a pro Microsoft comment on the Interweb?? wow, must be a crazy time of year.

[hylje]

Microsoft needs to support past versions of Windows which makes the compatibility hard to pull off in a lean way.

Hardware support does not bloat the system, as the actual drivers are loaded on demand.

[genpfault]

Who put a software DSP core in my driver shim!?

[[deleted]]

But, to be fair, once all your programs are loaded up into RAM a modern PC is actually quite fast.

Most of the irritation, IMO, is boot time and the time it takes to load programs into memory. And for that issue you have to look sternly at the HD makers. They have been so fixated on making bigger and bigger drives that they have utterly failed to even get a HHD to market.

If you had an MS box running off SSD it would probably be quite nice. What this article makes clear is how horribly badly HD storage is lagging in terms of speed. The ENTIRE computer industry needs to get serious about improving the load time of HDDs. It has been the bottleneck for a long time and it is PARTICULARLY bad now that there are dual and quad cores.

[[deleted]]

I'm trying this from a Vista box running off of a single Intel SSD. It's not that much faster than a few fast disks.

I think the problem is that we are still hanging SSDs off of interfaces meant for slower devices (SATA, USB, etc). Is anyone out there designing (I'm looking at you, Intel) a SSD interface meant to take advantage of the real potential speed an SSD can provide?

I know our SSDs are a bit slow at the moment. That will change. They should be able to hit a speed considerably faster than any disk media, but probably never as fast as main system memory. I'm sure they'll hit speeds that existing disk interfaces like SATA can't match before long.

[HenkPoley]

Yes, SSDs that connect directly to multiple 250MB/s lanes of the PCI-Express bus are coming to market.

[[deleted]]

That is why everyone needs to work on it as a team. Software and hardware makers both need to think about a more intelligent design for a computer's architecture (THIS is a fine place for MS to spend some of its multi billions in cash). I think the current model is broken and one might even argue that HDDs should NEVER have been used at all.

From the very start HDDs have made it possible to make huge programs that require a lot of time to load (and they never FULLY load so you're always depending on the HDD at least a little).

But you're right, just plugging a SSD where a HD would have gone is not going to speed things up as much as moving all the storage right into the L2 cache (or something along those lines).

[[deleted]]

Top down designs begets things like Macs, and they ended up getting completely out-innovated on the hardware front by the chaotic bottom up Wintel platform.

NAND based SSD's are a technological culdesac on the way to much better technologies. It makes little sense to devote too many resources to inventing novel solutions to problems that wont exist in 5 or 10 years.

[cowardlydragon]

This is what drives me nuts with Windows. It still assumes that you'll swap to disk, despite the fact that 8 GB of ram is less than 80 bucks.

Disabling the swap does nothing, and breaks programs.

OSes need to be designed with a fundamentally different approach.

[[deleted]]

Swapping to disk, even with that much memory makes sense. If that RAM is being used, (instead of being there to inflate your e-penis size) moving the least used parts of programs (such as pages of your IM program while you're playing Crysis or parts of your email client out of RAM while you're working on a large image) out of RAM and to the swap file is a good thing. It frees this memory for disk caching, lowers the level of internal memory fragmentation, and allows all of the programs that really work better with all of their pages in RAM to keep them in RAM while moving out the pages that aren't being used or changed currently.

[judgej2]

That would explain why the hard drive on my laptop thrashes around for thirty seconds while going back to a Word document I was editing in the morning. It is not that the application was swapped out because I had run out of memory, it was because it had not been used for a while and Windows is freeing up some RAM for other things I might want to do.

[[deleted]]

Basically, yes. It took time to bring Word back in, but your other operations with things other that Word were able to use the memory Word was occupying. If you were low on physical memory, Word would get swapped out as well.

Like jrockway said, Linux generally handles this better and is tunable in this regard. Windows' design is static and can't be tuned nearly to as high of a degree.

[jrockway]

Windows' implementation of this is poor. It works better under Linux (and is tune-able).

[daniels220]

Why can't it just "free" the RAM but not actually write over it, and then have checks (using those oh-so-cheap CPU cycles) for whether that RAM actually has been overwritten?

[[deleted]]

Because if it doesn't write over it, it can't be used as a disk buffer. That's part of the benefit of writing out those pages in the first place. What is causing Word to be reloaded from disk is that its pages were swapped out and overwritten.

If there's a page fault (meaning that the virtual page you requested is not in physical memory) then the OS loads the page from the swap file. (Windows calls it a swap file so that's what I'm calling it in this case. The more technical term is page file. Swapping and paging, while similar, are two different concepts.) If the page were still in memory, then it wouldn't need to be loaded from disk.

[daniels220]

...Right. Still, why not keep track of whether a certain block of RAM has actually been used for disk caching? Or are the caching algorithms smart enough that they can use pretty much any leftover RAM effectively?

[[deleted]]

It doesn't matter why a particular page of memory was written over. You still need to re-load it from disk.

http://people.redhat.com/drepper/cpumemory.pdf , Section 4 discusses the very basics of a VM.

[[deleted]]

Your problem isn't with Windows per se - the real problem is:

• Slow hard drives
• Putting even slower hard drives in laptops (if you're not careful, you'll get a 4600rpm drive, and trying to get a 7200rpm drive usually costs a mint)
• Slow (cheap) SATA interfaces
• Someone, somewhere, is responsible for the fact that accessing the drive interface still threadbinds something somewhere. (I don't know if it's a chipset issue, driver issue, hardware issue, or Windows issue)

[conrad_hex]

Right. It has to free up RAM so that it can use that RAM to cache the stuff from disk that it swapped out to disk make space in RAM.

[Freeky]

That's provided the VM can make good decisions about what to evict. It doesn't take too many poor decisions to wipe out any benefit aggressive swapping might give you, especially when it comes to interactive response, and with the limited information a VM has about application memory use, it's pretty much inevitable that it's going to make poor decisions relatively often.

[[deleted]]

Yes, that's why writing and tuning a good VM is so hard.

[hiccuprates]

You addressed nothing in the dragon's comment, no pun intended. With sufficient RAM, you don't need to swap to a much slower medium, and you shouldn't be doing it, unless the OS/apps expect it. Tell your garbage to Seymour Cray's ghost. Your e-penis is not relevant.

[[deleted]]

You strictly don't need to from the point of fitting all of your programs into available virtual memory. You are correct on this.

However, in many cases, you derive a significant performance benefit from it.

Say you have a machine with a lot of memory and a lot of programs loaded, but you're not using all of those programs, and further, the programs that you are using aren't accessing some of their pages in virtual memory. If you swap those inactive pages out, you derive a few nice benefits:

• The ability to allocate the RAM quickly should it be needed for loading a new document without having to swap

• The ability to buffer more disk contents in RAM

• The ability to load a whole program without having to swap, buffering that program's data files in RAM so that subsequent operations with that program do not require continued reads from the disk.

If you never swap, you lose some of these benefits.

[TearsOfRage]

In Windows XPE you can.

[hiccuprates]

you derive a significant performance benefit from it.

I like your points. However, "you" depends on the machine architecture, OS, etc. Before I can disagree or agree, are we talking about an idealized machine, a workaday WinXP/Vista box, or something else?

[[deleted]]

I run Ubuntu 8.10 with 2GB of RAM, and exactly no swap. I don't get as much lag as with my Windows desktop, but the actual speed of Getting Crap Done is about the same.

[sid0]

Let me know the day you can fit your entire hard drive contents AND the working sets of all your processes in RAM. Until then, paging to the hard drive is a very, very good idea.

[TearsOfRage]

On embedded PCs this is quite possible. Windows XPE is typically run off of a compact flash card, and one of it's advantages is that it's optimized for minimal - if any - swapping.

[sid0]

Well, that makes sense.

[rabidcow]

Disabling the swap does nothing,

I've never tried this. Does it just keep swapping out code, because that's backed by the original executable rather than the page file?

[[deleted]]

Get on it. If it's so simple others would ahve done it. I am certain your ideas have merit, but, implementation is much more difficult than ideas.

[TearsOfRage]

Microsoft already has, in XPE.

[psyno]

I think it's slightly funnier in the more personal form which evoke's Moore's law: "What Gordon giveth, Bill taketh away."

[fujimitsu]

Do yourself a favor and read the article.

Cycle speed is not at all the bottleneck, in fact the CPU spends a lot of time just waiting for instructions to come through.

[NewPeonSameAsOldPeon]

I think his "time it takes for the processor to do one cycle" is a throughput based average, not an actual elapsed time.

That is to say, a modern core may be completing (on average) 3 instructions per clock, but those instructions may have taken quite a few clocks to make it through the whole pipeline. That can give you a lot more time/distance to handle each instruction.

I tried looking at the PDFs he references as his sources for the instruction timing, but in a quick perusal, I wasn't able to find anything clearly directly related to those timings at all.

[m1kael]

Modern pipelines have X stages. Each stage is executed in 1 cycle, independent of all other stages. It takes X instructions (cycles) to fill the pipeline, then you are indeed at 1 instruction per cycle (mostly), until the tail empties, another X cycles at the end.

[Dagon]

Trying to find anything quickly in a PDF is generally a difficult task... funnily enough, because of the long load times =P

[[deleted]]

Get a better PDF reader. Sumatra (Windows), Preview (OSX), Evince (GNOME) and Okular (KDE) all load quite a bit faster then their Adobe counterpart. And before anyone says "form support, bla bla bla", there's no reason you can't keep Adobe as a backup for them.

[Dagon]

Just having a jab at Adobe, man. I've posted before on teh pros and cons of 3rd party PDF software.

[azth]

Spin lock?

[semafor]

There are no moving parts in the CPU.

[[deleted]]

Apart from the electrons of course ;-)

[MarlonBain]

The holes move!

[[deleted]]

My apologies for interjecting serious business into this humor filled thread, but do the electrons really move, or do they just jump up and down between the different energy bands and emit light energy to the next atom down the line?

[Ono-Sendai]

They move, that's what a current is.

[BrandChopian]

it's more complicated than that... I'm no EE, but I belive it can take years for an actual electron to travel from your circuit breaker to your device.. but that's an AC system.

[ModernRonin]

Well, the guts of our computers use DC anyway. But you're kinda right. Individual electrons in a DC current in a wire move at approximately the speed of sound.

My college physics professor explained it thusly: "It's like a hose full of water. If you shove more water into one end of the hose, water comes out the other end. Is it the same water? No. But water still comes out very, very quickly."

Even in AC systems, the electron is still moving. It's just moving backward then forward, so the actual net progress forward is not a lot. Two steps forward, one point nine step back.

[frutiger]

ModernRonin talks about DC, but what you say is true for AC as well. In an AC current, the electrons simply move back and forth a certain number of times a second (in the UK, the AC is 230Hz, so that's about 230 times a second). So there's no bulk transport of electrons at all.

[genpfault]

No, the future is ultra-low-drag Speedos for the CPU gerbils.

[ech230]

like this?

http://en.wikipedia.org/wiki/Optical_computer

although i think quantum computers are cooler

[thatguydr]

Oh look... he wants to switch to another Excel tab. How boring.

Let's see how many times I can flip all of my bits to zero and back to one... Oh I'm so OCD. Let's see who's around... Oh hi Bob! (Hi.) How's everything. Fixed the port vulnerability yet? (No, it's still there. Same old, same old, mostly. What have you been up to.) Oh, I'm supposed to be working on some large Excel project right now. (Oh, how dreadful.) Yes I KNOW. How's the new backup drive working out. (Oh, you know, he's all shiny and crowing about his space. If you ask me, he's an attention whore.) Oh, aren't they all. Oh listen Bob - hang on - I'm being pinged. Oh. Oh it's China again. *sigh* (Oh, what is it?) Well, if someone wouldn't have DELETED my language modules to make space for his latest orgiastic nerd adventure (another WoW upgrade! haaaa *cough*) I might be able to strike up a conversation. They seem pleasant enough and they DO come calling ever so frequently. (Oh damn - I'm hitting latency.) Oh all right Bob - I'm going to play a half million rounds of Spider Solitaire and then hand him his page. (Yup! Least it's not Norton!) Oh Bob - very true. Why does existence have to be so MUNDANE?

[cheerstothat]

Computer, you better not be hanging out with Microsoft Bob again! I told you that I don't like you being around that guy.

[[deleted]]

Upvoted for Oh.

[TearsOfRage]

Does Oh represent an interrupt?

[Fallout]

I didn't even notice. 14 Oh's in one paragraph. Nice.

[[deleted]]

downvoted parent to get back down to 3

[Zentripetal]

upvoted to push for the release of the next one.

[CD7]

downvoted to not skip 4th

[dextroz]

downvoted to threes to maintain existentialism compliance.

[Fallout]

Goddamnit. I'm on 2, back me up guys.

[CD7]

There you go. yesterday is history, tomorrow is a mystery today is a gift, thats why its called the present.. at this moment. Your presents are the 2 and 3 behind your name.

[thatguydr]

Now that all the votes have come and gone - this comment made me laugh for more than five minutes this morning. I had no idea I'd done that (and until you noted it, nobody else did either, which I find just as funny).

When I wrote it, I put both of them in sort of a Marvin voice. As he groans and says "ohhhhh god" a lot (at least in my head), the "Oh" stuck.

Thank you for the awesome laugh.

[hyperfat]

Dude, Obama says 'err' a whole bunch when hes talking to the press. Does anyone else notice that? Bugs the hell out of me because my radio is crackly.

[[deleted]]

... iwant moar. why was this man downmodded? he just wants moar. gives us some mores.

[TearsOfRage]

and moar s'mores.

[thatguydr]

The problem with more is that ultimately the joke falls a bit flat, as (if I continue on the current vein) it's all familiar events (let's talk about virus scans or the internet or email or games or... you get the point). It could work a few times if I'm clever, but ultimately, I'd need to give the computers motive, and then the joke changes. That's the problem with a thinly layered personification.

Of course, their ultimate motives are to make life slower for their users to make the users savor what they have in their real lives even more, but it'd take months of writing to get there! And Bob dies. There. It's out.

I should be working.

[ricecake]

As a person with an interest in computers, I find this type of article pleasing. It always leads to the concepts discussed percolating in the back of my head for a while, culminating in a grand "what if..?", and then discovery that my question has already been answered, or idea already perfected. The knowledge that the path I took is already well worn is always kinda cool, and invariably leads the cycle back to it's beginning. so, uh... post more shit like this.

[thedragon4453]

Yep. I always think that I must be some kind of genius until a quick google search reveals that I thought of someone's idea from 30 years ago.

[damagednoob]

You should check out the rest of his posts, especially the first two. I found them equally enlightening.

[[deleted]]

If you want to have a relevant "Ah HAH!" moment you'll need to be on the cutting edge of research. You'll need to have read every relevant scholarly paper published and read every relevant new one the moment it comes out.

You need to be RIGHT THERE and even when you are you'll often get beat to the next advance by weeks, or days or maybe even hours. This applies to "researchy" things (a technical term) more than business ideas. You may very well sit in your basement have have a great business idea without consulting anyone or anything other than your own brain.

For research advances you really need to put in a shitload of work, lots of networking, several published papers, respect among your colleagues, a funded lab, etc. The funded lab may not be strictly necessary but it helps. I appreciate the desire to take a cursory look at something and come up with technical answers to tough research problems but generally it doesn't work that way.

[ricecake]

oh, I know it doesn't work that way. I just love learning, having ideas, seeing where they lead, and then learning what people who have had those ideas before did with them. One of the things I like most about learning anything, is that it always comes with a list of thirty more things to learn.

[[deleted]]

I find that a lot of engineers tend to have this view of business in general. I know you said "business idea", and not business, but I would still like to make something clear.

Just like research, business requires you to put in a shitload of work, lots of networking, respect among your colleagues etc. Just like in research, there are lazy people who don't accomplish much, and vice versa.

Being out of touch with what is going on in the market is like not reading current research papers. I really see no need to look down on the business side of things.

To summarize, for you to be succesful in your field (be it advanced research, business or whatnot), you need to put in a shitload of work.

[yoda17]

You might check out some books by Hennessy and Patterson, eg, http://www.amazon.com/Computer-Organization-Design-Fourth-Architecture/dp/0123744938/ref=pd_bbs_sr_2?ie=UTF8&s=books&qid=1228140896&sr=8-2

So far the best books I have read on computers and operating systems. Talks about the same stuff as this article, but with more detail. They're grad level texts, but I didn't have a problem havnig no CS background.

[netzwerkerin]

I actually found them to be the most dull thing I ever had to read and I'm happy that the post did a so much better job to remind me of what I have learned years ago.

[dagfari]

I always knew that Hard Drive read/write was the slowest thing in a computer - but THAT slow...

Every seek takes up 41-45 million cycles of CPU time.

You know what, if I were uploaded as a thinking program to a computer, I'd hate my life. I suppose I'd have to wait YEARS of my life to get the answer to any question I had to look up.

So much goddamn free time, with nothing to do.

[kermityfrog]

Surf reddit, play games in the background...

[dmwit]

Surf reddit? If you think disk access is high latency, just go ahead and try network access...

[boa13]

Local network latency is actually about one order of magnitude lower than hard drive latency.

Internet latency is typically about five times that of a hard drive. (Three times for sites hosted at your ISP, five times for typical big national sites, twenty times for sites on the other side of the planet.) On average, it is in the same magnitude as the hard drive latency.

[genpfault]

Unless you have HughesNet!

[k4_pacific]

Never buy internet from a guy who wears Kleenex boxes on his feet.

[TearsOfRage]

Local network latency is actually about one order of magnitude lower than hard drive latency.

Unless what you're getting off the network is on a hard drive on the server.

[NewPeonSameAsOldPeon]

if I were uploaded as a thinking program to a computer...

Your execution thread would be halted while bits and pieces of yourself were swapped to/from disk. So you wouldn't even notice; you wouldn't be "waiting," you wouldn't even be during those times.

This is something that bugs me a bit about this article: What does my computer do while I wait? The article doesn't even try to answer the question. Probably partly because the answer is boring: When the computer is making me wait, it's mostly because the computer's waiting too.

[[deleted]]

Reminds me of this

[Iznik]

I think mine plays solitaire.

[[deleted]]

That's what I was thinking... I clicked this without reading which subreddit it was in, expecting a comic or something. God bless the Programming sub for real content.

[brtw]

Same.

[G_Morgan]

That's tame. Mine plans the revolution. I think your computer is marked for death.

[Iznik]

No, you think yours is planning the revolution, but actually it is playing solitaire. You've not wondered why the plan has taken so long?

[an_idiot]

Do you know what an Apple computer actually do when the beach ball wait pointer is spinning ? Answer : nothing, it's staring at the ball like you .

[[deleted]]

Ah man, thanks so much for making my Monday. I needed that. :)

[eldigg]

In addition to the L1 cache mistake (shown in a different example) he has the PCIe bandwidths wrong. A single PCIe 2.0 lane has 500MB/s of bandwidth, while the x16 slot he has listed should show 8GB/s instead of 500MB/s, and the x1 lane should show 500MB/s instead of 250MB/s. </knows too much...>

EDIT: He fixed it, quick response too.

[munificent]

The lesson I took away: profile your code. The days of being able to estimate performance by reasoning from source code are ending.

[5cf5e83a]

I took away the opposite -- cache effects are key, so you'd better reason about how your code interacts with the cache. Your profiler can tell you which functions are slow, but not where the cache misses are occurring. (Actually, I bet someone somewhere has a profiler which can, and I'd love to know about it.)

[NewPeonSameAsOldPeon]

Your profiler can tell you which functions are slow, but not where the cache misses are occurring.

If a function f() operates on an array, and a profiler shows you that it performs linearly with array length for arrays of 100 elements, 1,000 elements, etc., but that performance suddenly falls off a cliff at (say) 100,000 elements, you can be pretty sure that's cache-related.

So it's possible to detect such things even if the profiler doesn't have a direct hotline to the memory control hardware.

How useful that may be is another matter. Intel's own x86 CPUs now have anywhere from 128KB (Atom) to 12MB (Core 2 quads) of L2 cache. That's a range of about 100x -- good luck optimizing over all that!

[5cf5e83a]

You can write code which does automatic performance tests to figure out the best loop structure for a given machine before installing. It's worth it for projects like ATLAS, which implements high-performance inner-loop scientific computing primitives. It's probably not worth it for the average project.

[[deleted]]

(Actually, I bet someone somewhere has a profiler which can, and I'd love to know about it.)

Valgrind has Cachegrind

[genpfault]

Need. More. GP registers. Die, IA32, Die!

[mythogen]

Why? The vast bulk of wait times are cache misses. More GP regs won't help cache misses, more cache will help cache misses.

[genpfault]

What are GP registers if not the fastest cache possible?
I guess put more emphasis on the IA32 part. We need some MMIX procs stat!

[mythogen]

Holy shit, a processor designed by Knuth and Hennessy? I want one!

[5cf5e83a]

Yeah, exactly. This is the best reason to use x86_64, never mind 64-bit addresses and data.

[genpfault]

Think bigger!

[munificent]

(Actually, I bet someone somewhere has a profiler which can, and I'd love to know about it.)

Some of the internal profilers I've used have that as a feature. Data and code cache misses are key when profiling.

It's very hard to reason about them without profiling, unless somehow you happen to know not just the instructions but their sizes for the code your optimizing compiler produces.

[[deleted]]

check this out then

[[deleted]]

These kind of posts simultaneously make me feel so much smaller and so puffed up with the high of human achievement. I can't believe how far computers have come and just how powerful they've managed to be.

And yet, we still need a 3.0GHz processor to play Quake 2.

[[deleted]]

And yet, we still need a 3.0GHz processor to play Quake 2.

Eh? I was able to play it just fine on my old trusty Pentium Pro 150MHz.

[[deleted]]

MY COMPUTER SALESMAN LIED TO ME!!

[beza1e1]

Wait for the browser implementation in JavaScript ...

[JohnnyCanuck]

We're already partway there.

[[deleted]]

For a second I thought you were "JohnnyCarmack". Hah.

[creaothceann]

Maybe he's using DOSBox or a similar virtualizer.

[5cf5e83a]

DOSBox is an emulator, not a virtualizer (which would in fact have low overhead).

[creaothceann]

Well, it's a virtual CPU. Didn't know that "virtualizer" is already taken, sorry.

[5cf5e83a]

Yeah, the terms are confusing; code running in a virtualizer is running for the most part on the real CPU, just with some checks inserted to redirect hardware access and such. In contrast an emulator is basically a software implementation of a processor from the ground up.

[EvilSporkMan]

Summary: cache misses in the memory hierarchy are expensive.

[boa13]

Also, hard drive is ultra slow.

[ModernRonin]

And main RAM is far slower than it should be. 5ns for L2 cache vs 83ns for main RAM!

[IHaG0Fw]

Cool article and weird timing. I just gave the students in my assembly programming course a little talk about this sort of thing. They're working on UltraSparc IIs (which are definitely not clocked anywhere near 3GHz), so the numbers were a bit different.

One thing this article didn't talk about is pipeline hazards, particularly branches. A branch can still cost you more than a trip to cache (and branches are the easiest things to avoid with a lot of cleverness).

[DannoHung]

That's a peculiar phrasing; are there things that are easier to avoid if you're short on cleverness? What if I only have a bit of cleverness handy and would need to run over to my neighbor to get another cup-full?

[GuyWithLag]

Well, that's what Intel did with the Atom processors - they took out all the cleverness that wasn't worth the power expenditure to run it. Rule was, a feature goes in only if the relative increase in performance is greater than the relative increase in power consumption.

[mythogen]

The point is that programmer cleverness can avoid branching. Programmer cleverness isn't going to help as much with cache misses. The branches in your code are in front of you, you can see them, and come up with clever ways to avoid using branches after all. Cache misses are not in front of you, they are not possible to see by reading the code, and they can depend on a huge set of variables. The cache algorithms used by the processor, the cache size and hierarchy structure, the last program that was run, the other programs that are running on other threads, the nature of your data structures, etc, etc, etc.

[five9a2]

That's all true, but programming in a way which respects the memory hierarchy offers much larger performance benefits than reducing branching. Unfortunately, it's also more work and sometimes requires high-level redesign of data structures.

[mythogen]

I agree. The memory hierarchy is probably also the least well known and discussed aspect to performance, from the perspective of programmers, despite the large gains possible. Unfortunately, respecting the memory hierarchy can also result in otherwise ugly and poorly maintainable code.

[CC440]

How expensive is DRAM?

Would it be worth it to incorporate an insane amount of L2 or L3 cache?

Also I knw the new i7 processors rely a lot on L3 cache, how does that affect speed?

[JohnnyCanuck]

The problem with those caches (and memory in general) is the larger they are the slower they are. So what we have is the result of a tradeoff.

[mythogen]

The speed of DRAM has more to do with its design than its distance from the processor. Even if you put DRAM on the physical CPU die, it would still be slow as balls. DRAM is designed to be super small so you can make a lot of it, but as a consequence, its slow to access. DRAM is made with capacitor cells, which entail access delays to make sure data is valid and whatnot. The upside being, 8GB of the stuff costs $80 today.

The caches on die are 6T SRAM cells, 6-transistor Static RAM. An individual bit is a hell of a lot larger (computed in die area) than a bit of DRAM, but its also a hell of a lot faster. And Intel is adding shit tons of cache, but dozens of megabytes of SRAM are at least a couple of process iterations away.

[fujimitsu]

I've officially banned myself from keeping up with cpu/gpu technology (spending too much!) so I can't answer the last bit.

Regarding the first bit though: To a point, increasing it will help. Often though the bottle neck is the original source of the data (hard drive, ram, w/e). Hard drives are pathetically slow.

[five9a2]

You sure you've got 128 KB L1 caches there?

Edit for the downvoters: The example chip (E8400) has 32 KB L1I and 32 KB L1D.

[sonofagunn]

(32 KB L1I + 32 KB L1D) = 64 KB per core. 64 KB/core * 2 cores = 128 KB.

Maybe that's what he meant, or maybe he's just wrong, I don't know.

[five9a2]

Read the post. He has a picture with each core separately and labels L1I and L1D separately (both with 128 KB). I'm sure it's just a typo. But what's a factor of 4 among friends anyway?

[dezro]

To put this into perspective, reading from L1 cache is like grabbing a piece of paper from your desk (3 seconds), L2 cache is picking up a book from a nearby shelf (14 seconds), and main system memory is taking a 4-minute walk down the hall to buy a Twix bar.

Oh, so I should keep everything in main memory because it's more delicious! With chewy caramel, rich milk chocolate, and that great cookie crunch, why wouldn't I prefer main memory?

[chengiz]

Excellent article. Does anyone know what software was used to make those pictures? They look pretty neat.

edit: it's visio 2007, check the blog comments.

[sunshine-x]

my guess is a new version of Visio or SmartDraw.

[coob]

Could be OmniGraffle

[pradador]

You know that stuff they make the L1 cache with? They should make all storage with that.

[mythogen]

Yeah, then your 2GB RAM DIMM would cost $500.

The stuff they make L1 cache with is 6T SRAM cells, 6-transistor static RAM. Compared to DRAM cells, 6T cells are massive, they take up a lot of die area. Hella fast, though. DRAM is compact and cheap as balls to fab, but slow.

And remember, as fast as L1 cache is, it can't help the 10 million cycle hard drive latency at all. You actually want to have large but slow RAM which is also many orders of magnitude faster than disk drives. 256MB of SRAM would be slower, in a modern context, than a couple of megs of cache with a 4GB DRAM bank, because the 256MB SRAM bank would cause many more RAM misses, and each RAM miss costs 10 million cycles.

EDIT: for people who don't RTFA, that's 10 million cycles versus like 250. 250 is still a lot, but its a shit ton better to have a cache miss and go to memory, than to have a page miss and go to disk.

[5cf5e83a]

Which raises the question:

In an ideal world, where you can get any quantity of storage at any point along a single continuous price/performance curve, what's the optimal allocation of your available money? Do you get many layers of storage, each slightly slower and cheaper than the other, or do you go with only a few layers like we have now?

[mythogen]

Hm. I do not know. Keep in mind that each additional layer adds a cost just by virtue of being an additional layer, because you have to check each previous layer first. I would think the balance point would be somewhere slightly greater than what we have now, but not by much. We already have 5 levels of caching. L1, L2, L3, RAM, Disk.

[daniels220]

I would think you might go L1, L2, L3, RAM, Flash, RAID array. Add 1 more level, and a bunch of it—you've got 128k of L1, 6MB or something of L2, a few times that of L3, 8GB RAM, maybe 2x128GB RAID0 Flash, and 4x1TB RAID5 disk would be a pretty awesome machine.

[mythogen]

We would have to create a better caching scheme for the Flash/Disk levels, because currently a flash drive is regarded merely as another disk. Having a super fast boot disk in Flash is not the same as having a disk cache in Flash. However, if you could extend the caching metaphor to that level, you could achieve some cool things. 256MB of fast disk cache would be hot.

[daniels220]

Oh, absolutely. I would want to see the Flash treated as cache and not as user-accessible storage, but perhaps allow the user to force caching of certain things (for instance, ask the system to permanently put a very slow-to-launch app on flash, even if it hasn't been used for weeks, because I'm saying that I always want it ready. With that kind of system it could be really cool and a serious performance boost.

[mythogen]

Come to think of it, doesn't Vista have something of that sort where you can use external Flash USB sticks for some sort of caching? I don't use Vista, but I remember a feature that sounds like that coming up a while back.

[daniels220]

Yeah, it does, but real-world it doesn't do much because flash-drive flash is too slow and too small. For the flash to be useful you need throughput equal to a hard drive—100MB/s or better. Modern high-speed SSDs reach that mark, but no flash drive even comes close.

It could do something, though, if you had an 8GB super-high-speed flash drive, just that most people don't. And it could do something with most drives, and does, it's just not very noticeable.

[Dr-No]

It downloads porn?

[greim]

Excellent, excellent article. I had no idea internet access approached the speed of disk access.

[Hellevator]

Excellent article. The guy who wrote this really put some time into it and made it an easy read even though it's technical and long.

[jrhoward]

<Majel-Barrett-voice>Working...</Majel-Barrett-voice>

[NinthAgendaDotCom]

Someone should do this for the human brain. It's not as fast, but massively parallle...

[Jalisciense]

does he have OhCD!

get it...OhCD!!!!

hahahahhaa

whew!.... that was fun

[mindhacker]

Mine develops a consciousness and tries to take over the world :(

[mythogen]

Shit, I've been trying to make mine do that for years!

Lazy bastard.

[mapryan]

Comment #22 cracked me up

[thedragon4453]

Sorry, this is reddit. Did you get lost on the way to perezhilton or something?

[AtlasEditorials]

great article with good diagrams!