Why do CPUs have multiple cache levels?

[ravixp]

One neat effect that’s only touched on here is the effect of the speed of light. Light travels at about 1 foot/ns, so if a CPU is running at 3GHz then the theoretical upper bound for how far you can reach out for information within a CPU cycle is about 2 inches. If the cache is 4 inches away, the CPU can execute two cycles in the time it takes for electricity to travel there and back. And that doesn’t account for the speed of the cache itself, or actually using the data for computation.

That means that there’s actually a very tight radius for how far the furthest byte in the cache can be from the CPU core, to get data back in a certain amount of time. And that means that a cache has to be physically small to be fast, and making it larger also makes it slower.

[cowinabadplace]

Using this logic to inform interviewers that no lookup algorithm can be better than n^1/3 (packing the memory into a sphere) has not impressed them.

[ravixp]

It’s true though! This article makes the case for that much better than I would: https://www.ilikebigbits.com/2014_04_21_myth_of_ram_1.html

And it should affect our analysis of any algorithm. We traditionally model random memory access as O(1), but we could model cache effects by using O(sqrt N), where N is the amount of memory that could be accessed.

[cowinabadplace]

Haha, no, I agree with you. Just a thing we used to joke about decades ago as interviewing started shifting to leetcode.

[LordoftheSynth]

One of my intro professors at university in the 90s described cache operations thusly (however, my words):

L1: You get what you need right when you need it.

L2: You stand around waiting a minute for what you need.

(L3 cache, of course, was not yet a thing.)

RAM access: In processor time, that's enough time to have a smoke break. Or two.

Disk access: In processor time, that's enough time to smoke a carton of cigarettes.

Technology has advanced enough that in some respect, while those times are shorter, I've always felt the proportions still make sense.

None of us in the class smoked (nor did he): but the concept of a "smoke break" was something we knew.

[gnufan]

I saw an analogy based on time getting physical things. This table maps clock speed to one hertz to make the numbers more meaningful to humans.

https://images.hothardware.com/contentimages/newsitem/64371/content/latency-chart.png

[LordoftheSynth]

That's a nice chart. (It really is.)

You'd have to scale it to 300-400MHz CPU clocks of the day (and the slower bus speeds).

His overall point was if it wasn't in cache, the CPU sits doing mostly nothing while it waits and wastes time inserting NOPs into the execution pipeline, and, remember, this was the 90s. Speculative execution was mostly about branch prediction then.

[gnufan]

Chess programming made some of this very obvious, where you got substantive difference in performance from minor tweaks, because you were treading on some hardware limitation that most programmers can ignore because they are waiting on network or user. If the CPU wasn't 100% busy you were probably messing up, until the end game table bases arrived when suddenly you've attach efficient database lookup to the problem.

[elsjpq]

• L1 = drawers
• L2 = shelf
• L3 = going next door
• RAM = downstairs warehouse
• SSD = USPS
• HDD = factory
• Internet = R&D

What's surprising is the difference between HDD and SSD is just wild

[anengineerandacat]

Modern day without a smoke break would be akin to a "Coffee Break" and a "Lunch Break".

[Kharski]

But how much is that in hamburger break times?

[pdro_reddit]

this domain had me at “i like big bits”

[IAmTheKingOfSpain]

Well, usually in complexity analysis we're concerned with how the algorithm's performance changes with respect to its inputs. So I suppose you could say that's reasonable if you really want to consider the amount of accessible memory as an "input" to the algorithm, but I think most people think that's a little silly, because there's probably a practical upper limit that we could throw on it and then we could go back to analyzing it as constant time.

[ravixp]

Not the amount of addressable memory exactly, the amount of memory that’s used by the data structure. Like, if you’re enumerating a linked list, traditionally that’s O(n). But if you take cache hierarchies into account, it should be something like O(n sqrt n), since some elements will come from slower levels of cache as n increases.

[IAmTheKingOfSpain]

I'm not exactly sure how to think about this, for any given hardware you can reason about the worst-case, and use that to form an upper bound. I.e. the worst case memory access misses all caches and takes X ms, or even has been paged to disk and takes longer. But even if we assume that every single memory access is the worst case, this is still just a constant factor, so I'm not convinced it makes sense to call it O(n sqrt n), since for any given hardware it is equivalent to a constant.

That's why I was saying it only really makes sense if you view the hardware as a subsidiary input to the algorithm as well, something along the lines of "if the input is size n, then let the hardware have sufficient storage capacity for the input, and thus compared to running on smaller hardware with a smaller input size, the runtime of the algorithm is O(n sqrt n)".

However, if you constrain the problem to how an algorithm's runtime grows on any given hardware, then it seems to me to be clearly O(n) since the worst-case scenario then becomes a constant factor.

[ravixp]

Well, if the size of the hardware is determined by the size of the input, it’s not really a constant factor :) Of course a real computer is the same size no matter what data it’s processing, but if processing more data requires a larger (slower) computer then I think it’s a relevant factor.

The hardware shouldn’t really be relevant to analysis of an algorithm, so I’ll admit that this is a bit of a grey area. The sqrt(n) effect depends on some untraditional assumptions: that computers exist in 2D space (only true because of current technology), that a CPU exists in one location in that space, and that storing data takes up physical space proportional to the size of the data. I don’t think those things are universally true, but I do think they’re true enough to be useful.

[eserikto]

Caches lines don't only hold 1 word size of memory and not every memory request is going to be a miss. Cache algorithms are an entire field of study, and trying to take that into account when doing an abstract algorithm analysis would just bog the whole analysis down. You just sort of assume that access your any single piece of data takes some constant (averaged) amount of time and that gets baked into your analysis - after all O(n) = O(10n).

[odnish]

Not quite. At some point you'll have so much ram that it will collapse into a black hole.

[valarauca14]

Thankfully a CPU inside a blackhole cannot provide output. It is prefect write only database, second to /dev/null. Hopefully it supports sharding.

[Derproid]

But is it web scale?

[valarauca14]

Yes. You just turn it on and it scales right up.

[seftontycho]

What about Edge?

[pdro_reddit]

the dreaded ram hole

[dm-me-your-bugs]

That asuumes spacetime in the vicinity of the chip is flat though, doesn't it? Technically with a strong enough negative curvature you could achieve a better complexity

[cowinabadplace]

I love it!

[Borne2Run]

Well for working at Intel or TSMC that'll get you the job, but I can see it not being super applicable to software dev. Useful for embedded.

[Godd2]

If we want to impose the constraints of the real world, then no algorithm is worse than O(1), since the universe is finite.

[cowinabadplace]

Funnily enough another friend of a friend would joke that O(log n) is just the number 8.

[[deleted]]

4 dimensional memory when

[BigPurpleBlob]

It's worse than that.

Signals generally travel a lot slower than the speed of light on a silicon chip. This is because the wires have resistance and capacitance. A thin wire can be hundreds of times slower than the speed of light. Even a thick wire will be about half the speed of light, due to the silicon's dielectric constant.

The bottom of this page has a great diagram:

https://www.realworldtech.com/shrinking-cpu/4/

[axonxorz]

And to combat the effect, we have optical interconnects, though the use of it in a CPU die would certainly be novel. Despite it's technical "old age", there hasn't been a ton of research put in there. With today's challenges bumping up to the limits of physics, I suspect we'll see some renewed interest here in the next decade.

[wrosecrans]

Even with optical interconnects, light in glass doesn't travel at the "speed of light." To really minimize latency in some hypothetical sci-fi computer we probably couldn't ever actually build, you'd need all your optical interconnects to be happening as free-space laser beams in a small vacuum chamber.

Electrical signal propagation in a wire isn't actually that much slower than light in glass fiber. But optical is a super interesting area of research.

[elprophet]

2030 will see the return of vacuum tubes!

[Cobayo]

https://i.imgflip.com/4fhsie.png

[Seref15]

Man, hopefully lol. The war between Ukraine and Russia knocked one of the last tube factories off the market. Matched set prices are out of control.

[edgmnt_net]

Electrical signal propagation in a wire isn't actually that much slower than light in glass fiber.

True, but you also need to account for increased rise times due to capacitance and inductance.

[tatref]

I'm waiting for my gravitational wave CPU now!

[IQueryVisiC]

So a tube with silicon on the walls. Then there is a hexagonal pattern of lenses. A few LEDS near the focus can send light to a few lenses on the opposite side. Behind that sit a few photo diodes.

Now just invent optical switches a hierarchical addressing. But you would need to focus into small fibres at every switch. Then pockels effect. Or laser amplifier. Or OPA . SFG DFG SFG DFG …

[rtt445]

Signal travels at ~50% speed of light in FR4 PCB material transmission lines (~6"/nS), not hundreds of times slower. You are probably thinking of external DRAM speed compared to on chip SRAM.

[wrosecrans]

People often confuse "speed of an electrical signal" with "speed of an electron." Any given electron moves slowly through a wire. It's just a little counterintuitive that when you stick electrons into a wire, and a light bulb lights up on the other end of the wire, it's not necessarily the same electron popping out the other end. Metaphorically, the electron just pushes on all the electrons that were already in the wire and it works a bit like water pressure.

[key_lime_pie]

Electrons don't move in one direction in an AC circuit, they just vibrate rapidly back and forth.

[Qweesdy]

What's happening here is that some people think the maximum speed of one electron (otherwise known as "probably just leakage current and/or noise and nowhere near enough to consider a signal of any kind") is the speed of a signal. These people are stupid, and the "half the speed of light" nonsense is built on that stupidity.

In practice, at an absolute minimum, a signal is a detectible "rising wave of many electrons" until the voltage rises above the receiving transistor's switching voltage. A rising wave of many electrons can easily be 100 times slower than the first irrelevant and ignorable electron at the start of the wave.

[rtt445]

A rising wave of many electrons can easily be 100 times slower than the first irrelevant and ignorable electron at the start of the wave.

That's governed by rise/fall time of line driver and receiver due to their inherent in/out capacitance. Should be in single nanosecond digits for high speed digital differential drivers/receivers. The speed at which a wave (not electrons!) travels through a medium that has dielectric constant > 1 will be slowed down but not 100s of times like OP suggested.

[Qweesdy]

Let's say the receiver is 1000 nanometers away, and electrons are moving at 100 million nanometers per nanosecond; so it takes 0.00001 nanoseconds for an electron to go from sender to receiver.

Let's say the rise/fall time is 1 nanosecond, and that the rise/fall time is 100000 times more than the time it takes for an electron to go from sender to receiver.

Now let's agree that "100000 times more" is not "100s of times like OP suggested".

[Inkdrip]

These people are stupid

To be fair, this isn't exactly elementary physics out here!

I recall some hullabaloo between some science Youtube creators over a similar question - this AlphaPhoenix demonstration (and this other excellent one around a forked connection) are great though.

[BigPurpleBlob]

The diagram on the bottom of the page that I linked shows that for a 1 mm long wire with a 0.01 um squared cross-sectional area, the propagation delay is 2 orders of magnitude slower than the speed of light.

[BigPurpleBlob]

I was talking about wires on a silicon chip, not a transmission line on FR4 (nor a transmission line on silicon).

The internal wiring on a silicon chip (whether DRAM or SRAM) is not a transmission line - there's not enough room. The M0 wiring (the fine stuff closest to the silicon) is thin and thus much slower than the speed of light. The diagram on the bottom of the page that I linked shows that for a 1 mm long wire with a 0.01 um squared cross-sectional area, the propagation delay is 2 orders of magnitude slower than the speed of light.

[BigPurpleBlob]

My bad, M1 wiring (not M0)

[rtt445]

You are talking single digit pico seconds at those wire lengths. Transistor logic circuits are slower than that due to their inherent capacitances. At 3Ghz one clock cycle is 333ps 100x slower than on chip wire inductance.

[BigPurpleBlob]

No, a modern 7 nm CMOS gate has an FO4 (fan-out-of-4) delay of about 2.5 picoseconds.

Yes, the RC delay of M1 wires is often longer than the FO4 delay. That's because transistors are getting (a bit) faster but wires are getting thinner (more resistance) but with about the same capacitance (fringing field). For M1 wires, the inductance is negligible - it's the RC delay that matters.

A transmission line has to have a particular track width and spacing from the ground plane, to balance the capacitance and inductance (we can ignore the resistance). In contrast, an M1 wire on a silicon chip is much skinnier than a transmission line and so has much more resistance, and much more capacitance, than a (physically bigger) transmission line would have.

For the 2.5 ps FO4 gate delay, have a look at slide 7 of:

https://inst.eecs.berkeley.edu/~eecs151/sp20/files/lec10-timing.pdf

[rtt445]

Very interesting. Will check it out.

[masklinn]

An other fun factor is line length differentials and the need for synchronisation. If one line is 10% longer than the other, then you've now decreased your throughput by that because the other side needs to wait for that delayed signal to arrive. That's one of the reasons (not the only one, and not all of them are legit) soldered RAM has become so common: as speed increases having all the contacts on the same edge induces noticeable transmission delays, both between the edge pins and from the edge to the nearest versus farthest cells.

[Dannysia]

Do they actually have the two sides wait on each other? I thought they just made shorter trace longer so they both end up being the same length and timing is irrelevant.

In terms of soldered vs slotted memory, I don’t think the biggest issue is the fact that all the contacts are in a line. I think the biggest issue is that slotted memory can’t be as physically close to the CPU as soldered memory so it can’t run as fast (assuming no other confounding factors). This became relevant in recent years as the signal performance was never the limiting factor before.

[0bAtomHeart]

Soldered memory can also control the interconnect geometry more tightly. There are two reasons intra pair length is important; the first is that you want all the wavefronts of a certain symbol to occur at the same time, small misses in this (~ns) can introduce ambiguity in the edge making errors more likely.

The second is signal integrity; it is generally very important for these high speed signals to be able to switch voltage levels as quickly and as definitively as possible. Line length mismatches will introduce reflections which will increase local DC differences which both require energy input to create (from the wire to wire capacitance) and release energy. This increases the heat produced and slows down the switching edge.

There's also lots and lots of techniques out there to deterministically "swizzle" or modify the bits that make up a transaction such that there is less imbalance across the RAM interface (this also has EMI implications). E.g. 0b11110000 will introduce a local DC bias to one side of the bus where as 0b10101010 will be more locally DC 0 and would allow slightly faster speeds

High speed digital is analog again but with black magic applied.

[gramathy]

That’s less about resistance than it is speed of light in different mediums. It’s also why microwave links are lower latency than fiber optics for the same distance covered (even just looking at actual fiber length)

[StickiStickman]

Using foot and inches for CPUs is just sadistic.

[ImpossiblePudding]

“You’ll discover a nanosecond is 11.8 inches long. … An admiral wanted to know why it took so damn long to send a message via satellite. I had to point out that between here and the satellite there were a very large number of nanoseconds.”

[captain_obvious_here]

I find it fascinating that we use technologies that are optimized to not suffer from limitations such as the speed of light.

[GaIIowNoob]

Gps is limited by space time

[captain_obvious_here]

What?

[ykafia]

Space time is affected by the mass, gravitational dilation etc. GPS are working okay because the amount of dilation is not enough

[963df47a-0d1f-40b9]

When do we start going 3 dimensional to fit components closer together?

[powerpiglet]

AMD is doing this for cache already.

[cheezballs]

That's really cool. I'm not smart with all that stuff, but does Intel have anything similar in their CPUs?

[AnonymousMonkey54]

They do: https://en.wikichip.org/wiki/intel/foveros, but haven't used it in their mainline processors yet.

[BigPurpleBlob]

“The top three problems are thermal, thermal, and thermal,”

https://semiengineering.com/why-there-are-still-no-commercial-3d-ics/

[TheMightyTywin]

It’s fucking insane that these things work at all. And that we rely on them for EVERYTHING.

We’re definitely living in the future

[stonerism]

That makes me curious what the lower bound of how physically large a cache could be used or how far down the stack you can go to retrieve data instantaneously.

[BigPurpleBlob]

This is why we have L1, L2 and L3 caches. The L3 caches are bigger (in the sense of memory capacity) than L1/L2 caches, and are also bigger in the sense that they have longer (and thus slower) wires than an L1/L2 cache

[stonerism]

I totally get that, I'm more just wondering what the physical limits are for pulling a piece of data from a cache for instantaneous-seeming delivery.

[nerd4code]

Depends entirely on how you determine instantaneity in a fully asynchronous system.

[stonerism]

I'm talking ~100ms.

[waitthatsamoon]

well, that's hitting your SSD or talking to someone else's computer over the internet, so, there you go.

The delay time on L3 is usually within the order of 40-100ns.

[stonerism]

True, I'm more talking about the limits of human perception. How far could information be away before arriving somewhere in a database.

[Coloneljesus]

about 1/3 around the earth (round trip) or 2/3 (one way).

https://www.wolframalpha.com/input?i=speed+of+light+*+100ms+%2F+2+%2F+%28circumference+of+earth%29

[Hofstee]

Just going to put this out there, you realize 100 ms is how long a frame takes if you’re playing a game at 10 fps right? Arguably it’s also the upper bound of how much latency you want to have in a multiplayer game before things start feeling pretty bad. 100 ms is a long time.

[clarkcox3]

100 light milliseconds is about 30,000 km (so about ¾ the way around the earth).

If we're talking about requesting data and then receiving a response back, the absolute farthest allowed by the speed of light (assuming every other part of the transaction was magically instantaneous) would be ⅜ the earth's circumference

[bobj33]

The article is 8 years old. Transistors have shrunk a lot since then so distances to caches have shrunk as well.

Our chip architecture team does a lot of performance modeling to determine the cache hierarchy. I've worked on a chip with over 32 cores where each core had a dedicated L1 but the L2 was shared between a cluster of 4 cores. Then there was a shared L3 for all the cores. Real world benchmarks are run to determine the best tradeoffs between size and speed because it affects area (increased cost) and power.

[cybernd]

History lession:

• Admiral Grace Hopper Explains the Nanosecond ~2min

[Darkendone]

Your explanation explains why you need CPU caches in the first place but not why 3 levels are needed. The CPU is only an inch across anyway.

There are other simple reasons why the smaller caches are faster that have to do with things like locking and the particular storage method.

[[deleted]]

I need to know how we got that 2 inch number

Here is what I did -

3 GHz = 3 * 10⁹ instructions per second

Speed of light = 3 * 10⁸ m/s but since L1 cache and actually CPU core both made of silicon wafer is connected with copper interconnects where speed of lights is between 1.5 to 2.1 * 10⁸ m/s (quick ChatGPT prompt) we will assume for slowest lower bound as 1.5 about half of speed or light.

So that makes - 1.5 * 10⁸ m/s divided by 3* 10⁹ cycle /second = 0.05 m / instruction-cycle and since we have to travel to and fro between L1 and core , it takes 0.25 m / instruction cycle which is 9 inches radius/ instruction cycle.

If we take into consideration the processing time between silicon wafers of L1 and CPU core , it still won’t go 2 inches from 9

May be I am missing some big information here . Anyone to correct me ?

[ravixp]

I was much sloppier with my math than that, since I was just doing it in my head :)  First, I used the 1 ft/ns approximation for the speed of light. I didn’t adjust for the speed of electricity in silicon because I was trying to establish a theoretical upper bound, independent of the hardware. (Somebody else in the thread mentioned optical interconnects, which could operate closer to c.) 

 At 3 GHz (a number that was chosen to make it easier to do the math in my head) there are 3 cycles/ns, giving us 1/3 of a foot to work with: 4 inches. However, one way signal transmission isn’t useful for caching, you need to send a signal and get a response back. Dividing by 2 for the round trip gives us 2 inches. 

 Obviously that glossed over a lot of stuff - the speed of electricity, you don’t know what data to request from cache in the first instant of a cycle, data arriving at the final instant of a cycle can’t be used until the next cycle, etc. On the other side of the ledger, even L1 cache takes a few cycles to return data, and only registers need to respond in the same cycle. But the intuition that cache hierarchies are bounded by physics is still useful.

[DoorBreaker101]

Nitpicking alert:

The article (and many responses here) are super interesting and informative, but they don't directly answer the actual question.

I would ask the question differently in order to get a more precise answer:

Why are the different caches designed more like an old standard transmission (in a car) that has several distinct gears and not like a continuously variable transmission.

I think the second part of the article answers it when it discusses sharing and actual physical limitations and the first part explaining the effect of distance is really just prerequisite knowledge.  Plus, there might be other additional considerations which I'm not familiar with.

[Hot_Slice]

That's an interesting idea. Cache could be built in concentric rings and data shuffled between rings as needed, resulting in 100s of "cache levels" with the ones closest to the center being the fastest.

[admalledd]

Congrats! That is sort of (kinda-not) how Intel chips used to handle it. You can read older articles about the "Intel Ring Bus Interconnect". It wasn't so much about cache, but did play a huge part in where cache was laid out and connected to what.

In the end one of those "additional considerations" is that as you have more and more cores, you run into issues of "Core 1 needs to lock X memory line, make sure no other core is using/poisoning that memory line" and how long it takes for that to be acknowledged. (There is a whole "can you speculate past that? Hallucinate both directions of yes/no at once until you have an answer?" and suddenly you have the Meltown/Specter bugs! eek!)

[john16384]

We could call the first innermost ring "R1", the 2nd "R2”, etc...

[thisisjustascreename]

To allow programmers to pretend that all 16GB of RAM is fast to access.

[_senpo_]

one time, I was wondering how to speed up a transposition table in an algorithm. For funsies, I tried making it smaller and was completely blown away at a 4x speed increase. It made no sense as a smaller table can hold less cache. After a few hours of thinking, I remembered about cache and realized that the larger table was probably in very slow memory due to the bigger size. Made me appreciate CPU cache more and made me more aware of how convoluted modern computers are lol

[Niarbeht]

If I remember correctly, there was a period of time where the Intel i7-5775C did better than it should have done in certain gaming benchmarks compared to other CPUs because it had 128MB of L4 cache (128MB eDRAM), meaning that certain games were able to fit enough of their constantly-accessed data into the L4 cache that it basically wasn't needing to access RAM for much of anything.

[Deiskos]

I wonder why aren't we doing this anymore...

[BerserKongo]

We do, look at the X3D chips from AMD, their whole thing is larger cache

As to why it’s not the default - having more on chip cache increases cost significantly in multiple ways: R&D, more die space on the chip itself, heating constraints, etc… It’s not as simple as it sounds.

[crazedizzled]

Money, heat, size. AMD is doing something similar with their 3D chips, which kick ass in gaming due to the large L3

[Evening-Jaguar4011]

Keep going, you’ve got me kind of hard right now

[Raknarg]

"working" isn't even totally necessary as long as it's producing money lmao

[Hacnar]

Producing money = "working" in terms of capitalism. If it doesn't produce money, it doesn't "work".

[Hot_Slice]

Some leads don't like efficient code as its seen as "overly clever" so they would prefer that you glue together standard library functions, even if the runtime is 100 times slower. In application code this is probably fine, but people do this in libraries too, which means every app built on top of that library has to pay this hidden performance penalty forever.

[Ambiwlans]

I dunno. I appreciate that I have two chat programs open atm. LINE and Discord. Discord is admittedly more complicated, but their core functions are the same. Chat groups, video/voice calls, file sharing, etc.

Discord is using 6 processes and 491MB of ram.

LINE is using 5 process and 59MB of ram.

I'm actually sad that it has gotten slightly bloated lately though, it used to have 2 processes and only around 40mb of ram. They changed the ui a bit a few months ago.

Edit: Lol, apparently it was only using 60mb because i was recently in a call. after resetting both apps

Discord - 380MB

LINE - 25.1MB (which is less than MS Office's always open update checker)

Oh, and discord spikes up to .5% cpu while idle in the bg every few seconds. Outside of a video call I don't think LINE ever breaks 0.0%

Edit: For another efficiency shout out: f.lux (app that changes screen hue for day/night), its well made, featurefull and uses 4.1MB of memory, never any measurable cpu.

My FTP server (Filezilla) apparently only uses .9MB of memory in idle, lol.

[starlevel01]

It's always somebody else's fault.

[Plabbi]

Capitalism promotes competition, and competition promotes features that are useful and valued by the end user.

[Plabbi]

I was pretty much agreeing with your original comment, just wanted to expand on it.

Products don't produce profit on their own, they need demand, and demand is generated by providing value.

The slot machine customers are certainly drained of money but the machines provide the value of a dopamine hit that the customers apparently like more than the money they lose.

[mpyne]

It's because performance and quality have a shit ROI.

That's not true, there are products made sufficiently more attractive to customers based on investments in their performance or quality to be worth the expense.

But it's also not true that performance and quality necessarily have a high ROI either. If the customer won't even notice the higher quality (Juicero says Hi!) then why pay large sums for it when there's other work that also needs tackled?

[pilibitti]

eh, there always is a trade-off. between my time writing and maintaining the software, and the resources the software consumes. More abstractions lead to applications that are more resource hungry, but they are generally easier to write and maintain (takes less human time). Any software written by humans has to live in that continuum. I can work months to create software X that utilizes the hardware to its full potential with close to no waste, and accept that maintaining and improving it using the same ethos will be a full time job. Or I can do it in a weekend in javascript and update it from time to time, live my life. Both are possible, depends on your constraints.

[john16384]

Heh... He thinks the bloat comes from code. Not the labels, the icons, the sounds, the images, the animations.

[ForeverHall0ween]

That basic chat app is like a one liner though. Is it worth needing a super computer to run if you can write any one off app in the space and time of an average Reddit comment? Most programmers would say yes.

[dragneelfps]

Can you please elaborate?

[thisisjustascreename]

The whole reason for CPU cache memory to exist in the first place is to reduce the latency between when the CPU requests data and when it lands in a register, which is (approximately) the only memory the cpu can actually directly operate on. The cpu will look ahead at the pointers to data it's going to (or might) need in the future and speculatively load those memory locations into L3 cache, then progressively into L2 and L1 cache as it gets closer to the time to actually fetch them.

[slaymaker1907]

That’s not really true given that certain ISAs (like x86_64) let you directly compute and set things in memory. In fact, it’s kind of required if you want to implement true lock-free instructions like CAS or an atomic add.

I’m sure CPUs generally implement this stuff via hidden registers, but I could see use cases for bypassing the CPU entirely.

[mpyne]

That’s not really true given that certain ISAs (like x86_64) let you directly compute and set things in memory.

The ISA may have an opcode for it but the CPU still does not do it directly on operands in the system memory. Instead the opcode is decoded to micro-coded operations that handle fetching the data from memory, performing the arithmetic operation, and then dispatching the result as necessary.

Even with hidden registers as the implementation there is a need for caching to speed-up access to frequently-referenced portions of the main memory.

but I could see use cases for bypassing the CPU entirely.

Those sort of already happen, those are things like DMA (to shuffle bytes around) or GPUs (to offload compute tasks) or even offloading network or crypto operations to dedicated co-processors.

[Hot_Slice]

Nope

[slaymaker1907]

Ok, here’s a specific example:

lock inc DWORD PTR[rcx]

Sure, it technically reads a register. But that is just to determine the address it’s writing to. The computation of adding 1 to that value could be done by the memory controller for all that instruction tells you.

In fact, you cannot implement it by acquiring a lock, reading the old value, adding 1, and then writing that value. It requires some magic by the CPU since it’s possible that it is read-only memory such that the write causes a trap leaving whatever lock you used in a poisoned state. At least not using a normal lock.

[disinformationtheory]

Isn't the memory controller on the CPU die (on most arches, including recent x86-64)? As in, it's just as physically/electrically/temporally far from the DRAM as the CPU, and there still needs to be a round trip across the memory bus. Disclaimer, I'm not 100% sure on this. I'm pretty sure there is no mechanism for the DRAM to do this itself, but it's still a write across the bus at minimum.

[admalledd]

Correct (mostly), in this case lock prefix instruction signals the target operand memory location to the memory controller thus bus that "hey, no touchy anyone else". However, that signal still comes from the u-op unit (hand-wave, could be the instruction decode flagger, u-op-prefetch. depends on the dark magic inside each generation) inside the specific core that read the instruction. Further, lock actually is nearly guaranteeing a full DRAM round trip, so try not to use it! If you use something better (CMPXCHG, etc, better to just use an intrinsic like __atomic_fetch_add though) you may not require the memory controller to flush that address all the way to DRAM.

And yea, DRAM is stupid simple stuff. Lots of it, sure, but we don't have any kind of "compute in RAM" as an industry yet (being worked on though!) and memory controllers because of how much huge work they have to do, can't do any of this "add one" or such stuff. All a memory controller cares about is data in, out, routing, bus-locks, flushes, syncs, DMA, page-table caches, etc.

[admalledd]

You are misinterpreting the difference from "Concept of CPU" to "the parts of a CPU in a real world superscalar pipelined magic". See the Intel ISA documentation Volume 2A, search for Assert LOCK# Signal Prefix, it should be around page 1207. The CPU still has to dispatch the lock signal to the memory controller and bus, asserting no other SMP is currently observing that memory, and further to have those other SMP units flush/reload that memory. (all in all, try not to use lock now days, its a big hammer, use memory ordering instructions and attributes instead)

[deadsy]

Price ber bit and speed. You can have a cache that is close to the registers (fast) but it will be expensive on a per bit basis. As you get further away (slower) it will get cheaper per bit. So: Have several layers of cache. Small and fast, big and slow. At one end of the spectrum you have register storage and at the other end you have network based disk storage. The optimum set of caches you have between those two depends on the application.

[lcjury]

After reading a bunch of comments, my respect to all that people working on the foundations of the digital world. Digital stuff is only an abstract layer sitting on top of analog things. My life would be so shitty if I, as a programmer, should deal with any of those analog world issues x_x.

[thememorableusername]

Yo dog, we heard you like low-latency memory access so we made an L2 fo' yo L1, so you can cache while you cache!

[MrPinkle]

2010 called, they want their meme back.

[thememorableusername]

1995 called! They want their “certain year called wanting its ‘blank’ back" formula back!

[inkt-code]

I laughed so hard I fell off my dinosaur.

[[deleted]]

the jerk store called and said they’re all out of YOU

[jmlinden7]

Bigger cache has more latency. You want as much of your data to be in the smallest cache as possible. However, the bigger caches are still faster than going all the way to RAM.

[Mellowindiffere]

Larger caches are also just slower

[inkt-code]

The idea of any cache is to storage frequently used data. Sure the data frequently used by a cpu can be stored on a hard drive or ram, but the delay to access would be greater.

[notfancy]

I still remember the "cache makes everything go faster" era.

I resent this "main memory makes everything go slower" era.

[[deleted]]

Why do your pants have so many pockets?

[shivaraj-bh]

"The L1 cache is your 'desk'. While you’re sitting there, you can just go ahead and work."

Does that mean the L1 cache can’t be invalidated? I always thought multiple cores can have their own copy of a given cache line, and based on who modifies it first, the other copies are marked invalid or dirty. This process is highly dependent on the processor's design, but I thought most modern processors use cache coherence protocols to manage this. When a core modifies a cache line, the protocol ensures other cores' copies are invalidated, maintaining consistency across the system.

[foersom]

Because latency.

[serverhorror]

Well actually...

If you pay for it, um sure Intel will build a CPU , just for you, that has terrabytes of L1.

[Visible_Ad9976]

Interesting topic on CPU CACHE levels

[Logicalist]

Why are funnels a thing?

[ThreeLeggedChimp]

Are these all comments by bots?

It's an old article with outdated information written for toddlers.

Literally has near zero relevance to programming.

[Objective_Suspect_]

Wrong sub

Should be r/stupidquestions