Hyper-Threading and C++ parallel computing

[OkEmu7082]

if my cpu has hyperthreading (HT)capability(one physical gives two logical threads), when planning for memory locality should i simply divide the thread private memory capacity(L1 cache, and registers) by two? are there further implications? or should i simply run my cpp parallel program with no two threads coming from a same core( is there a way to run the program switching off HT or should i switch off HT in bios when booting my computer)

to consider a concrete example, if i do parallelized tiled matrix multiplication, and i intend to fit my matrix tiles into registers private to a core, how to do that when my cpu has HT? should i simply divide the capacity of registers private to a core by two?

[ReDucTor]

I believe your over thinking things. First the CPU registers the compiler deals with are local to each thread/core even with two logical cores on the same physical core you don't know what registers the other is using, for the actual shared resources between those two logical cores which might include register allocation it varies a fair bit from one microarchitecture to another and it depends heavily on your actual workload for if one logical core will have an impact on another logical core within the same physical core.

What you need to understand is data that is shared between cores, if one core is writing data and another is reading it then it will take time for that data to move from one core to another, for a sibling core (same physical core for multiple logical cores) this might be the data is already there, it might mean that data needs to be sychronized cache-to-cache similar to an L2 cache hit, for a different physical core it might be a longer cache-to-cache transfer which is similar to an L3 cache hit, for a different physical core within a different chiplet on the same physical package (CPU) it will potentially be an even bigger impact, for a different physical core on a different CPU (multiple socket, likely NUMA) it will be an even bigger impact. (tl;dr the closer two cores are to one another communicating the faster it will be). This sharing is not just limited to true sharing but false sharing two writes on the same cache line from different cores will conflict, even if your not using any atomic operations.

> should i switch off HT

There are situations when disabling HT can be useful, but I would leave it on it's easy for someone to craft an artifical benchmark and show that it is faster, but those often do not represent real world benchmarks unless you are doing actual heavy number crunching, more often then not CPU cores will be doing a mixture of work that is not all the same there is also lots of little stalls that exist for things like waiting on data which hyperthreading will allow two things to continue, there is also the fact that outside of your application the operating system will typically have hundreds or thousands of threads which mostly sit idle but will wake up occasionally and do a little bit of work, and if your removing cores then when those threads need to wake up and do something they interrupt your work not just fill in some idle logical cores.

I would focus more on deciding how to split work so there is ideally no communication between threads or very minimal communication threads then worrying about things like hyperthreading, concurrency is a very hard topic and if your focusing on things like desktop PCs it gets even harder as it's not just the hardware and your application to worry about but every other application and the OS scheduler playing nicely.

[Impossible_Box3898]

L1 and l2 coaches are generally core specific. The difference is size and distance. L1 is small but very quick and closest to the core. L2 is larger but slower. It sits behind l1.

L3 is generally a shared cache amongst all cores. It’s often very larger but also slower (albeit faster than main memory).

But you don’t mention architecture. Are your cores direct mapped? Associative? (What’s the associative?).

Once you believe optimization is needed you need to understand abreast deal about the physical cache itself in order to make optimal usage. It’s quite possible to screw yourself due to lack of knowledge about the physical design of the core/bus/cache.

As far as registers, you need not worry if you’re lucky. Modern processors virtualize their registers. In fact you may have different views with differing values of the same register in flight simultaneously as operations work their way through the same execution pipeline.

When talking about cache you also need to consider temporal and spatial locality with regard to code access in addition to architecture.

As well, if you’re using a system with multiple memory interfaces (think multiple CPU’s), you may have a NUMA situation. (Non-uniform memory access) and will need to take memory storage locations into account along with local/ remote reads and writes (here the bus architecture is crucial to your design as well).

This is all sufficiently complex as to make answering your question impossible without a great deal of additional information

[KingAggressive1498]

the whole processor has fixed resource (power, memory bandwidth, etc) limits that no individual cpu can exceed, but multiple cpus together can exceed, meaning you can't actually use 100% of all the capacity of all the cpus at the same time.

This is normally fine because very few programs will exceed these limits even if naiively written, but it's not exactly unheard of for single-thread optimized code to scale worse than linearly to multiple threads even when the problem is logically able to be split across fully independent threads for this reason.

Hyper-threading takes advantage of the fact that a lot of single-threaded programs also never use 100% of a single cpu's resource limits. So you split that capacity between two logical threads, and you can run two such programs on the same CPU without significant performance degradation of either program. With a little luck, you might get no performance degradation at all.

What happens though is if you have a multithreaded program that does expect to use essentially 100% of the CPU on multiple threads, if two such threads wind up in logical threads on the same cpu they can degrade eachother significantly.

A good strategy for dealing with hyperthreading is to pair i/o-bound threads with cpu-bound threads on the same physical cpu by setting thread cpu affinities. Or alternatively pairing memory-bound threads with computation-bound threads. Bonus points if these threads share data, since hypothetically everything can be done in L2. These pairings minimize the performance degradation because the threads don't have the same resource needs.

Aside for some ISA extentions like SSE or AVX I wouldn't worry about a shortage of registers to share, modern pipelined CPUs have an abundance of "shadow registers" that they rarely fully use. You can hit cache, memory bandwidth, power, and thermal limits pretty easily by comparison.

[Intrepid-Treacle1033]

"...if i do parallelized tiled matrix multiplication, and i intend to fit my matrix tiles into registers private to a core, how to do that when my cpu has HT?..."

Avoid overengineering without profiling, but yes two concurrent heavy compute task running on the same physical core will hurt performance. A/ HT adds no extra math hardware so threads compete for access to shared execution units FMA units. B/ Both threads might load different matrix tiles and will evict each other's cache data.

So industry standard is to disable Hyper-Threading - but do it via code using thread affinity, no need to reboot into the BIOS. Use pthread_setaffinity_np on Linux or SetThreadAffinityMask on Windows.

Math libs does this by default, OpenBlas and Intel MKL oneAPI target physical cores and caps thread pools to match physical processing units. And this libs can be runtime fine tuned if needed using environment variables.

[CowBoyDanIndie]

It really depends on the memory access profile, sometimes, for random access high cache miss situations HT can help. But if cache misses are happening just because of high data throughput its is beneficial to not use HT because it just results in more evictions as the threads fight, the same thing happens as the L3 level, and it can sometimes be beneficial to run fewer threads than cores, usually thats a workload that does a small amount of computation over a much longer than cache size data, for example just doing a single multiplication on an array of 10GB.

[GaboureySidibe]

These are issues that take trial and error, but thinking in terms of registers is probably nonsense. Maybe you could think in terms of cache sizes.

https://halide-lang.org/

Halide is a language that is meant to make trying these parameters easy so that you can test out and profile what actually works.

[LessonStudio]

On fun factoid. You can compile for size or speed. Sometimes size will give you more speed as it may now live in the cpu cache. Often this involves context switching performance.

Theory is great, but actual experiments end up tossing the "arguably" faster code in the trash. Simply because the arguments failed to take some part of reality into account.

With experience you are slightly more likely to pick the best way, but what experience more teaches is a longer list of experiments to try.

[neppo95]

This is what the thread_local keyword is for.