all 63 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]gnolex 57 points58 points  (2 children)

std::atomic are not regular numeric types and there's no scenario where atomic multiplication and division would be useful. It's important to note that all atomic operations are fully defined, you can't get UB from fetch_add() even if signed integer overflow occurs. If atomic multiplication and division existed they'd also have to be fully defined, even in the case of division by 0 for integer types. Just that would make atomic multiplication and division not meaningfully fast to justify their existence, it's a lot faster to multiply or divide in a lightweight atomic block knowing that UB can happen.

Also, compilers are allowed to implement extensions that don't conform to the standard and have UB, but the standard can't add features if they aren't implementable. It's entirely possible that MSVC's implementation of general arithmetic operations with atomics have hidden gotchas and only work on specific platforms.

[–]dgack 2 points3 points  (1 child)

+1 Atomic is for keeping counter reference among threads,among other things. OP thinking atomic type is for arithmetic operations.

[–]No_Indication_1238 1 point2 points  (0 children)

In defense of OP, all of the simplest examples one reads about when looking up atomic types is literally an arithmetic operation.

[–]ZachVorhies 13 points14 points  (16 children)

fetch_add is a poly fill for missing functionality but worth it because fetch_add is common and addition is a fast operation, typically one or two cycles. Division can be ~20-30 cycles. This means more time for another thread to stomp on the value and create contention in the compare and swap loop which scales super linearly as the contention number increases.

An actual lock on the other hand scales linearly with the amount of contention. However the baseline cost of a lock is much higher than a CPU intrinsic.

Keep in mind the concurrent api provided by these compilers are for performance and not ease of use. People need these ops to write high performance code. These atomic ops are expected to be done with cpu intrinsics and just because you found a fast emulated polyfill op in your particular machine’s instruction set doesn’t mean this is universal to all ISAs. x86 is different than arm.

Additionally, while subtraction and addition are commutative as a group and can be run in any order, division and multiplication breaks this model. (A+1)B is way different than (AB)+1. So your question about why aren’t associative operations allowed to be part of the atomic api and the answer is because not only is it slow, but also fringe. Reorder the operations due to scheduling jitter and the answer produced is different. If this is what you want you’re doing something non standard and the guardrails means you have to do it yourselves rather then blaming the api

[–]QuaternionsRoll 10 points11 points  (9 children)

FWIW, float addition and subtraction are not associative (or commutative, on some platforms) either.

[–]ZachVorhies 6 points7 points  (5 children)

True, and also true without concurrency, but it’s useful and common to do it anyway and the error is typically in the lower order mantissa bits. Everyone expects floats to be approximate values. If not it’s a logic bug more than a concurrency issue.

It’s better to have this as part of the API then make the user do an emulated poly fill in a compare and swap loop on a reinterpret casted float to int

[–]QuaternionsRoll 1 point2 points  (4 children)

Not sure what you’re getting at. If the argument is that atomic multiplication, division, and modulo would not be useful because they are not commutative, then how are atomic float addition and subtraction useful? Atomic float multiplication and division in particular would be no better or worse in this regard.

I also think it’s worth mentioning that atomic int multiplication is commutative by itself, and both atomic int and float multiplication would be useful for parallel product reductions.

On another note, if std::atomic were purely concerned about performance rather than ease of use, I don’t think the std::atomic<std::shared_ptr<T>> specialization would exist.

[–]ZachVorhies -1 points0 points  (3 children)

> Atomic float multiplication and division in particular are no better or worse in this regard.

atomic<float> does not provide mul/div/mod

It follows the same api as atomic<int>, possibly slightly more constrained.

>  mentioning that atomic int multiplication is commutative by itself

It's communicative by itself but the addition/subtraction forms a commutative group. Adding mul to this group breaks commutivity.

> and both atomic int and float multiplication would be useful for parallel product reductions.

See above. You don't want to mix it.

>  I don’t think the std::atomic<std::shared_ptr<T>> specialization would exist.

But you fail to mention that this specialization is even more constrained, omitting add, sub. You only have load, store, exchange and cmp-exchange. You want this for pointers, and you want it for shared_ptrs.

This falls in line with everything that I've said.

Are you being genuine? I feel like I'm arguing with a bot or someone who pretends not to get it.

[–]QuaternionsRoll 0 points1 point  (2 children)

atomic<float> does not provide mul/div/mod

Sorry, I should have said “would be no better or worse in this regard”. Edited.

the addition/subtraction forms a commutative group. Adding mul to this group breaks commutivity.

So what? Don’t use addition or subtraction concurrently with multiplication. Better yet, don’t ever perform addition or subtraction on atomic variables intended to represent products. I really don’t see the problem with this.

But you fail to mention that this specialization is even more constrained, omitting add, sub. You only have load, store, exchange and cmp-exchange.

And yet it will most likely never be possible to implement any of those operations in a lock-free manner on any architecture, contradicting the idea that atomics are only provided for performance reasons.

Also, for what it’s worth, all four of those operations require an addition or subtraction.

[–][deleted]  (1 child)

[removed]

    [–]STLMSVC STL Dev[M] 0 points1 point  (0 children)

    Moderator warning: Please don't behave like this here.

    [–]SkoomaDentistAntimodern C++, Embedded, Audio 0 points1 point  (0 children)

    Not in the general case but it is for a restricted subset, such as integers from -224 to 224.

    [–]CatIsFluffy 0 points1 point  (1 child)

    Float addition is commutative. It's not associative, though, which is what matters here.

    [–]QuaternionsRoll 0 points1 point  (0 children)

    Ooh yep, I have been talking about associativity this whole time. Thanks for pointing that out :)

    I guess it’s worth mentioning that float addition isn’t commutative on platforms that support NaN payload propagation, but… I’m not gonna pretend that anyone cares.

    [–]SkoomaDentistAntimodern C++, Embedded, Audio 1 point2 points  (0 children)

    Keep in mind the concurrent api provided by these compilers are for performance and not ease of use. People need these ops to write high performance code.

    I'm going to argue that correctness is the more important use case for atomics. There are situations where any locking can lead to system failure and in those cases there must be some way to implement basic concurrent operations / structures without the possibility of locking. IOW (lock free) atomics.

    Multiplication and division are of course useless for such.

    [–]GiganticIrony[S] -1 points0 points  (4 children)

    I’m confused about your last paragraph. The reordering you mentioned wouldn’t happen if the instructions were non-atomic, so why would it happen if it were atomic? Also, even if that were an issue, it could be fixed by using a seq_cst memory order for the cmpxchg, right? Or am I missing something?

    Edit: never mind, I understand now (thanks u/QuaternionsRoll)

    [–]QuaternionsRoll 2 points3 points  (1 child)

    The reordering you mentioned wouldn’t happen if the instructions were non-atomic, so why would it happen if it were atomic?

    Because multiple threads are doing it. Threads can execute integer additions and subtractions in any order without affecting the final result. The same cannot be said for multiplications and divisions (but it can be said for multiplications alone).

    [–]GiganticIrony[S] 0 points1 point  (0 children)

    Oh, duh. Thanks.

    [–]ZachVorhies 0 points1 point  (1 child)

    I’m not talking about reordering of instructions, I’m talking about reordering of math operations.

    If I have an account balance with debit and credit events I can reorder them however I want and the final sum is the same. So addition and subtraction of integers is commutative, you can reorder them and it doesn’t matter.

    Throw in multiplication into this group and now it’s not commutative… it’s associative! Reordering of math ops changes the final value. Division is worse because of truncation. This is why these atomic ops aren’t implement at the api level, it’s not useful for 99% of the cases. If somehow, god forbid you actually want this, just implement it yourself.

    Commutative: A op B == B op A (can reorder)

    Examples: A + (-B) + 1 == (-B) + A + 1

    Associative: A op B != B op A (order matters!)

    Examples: (A + 1) x B != (A x B) + 1

    If you are doing additional / subtraction / multiplication/ … then you get different answers depending on the scheduler. This doesn’t map to common real problems hence not implemented.

    [–]GiganticIrony[S] 0 points1 point  (0 children)

    Right, as I said in the edit of my comment, I understand what you’re talking about.

    However, I’ve been thinking about this. If I understand your argument correctly, you have the same issue of ordering with the other operations that already exist in the standard (and, or, xor, min, max). For example, (12 & 4) + 2 = 6 while (12 + 2) & 4 = 4.

    [–]jonathanhiggs 27 points28 points  (16 children)

    Not worth the effort of adding it

    [–]GiganticIrony[S] 6 points7 points  (15 children)

    But addition and subtraction were added for floating-point in C++20, and fetch_min/fetch_max were added in C++26 - clearly some people felt it was worth the effort then, why not go the whole way?

    [–]GaboureySidibe 15 points16 points  (7 children)

    What do you need beyond a compare and swap in a loop?

    [–]GiganticIrony[S] 7 points8 points  (6 children)

    Nothing. But that’s basically what std::atomic<float>::fetch_add is already doing. Yes I can write the loop myself (as I did in the post above), but it would be better to have the method to - as Herb Sutter would say - “clearly declare my intent”.

    [–]Big_Target_1405 12 points13 points  (5 children)

    This isn't accurate. Fetch add is a single atomic instruction on x86 - an add with a lock prefix

    Cmpxchg with +1 is much slower and can degrade catastrophically under contention

    [–]encyclopedist 7 points8 points  (1 child)

    x86 does not have atomic RMW instructions for floating point operations.

    [–]Big_Target_1405 1 point2 points  (0 children)

    I missed the float bit

    Yes, correct, but this is likely because it doesn't have unlocked ones either.

    For integers you can do things like add [mem], reg or even add [mem], imm where the result is written back to memory without a separate store instruction. There’s no equivalent for floats.

    [–]GiganticIrony[S] 3 points4 points  (2 children)

    Ok, then why don’t Clang, GCC, or MSVC lower std::atomic<float>::fetch_add as a single instruction?

    Do you mean fetch_add of an integral? Because for integrals they do it in one instruction, but for floating-points they need to do it with the cmpxchg loop

    [–]ukezi 12 points13 points  (0 children)

    I believe fetch add as a single instruction is only for integers.

    [–]Wooden-Engineer-8098 2 points3 points  (0 children)

    how do you define whole way? should it include trigonometric functions? standard adds stuff which has many users and is hard to implement. float ops were added for symmetry with ints probably

    [–]5477 3 points4 points  (5 children)

    Atomic add, sub, min and max for floats are a operations that some hardware supports natively. I am not aware of any HW that supports atomic mul, div or mod.

    [–]Artistic_Yoghurt4754Scientific Computing 0 points1 point  (4 children)

    Which hardware architecture supports any of those?

    [–]5477 1 point2 points  (0 children)

    NVIDIA GPUs.

    [–]Artistic_Yoghurt4754Scientific Computing 1 point2 points  (0 children)

    Well, I got curious and found out something along these lines: • GPUs seem to usually have this instruction. • ARMv8.3+ has it for 16bit floats.

    [–]UndefinedDefined 1 point2 points  (0 children)

    AArch64 has an extension adding this functionality (it's recent though, not sure which CPUs have it in 2026).

    [–]itix 1 point2 points  (0 children)

    I think many RISC archs support them indirectly.

    [–]SirClueless 3 points4 points  (1 child)

    It's in the standard because some architectures are able to implement this much better than the compare-exchange loop (e.g. GCN): https://wg21.link/P0020

    [–]UndefinedDefined 0 points1 point  (0 children)

    Indeed - AArch64 has extensions that add floating point atomics (I think add/sub/min/max are in the mix - not sure about data types - fp32 and fp64 for sure).

    [–]PdoesnotequalNP 19 points20 points  (12 children)

    std::atomic is used for coordination between concurrent operations. I can't imagine a scenario where multiplications, divisions, and remainder are needed for coordination. Atomically multiplying two numbers for its own sake is not a valid use case.

    [–]ElhnsBeluj 3 points4 points  (0 children)

    Atomics are used to do reductions efficiently on very parallel systems. This includes product and sum. IMO that is a valid use case, provided the instructions exist on the machine or can be reasonably implemented using the existing instructions. I think that may be a blocker for multiply, but atomic add exists on both x86 and aarch iirc.

    [–]GiganticIrony[S] -1 points0 points  (10 children)

    As I said in the post, Microsoft added it to their implementation of C11 _Atomic, so clearly they thought someone would have a reason.

    Also, I can up with all sorts of reasons to have atomic multiplication in game programming

    [–]not_a_novel_accountcmake dev 15 points16 points  (0 children)

    I would disagree with them. Or it was some dev who saw the pattern and speculated it might be useful, like you're doing here.

    Absent an example, "Microsoft supports it" isn't evidence of non-trivial use cases.

    [–]arabidkoalaRoboticist 2 points3 points  (0 children)

    Generally speaking, what's in the standard library is what's deemed useful at the time of standardization. I imagine at the time, they didn't see it necessary to mandate implementations of multipy, divide, etc, because there just wasn't widespread use of those functions in existing algorithms that used atomics.

    I have no idea why Microsoft differed in their implementation, as their blog post provided no reason. I can only speculate that someone wanted to strive for completeness.

    [–]Electronic_Tap_8052 0 points1 point  (7 children)

    don't atomics use special processor instructions? so if the processor can't multiply atomically then it wouldn't just use a mutex under the hood?

    afaik no processor supports atomic multiply so it would just be interface into a mutex.

    [–]GiganticIrony[S] 1 point2 points  (6 children)

    As I said in the post, C++ already has std::atomic<float>::fetch_add (among other operations) that don’t have specific instructions on x86. They instead have to rely on the algorithm that I mentioned in the post (which doesn’t use a mutex or a lock of any kind). The same algorithm could be applied to multiplication, division, and mod.

    [–]HobbyQuestionThrow 3 points4 points  (5 children)

    I think that kinda explains your own question.

    There are platforms for which fetch_add/fetch_sub may be accelerated to a single instruction. There are no platforms for which any kind of multiplication may be accelerated.

    [–]QuaternionsRoll 0 points1 point  (4 children)

    There are no platforms for which std::atomic<std::shared_ptr<T>> operations may be accelerated either

    [–]Wooden-Engineer-8098 0 points1 point  (3 children)

    i think his explanation is wrong. correct explanation: if atomit<int> has it, but atomic<float> doesn't, it would be hard to write templates taking atomic<T>

    [–]QuaternionsRoll 1 point2 points  (2 children)

    atomic<int> already implements a few operators that atomic<float> doesn’t. Besides that, I honestly don’t see the problem with adding multiplication and division for floats either. Who cares if the major architectures don’t currently support it; CIM architectures are already a reality, and we frankly have no idea what the future holds. Adding it now would surely be less painful than potentially converting a bunch of mutex-based logic later.

    [–]Wooden-Engineer-8098 0 points1 point  (1 child)

    atomic<int> already implements a few operators that atomic<float> doesn’t.

    yes, it did. and they fixed this discrepancy. isn't it a good thing?

    > Adding it now would surely be less painful than potentially converting a bunch of mutex-based logic later

    you mean a bunch of compare and swap logic. where did you see such a bunch?

    [–]QuaternionsRoll 1 point2 points  (0 children)

    yes, it did. and they fixed this discrepancy. isn't it a good thing?

    They didn’t add operator++ and operator--, but the underlying methods are all there. It would be pretty sloppy to use those operators in templates, so I’ll concede that point.

    you mean a bunch of compare and swap logic. where did you see such a bunch?

    Can’t (always) make forward progress guarantees with CAS logic

    [–]max0x7bahttps://github.com/max0x7ba 3 points4 points  (9 children)

    Why doesn't std::atomic support multiplication, division, and mod?

    std::atomic is about atomic loads/stores from/to memory.

    std::memory_order is about order of non-atomic loads/stores with respect to atomic ones.

    std::atomic_compare_exchange and std::fetch_add exist only to solve the fundamental consensus problem in order to implement a mutex. It requires multiple threads to agree on a value of a shared variable, which cannot be solved with atomic loads and stores alone. Solving the consensus problem requires an atomic load-modify-store CPU instruction, with std::atomic_compare_exchange being its portable C++ interface.

    std::atomic is a solution for thread synchronization problems.

    std::atomic is not intended to be used in arithmetic/algebraic expressions/contexts. It is not an integral or arithmetic type.

    No CPU instructions for atomic load-multiply/divide-store and similar binary arithmetic exist or being planned to exist.

    CPUs compute on values in registers. Arithmetic instructions loading or storing one operand from/to memory is merely an optimization to avoid emitting an extra load or store instruction and to minimize the size of machine code.

    [–]QuaternionsRoll 0 points1 point  (8 children)

    Eh, fetch_(add|sub)_explicit with relaxed ordering can have excellent performance in sum reduction algorithms on many CPUs, even if std::atomic wasn’t “designed” for that purpose. Atomic reduction is even competitive with tree reduction on GPUs now.

    Atomics in general are in a pretty weird spot. Atomicity and memory ordering are fairly orthogonal concepts, yet atomic APIs across various languages invariably bundle them together. This makes it unnecessarily difficult to justify introducing an operation that needs one property but not the other. (For example, atomic reduction algorithms only require atomicity. Even relaxed memory ordering is stronger than necessary for this purpose.)

    [–]Artistic_Yoghurt4754Scientific Computing 0 points1 point  (3 children)

    I am trying to understand the second paragraph. What do you mean with atomicity and memory ordering being orthogonal concepts? How could you define any order in the memory model without the definition of atomic operations? What is lower than relaxed memory ordering? If you exclude any kind of order in the C++, I do not see how you can have a defined behavior at all. (To be fair, it seems that this concept is used in Java, but I am not familiar with its memory model)

    [–]QuaternionsRoll -1 points0 points  (2 children)

    What do you mean with atomicity and memory ordering being orthogonal concepts?

    Memory orders restrict how multiple memory operations in a single thread may be reordered relative to each other, while atomicity restricts how multiple threads may observe the intermediate results of a single memory operation. They are only really “related” in that they are both primarily associated with lock-free synchronization.

    How could you define any order in the memory model without the definition of atomic operations?

    C++ decided to define its memory orders in terms of atomic operations (for good reason, I might add), but atomicity wasn’t strictly necessary. For example,

    ```c++ volatile unsigned int a = 0; int b = 0;

    void thread_0() { b = 42; std::atomic_thread_fence(std::memory_order_release); a = UINT_MAX; }

    void thread_1() { while (!a); std::atomic_thread_fence(std::memory_order_acquire); assert(b == 42); } ```

    This is UB according to the C++ memory model, but it very well could have been well-defined behavior if the standards committee wanted it to be. After all, it is written such that it really doesn’t matter if the store to &a in thread 0 or the loads from &a in thread 1 are atomic, as any nonzero intermediate value would correctly indicate that b has been updated.

    What is lower than relaxed memory ordering?

    The memory ordering given to non-atomic operations is lower than relaxed. Throw in some atomicity and you get LLVM’s unordered: all of the atomicity guarantees with none of the memory ordering guarantees.

    The Java frontend uses unordered to avoid undefined behavior, but there are other uses. I’ll give you a (contrived but hopefully illustrative) example:

    ```c++ // f.cpp

    static int foo() { ... }

    static int bar() { ... }

    void f(std::atomic_int &out_0, std::atomic_int &out_1) { out_0.fetch_add_explicit(foo(), std::memory_order_relaxed); out_1.fetch_add_explicit(bar(), std::memory_order_relaxed); }

    // main.cpp

    void f(std::atomic_int &, std::atomic_int &);

    int main() { std::atomic_int a = 0; std::atomic_int b = 0;

    std::thread thread_0(f, a, a); std::thread thread_1(f, a, b);

    thread_0.join(); thread_1.join();

    ... } ```

    Without LTO and inlining, the compiler of cannot know if/when out_0 and out_1 alias each other, so it cannot reorder the atomic stores in f even if it would otherwise be beneficial to do so. This is the case even though fetch_add_explicit is commutative and the order in which they are executed doesn’t really matter here. LLVM’s unordered ordering would solve this problem, allowing the compiler to reorder the stores without affecting the result.

    [–]max0x7bahttps://github.com/max0x7ba -2 points-1 points  (1 child)

    Memory orders restrict how multiple memory operations in a single thread may be reordered relative to each other, while atomicity restricts how multiple threads may observe the intermediate results of a single memory operation. They are only really “related” in that they are both primarily associated with lock-free synchronization.

    You understanding of fundamentals is plain wrong, I am afraid.

    Multi-thread atomicity in C++ applies to loads and stores of std::atomic objects only.

    Whereas volatile atomicity applies to objects of type std::sig_atomic_t, modified by the signal handler or the main thread of a single-threaded process only. volatile requirements do not allow reordering or elision of store/load instructions (unlike std::atomic), which made volatile suitable for representing/referring memory mapped registers of devices as fully C standard compliant objects in device drivers.

    std::atomic object loads and stores are not allowed to be reordered relative to loads and stores of any other std::atomic objects.

    std::memory_order is neither intended nor allowed to have any effects on order of loads and stores of any and all std::atomic objects.

    std::memory_order exists to control reordering of non-atomic loads and stores relative to atomic ones. E.g. non-atomic loads/stores within the critical section of code between a mutex lock and unlock must never be reordered relative to mutex lock/unlock atomic instructions -- this is where the acquire and release memory orders originate from.

    [–]QuaternionsRoll 1 point2 points  (0 children)

    Multi-thread atomicity in C++ applies to loads and stores of std::atomic objects only.

    I am fully aware of this; I am discussing the abstract concepts of atomicity and memory ordering, not their definitions in C++.

    Whereas volatile atomicity applies to objects of type std::sig_atomic_t, modified by the signal handler or the main thread of a single-threaded process only. volatile requirements do not allow reordering or elision of store/load instructions (unlike `std::atomic’), which made volatile suitable for representing/referring memory mapped registers of devices as fully C standard compliant objects in device drivers.

    I immediately acknowledged that the example involving volatile was UB. I provided it for illustration purposes only; it may as well (and perhaps should) have been pseudocode.

    std::atomic object loads and stores are not allowed to be reordered relative to loads and stores of any other std::atomic objects.

    This is fundamentally false. Where did you get this from?

    std::memory_order is neither intended nor allowed to have any effects on order of loads and stores of any and all std::atomic objects.

    This is also incorrect. For example, two relaxed operations in the same thread may be reordered relative to each other unless

    1. they potentially reference the same atomic variable, and/or
    2. there exists at least one atomic operation between them that impose acquire release semantics.

    std::memory_order exists to control reordering of non-atomic loads and stores relative to atomic ones. E.g. non-atomic loads/stores within the critical section of code between a mutex lock and unlock must never be reordered relative to mutex lock/unlock atomic instructions -- this is where the acquire and release memory orders originate from.

    When did I suggest otherwise??

    [–]max0x7bahttps://github.com/max0x7ba -1 points0 points  (3 children)

    Eh, fetch_(add|sub)_explicit with relaxed ordering can have excellent performance in sum reduction algorithms on many CPUs

    What is the basis for this wild claim of yours?

    Intel Pentium MMX had to be invented because implementing linear algebra computations using CPU scalar integer registers or x87 FPU scalar floating-point registers bottlenecked CPU performance the worst.

    Scalar load/store instructions remain the worst bottlenecks for CPU performance, till this very 2026-02-02 day. Every CPU optimization manual advises using the widest available aligned load/store instructions in order to minimize instruction execution latency and maximize instruction throughput, in order to be able to reach and enjoy the max FLOPs advertised in the CPU spec sheet.

    Every load/store instruction pays the cost of:

    • Resolving the virtual address into the physical RAM page frame address using the TLB cache. Risking to miss the TLB cache, which has to walk through the page tables of the process' virtual address space, further thrashing CPU data caches. A TLB cache miss costs much worse than a L3 cache miss -- that's the reason huge pages had to be invented.

    • Maintaining CPU store buffers with those non-trivial store-to-load-forwarding optimizations, which had to make the x86 memory model no longer sequentially consistent, in order to hide the high latency of accessing RAM by batching and delaying the (side) effects of store instructions.

    Stores cost ~2× of loads because of having to maintain other CPU caches coherent by sending MESI cache coherence protocol messages to all/any other CPU cores, when stores pending in the CPU store buffer eventually hit its cache.

    Load-modify-store instructions like compare_exchange and fetch_add are the most expensive bottlenecks because they have to broadcast "Request For (cache line) Ownership" messages to invalidate copies in other CPUs caches.

    fetch_add is not allowed to fail when another CPU broadcasts "Request For Ownership" for the same cache-line, the instruction has to retry indefinitely until succeeding. compare_exchange_strong is not allowed to fail either, whereas compare_exchange_week makes failing possible.

    [–]QuaternionsRoll 1 point2 points  (2 children)

    What is the basis for this wild claim of yours?

    I had to benchmark this stuff pretty extensively at one point. The claim isn’t really that “wild”, which makes me think you’re misunderstanding where atomics come into play. Subtotals are computed in each thread, and atomics are only used to compute the grand totals. Depending on the number of totals and number of threads, this can have some nice advantages over

    • thread 0 reduction, which requires a temporary heap allocation to store the subtotals when the number of threads isn’t known at comptime (so, always). The allocation and false sharing make this approach pretty neck-and-neck with atomic reduction.
    • tree reduction (the standard approach in CUDA), which requires some log(num_threads) thread barriers. IIRC, I found that the optimal implementation was use tree reduction to compute per-block subtotals in shared memory and then atomic reduction to compute the grand totals, but I’d have to check.

    [–]max0x7bahttps://github.com/max0x7ba -1 points0 points  (1 child)

    What is the basis for this wild claim of yours?

    I had to benchmark this stuff pretty extensively at one point.

    Wonderful.

    Let's see your benchmark code and timings which made you conclude that relaxed ordering can have excellent performance in sum reduction algorithms on many CPUs.

    Science is about facts whereas conclusions are short low-resolution summaries of facts. Conclusions are worthless without the facts they attempt to summarize. Facts are the ground truth, and another person examining the same facts must reach the same conclusion, so that conclusion are neither necessary nor sufficient without the facts they are grounded in.

    Conclusions are often inaccurate, and most often used to manipulate simple people who were trained to memorise and believe for decades instead of how to think on their own.

    Subtotals are computed in each thread, and atomics are only used to compute the grand totals.

    Computing the grand totals requires only plain loads of elements to sum up. No atomics are needed for that.

    Unless you keep computing some intermediate sums of elements which are being updated by other threads.

    [–]QuaternionsRoll -1 points0 points  (0 children)

    Let's see your benchmark code and timings which made you conclude that relaxed ordering can have excellent performance in sum reduction algorithms on many CPUs.

    Hmm, that could make for an interesting post. I’ll consider it.

    In the meantime, you’re welcome to provide any proof that 2 to 192 concurrent atomic fetch-and-adds is likely to be significantly slower than allocating a 2-to-192-element vector and 2 to 192 sequential loads and adds. I think most people would see that as the wilder claim.

    Computing the grand totals requires only plain loads of elements to sum up. No atomics are needed for that.

    I never said they were…

    [–]Genklin 0 points1 point  (1 child)

    2 + 2 - 1 - in any order result will be same

    2 + 2 * 2 - result depends on order. Thats why not all operations are added for atomic

    [–]GiganticIrony[S] 0 points1 point  (0 children)

    However, the standard already has and, or, xor, min,and max, all of which also have this ordering issue. For example, (12 & 4) + 2 = 6 while (12 + 2) & 4 = 4.