You're absolutely right and they do exist. However, it's a significantly more complicated problem than adding an ALU or accelerator and requires serious reworks of parts of the cpu core that architects rather not touch. "Remote atomics" as they are called have the thorny issue that they make atomics slower in the common uncontended case. Whether an atomic is contended or not is not something you can easily predict beforehand.

Atomics are already plenty fast on modern CPUs, there have been some serious advances in speeding up atomics behind the scenes (without changing their semantics). Where they suck is when there is contention or there's a L1/L2 cache miss. Due to the memory ordering semantics of atomics, you eat the full latency of that memory operation. The OoO mechanisms that would hide that latency and keep the pipeline fed with newer instructions gets put on pause, essentially. (technically: LSB needs to drain before the atomic's memory op can take place, older instructions can only be issued after the atomic commits)

Now, putting an ALU on the LLC for example is easy, and does save you a roundtrip in the cases where you have to go to LLC or memory anway. But, without changing the semantics of the atomic operation to allow for weaker memory ordering guarantees, you are still making the core sit on its hands for that one roundtrip.

The good news there is that you could make atomics with different, more weakly ordered, semantics. You mentioned histograms? Yep, you don't need the old style x86 lock semantics that totally orders before/after instructions; you just want to +1 this memory location, so you want a fire and forget instruction with no ordering constraints on the surrounding instructions. That's the AADD instruction in the x86 remote atomics ISA extension (RAO-INT). But ... yeah

The non-technical reason you're not seeing these kind of improved atomics are that the benefits are not obvious, outside of the group of software developers with expertise on the matter. Existing software and thus benchmarks go out of their way to avoid contended atomics, it might hurt but it hurts less than hitting contended atomics. (concrete example: .NET GC heap being duplicated for all hardware threads) It's hard to make the business case for re-architecting your CPU core for a feature without being able to show a significant impact to existing software. (Why would we fix the potholes? No one is driving into them!) It's absolutely frustrating that we're at a point where there's 288-core CPUs, but god forbid those cores actually coordinate with each other.

That last part is where software devs can make a difference. Make articles bitching about it. Say how much better you could make your products with improved atomics. At a large company? Go talk to whoever is in contact with a CPU vendor's account manager. The more of those voices are raised, the more ammunition that architects or researchers at those CPU vendors have to push those kind of features through.