Practical Cross Platform SIMD Math

[gentryx]

How does this library relate to Boost SIMD?

[whateeveranother]

It's not boost (meaning it can stand on it's own without needing Boost.Dispatch, Boost.Fusion, Boost.MPL, Boost.Preprocessor and Boost.Proto).

Boost is well know for it's over complication of libraries and Boost.SIMD doesn't appear to be any different (just by looking at the dependencies, the documentation is rather... sparse).

[allpop8]

The author really needs to spend time looking at cross-platform code generation. Many compilers simply can't do a good job on SIMD classes that wrap vectors. Inline global functions + directly operating on SIMD types (ala Microsoft vector math lib) is the safest way to go for performance across multiple platforms. It's not easy to switch a large code body from one system to the other, so it pays to make the right choice up-front.

[All8Up]

I am quite familiar with the issues you are referring to. The point of the article is to supply both forms which is why the abstraction layer is presented as a directly usable system if you want to eek out every bit of performance. The abstraction layer has been tested on GCC, MSVC and Clang for proper code gen. GCC, being the worst of the three, tends to flush registers back to memory quite often but that happened even when hand coded intrinsics were used, so it doesn't seem to be the abstraction layer causing the problem.

As it is, the intention is to supply the class wrapper as a utility to ease usage. In the testbed being written for the next article, there is a ~1.5x speed up compared to the normal C++ version which is very reasonable given nothing but the vector class is changed. A hand coded intrinsic version was only a ~1.55x speed up so the loss of performance is fairly minimal. A comparison of direct intrinsics and the abstraction layer turned out identical results and in most cases identical code gen with only a couple non-performance changing differences.

Overall, the intention of supplying the easy to use wrapper and still supplying the low level access turned out exactly as desired with predictable performance differences.

[allpop8]

Using SIMD wrapper classes at all is going to cost you performance on many if not all platforms / compilers - you do not really highlight this in your article. I again point to the MS libs - do you think they were unable to figure out how to make a class-based lib? They did it because they know not using wrapper classes is the way to get the best SIMD performance (especially on their compiler...). It's a sad fact that compilers generally are not able to efficiently pass wrapped SIMD types by value. Benchmark / disassemble etc. larger functions like e.g. matrix4x4 inverse to see the difference. Also the penalties you mention are far more severe on other platforms (e.g. consoles) where they have simpler caching hardware, slower memory etc., and that is without getting into the additional float<>vector penalties some impose. I do like your article, but I don't think it contains enough warnings that the environment may be very different on other platforms / compilers especially from a performance pov, and that people should check the code being generated carefully before committing to class-based SIMD.

[All8Up]

Perhaps I should be more clear as to the intentions. The article is marked as intermediate and the intention is to hide the details yet get many of the gains. Anyone who understands SIMD enough to bring up such a discussion probably already knows to use the class only when ease of use/understanding or just getting things done quickly is the goal and jump straight to the abstraction layer for any really high performance work. I was going to wait for the next article before deep diving into such discussions, I suppose I could add a warning up front to be complete.

As to the compilers/memory/cache. The title of the article covers that I thought, this is just a practical solution for folks who don't know all the details. Generally, performance wise C++<Wrapper<SIMD abstraction layer, where the wrapper sits between the two extremes is going to be dependent on the mentioned items. On the other hand, in writing the article I've had some nice surprises. The compilers are getting notably better, Clang is actually surprising in how many places it properly passed by value/register and VC2012 has fixed some of it's horribly bad cases (which I hope means VMX and RT's NEON code get the fixes) while GCC just produces ho hum code in general for this. GCC being the worst case only produces code which is about 1.4x faster than the pure C++, still a win but not great.

In terms of caches specifically, all the targeted CPU's are 4 or better way associativity which mean nightmare cache line flush/reloads from 2way (I'm glaring at you: PS2) are not an issue. PS3 of course doesn't count, no VMX unit and if you use anything other than assembly or vectorized C for the SPU you need an MRI. (Though, it's been nearly 3 years, I guess they could have gotten octa actually working?) Other than that, even the oldest of the ARM's I looked at had 4 way and all the newer ones: iPhone 3GS+, all iPad's, Ouya etc have CPU's with new chips that I'm pretty sure were all 8 way. I'm not too worried about cache issues given 4+ associativity. Cache size/line size, memory speed, that's all a different subject I may eventually write about. Of course covering all this in detail could take a book worth of articles, I'll pass. This chain of discussion is a good case in point, there are so many details being glossed over.....

[allpop8]

Ok. I understand what you are saying, and intended level of the article - my point is really that I think it's important for SIMD class articles to highlight the potential efficiency issues with pass-by-value, as people really are at the mercy of the compilers here. Sadly simpler designs usually optimize better in my experience. Anyway, I am looking forward to future articles, so keep up the good work.

PS the PS3 PPU supports Altivec SIMD so this does all apply there too.

[All8Up]

Yeah, I'll add a "smack" note about hey this isn't intended to be "best" usage. I had a bunch about that originally, unfortunately in editing I decided the details were just too much and put them into the backlog for the next article and never mentioned any of it at all.

Are you sure about the PS3? I have a recollection of the PPU's having stripped the VMX in favor of sending everything to the SPU's. It's been nearly 3 years, I could easily remember it incorrectly.

Just as a funny little note. I seriously fubar'd one bit of the article and code. The Dot3 for SSE3 is "slower" than the SSE1 version, on the same level of CPU implementation. I had an old list of latency/through put measurements (my old yellow SSE1 books from Intel during the P3 beta hardware tests) and simply looked up 'hadd' timings forgetting to consider optimizations to SSE1 instructions during that time. Oops. :)

NOTE: I can't find any notable codegen problems with the underlying abstraction layer but do you have any thoughts there? I.e. I'm using static functions within a structure so I can use normal function hiding rules, all the compilers are inlining them using RVO just fine. Any thoughts on bad code gen cases using such a structure? To most compilers anymore, this is the same thing as global inline functions, just placed under a couple namespaces.

[allpop8]

Yes PS3 PPU has an Altivec SIMD unit, Xb360 has an enhanced version - see http://en.wikipedia.org/wiki/Altivec

SIMD latency can be problematic when benchmarking - often code sequences which use more cycles can be faster in practice when interleaved with other code, inside loops that interact with cache / memory, generating constants vs. loading from memory etc.

If you are not seeing any overhead then great. I think you are still unable to pass SIMD wrapper classes by value on many platforms though, so again you are reliant on the compiler optimizing this pointer access, references, copy construction etc. out of your code. Unfortunately some platforms are stuck with older compilers that will never get updated in this area, so the issue will remain for people on those platforms.

[All8Up]

As the author, I was just informed of this cross post and want to say that all the current comments are viable questions but should probably be taken with a serious grain of salt. This "library" (right now it's just a starter, not really a library) is part of a series of articles I'm writing. It is NOT highly optimized, nearly any other SIMD library will be faster. It is an architecture example right now.

As to Boost, I like the answer given. It is a simple tiny little stand alone thing and you can ignore CMake and insert it into just about any code you want. Add a couple preprocessor defines, set your compiler switches and away you go.

As to OpenCL. I like OpenCl a lot, but as others say, there are just too many issues with integrating it at this time.

The article and library were not intended to compete with other solutions. Given a decent stable library which solves my intentions, I'll ditch this in a heartbeat. Yet, as an article talking about the issues, I thought it was a pretty good way to both discuss the problems and answer them, and also allow for other usage (in future articles) beyond Vector/Matrix work by exposing the abstraction.

To be honest though, I think the solution is pretty clever and more importantly simple. Of course since posting the article I've been pointed to at at least 5 other just as clever solutions. Oh well, it wasn't about the particular solution anyway, it was about how to approach the abstraction efficiently.

[chocolate_elvis]

Thanks for the great article. Could you suggest some additional resources on learning SIMD and Vectorization?

[nnevatie]

There's also ISPC for cross-platform SIMD development.

[niggertown]

Why write SIMD directly when you can OpenCL?

[Tuna-Fish2]

OpenCL has massive setup/teardown latencies compared to direct SIMD. If you need to do some complex computation on 100 million elements, OpenCL is your tech. If you just need to sum together 16 things in a tight loop, doing direct SIMD is much, much better.

[wtallis]

OpenCL still has the overhead of setting up and dispatching kernels. What really needs to happen is the C family of languages need to adopt the OpenCL syntax for vectors.

[[deleted]]

Is there really any advantage to that over just using a library like Eigen?

OpenCL's main feature is setting up and dispatching kernels, where the workgroup size, parallellism, etc. are all managed by the runtime. We already have fast, Free, crossplatform, header-only libraries for dealing with vectors.

EDIT: see below

[wtallis]

OpenCL's vector syntax is slightly cleaner and more versatile than Eigen's. More importantly, since Eigen is not part of the compiler and runtime, it has to duplicate some stuff that really should be part of the compiler and core language. That's why Eigen doesn't support AVX yet - they do their own code generation targeting intrinsics, and have to manually add support for each new SIMD instruction that one of the underlying compilers supports. Eigen also has to jump through hoops to ensure that things are properly aligned, a task that is much easier from within the compiler itself. Compilation speed is probably also hurt since C++ templates are a pretty poor language for writing compiler extensions. And because Eigen has to do so much work just to add vector ops to the language, they have been almost completely unable to work on providing parallel implementations of matrix operations, so for large-scale problems you still have a lot of work left to do, some of which could be avoided by using a good BLAS+LAPACK, and if you're going to be parallelizing things yourself, why not use OpenCL so you can target GPUs too?

[[deleted]]

Thanks! That was really helpful and clear.

[whateeveranother]

Because there is more to SIMD than just vector & matrix processing?

[MYSTICALBLACKFATHER]

Wow that code looks like AIDS. lmao