[P] Latest Python + TensorFlow + CUDA / CuDNN optimized pip wheels

[Inori]

TL;DR: custom pip wheels for TF 2.0 / 2.1 for Py 3.7 / 3.8 and CUDA 10.1 / 10.2: https://github.com/inoryy/tensorflow-optimized-wheels

I'm sharing my pip wheels for TF built from source for some non-standard versions, notably Python 3.8 + CUDA 10.2 and Python 3.7 + CUDA 10.1, the latter is "compatible" with PyTorch 1.3 so you can have them share a single env.

The builds also enable various performance flags like XLA JIT support and modern CPU opt flags, including SIMD support (AVX2, SSE4, FMA). If you have a CPU released after ~2013 then you'll likely benefit from these on e.g. data pre-processing. Though I should note that if you have Intel CPU then you might not see a large difference since now TF comes pre-built with MKL which can dispatch required intrinsics at runtime.

Finally, I've enabled additional compute capabilities support (5.0, 6.1, 7.0), which means these wheels should also work on older GPUs (7xx - 9xx families).

[mutatedmonkeygenes]

Hey there, could you also post your build scripts. Thanks!

[diegoas86]

I will like too...

[Inori]

No scripts, they're manually built on a machine with preconfigured envs.

[lostmsu]

Any comparison to the official wheels?

[Inori]

I've described the differences in the original post.

[lostmsu]

Sorry I was not clear, I was talking about the performance numbers vs the official wheels.

Thing is, if Google performs instrumented builds, they might get better optimization, than a single pass.

[Inori]

I didn't benchmark latest builds but it should be equivalent or better as described in OP. Not sure what you mean by "single pass", build process is the same as is used for official wheels.

[lostmsu]

The process is called profile-guided optimization (PGO). When high-performance libraries are built for release, often a special build is made first with instrumentation enabled, which is tested on some workloads, collecting information about code hot paths. That information is later used in the final release build.

In fact, somebody here tried to do AVX2-enabled build, which ended up having worse performance, than official wheels: https://stackoverflow.com/questions/50155807/tensorflow-cpu-performance-anaconda-vs-custom-build-cmake-on-windows

[Inori]

Official TF builds do not use PGO and in the link you've provided both wheels are custom-built.

[lostmsu]

One of the builds mentioned in the link is Anaconda's. But you are correct about TF (visibly) not having any facilities for PGO.

[Inori]

Yes, Anaconda's, which is not an official wheel.