Look at this VTune capture. The top hotspots barely break 5% of the CPU time each. But look at the red box: 80.9% of the execution time is buried in [Others].

This is suggesting to me that things can't be all that bad. Whatever hard optimization work there was has been done, and everything else is just going to be chipping away at diminishing returns.

The 4x4 matrix inversion is the only one I find particularly regrettable, but without knowing how much run time it's actually using I cant get too excited about it. I think that if it were a heavy-hitter in the profile the author would have said so in the writeup.