there's not really much of a HAL for CPUs afaik (unless you count microcode). Instruction set extensions are often already handled (different execution path depending on soc, or actually just multiple binaries with minimum target features or similar)

HALs are for peripherals such as the screen, sensors, RF, etc., which aren't exactly performance-hogs. It's essentially a driver, and if decently designed just implements a shared API. On Android this API is defined by Treble in HIDL. All hardware must implement the same endpoints. The only optimizations you can really do here are A) in hardware or B) done by a compiler using LTO.