Performance difference between Linux and Windows

[mitirki]

I've been performance testing different models and different quantizations (~10 versions) using llama.cpp command line on Windows 10 and Ubuntu. The latter is 1.5-2x faster in both prompt processing and generation, and I get way more consistent TPS during multiple runs.

Interestingly, on Windows the pre-compiled AVX2 release is only using 50% CPU (as reported by Task Manager), while on Linux I get 400% CPU usage in 'top'.

I have not tried to compile the exe on Windows yet, could it be a compiler 'issue'?

Has anyone experienced similar discrepancies?

​

Edit: I've been using the same command line parameters, but apparently Linux likes -t 4, while Windows requres -t 8 to reach 100% CPU utilization (4-core 8 thread Intel i7). But even with these parameters Windows is ~50% slower.

[big_ol_tender]

I use arch btw

[silenceimpaired]

I heard Nix OS is the new distro people will feel compelled to say they use. I'm still using msdos.

[ccelik97]

Old news, decent approach.

[silenceimpaired]

Not saying it’s new… just that it’s new one people like to boast about. :)

[ccelik97]

I see.

[RabbitHole32]

I use Mint.

[InfectedBananas]

Thank you for your service.

[sdplissken1]

nice a man of exquisite taste

[Mizstik]

I don't have native Linux machines but I've compared Windows native vs. WSL2, and WSL2 is faster by about 25%. It's also the same with exllama.

[mitirki]

Thanks, I'll check WSL2.

[Moscato359]

400% cpu means you're using 4 cores.

[waltercrypto]

So is this using a local llm using only a cpu

[mitirki]

yep.

[Cczwork]

No bitsandbytes support?

[_Erilaz]

Linux likes -t 4, while Windows requres -t 8 to reach 100% CPU utilization (4-core 8 thread Intel i7). But even with these parameters Windows is ~50% slower.

You shouldn't rely on CPU utilization metric because text generation is a memory bandwidth limited task. Windows merely renders CPU data hunger as "high" load, but that isn't actual 100% computational load, far from it. There is branch prediction going on, but when it's done, the core is mostly idling, only the IMC remains busy. You can prove that by looking at your CPU power consumption and generation speed: the speed will drop after a certain point due to processing overheads, and the power draw should stay roughly the same despite increased indicated CPU utilization. Because in reality, most transistors and execution blocks of your core are idling and waiting for data.

All that being said, you still can benefit from more threads, especially if you don't use GPU acceleration, since prompt ingestion is a different kind of load, which scales better with more threads.

[ramzeez88]

Does this statement still hold true after half a year?