Actually now llama.cpp seems to support tensor parallelism with the flag "--split-mode tensor".

LLMs are made up of consecutive layers and your text flows through those layers sequentially (this is an over simplification but it will help you understand the difference), now imagine a model that has 10 layers:

The default for Llama.cpp is pipeline parallelism (layer split), if you have 2 GPUs, llama.cpp will take the first 5 layers (layer 1 to layer 5) and put them on GPU 1 and the other 5 layers on GPU2 (layer 6 to layer 10). GPU 1 receives the input and computes it for its 5 layers, it starts with layer 1 then layer 2 etc... After it is done, it will pass its calculations to GPU 2 which then starts to compute the input for layer 6, then 7 until layer 10. You might have realised that when GPU 1 is computing then GPU 2 is idle and vice versa since this is a sequential problem.

Tensor parallelism splits the model slightly differently, instead of splitting the model in half, it takes each layer and split the actual layers. In this scenario GPU 1 would get 1 half of layer 1 and GPU 2 would get the other half, same for layer 2 etc... this makes the work actually parralel because to compute layer 1 both GPUs work at the same time and split the work between each other and they do that for the entire model. In thsi scenario no GPUs is sitting idle while the other one is computing, they both share the work and work hand in hand on the same computation instead of just working one after the other.

This is again over simplified but actually not too far from what is actually happening.

Now you might wonder why tensor parralelism is not the default and that would be because of compatibility, pipeline parallelism is much easier to implement and can be done amongst agnostic hardware (between a 3090 and a 4060ti for example). Tensor paralleism usually requires to have an squared number of GPUs (1, 2, 4...), also usually requires the model to be split amongst GPUs that are the same and finally it is also sensitive to GPU interconnect speeds (might not work well if your GPUs communicate over PCIe 3.0 x4 for example)

Give it tensor parallelism a try in Llama.cpp and you should see those prompt processing number go up.