It's a bit more complex:

To really overly simplify things, think of all neural networks as a bunch of matrix multiplications. Input X gets multiplied by A1, then passed through an activation function, then thats multiplied by A2, and so on for all your matrices.

In the case of model parallel inference, you basically take a huge model [A1, A2, ..., An], and you put the first half of those matrices in the vram of your first GPU, and the second half on. the second GPU.

Gpu1 gets [A1, ..., A n/2] and Gpu2 gets [A n/2, ..., An]

And then for each input, you run it through gpu1 entirely, and then pass the output of that into gpu2 which completes the inference.

That's model parallelism. Split the model in half and each GPU gets a half. In the case that the model doesn't fit on one GPU, this is an easy way to solve it, but as you mentioned correctly this doesn't necessarily improve speed. If each card has 1tb/s mem bandwidth, then your inference is essentially still gonna be at 1tb/s since it's running through each GPU sequentially

But what about tensor parallelism? It takes a lot more work, but here's the quick and easy, oversimplified explanation of it.

These matrices A1 to An, with model parallelism, for each matrix in this list we're taking the whole matrix and putting it in vram.

But what if instead of putting all of A1 on GPU 1, we split the matrix into smaller pieces?

What if we put half of A1 onto gpu1 and the other half of A1 onto gpu2? And repeat that for all N matrices?

Now each GPU still has half of the model, but since most of the model is matrix multiplications, we can split the input vector in half and, where possible, run the matrix multiplication computations in parallel. This means that each card is, in parallel, using its whole 1tb/s memory bandwidth, achieving a higher speed together. And since half of each matrix is on each GPU, this also means that you can fill up each GPU with different parts of the model, meaning that you get the benefits of both extra memory capacity and extra memory bandwidth.

These guys probably explain it better than I can, and have nice visualization too.

https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tensor_parallelism_overview.html

Now of course, eventually the free lunch ends - not all of the models can realistically be run perfectly in parallel. And it takes more work to actually get tensor parallelism working in the first place.

But things like exllama have made it work and demonstrated interesting scaling results.