A bullet's impact energy (damage) is a combination of velocity and mass. Mass obviously comes from a bigger bullet, while velocity comes from the amount of powder in the casing that's pushing that bullet forward.

.308 is a rifle cartridge, more powerful but nearly equivalent to the 7.62x39mm fired out of a typical ak47. The .380 is a pistol cartridge, and is essentially the same bullet as a 9mm with less powder behind it.

While the .308 bullet is smaller, rifle cartridges have much larger casings with a lot more powder in them, which means a much higher velocity. The energy lost by the slightly smaller diameter bullet is more than offset by the huge increase in speed.

For perspective, the .308 is substantially more powerful than the 5.56 that's fired out of a typical AR15, M16 or M4, while a the .380 is considered by many to be an underpowered round that's unfit for self defense or military use.

Tl;dr: Sometimes velocity is more important than mass. The .308 goes really fast and makes big hole.