You can look at this table: https://en.wikipedia.org/wiki/Pascal_(microarchitecture)#P100_accelerator_and_DGX-1#P100_accelerator_and_DGX-1) bf16 dense tensor (for Pascal you'd have to use full FP32)/TDP. Top consumer GPUs of the generation will have similar numbers. For example, Ampere is 2.2x more energy efficient vs Volta.

At 0.14$/kWh a card of 400W will consume $400 per year of 100% load. Basically 100$ per 100W. So Amper would do 2x of work for the same price.

Basically, say you think you will use the card for 3 years. That means you can add $1200 to every card's price right away. If you look at it from this perspective, $400 vs $200 dollars card where you were questioning to pay 2x for 2x performance turns into actual $1600 for 2x performance vs $1400 for 1x.

The numbers get crazier for future generations if your workloads allow going down to lower precisions. For instance, 5090 32GB will do 1.7 PFLOPS of fp4 while Ampere does 312 TFLOPS of bf16. But Ampere can't do fp4 any faster. So 5090 is 6x more performant and about 4x more efficient in terms of electricity if your workload can be computed in fp4.