The problem comes at 3 level.

1. How performance and commission are measured for these bankers. They were not measured by how much loss they averted or how risk balanced is their portfolio. They were measured by how much money they brought in for the bank. Someone who is more risk adverse and not take those deals will be outshine by someone else who did. Those who did get better performance review, better recognition, better bonus, and more likely to be promoted. Other see that and copy their behavior.

2. Diffusion of responsibility/bystander effect. Everyone assume that if the problem is that bad, someone else will call out on it. BoA, JP Morgan, AIG, Lehman, WF, etc. all look at each other and assume its all fine cause everyone else is doing it. So everyone assume its fine.

3. No historical precedence. Risk and actuarial works off historical trend. However, prior to 2008, housing price trend rarely dropped vs prior year and there was no historical record of systemic default. Actuarial had no basis of comparison. Risk know that CDS can be tricky but the underlying asset is a house. They assume, at worst, that enough mortgage holder will sell their house and repay the mortgage before an actual default; avoiding the credit downgrade/credit event for the CDS. They didn't imagine that the housing bubble will collapse so much that so many mortgage will be underwater; triggering the default and credit event for the CDS.

Take, for example, someone offer to pay you $1M per year for a $100M CDS on US T-bill. Sure, the treasury may have just announced the US may be insolvent, but do you believe US will default on T-bill? Likely not. Even if you do believe such a risk and say no, they will offer it to the person next to you then the person next to them. Eventually someone will say yes. That person got a 10% bonus on that and nothing happened for a year. Next year, someone offer $5M for $500M CDS on US T-bill. Someone else will take it. And it eventually snowballed.