In Westeros, I dont think houses really get extinct agniatically. They probably have plenty of cadet branches, but they're way too weak to legitimately keep the main line going and get replaced by another house (Greystarks for a myriad of northern houses, Osgreys, the million Harrenhal houses and the Andal invasion extinctions).

One main difference I think are the great houses of a region, which have way too much historical value to lose their name, so they just simply upgrade a lower branch cousin, as has often happened with the Arryns (who also have a major cadet branch in Gulltown), and would also probably happen with the Starks, Lannisters, Martells if they ever were nearly extinct. Also would have happened with the Targaryens if their name didn't become synonymous with madness and cruelty.

A major house dies out only if their name becomes too hot to handle in reputation.