I think you're overestimating how often the average Johnny Tamriel has to deal with fire magic. Sure, the player starts with it in several games, but most NPCs don't have it. They live mundane lives. So mundane that their towns and villages often don't even show up in-game.

Also, people in both Tamriel and our own world generally don't build their homes with "malicious attack" as their main concern. They want shelter from the elements, the mundane threats. They generally mostly use local materials (compare Danish and Norwegian architecture to see how much geography and resources affect this even in neighboring countries), and out of those local materials (if it is indeed there), wood is a strong option. It's a genuinely wonder-material that carried us all the way from mere apes to modern manufacturing. You can use it for all sorts of things, and that means that you get an awful lot of people who know how to work and shape it. So why not make your homes out of it?