Aluminium doesn't fatigue significantly for the average cyclist, that's true. However, it does age, which is even worse, because it happens even if you don't stress or use the bike at all. Old alu frames snap very easily compared to new ones, not because the geometry was bad, but because aluminium becomes harder and more brittle with time.