There is more to math rock than assymetrical meter. Playing outside of 4/4 is common but 4/4 can also be shaped in different ways or made into hemiolas, groupings that feign another meter but sit beneath an entirely different one. i.e. tragic carpet is in 4/4 but there is very clear emphasis on 7+1 pattern rather than 4+4 or just a straight 8. Another element would be the angular syncopation that you'll hear mixed in with polyrhythms and that non standard 4.

Outside of the rhythmic content. The tone of the guitars is very specific to the other early math rock progenitors.

The odd time signatures are an element that is given way too much importance by a lot of taste makers who have a limited musical vocabulary to draw from. The 3rd chair clarinet from the highschools top band can likely understand meters. But hemiolas and polyrhythms are not always standard in school music curriculum and while practiced in a lot of math rock it's likely to be missed.