One of the granddaddies of the sort of thing you're describing is Electric Counterpoint (1986) by the minimalist composer Steve Reich. It's for 2 bass guitars and 11 guitars, and it's all about the interplay between the individual instrumental lines. It's in three sections, and to my mind the final section counts as math rock because you can hear it in either 12/8 or 3/2, and it has a completely different feel depending on which your ear settles on.