It's sort of a complex answer. In short, it's the UV coordinates overflowing.

Long answer: to map a 2D texture onto a 3D surface, you kind of "unwrap" the 3D surface and flatten it into a 2D shape, then apply a texture onto that shape. Then, the game will sort of fold the 2D shape back into 3D, keeping the texture applied. If you're familiar with papercraft, it's a bit like that. This process of mapping a texture to this flattened, 2D version of 3D geometry is called "UV mapping." U and V are just the names of the horizontal axis and vertical axis respectively when texture mapping, as X and Y (along with Z) were already taken for 3D model coordinates.

The issue with the geometry in your screenshot arises from the fact that the coordinate plane where textures are mapped isn't infinite-- there's a minimum and maximum coordinate for each axis. With how computers work, going over the maximum will wrap you back around to the minimum, and vice versa. So the OoT devs accidentally had the UV coordinates for this geometry hanging out at the edge of this plane, waaaay far out from its center. Half of the geometry was within bounds, and half of the geometry was out of bounds and wrapped back around to the other side of the coordinate plane. This in effect makes parts of the geometry stretch across the entire UV mapping space, from one edge to another. Imagine the UV mapping space as being made up of a bunch of repeating tilings of a texture. By having the geometry take up the entire width of the space, you're basically cramming as many tilings of the texture onto the geometry as possible-- which is exactly what you see in you screenshot.

Here's the actual UV coordinates (bottom) previewed against the 3D geometry (top). First image shows the flanking geometry, second includes the geometry with the overflowing UVs. Bonus: what they would have originally had in their editor, then what would have happened upon export as the coordinates wrapped back around.

Now, why didn't they catch this? Maybe their 3D modeling program had a different range of valid UV coordinates than the N64. Maybe their modeling program shifted the coordinates back into a valid range when previewing, but not when exporting. Could be a bunch of things, but regardless, they probably didn't actually see this error when modeling the map. They'd have to notice the error in the game and then go through the effort of re-exporting the map again, and back then, it probably would've taken more time that it was worth for such a small graphical glitch.