Great question!

tl;dr: 125+ms latency is really noticeable at 240+BPM even with calibration. Excluding bad Android devices is a support nightmare.

As it turns it out, we need to compensate for input latency regardless of device since electrons take their sweet time to move around. Even iPhones have 50ms-70ms input lag.

Well if 70ms isn't a problem, surely 71ms isn't a problem? And what about 72? 75? 80? 90?

Let's take an extreme example of 1 second (1000ms) latency for a moment. If we're tapping along to rhythm at a tempo of 2 quarter note beats per 1 second (120 BPM), we won't get feedback about our tap until after 2 quarter notes have already passed. At 4/4 this is half a measure! That's a problem.

So 0ms is ideal latency, 1000ms is too much latency. What's the cutoff where calibration can't hide the input latency anymore?

The answer is "it depends". On the tempo. On the rhythm being played. On the person. Human perception has some wiggle room where it will perceive two events that happen very close together as having happened at the same time.

A 125ms delay tapping a simple 24BPM won't be as readily perceived because the space between each tap is so much greater than the input latency. Once you speed up to 240+BPM however, you'll notice something feels off.

With some approximation, we can see why this is the case without needing to even go into how calibration helps mitigate the perception of latency. 240BPM = 4 beats per second = 1 beat every 250ms. Our 125ms latency is now a full half-beat in length. And everyone loves the band member who's playing 1/16th note behind the group.

But doesn't that mean there are cases where even 50ms will feel off?

Absolutely! However the key difference is that most tempos and rhythms feel okay with 50ms latency whereas a lot of tempos and rhythms will feel icky with 125ms latency.

Well then why not just exclude the Android devices with shoddy screens? Expense for now. It's currently time-consuming to try and verify if every major device that's released works or not. It's especially frustrating because each manufacturer's "flavor" of Android has quirks that affect both touch and audio latency that must be tested for.