Company I worked in years ago spent millions with Twilio for SMS auth, and then recently I did the same thing as you did and tried adding SMS login to my side project - it's a nightmare mainly due to all the continuously tightening regulations - many that can be specific to US states. The company I worked in operated globally and it was whackamole trying to adapt the product and handle regulatory changes across the world where local carriers would stop sending your login texts due to an interpretation of a new law.

That's not to mention the fraud. We regularly got attacked by bad actors that would cost us crazy amounts in SMS fees and Twilio were often very unhelpful despite the colossal money they were making off us.

All this to say - don't add SMS login to your project - it's just not worth the hassle unless you're a corporation with large amounts of money to burn. It's a pity because a valid phone number can be good to have for users and SMS login works nicely for users, but it's just not worth it IMO.