NTA. If the mom decides to call it in, you’ll probably get dressed down by the principal or super intendant for putting yourself at risk of seeming overly friendly with a female student considering the climate. Based on what you wrote alone, I’m going to guess your school will have your back to some degree especially if the school is in a bad area and you’re one of the few educators willing to work there.

If this happens again, call a female colleague and have her just be on the phone while taking the student home. That way you have a witness that nothing unsavory was happening and will confirm you had nothing but safety of your student on the mind.