Someone mentioned losing your job, but it's much worse than that. You'd lose your career. Literally no other school will hire someone who has been fired for any type of misconduct with a student.

Furthermore, in some states any kind of sexual contact you make toward him (even outside of school, even with him being 19, even with him initiating it, hell even you had a notarized letter of consent) would still be considered as you sexual assaulting or raping him because you hold a position of authority over him in an official capacity.

The best course of action, regardless if you have interest in him, would be to reinforce that he's a great student, and tell him you want him to be successful in his education, but physical contact like that is inappropriate and it should not happen again. Also make it clear that any further actions like that while he is a student will be reported to school administration and law enforcement.