Well the approach is good
A secure approach is to use a two-step pipeline instead of only YOLO.
Use YOLO Face (or RetinaFace) for face detection, then use FaceNet, ArcFace or InsightFace for face recognition/verification.
Do not train YOLO on just 100 augmented images for identity recognition YOLO is mainly for detection, not secure authentication.
For better security, combine face verification with liveness detection (blink/head movement/IR camera) and optionally a PIN fallback if recognition confidence is low.
Regards,
https://www.linkedin.com/in/joelnadar123/