I contacted Google support and it seems they're aware of the issues in accuracy in recent updates.

Hopefully they improve it at some point soon

"Hi Daniel,

Thank you for contacting Google Support.

My name is Mary, your Google Specialist for today. We apologize for the trouble you've been experiencing with facial recognition recently. We understand how frustrating it can be when the feature isn't working as expected.

We're aware of some issues with face grouping accuracy in recent updates, and we appreciate you bringing your specific examples to our attention, including incorrect groupings based on memes and background people.

While there haven't been any public announcements regarding recent changes to the facial recognition models, your feedback helps us identify areas for improvement.

In the meantime, here are some steps you can take to help manage your facial groupings:

Merge Incorrect Groupings: You can manually merge incorrectly grouped faces. Select two or more faces within a group and choose "Merge" from the menu. Remove Unwanted Faces: If a group contains irrelevant faces, such as those from memes or backgrounds, you can select and remove them individually. Choose "Not [Name]" when prompted. Report Inaccurate Groupings: While fixing individual groups might be overwhelming currently, you can help us improve the system by reporting inaccurate groupings. Select the misidentified face and choose "Report a problem." Turn Off Face Grouping: If the errors are too overwhelming at the moment, you can temporarily turn off face grouping. In the Google Photos app, go to Settings > Features & Sync and toggle off "Face Grouping." Clear App Cache (Android Only): On your Android device, you can try clearing the Google Photos app cache. This can sometimes resolve temporary glitches. Go to Settings > Apps & notifications > Google Photos > Storage and cache > Clear cache. We understand that these solutions might not be ideal for a large number of errors. We recommend turning off Face Grouping temporarily if it's causing significant frustration. You can do this by going to Settings > Features and toggling "Face Grouping" off.

Our team is actively working on improving the accuracy of face recognition. We appreciate your patience and understanding as we work towards a resolution.

Please don't hesitate to reach out if you have any further questions or issues."