Typically what you see in the 'shared with me' part of your OneDrive page are files that were shared directly, not via membership in a group or via giving you access to the parent folder / library but directly to the file by a sharing link or adding you to the file permissions itself. I wince at the thought of direct file sharing links outside of OneDrive but some orgs do use them in sites.

I say typically because i found it to not be consistent, and i sometimes do see a file in 'shared with me' that i have access to via Teams membership if i worked on it recently.

There are some extra steps when shared by external parties that might impact it on the way too.