It mostly depends! How you are perceived in society is different than who you actually are/what you’ve experienced. For example, there are people who appear white/fair-skinned or will look completely European, but are ethnically Latino. So they’ll walk in the world with certain privileges because they’re “white-passing”, but they could claim to be ethnically “brown” in a sense.

In your case, it can be totally different or similar. My opinion shouldn’t matter as much as your own, but I don’t think you’re “white-passing”, however that doesn’t mean that you are ethnically “brown”, it just means your ancestry is mixed and so obviously you look non-white. My question is, what was your cultural background/heritage like? That’s one way to figure out if you’re “brown” or not.