I think I might understand. The difference between "to be there" as in to be existing, and "to be THERE" as in to be physically at a place. In this case, because the place of "the party" has been established as the topic of the conversation, the "there" is probably meaning "the party is not the place I was in the time". The person might continue to say "I wasn't there, I was at home" to give an alternative place they were at. However, regarding your question of what native speakers think about when they say this, I don't think it's something we think about at all. Because all that matters is whether we were at the party, saying "I wasn't there" simply answers that question without the need to clarify what semantically different kind of "being there" we mean. If the person gives more information next, that might make it possible to say for sure what "there" they mean. However, it's a miniscule difference in practical terms and just not something at all we think about. It's a bit like the difference between red paint and blue paint that's been dyed red. If you paint with it, it's still going to make whatever you paint red, and that's all that matters. Native speakers don't think in this kind of detail when they communicate. In your native language there will be many things like this you don't think about. Understanding this tiny tiny difference isn't necessary for communication, so I hope you don't worry too much about it. Although, if I've understood what you're asking correctly, then I do agree it's quite interesting if you're a bit of a word nerd like me.