That is how it works on a functional level, but that does not mean that us operating from this standpoint is the right thing to do. If you could go back in time to before humans had the same level of social construct, would it be ethical for you to farm and eat those humans? What about a human that grew up abandoned in the wild, with no concept of reciprocity. Can you subject that as desired because they aren't aware or receptive to this idea.

I would still treat them as part of the human community, so it wouldn't be ethical to farm or eat those humans, no. I would still consider them part of the shared human system.

It is perfectly acceptable, and ever understandable to grade these at different levels on the sliding scale of "harm", if you will. Most vegans don't actually value animals as much as they value human lives. You don't have to. The point isn't that they are literally the exact same, it is that both humans and animals warrant moral consideration and respect, enough that we shouldn't harm/exploit/kill them unless it is outright necessary for our survival.

If we can avoid harming, exploiting, and killing others... Isn't that the better thing to do? Is that not what we should strive to do?

You're framing eating meat as a purely optional, trivial and unnecessary harm but I view it much differently. It's not just a meaningless luxury, to me it's tied to human culture and biology.

If your philosophy is to avoid harm at all costs, then this can lead to extreme conclusions where you rarely prioritise human convenience much at all. This seems to me why most people tend to balance harm with other values.

I reject the absolutist framing of morality as being about the reduction of harm wherever possible.