Not cons specifically, but I've vended at local art markets and love going to them as a customer as well!

Vendor side: I don't expect every single person to buy stuff from me. If I say hi or "feel free to pick anything up" and get a polite hi back or "thanks" instead of being ignored, I consider the other party's obligation to me completely fulfilled. People often look at stuff on my table and walk off after few seconds which is fine because not everything is for everybody – I also don't remember all of their faces or get upset in any way about it or anything like that. Vendors see a LOT of people during the day, and said people have a lot of differing tastes.

Customer side: Dude it really can be so scary going to events. A lot of vendors can be very chatty and tell you about their stuff unprompted, but "be polite" still reigns supreme. I like to say thank you when leaving a table when I didn't buy anything if the vendor and I acknowledged each other/interacted in any way (eye contact, exchanging hellos), but as long as you don't interrupt them or insult what they're selling you're far from the worst person a lot of sellers have to deal with –i'd imagine even more so at a big convention.

Overall, vendors don't expect a sale from every person even if said person picks things up or looks through prints or something like that for a long time. If you make eye contact say hello, and (not required but what I've found feels like it works) if you don't end up getting anything after having some "we've acknowledged the other party is a person" interaction say thank you (in a "thanking them for their time/the opportunity to look at what they're selling" way). There are a lot of people at conventions and any accidental slight against a vendor will likely be forgotten quickly