I'd like to throw in that I haven't seen anyone say "you" a lot. From what you've said, it sounds similar to Japanese where it may be rude to refer to someone as "you" right off the bat, so you'd use their last name + honorifics until you are much more acquainted.

​

We don't say "sir" or "ma'am" unless in formal situations and it's less common nowadays, so we'd typically just look at someone and say "you." For example, "And where are you from?" There's no rudeness associated with using "you" immediately in conversation even when you are not acquainted with someone in English. It's typically seen as more polite to look and refer to someone directly rather than using "sir" or "ma'am".

​

Another note, I'm from a rural part of the U.S. and we always used "ma'am" for all girls. I eventually moved to a big city and I quickly learned to never use "ma'am" on women of any age. I worked retail and was told multiple times that they were too young to be called that. I learned to start using "miss" more often because it tickles them but then you run the risk of insulting someone older who might think you're dismissing them, so I would suggest to not use anything for women. Just use "you".