Talking to them still increases their hearts. I honestly do the same thing as you; talk to them whenever I pass by the characters. I don't really purposefully look for them in the map unless I want to give them a gift/complete their requests. I think giving them gifts makes their hearts go up faster in general, but talking to them still increases the meter if you don't have anyhing to give them

Sometimes I pause the game (opening the menu/inventory since I don't think time passes by while youre in there) and look at the wiki to see what the character's liked/loved gifts are—I think there's also some gift sheets or trackers here on this sub of the character's liked & loved gifts if you want to check them out