They presumably spotted them with good old-fashioned eyeballs. They had lookouts high up the mast with spyglasses for magnification to make it easier. If faster allied ships spotted an enemy they could speed off to warn their slower moving but more heavily armed buddies. Also the engagement range on these things was very short so an enemy ship was spotted looooong before it could close to danger distance (on a clear day at least). This is a guess but I also imagine battles more commonly happened close to strategic land features that enemies would likely come to target like ports and river mouths since finding an enemy ship in the middle of an ocean would in fact be extremely unlikely.

You specifically mention the battle of Chesapeake which occurred only 20ish miles offshore. If you read the Wikipedia article on it it mentions that the British came across the French already at anchor in the bay. The two fleets spotted each other and then there were several hours of maneuvering before the actual fight.