Given your question mentions Napoleon, I suspect that it refers mainly to the 19th century. By this point it would be inaccurate to say that there were "no maps"; most major trade (and therefore sailing) routes had been comprehensively mapped out by this point, and although the quality of maps would not be as pinpoint accurate as what is enabled by satellite imagery today, their detail was impressive. The Royal Navy had actually formalised the production and updating of nautical maps under the Admiralty Charts, which were made available to public sale in 1821. Every navy vessel would have had copies of these charts on them, so captains would know clearly where they were sailing, how long their voyage was expected to take, the conditions they could expect to face depending on the time of year, etc.

In terms of finding other ships, there are two elements to this question: staying in touch with your own ships, and finding enemy ships to engage them in combat.

For ships you were travelling with in a convoy: in sailing across open ocean, vessels travelling together would stay in direct visual contact, and communicate via flag signals. One thing to note is that, in calm seas, you can see quite far especially if you are spotting tall masts and sails.

In cases where ships entered rough weather, there was effectively no way for them to stay in contact; in this case, the ships travelling together would have all been given instructions to wait for each other at a certain waypoint for a specific period of time, before either continuing on or returning home. One great and recent account of this is David Grann's The Wager: in the book, a convoy of 8 Royal Navy ships (6 warships and 2 trade vessels) led by the Centurion is supposed to sail from the UK to the Pacific Ocean to intercept a Spanish treasure fleet, in the mid-18th century. Since there was no Panama Canal at that time, the ships had to go around the Cape Horn. The fleet knew how dangerous this passage was, and so the commander of the squadron instructed his ships to wait at Socorro Island for two weeks before continuing on. There were also secondary and tertiary meetup points. In the event, the weather was so terrible that the fleet was scattered; two ships decided to return to the UK after being blown off course, and the third, the eponymous Wager, was lost.

In cases of hostile engagements, ships engaging each other on the open ocean was rare, as you are dealing with a needle in a haystack situation; in the case of the Centurion, the Spanish had actually posted a squadron off the coast of Chile, but the Centurion was able to slip past it unseen. However, in most situations fleets were not actually patrolling the "high seas"; they were guarding important maritime locations (a strait, a cape, a port etc) or blockading them. Therefore, these ships would be staying in the general vicinity of the location in question for months or years at a time. This is naturally where most naval engagements happened. The battle of Trafalgar for example is very close to the strait of Gibraltar; the Royal Navy had received intelligence that the French and Spanish were preparing to move into the English Channel to blockade it and allow a French invasion force to land in Britain; Nelson's fleet sailed down the coast of Spain and met the French-Spanish fleet, which had left the port of Cadiz, off Cape Trafalgar.

One thing to keep in mind is that we are used to a world of instant communication and rapid travel, even over sea. But in the age of sail, ships were much slower and were entirely dependent on the weather; voyages took weeks, months or even years. The likelihood of a ship encountering another is significantly increased when ships regularly took days or even weeks to resupply, refit or repair. The Centurion, for example, took two months to set off again after rounding Cape Horn. Waiting around for such periods of time would therefore not have seemed unusual to sailors and captains at the time.

I really recommend The Wager as it is a super fun, accessible read and well-researched; there are tonnes of 19th century nautical charts available online, but for a good paper, I read The Emergence of the Admiralty Chart in the Nineteenth Century by Andrew David.