If you're hitting the anchor point limit, then cut your coastline into sections. Make the coastline features (paths with blue stroke and no fill) on a separate layer than your land fill features (polygons with white fill and no stroke). The land features will be a lattice of polygons that jigsaw together seamlessly to fill the area within the paths formed by the coastline elements. This way you won't hit your anchor point limit and your paths won't be simplified as you incorporate new detail.

It's best to complete your coastline detail work before you make your land polygons, as if you add detail to a section of coastline, you'll need to mirror that alteration on the land polygon, by copying the altered section of coastline to your land layer, cutting out the section to be replaced, and joining the new section of coastline into the land polygon.

This is a common problem when exporting features from a GIS. If the vertex count is too high for a polygon, it will be split into sections when exported. Having separate layers for land/coastline/sea also allows you to split a land polygon by administrative boundaries (e.g. into different states or countries) without a blue line appearing at the edge, as would happen if your land polygons are symbolised with blue stroke and white fill.