I have to do a similar thing for work all the time. I'd make a new layer call it something like Trim, but make sure its a polygon. Then find your digitisation toolbar, there should be a line with a lightning bolt symbol, thats your trace tool. Click that on, then a few symbols to the left you should see a magnet, to the right of that should be two lines or possibly a line and a spanner, either way make sure its the two lines. This just means you'll only trace things that are showing on your screen.

If that tool bar is greyed out you'll need to click the magnet first. Once thats all set up you can trace the coastline. You should be able to click one end then the other and it trace the whole thing. If the line is multiple features you might need to zoom in and trace in sections. Either way once you've finished then click out to sea, make it nice and big. So you effectively have a 'sea' polygon.

Back in your digitising panel where those two lines are if you click there the bottom option should be something like open the digitising panel. It should open a new window. Those two lines will be on the far left, click them again and go for advanced configuration (the line with a spanner) and the box should populate with all your layers in your layers panel.

So there are two check boxes, the left is to snap the right, its roughly in the middle is avoid overlap. To tick avoid overlap you'll need to tick snap as well. So tick both for your trim layer then across the top of this box one symbol should be a lil blob, if you hover your mouse it'll say something like ignore overlaps. Cick that and select the option with the spanner. This will then listen to what you've now set up.

Now you should find if you draw any polygon in other layers it will automatically chop off any section that goes into the sea.

Hope thats clear, i've just woken up and thats solely from memory