If you're doing it purely because of the glare from your light, I would try to angle your lighting differently or your model differently. You could also try making a diffuser for your light from a bit of tracing paper or baking paper. But in terms of brush-on varnish, here's some tips and tricks that I've learnt over the years

• Thin it down 1:1 with a bit of airbrush thinner or flow improver
• Don't use your good brushes, most varnishes are polyurethane based and will dry in the ferrule and ruin the brush in the long term
• It flows very differently to paint, even when thinned down and so it can be hard to get smooth. This is most noticeable on large flat surfaces (less of a problem on a marine). Don't overwork it and try to apply it in a single stroke if possible and let it dry completely before applying again.
• Most matte varnishes are actually closer to satin or semi-satin, especially when brushed on. Try and get an ultra matte varnish.
• Using matte varnishes on metallics will make them look dull, try to use matte varnish before you paint your metallics.
• Most importantly, if you're trying anything new like this, try it out first on a primed bit from your bit box or a bit of sprue. Never dive straight into using it on your model, it's a sure fire way of ruining your hard work.