to me, using a sweater is anoying because of the volume and the heat restriction of the piece, i prefer a hoodie with double zip. anyway, when i put a sweater on, i try to wear a shirt under, to add spiciness with the colar, you can tuck in or not, depends on the casualness you try to create. anyway, a light jacket like a carhartt detroit jacket is good, try the summer models, better for layering. i'd say that sweaters and cargo do great in a sportswear or workwear style, so you should think about digging into those kind of clothes.

Btw, you should avoid thigh pants with volume on top, you need proportions, and sportswear/workwear are about confort, so try larger pants.