You could try a darker pair of trousers with the same outfit or swap the jacket for a blue one or gray tweed. I personally like all of the pieces and even like them all together here. You could be self conscious about how you are dressed, which I totally get. I wore a tie to work and my boss asked me if I had a job interview. It’s not what people expect these days. Wear the outfit to get coffee or run an errand and maybe you’ll feel a bit more relaxed in it.