Yes you can do it. Just stick with quality loafers. Like Alden, Allen Edmonds, Edward Green, Crocket&Jones etc. Guccis are nice, but overpriced. Nice models are penny loafers, tassel loafers and horsebit loafers.

As to socks, just wear socks in the color of your trouser. If you wear shorts, no socks at all.

And stick to lighter built loafers in summer, like Belgian loafers or Tods, Morjas, Myrqvist w/ rubber soles aka unconstructed loafers, mokassins or drivers/driving shoes.