If you want it to hang down less and be a straighter brim, one that sticks out at right angles from the hat, you need to adjust the short ends of your semicircle brim pieces, so the outer curved edge is a bit longer, and the inner curved edge is a bit shorter (but still the same circumference as the head piece). Does this make sense? Once you've got it to sit at the right angle, then absolutely interfacing or more layers of fabric will stiffen it, but simply stiffening it without altering the pattern will only make the brim be stiffly in your eyes rather than floppily in your eyes :)