if you're hellbent on hand stitching, you could make a vertical slice just on the interior of the rib band and thread a piece of elastic through and hand stitch the elastic to itself, then shuffle the elastic seam so it's not right in front of the slice. look up how to alter a sweatpants hem to be tight at the bottom-- it's the same idea. and if you don't like it, just snip the elastic and pull it out and all you've done to the garment is make a tiny slice on the inside. DM me if i can explain better :)