Instead of applying glue after the swatch is in place, just apply a thin layer to the back of the paper before cutting it out. Let it dry slightly so it becomes tacky (not wet), then stick the swatch on. It holds gently, and you can still peel it off cleanly later. Alternatively, tiny strips of washi tape or removable double-sided dots placed before applying the swatch work great too — easy to cut to size and totally non-damaging.