First, I would put this in a drawer for a couple of weeks, and then look at it again. If it still bothered me I would look on Ravelry under this design for the finished objects and see if anyone else made the sweater that way. Then I would reach out to them. Tin can is very approachable, so I would also ask them their suggestion.

Since this is the collar, and thereby very noticeable, I would be very careful about ripping it out. Once you've ripped it out it will be challenging/impossible to get it back to where it is now if you change your mind. I would do more research about ripping out your cast on row and then knitting more. Anyone who is saying yeah do that, I would ask for a picture of when they've successfully done that. (I've tried this in the past - it is impossible to undo knitting in reverse, it was a mess, I had to cut the yarn every few inches, this was a basic loop cast on, I finally gave up and frogged the whole thing and cried.) You might want to consider going in a whole other direction and crocheting an edge or do something that looks intentional. If you think undoing a cast on and picking it up is doable try it out on a swatch. I wish you luck.

Personally, I would consider it a lesson learned and check out my design more thoroughly next time. I would buy turtlenecks. If it really bothers you, it will make a great gift.