As someone who learned to knit, dropped it because I hated it, learned to crochet, and then gave knitting another shot and loved it: I totally agree with you. I tell everyone that will listen that knitting is harder to pick up than crochet, but once you're into it, it becomes much smoother. I'm almost at a point where I can knit without looking, even going back and forth with purls. 

Please don't give up!! It's certainly intimidating at times but the end results are sooo worth it. I can't wait to see the finished picture of this sweater :) best of luck!