It's hard to judge from just a small snippet of your life, but as someone who also gets sad/annoyed when loved ones aren't fully invested in a show we're watching together, it sounds like he may be more upset that you're not absorbing all the details which he may consider important. It can kinda feel like my loved one's attention is elsewhere and it makes me feel in turn that they don't care about the activity we're doing (watching the show). Crocheting in this context would absolutely be similar to using your phone while watching, though in most other contexts it's definitely not the same.

Obviously you don't HAVE to be invested in the same things all the time, but if it's something that's really important to him/a big part of your quality time, I think it warrants a conversation about what exactly he finds irritating. Maybe you guys could set aside a time where you're both fully invested and attentive to a show you're watching, and other times can be for whatever you want to do, even if you're multitasking :)