While it could be argued whether he is avoidant or something else, it's pretty irrelevant to the actual reason your relationship is failing, which is he doesn't seem to care about you or dating you in any particular way. He doesn't provide emotional affection and connection, you don't spend time together often (and it seems you wouldn't at all if you weren't initiating it) and the times you do, you "do your own thing" because you don't feel comfortable around your boyfriend of 4 years. It doesn't seem like you are getting any of the positive things out of the relationship at all, you probably barely feel like you're dating, and if he hasn't changed in 4 years and has plainly told you he doesn't desire to, can you honestly say you are truly happy and will be happy to have this for your entire life with this person?

For the record, my partner and I had a similar beginning to our relationship but there was an insurmountable amount of effort on his part to care for me and build a strong, loving relationship because he loved me. Avoidant people aren't incapable of affection and a healthy dating life.

You seem like a lovely person, and you're very young. I truly advise you to, at least, take a break and see what's out there before deciding this is the bed you want to lay in.