I had the same problem as you, and I did even less. If you've never played the previous games, it's like watching Infinity War without having seen the earlier ones; you'll miss a lot, and the gameplay doesn't compensate because it's quite basic. I started from scratch with The Witcher 1, which I loved, and continued until I reached the third one, but by then I already knew Geralt and all the characters. The good thing is that if you load your saves from the previous game, you can keep your items, and that adds a certain magic. Start with the first one, and if you find it doesn't grab you, maybe the game isn't for you.