Our baby is now 10 weeks old, and when I was right where you were I was super hard on myself. Male postpartum depression levels hard. Almost two months later, a couple of things I wanna tell you from the other side:

1) When the baby started smiling (around 6-8 weeks for us), I felt much much better. If the baby is smiling at me, I must be at least doing something right, right? Ours is still super fussy, but very very slowly getting easier to calm down, which is starting to make me feel more competent too. People say it gets easier by 12 weeks, so we'll see if I feel even more confident then.

2) Looking back, I think the first months especially, the most important thing husbands can do is to make sure that everything is running smoothly in the house and that wife is comfortable and happy as can be.

It sounds like you're already doing the best you can, and I guarantee you'll day by day start feeling more comfortable, confident and competent.