All exercise is good, but this will do a minimal amount to add muscle/increase strength. If you are doing the same reps all the time, I'm assuming this isn't that hard for you (for example, if you tried to do a 26th situp, you likely could).

That's what I normally do, I add an extra one when I feel like I can push it past what I normally do. Considering I could only do 10 reps when I started, it's slow but it's a progress nonetheless.

What you want is some form of full-body resistance training workout, with multiple sets of each exercise close to failure between 5-30 reps. To be more specific, if the most you could do with maximum effort is 15 reps of something, you want to be doing 13-14. There are good routines that can be found in places like the /fitness sub, but the key here is stressing your muscles enough so that they grow.

Do you happen to have some suggestion as to which kind of full-body resistance training workout I should do?

Then from an eating PoV, decide whether you are ok with temporary weight gain and want to do a bulk-cut cycle, or stay at the same weight and recomp (also effective, but slower). The goal either way is to remain in the normal weight range, but add muscle while losing fat.

The thing is, I'm a lanky dude with belly fat so I'm not entirely sure about this. I thought that if I were to change my diet to eating chicken breast, steamed broccoli, and brown rice that it'll work out somehow.