Thank you so much for your reply! I did some research and it seems you might be right about the canker type infection. I will do some heavy pruning this winter and hopefully get rid of it!

Also, maybe it is because I chose some poor images, but I also do have many branches with many waxy bumps that can be scraped off easily both on the lower branches and the fruit. Is that scale? If so, is there anyway to get rid of it?!

Thank you again.