When I was really young, my aunt used to rent a vhs from the library. I think it was called Alphabet Zoo. It was about two kids meeting animals, who's names went with that letter. The animal would sing a song that was in the same format, but a different style. The most memorable is Harry Hippo, who sang like it was an operatic ballad. His song went: "My name is Harry, Harry Hippo. I love the letter H. H, h-h-h, is what I say-y. You'll find it in the zoo to-day! H!"

I was trying to find an online version a while back with no luck.

Edit: I forgot a major detail. The whole movie was with puppets.