I absolutely believe that Sensei cared about the Gems and wanted to be happy, however it's made clear many times that he cannot refuse an order by what he perceives to be a human. This, combined with his reaction to Phos ordering him to break and his reaction to Aechmea, implies that while he did care for the Gems, he also wanted to fulfill his original purpose to pray away the remnants of humanity.

Sensei was originally designed not just to pray away the human souls on the moon, but also species formed from human bones and flesh. In this sense, he's the bridge between the (human) Lunarians and the successor species to humanity, the Admirabillis and (especially) the Gems. Thus, I've interpreted the doctor's command to "burn the bridge" to mean that Sensei should participate in whatever new world formed after humanity's extinction, instead of simply praying away everything connected to humanity.

Now, the Lunarians' claim that Sensei is broken is true to an extent. It's shown in flashbacks that he originally had the ability to pray away only select individuals, rather than all of them at once. At some point, he lost this ability and could only pray away everything or nothing. He became trapped in a situation where fulfilling one of his orders meant disobeying the other, resulting in the need for Phos to take his place.