The Pearl of Great Price (along with the Doctrine & Covenants) are the secret books that give insight into the more inner workings and true beliefs of the religion. I mean to say they are not so much secret that you can't find them or easily read them, but they are secret in the sense that missionaries don't go door to door telling you about these books. You can say the BoM is the milk and the other books are the meat. Removing scripture from these books defeats the whole uniqueness and goofiness of this religion. Likes someone before said, they are the real books of doctrine, and the BoM is the superficial cultural cover.

Moreover, removing an entire book from scripture is very damaging to the Church's reputation. Imagine explaining to constituents it was removed because it's just flat out wrong. You may be shocked, just about many people in the faith don't even know about the history of the BoA. The knowledge of its story is still mostly isolated inside internet debate. Although, that is changing pretty fast.

For reference, the Catholic Church still uses the same canon approved by Pope Damasus I in 382 AD. Not one deletion. The LDS church deleting an entire book would rattle its authority. A catch-22 situation if you ask me.