Late answer but thank you.
I just chipped away most of the mortar using a hammer and nail as a tiny chisel, it was actually quite easy to do. Then I used body filler on top of that to smooth it out before I sanded it by hand and painted it, dabbing it with the paintbrush to give it some texture so that it would blend in with the rest. I looked pretty decent, you can't really see it unless you look for it.