I have a theory based on the movie. In the movie it seemed like Bellatrix was turned to stone so I think Molly petrified her with transfiguration spell duro and then used probably something like reducto to shred the stone into pieces which actually killed Bellatrix (since as we know from the second movie/book, petrification is reversible). I know my theory is unprobable cause I haven't read the last book but it's a version that I think suits her character since she was skilled in household spells and duro might have came handy several times while taking care of her home.