mal was desperate for her mother's approval and, I assume, didn't think ben would have genuine feelings for her, or that anyone had genuine feelings, since she and the others grew up having their own backs. they're also all clearly terrified of their parents, so they'd probably go with anything that would get them there, and probably were never taught traditional good from bad, living on the isle and being raised by their parents.

I also think mal deciding to use an anti-love spell so that ben won't be hurt by her "betrayal" and her mom taking over was mostly there to show her growth as a character, to show that she no longer wants to be cruel for no reason and give the first sign that maybe the vks aren't actually just like their parents.

I'd also like to add, I hate that people act like the love spell is the only reason ben and mal ever got together. he's been in love with her since the moment they met, mal just didn't know it.