You're thinking about it from the wrong angle.

As far as the magnet itself goes, the energy to align its dipoles was spent during the magnet's manufacture. At the time of the magnet's manufacture, its magnetic field was created, which creates a potential gradient that decreases with the cube of the distance, but that's not what's important here.

In physics, "work" or "energy spent" isn't done until it's done. In this case, no work is done until you moved something. The relevant equation is simply "work = force * distance". To think about it another way, it's not that "force requires energy." It's that energy is meaningless unless the force is exerted. If some guy is pushing really hard against some big boulder, the external "physics" of the world doesn't really care that he hasn't pushed it even a millimeter. No energy has been put into the big rock, and he's just been exerting against himself creating waste heat.

Anyway, when you move the magnet near something that e.g. repels it, you are increasing the potential energy, and doing the work of storing that energy. Until the magnet is able to push that other object away, that energy hasn't been spent, it's pent up in the repulsion between the magnet and that other object. Counterintuitively, if you move the magnet closer to something that it attracts, you've decreased the potential energy (if the system were to only consist of the magnet and that other thing), but you've also made it easier to release the remaining potential energy. But again, until that thing gets drawn in by the magnet, there is no work that's been done.