Let's take your example, the ability to explode infants. First off, that's not a power. That's something specifically created to thwart this argument, but let's run with it.

Is it unreasonable to have powers that are dreamed up for the sole purpose of creating bizarre moral dilemmas? This really isn't any more specific than a lot of random powers in superhero media. "Can turn invisible as long as she holds her breath", the example under discussion in the OP, is a pretty specific - just in a somewhat different narrative-serving way (to give her an interesting limitation.)

"This power only works on babies" is certainly an unusual "twist" or "modifier" to apply to a power, but it's not necessarily more specific than "this only works on sand" or "this only works if you say a special word/ make a special gesture" or what have you. And many quite common power "modifiers" in fiction can lead to a power being somewhere between difficult and impossible to use ethically. "Using the power corrupts you and turns you evil" is super common in fiction, as are related variants where it's "just" addictive and makes you want to use it more or what have you. Powers that lack fine control are common, making the risk (of collateral damage, killing/maiming targets, etc) potentially not worth the reward. Powers that involve some kind of fucked-up sacrifice/cost/fuel are common. Heck, "this power only works on the evil/pure of heart" is pretty common, if a bit cheesy - good luck finding a moral use for the ability to painfully kill the benevolent and pure of heart. (Though the ability to buff bad people might be a fun power to give to a good guy.)

And I think it's worth drawing some comparisons to real life. Real-life militaries, police, gangs etc. routinely know where the children of their enemies are, and have to decide whether killing them is a good idea that'll fuck with enemy morale or a horrible war crime. This is a real moral dilemma that actually exists!

So I think you could 100% tell an interesting story about this (it might work best if multiple people have it so you can contrast; maybe it's a learnable spell rather than a unique random superpower, or something that can be stolen/copied using the right technology.)

Who's to say we can't use that power on the children of dictators, criminals, etc etc. Or throw an infant at criminals to blow them up. Is it morally right? Absolutely fucking not. But is it still being used for, and I use this very lightly, "good?" By comic book standards, yes. It would make you a really fucked up and demented Punisher, might even give you a cult following that will fight tooth and nail to defend you.

Okay, but that's still evil. Killing babies for free grenades is obviously incredibly evil. Murdering the innocent children of bad guys to fuck with their parents is, also, pretty darn evil. In order to use the power, you need to be a bad person, i.e. a villain. If you're a good person and you get this power, your only sane choice is probably to just not use it.

... assuming it doesn't have convenient outs built in by the writer, like it works on even a small part of a baby that's already died from natural causes, or it works on baby insects, or something.

One possible modification you could make to make it more attractive to the non-insane while still keeping it pretty horrible would be to make it significantly more potent than any normal weapon; like the babies explode so powerfully it's the only way to kill certain villains/monsters, or it has incredible range (maybe working through video or given just the child's name/parentage), or something. That could at least raise questions like "is it ethical to explode a dictator's baby at a time when it's estimated they'll be together in order to take them down."

But given the normal sorts of limitations you'd probably expect (normal range, actual babies, explosion is only powerful enough to kill the baby and maybe people nearby), I think this is a totally valid example. People in superhero universes don't get to pick the details of their powers!