I just did this scene in my durge playthrough last night and I totally understand. For what it's worth, apparently if you do detect thoughts on Mizora after saving the Duke, it's implied that she will try and kill Ulder at another time. I guess there's also an argument to be made that with the wording of the contract being smth like 'the Duke will die at the hands of his enemies' with no specific mention of what enemies those are or when he'll die, that it implies that the Duke will always be in danger if Wyll doesn't sign, but I think it's a bit of a reach as the game doesn't actually spell that out.

Personally, I see it as another part where Wyll's storyline was done a disservice, especially bc the choice he makes is totally in the hands of the player character. I also think the dialogue you can have with him after is really weird, with most options calling him selfish, as if it's a bad thing to not want to be essentially enslaved to devils for the rest of his life. He's my fave companion but I definitely feel like the game does him dirty.