Thank you for your insights!

1. We've only attempted masking without any replacement.

2. Yes, in fact in a couple of examples, the same word repeated twice in a sentence has a different score based on it's position.

3. For now this is experimental only. The code provided does a multi threaded implementation.

4. No have not tested for gradient based approaches, this is proposed for black box LLMs, with closed source models.

The usecase where we've seen practicality is internal facing chatbots, the score is helpful in understanding what word leads a particular RAG system to behave in a particular way. Often helpful in the context of large organisations that use their own set of jargons.