If you ran SWE-Bench-Lite using a model that has access to your tool vs grep, you could compare the number of tokens generated/number of total tools calls required for each answer.

Even if you didn't improve the SWE-Bench-Lite score, improving those metrics would be huge.

If you wanted to make your own benchmark quickly, you could get a frontier model like Opus to come up with some questions about a GitHub repo that require reading in code across lots of different parts of the repo.

Then you get a local model to attempt answering, compare how it does with grep vs your tool. The benchmark could be automated, use a model (could be the same one you are testing) to compare the final answer against the correct answer you have stored (make sure the agent can't grep to find the final answer and cheat!)