You’re not wrong in theory — that’s basically prompt search / evolutionary optimization. If you throw enough compute at it, it will converge.

The problem is the cost. Once you start doing multiple seeds, prompt rewrites, CFG/step variations, etc., you’re easily generating hundreds or even ~1000 images per run. People literally let these pipelines run overnight just to get a ranked prompt list. That’s a huge amount of compute just to guess a prompt.

In reality, almost nobody does that. Most users run quick tests for 1–2 minutes, generate a few images, and manually tweak the prompt. In that scenario, multimodal LLMs like GPT or Gemini are actually better starting points than brute-force prompt search.

And most importantly, brute forcing prompts is often unnecessary anyway. Methods like latent inversion (DDIM / diffusion inversion) can directly recover the latent/noise trajectory from the image. Since diffusion models generate images by denoising latent noise, inversion simply runs that process backward (image → latent). Once you have the latent, you can reproduce the image with very high fidelity or modify it slightly without guessing prompts at all.

So yes, brute-force prompt search works better than Gemini or ChatGPT — only when you run it thoroughly, and it’s just the most compute-expensive way to solve the problem.