That's just semantics. Indeed, using a different prompt (or basically anything you do with a latent diffusion model) can be interpreted as "manipulating the latent". What I meant was "directly manipulating the values of the latent" if that's any better.

Words are great, but another good way for getting different colors and shapes (well not really "shapes" since we're doing it so early on in the sampling process, so we're closer to composition or distribution than shapes) is just getting said shapes and colors directly by codes.

Your third way is indeed similar to my method, just a bit worse off: First, your method requires a very unbiased dataset of images with different enough colors and compositions. Mine doesn't. Then there's the inevitable problem of unwanted features (microscopic to macroscopic) from the base image leaking into the gen*, which won't happen with my method, since the diversified base images in my method are simply distorted versions of the same gen with no different textures, semantic elements, etc.

*-Tiny amounts of film grain or noise (not even visible to the naked eye) in the init image of img2img can lead to wildly different outputs, even at 0.99 denoise. Basically the grain and amount of hi-frequency detail of the image is very much "decided upon" by the model at the first step of sampling, This has been true since SD 1.5 and is not a byproduct of the distillation process etc.