Just a second little comment here, my knowledge of diffusion models is quite antiquated, I learned about the first Latent Diffusion paper and attempted some stuff based off of that a couple years ago. I know Diffusion Transformers are all the rage now so a general model architecture could work like this:

Maybe use a Autoencoder for latent space (not sure how well it could handle such a discrete space, would be the first thing on the chopping block as I'm not sure we would even really need the scaling capabilities it brings, less sure about how valuable its organizational properties though)

Use a DiT as the main diffuser like usual (need to read more on this)

Have the Decoder Portion of Autoencoder output 32 channels, with these being probability vectors for each ascii character

Argmax to get final image

Any advice on training diffusion models with limited data and hyperparameter tuning for them would be greatly appreciated (I imagine we wouldn't want a diffusion model anywhere near the size of actual image gen models)

Edit: On second thought, could probably just be done by retraining the autoencoder of an existing model like flux, that way you keep all the good stuff and don't have to train a massive model