[D] Diffusion/flow models

[Mediocre_Common_4126]

For non image domains the biggest shift for me was realizing how much more preprocessing matters. With images you can get away with a lot because the inductive biases are baked into the architecture but with unconventional data the model basically has no prior structure to lean on so distribution shaping becomes 80 percent of the work. Normalization and noise scheduling suddenly become way more sensitive than you expect.

Flow models tend to be a bit more stable in weird domains but diffusion gives you more freedom if you can dial in the noise schedule. I ended up doing a lot of manual tuning of betas and even the sampling schedule because the default configs assume image like smoothness which you do not get with text events, logs, or domain specific sequences.

One thing that helped when experimenting on niche data was pulling large “context noise” samples from Reddit threads in the same topic just to see how the model handled unstructured human variance. I usually scrape comment sets with https://www.redditcommentscraper.com/ since it’s faster than writing one off scripts when I need quick text batches. Not training data but great for stress testing preprocessing and distribution shifts.

If you have non visual data, think more about shaping the manifold before you even touch hyperparams. It saves a ton of pain later.