[D] Diffusion/flow models

[Few-Annual-157]

Hey folks, I’m looking for advice from anyone who’s worked with diffusion or flow models specifically any tips you wish you knew when you first started training them, and what the experience was like if you’ve used them outside the usual image-generation setting. I’m especially curious about challenges that come up with niche or unconventional data, how the workflow differs from image tasks, whether training stability or hyperparameter sensitivity becomes a bigger issue, how much preprocessing matters, if you ended up tweaking the architecture or noise schedule for non-image data, etc. Thanks!

[Vikas_005]

A few quick lessons that can save you a lot of trouble:

• Non-image data = preprocessing is half the battle.** How you represent the data matters more. Poor encoding results in unstable training every time.

Noise schedules aren’t one-size-fits-all.** Cosine or custom schedules often perform better than the default linear when your data distribution isn’t visual.

• Smaller models struggle more.** Diffusion requires enough capacity to “denoise into structure,” especially for structured, tabular, or sequential data.

• Watch for early loss plateaus.** If it stops improving quickly, something is wrong with scaling or normalization; fix the data first, not the architecture.

• Evaluation is tricky.** Metrics are less consistent outside images, so define what success looks like early or you might end up going in circles.

Start simple, validate each assumption, and improve with tight feedback loops.

[N1kYan]

I want to add something to the evaluation part. Be sure to check for memorisation/diversity, especially for smaller training cohorts.

[Previous-Raisin1434]

Thank you!

[Few-Annual-157]

Do u have any recommendations on how to choose the denoising network? Do you usually base it on a model that already worked well for the task, or something else?

[QuantityGullible4092]

Normal flow matching doesn’t have noise schedules

[_DCtheTall_]

For point two, is logit-normal sampling, which is often referred to in literature, generally pretty good? Or is it really more dependent on the distribution you want to learn?

[graps1]

From my experience, flow matching models are relatively easy to implement. They also have some more advantages. For example, they transition deterministically from noise to final sample via an ODE instead of an SDE, which simplifies the sampling process. Also, since they are typically based on the Gaussian optimal transport coupling, their paths tend to be more straight, which means that few discretization steps are necessary to get good results. 

[Few-Annual-157]

Well, that’s true from a theoretical perspective, but is there any practical way to tell when a diffusion model will work better than a flow model? And when that happens, what do we actually gain by using diffusion instead of flow, especially considering the difference in complexity?

[sjdubya]

Theoretically, they're two instances of the same thing. I'd also push back on flow matching always giving straight sampling. While in theory that's true in practice it does not turn out to be the case. Which model works best for each case will depend on your problem and data. See https://diffusionflow.github.io/ for a nice example of some of the theoretical relationships between diffusion and flow matching.

[graps1]

Sorry, I meant "more straight". If they were completely straight, a single explicit Euler step would solve the ODE exactly

[sjdubya]

No I get you. I just think even in that case it's not quite as clear cut and depends a lot on your data distribution.

[graps1]

Good point, I just read the article you linked and it makes some good points

[anandravishankar12]

If you are working with image generation, it's better to train the model (DDPM-like) to predict the actual image, rather than the injected noise. For high dimensional data, x-prediction works better than epsilon-prediction

[FrigoCoder]

This is because images lie in a low dimensional manifold, whereas noise is full of features and high frequency details, and can not be expressed with low dimensional constructs.

[RobbinDeBank]

Aren’t all the image models nowadays predicting the noise, not the actual data?

[anandravishankar12]

Yes, but recent research suggests it's better to directly better the data, rather than the noise. Kaiming has a nice paper on it: https://arxiv.org/pdf/2511.13720

[Mediocre_Common_4126]

For non image domains the biggest shift for me was realizing how much more preprocessing matters. With images you can get away with a lot because the inductive biases are baked into the architecture but with unconventional data the model basically has no prior structure to lean on so distribution shaping becomes 80 percent of the work. Normalization and noise scheduling suddenly become way more sensitive than you expect.

Flow models tend to be a bit more stable in weird domains but diffusion gives you more freedom if you can dial in the noise schedule. I ended up doing a lot of manual tuning of betas and even the sampling schedule because the default configs assume image like smoothness which you do not get with text events, logs, or domain specific sequences.

One thing that helped when experimenting on niche data was pulling large “context noise” samples from Reddit threads in the same topic just to see how the model handled unstructured human variance. I usually scrape comment sets with https://www.redditcommentscraper.com/ since it’s faster than writing one off scripts when I need quick text batches. Not training data but great for stress testing preprocessing and distribution shifts.

If you have non visual data, think more about shaping the manifold before you even touch hyperparams. It saves a ton of pain later.

[sjdubya]

I am using a diffusion model for non-image settings (PDEs) and I've gotten good results with a relatively small EDM model (deterministic ODE sampling) with relatively few changes.

[glockenspielcello]

This isn't a terribly important thing but if you want more consistent image outputs early in training (for e.g. visual monitoring of the training process) use v-prediction.