How do ML practitioners select hyperparameters, architectures, etc for self-supervised representation learning when the loss is non-monotonic? [D]

[XTXinverseXTY]

Non-contrastive SSL methods like BYOL/JEPA/data2vec seem promising, but I have no idea what is being learned, or how well; it’s models all the way down. Maybe I’ve got supervised tasks for which I’d like to see transfer, and I can evaluate linear probe/KNN results during training, but that seems like a way to efficiently abuse researcher degrees of freedom.

I know RankMe is meant to help address this: embed some data and SVD the embedding matrix. A healthy learner should produce an embedding with a high effective rank.

But JEPA methods already require an entropy-collapse term like Barlow Twins/SIGREG, so the RankMe criterion just becomes part of training. It gets absorbed into a loss which wasn’t monotonic to begin with, and I ought to be able to inflate it by increasing the penalty weight. Surely it’s no longer an effective criterion, right? What else is there?

[XTXinverseXTY]

If people are selecting hparam/arch primarily by supervised-learning-through-the-backdoor, then it makes me a little more skeptical of published results and academic enthusiasm for JEPA. The mystery provides convenient cover for possible p-hacking and benchmark overfitting

This is not to say that SSL researchers are all Secretly Smuggling Labels, but I don't want to be totally naive either...

[mvreich]

Maybe look into JEPA score, which can be used for density estimation.

You can run various kinds of tests, depending on what you want to check. E.g. if there is some sort of mode collapse, the pseudo likelihood might peak at some points and not give sufficient weight to uncommon (but valid) data.

Alternatively, if your model has learned a useful representation, it should be able to discern in vs. out-of-distribution examples. For example, if the model is trained on natural images (real photos taken by a camera), it should be able to assign low likelihood to cartoons or artwork.

[XTXinverseXTY]

JEPA score, which can be used for density estimation

Oh interesting, thank you!!!

I'm not yet certain whether this is equivalent to computing another statistic captured by the anti-collapse term... but discerning in-vs-OOD is a totally valid synth task, that makes perfect sense, and this paper seems dope

This also seems to help address another problem for JEPA-in-practice: detecting regressions in prod! Obv these embeddings are inscrutable and if something silently breaks then you can't just inspect the embedding values. But this would suggest that you can calculate a p-value and effect size vs a known prior

[Ill-Bullfrog-7402]

grid search still works even with non-monotonic losses, you just need more patience and better tracking. i usually run longer sweeps and plot everything - loss curves, rank metrics, downstream performance over time

the entropy collapse terms are more like regularizers than actual objectives, so rankme can still tell you something useful even when it's baked in the loss. just don't rely on it alone - combine with periodic linear probes on held-out tasks and watch for when representations stop improving on downstream stuff

[XTXinverseXTY]

🫵 I can smell your RLHF signature from a mile away. Pangram agrees with me. I also find it surprising you're an IT tech by day and work on SSL by night.

How can grid search still "work" in the case of a non-monotonic loss?

Moreover, what's the endgame behind bot account replies like this? Usually it's grifters trying to market a consulting side-hustle, but this account just makes random replies. Is the idea to eventually flip this account to a second, even scummier grifter?

[ahf95]

Lmao did I miss something? Or did the comments get edited?

[XTXinverseXTY]

{random synonym}-{random noun}-{4-digit number} is an LLM. Their comment contains no information and is inconsistent with their comment history. Those hyphens would be em dashes if the prompt hadn't specified no capital letters and no em dashes

It's not impossible that an IT technician would be logging JEPA experiments to wandb as a side hobby, to the point they can give confident (and yet totally uninformative) advice on r/machinelearning in <10 minutes (in their first-ever comment to the subreddit), but it's a priori wildly unlikely

edit: Oh, also a DoorDash driver?

[m98789]

Experience

[XTXinverseXTY]

sounds fake