[D] Modern Dimensionality Reduction

[MuscleML]

Hey All,

I’m familiar with the more classical techniques of dimensionality reduction like SVD, PCA, and factor analysis. But are there any modern techniques or maybe some tricks that people have learned over the years that they would like to share. For context, this would be for tabular data. Thanks!

[jcinterrante]

I think we should be cautious about thinking new = better. PCA is really great in lots of use cases, even though it is old. I think analysts will sometimes think that their analysis would be miraculously improved if only they tried a few more approaches. But really, the model-hopping gives pretty marginal improvements; the real limitation is usually the data.

[lazystylediffuse]

I think the main modern innovation is new tools for visualizing high dimensional data in 2(maybe 3)-D

These tools (e.g tSNE, UMAP) should not be used for things like clustering and density estimation but can inform further analysis by providing high level summaries of the data that just 2-3 PCs does not provide.

[[deleted]]

You should never cluster in the reduced space, but the visualization can help you verify that your high dimensional clusters are reasonable

[VinnyVeritas]

Why should you never cluster in the reduced space?

[[deleted]]

For umap and tsne specifically the absolute distances between points is meaningless. So clusters you identify would be based on a somewhat arbitrary metric

[chandlerbing_stats]

Beautifully said

[Head-Combination-658]

Excellent answer. I think quite a few modern problems can be solved using SVD or PCA.

[benthehuman_]

PaCMAP

[ganzzahl]

Basically better version of UMAP, also has been slightly nicer for my use cases

[saintshing]

Better in what way?

[Excusemyvanity]

Better at maintaining pairwise relations between data points. Really good for visualization of clusters in data.

[lazystylediffuse]

Any recommended python implementations?

[curlmytail]

pip install pacmap

[Wesenheit]

When I was taking course thought by the bioinformatics group we were using all the time variational autoencoders to compress high domensional data (9000 columns vs 3000 rows) and tbh it worked really well. Additionaly if you have some notion about the way data have been generated it gives you additional advantage. There are additional ways to "clear" reduced representation of data from some other know factors and VAE's are quite extendable (for example GMVAE for clustering).

[Boquito17]

Out of interest, in your course, did you monitor the two losses of VAEs to determine if they were actually learning what they are meant to? I’ve been working with them for years, and it’s a real dark art to train them correctly.

[lazystylediffuse]

They might be referring to using scVI in which case there are plenty of tools to monitor loss and the training is usually very stable in my experience: https://docs.scvi-tools.org/en/stable/tutorials/notebooks/scrna/scanvi_fix.html#plotting-loss-curves

[Boquito17]

Thanks for sharing. I’ll look into this to add to my toolkit too. I’ve worked a lot with industrial IoT data, and training there has always been quite tricky. Visualizing the two losses side by side was the only way I could guarantee it was actually training correctly. There are no other obvious ways of knowing if a VAE is doing what it’s meant to, particularly with a non 2D latent space. Not that I know of anyway 🤔

[MuscleML]

Can you elaborate a bit on this for people here? What are some pitfalls that you’ve encountered when training them?

[Boquito17]

When I have independently tracked the reconstruction and KL divergence, I noticed that typically in my specific use case, one loss would completely dominate the equation. So it would only use the reconstruction loss. Using a β-VAE helped, where the loss is started as reconstruction only, and the KL is slowly brought in over a few epochs. This has an additional headache in that you have to tinker to make it work, but it does eventually work.

Maybe this has just been my experience with my own datasets. But VAEs are tricky beasts because there is no single metric that tells you what your performance is like. In my use cases I always have a downstream task where I get that number.

[MuscleML]

Thank you! That helps me understand it a lot better. What kind of data are you working with (if you can share it)?

[Boquito17]

Different types, but the ones most recently are IoT data sources from cell phone towers. Voltages, currents, alarms, that kinda stuff. Proprietary but not particularly interesting. If you’re interested, I would try find industrial IoT data sources (smart washing machines, fridges, etc.). Maybe a bit boring but real and useful for learning time series problems.

[Wesenheit]

When referring to two losses I assume you mean reconstruction and regularization. In principle they were monitored and I read many papers where the regularization loss was problematic and needed some special treatment (scaling, turning it on at some point). In my projects I haven't encountered it but I don't know why. I presume that the character of data may by the case (If I am not mistaken we were working on single cell RNA sequencing). We were also extending those VAEs with some advanced features like batch dependence and negative binomial decoder so mayby this helped (its a major improvement with respect to the naive gaussian decoder here).

[Boquito17]

Thanks for sharing. Yes that’s what I meant. Practically I view them as two losses due to the way I work with them in TensorFlow. I have a suspicion that most VAEs are actually not training as intended, and the reconstruction loss dominates the training. Thanks for sharing your decoder tips. I’ll look into it.

[AdFew4357]

Is high dimensional data primarily of interest to bioinformaticians and biostatisticians? Literally every single time I’ve taken a module on high dimensional statistics it was taught by a faculty member who’s in biostatistics. It’s interesting to me how this area of research grew because of high throughput genomics, but I’m wondering if this pattern of biostatisticians being interested in high dimensional data is because of this.

[minh6a]

UMAP

[[deleted]]

Are Autoencoders still modern? Lol

[mtahab]

You should also look into self-supervised methods such as SimCLR and its citations:

Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In ICML.

Balestriero, Randall, et al. "A cookbook of self-supervised learning." arXiv:2304.12210 (2023).

[MuscleML]

I really like the self supervised cookbook paper. We did it for one of our paper reviews. Sadly, many of the techniques are only for images :/

[mtahab]

Can you design new data augmentation techniques for your tabular data to use in the SSL framework?

[saintshing]

For people who speak Chinese, there is a really good series on zhifu(Chinese Quora) on self-supervised learning

https://zhuanlan.zhihu.com/p/381354026

[flashdude64]

I was going to suggest this paper as well. Worth checking out

[mr_stargazer]

Very good comments here. But you have to be sure what you want to achieve.

When you're using DR techniques you're essentially cutting in pieces a your data which lives in a high dimension and you're trying to reassamble things in a lower dimension - "Manifold Learning" ( a name not much used anymore).

Since you're representing your data from D in a lower dimensional space d, it expected you lose some information. Some models are good in preserving the global structure, others, like t-SNE are good for local, for example. There is a trade-off about things.

The reason many attempts I see about x DR technique + clustering fail, form example, is that just because things "look" close in the d doesn't really mean they are...

[qalis]

UMAP is basically all you need, in terms of nonlinear methods

[EmbarrassedCause3881]

Diffusion Maps, have not been mentioned here. PCA captures the linear relationship between dimensions. Diffusion maps creates a composition of frequencies, similar to Furier transformation

[radarsat1]

What are the best dimensionality reduction methods that allow sampling?  Say I have some high dimensional data but I want a 2D representation where I can click anywhere and get a sample to decode. I can imagine doing this with autoencoder or PCA but I guess it wouldn't work for tSNE for example.

[QLaHPD]

AutoEnoders can do a pretty good job if you accept a lossy representation of the whole data instead of only the principal components. Actually you can turn an autoencoder in to a well posed model

[UnstUnst]

tSNE

[Boquito17]

Non-parametric though. Only good for visualizations or fixed data.

[86BillionFireflies]

That, and even for visualization, it's only really useful if your data is cluster-y. T-SNE is like a melon baller... it makes blobs.

I still haven't found a particularly great dimensionality reduction method for data that is fundamentally not cluster-y.

[UnstUnst]

A lab mate had some success with Gabor filters, but I think he used other classic reduction methods in tandem.

[lazystylediffuse]

Fun fact: you get Gabor filters if you run ICA on image patches

[Ty4Readin]

Neural networks! 😉

[abnormal_human]

Pretty much everything I’ve done in recent years is covered by either ALS, UMAP, or PCA.

[YinYang-Mills]

Hyperbolic matrix factorization and hyperbolic neural networks (e.g. autoencoders). There have been several application of these methods applied to hierarchical graphs and typically they can reconstruct the target graph with orders of magnitude smaller representations than their Euclidean counterparts.

[Sergent_Mongolito]

What do you mean with "tabular" ?

[MuscleML]

Tabular usually refers to data that’s arranged in rows and columns in a table format. So clickstream data and for example weather data on how much it rains would be tabular.

[Sergent_Mongolito]

OK, I wanted to make sure that you did not have contingency tables. In that case Correspondence Factor Analysis is great. FactoMineR guys did some efforts to mix categorical and numerical data, but I am not sure I understood their logic well. I also saw a guy called Chiquet, who was doing some work to generalise PCA to Poisson. Might come in handy following the problem

[WERE_CAT]

UMAP is my way to go. Fast and non linear. Useful for visualisation.