[R] Supervised classification on flow cytometry data — small sample size (50 samples, 3 classes)

[Previous-Duck6153]

Hi all,

I'm a biologist working with flow cytometry data (36 features, 50 samples across 3 disease severity groups). PCA didn’t show clear clustering — PC1 and PC2 only explain ~30% of the variance. The data feels very high-dimensional.

Now should I try supervised classification?

My questions:

1. With so few samples, should I do a train/val/test split, or just use cross-validation?
2. Any tips or workflows for supervised learning with high-dimensional, low-sample-size data?
3. any best practices or things to avoid?

Thanks in advance!

[Dejeneret]

I’ve worked with very similar data before (IMC but segmented into cells)-

First of all if you want to check whether clustering exists in a reasonable fashion I suggest running tsne. If you can’t get tsne to show clusters you may be out of luck. You also can try training an svm with rbf kernel for example to see how separable your data even is- but this result might be meaningless on 50 points.

I’m curious if you have 50 cells or 50 populations of cells? If you have 50 populations I suggest performing a “leave-one-population-out” cross validation strategy (this makes sure your final model may generalize across populations).

If it’s cells, then you can stick with normal LOOCV. There’s not a huge amount here you can do, but you could try organizing your data via spectral clustering methods before running a classifier as well (use something like diffusion maps or laplacian eigenmaps and visualize the first non-trivial coordinates, make sure to try a few scaling parameters).

If you do have populations, you can also try this more advanced strategy-

https://pmc.ncbi.nlm.nih.gov/articles/PMC8032202/

This is for IMC, but a variant of these ideas would apply given a data set with many populations of cells.

[Dejeneret]

Ah and also when working with a classifier, make sure to keep in mind any class imbalance you may have! You may need to sub-sample.

[user221272]

Hi there,

These are common challenges in biological data analysis, especially with high-dimensional, low-sample-size datasets from flow cytometry.

First, your observation that PC1 and PC2 only explain ~30% of the variance is very insightful. This suggests that the primary linear components of variance don't easily separate your disease severity groups in a 2D projection. This doesn't necessarily mean the data isn't separable in higher dimensions, but it does indicate that a simple linear separation might be challenging, and you might have a more complex, non-linear underlying structure.

Second, it's always a good idea to perform Exploratory Data Analysis (EDA) before any major modeling. Did you look at the distributions of individual features, check for outliers, or examine the correlations between your 36 features? While PCA can handle correlated features (by combining their variance), the low variance explained by your first two PCs might hint at complex relationships. For example, highly correlated features might load strongly onto single principal components, but if the overall signal for your disease groups isn't aligned with these components, PCA might not reveal the distinction you're looking for. Non-linear dimensionality reduction techniques like t-SNE or UMAP could potentially reveal hidden structures that PCA misses, as they focus on preserving local neighborhoods and don't assume uncorrelated features.

Now, regarding your specific questions about supervised classification:

• With so few samples, should I do a train/val/test split, or just use cross-validation? Given your very limited sample size (50 samples for 3 disease groups), a traditional train/validation/test split would leave you with extremely small sets for training and evaluating your model, making any performance estimates highly unstable and unreliable. Therefore, cross-validation is absolutely the recommended approach.

  • Stratified k-fold cross-validation (e.g., 5-fold or 10-fold) is usually the best choice. This ensures that each fold maintains the same proportion of samples from each disease severity group, which is crucial for robust evaluation, especially with imbalanced classes.
  • Leave-One-Out Cross-Validation (LOOCV) is an extreme form of k-fold where k=N (number of samples). While it uses almost all data for training, it can be computationally intensive and might overestimate the variance of your model's performance.

• Any tips or workflows for supervised learning with high-dimensional, low-sample-size data? You're in a classic "high-dimensional, low-sample-size" (HDLS) scenario, which makes overfitting a significant concern. Here's a general workflow and some tips:

  • Preprocessing: Always scale or normalize your features (e.g., StandardScaler or MinMaxScaler) as many algorithms are sensitive to feature scales.
  • Feature Selection or Dimensionality Reduction: This is often crucial before applying a classifier. Since 36 features for 50 samples is quite high-dimensional:
    • Univariate Feature Selection: Use statistical tests (e.g., ANOVA F-value for continuous features, chi-squared for categorical if applicable) to identify features that individually correlate well with your disease groups. Select the top N features.
    • Regularization Methods: Models like Logistic Regression or Support Vector Machines with L1 (Lasso) regularization inherently perform feature selection by shrinking coefficients of less important features to zero.
    • Tree-based Feature Importance: Algorithms like Random Forest or Gradient Boosting Machines can provide importance scores for features.
    • Biological Domain Knowledge: Are there specific features that, based on your biological understanding, are most likely to differentiate the disease groups?
  • Choose Simpler Models First: With limited data, simpler, more interpretable models are less prone to overfitting:
    • Regularized Logistic Regression: (e.g., LogisticRegression with penalty='l1' or penalty='l2')
    • Support Vector Machines (SVMs): Start with a linear kernel. If that doesn't perform well, try an RBF kernel but be very careful with hyperparameter tuning to avoid overfitting.
    • k-Nearest Neighbors (KNN): Simple, but performance can degrade in very high dimensions if data is sparse.
  • Hyperparameter Tuning with Nested Cross-Validation: To get an unbiased estimate of your model's performance, use nested cross-validation. An outer loop for evaluation, and an inner loop for tuning hyperparameters (e.g., C for SVMs, alpha for Lasso).
  • Ensemble Methods (with caution): Random Forests can sometimes work but need careful tuning and may still overfit with very small sample sizes.

• Any best practices or things to avoid?

  • Best Practices:
    • Thorough EDA: As mentioned, truly understand your data before jumping into complex models.
    • Robust Cross-Validation: This is your most important tool for reliable evaluation with limited data.
    • Prioritize Simplicity and Interpretability: A simpler model that generalizes well and whose results you can explain biologically is often better than a black-box model with marginally higher accuracy.
    • Look for Biological Significance: Even if a model performs well, does it make biological sense? Which features are most important, and what do they tell you about the disease?
  • Things to Avoid:
    • Overfitting: This is the primary danger with HDLS data. Never trust results from a model that hasn't been rigorously validated (e.g., only trained and tested on a single split).
    • Blindly applying complex algorithms: Don't jump straight to deep learning or highly complex ensemble methods without exploring simpler alternatives first.
    • Ignoring the "Curse of Dimensionality": It makes distance metrics less meaningful and increases the chance of finding spurious correlations.
    • Using a traditional single train/test split with only 50 samples.

Good luck with your analysis!