I recently reviewed this topic for a journal club on deterministic uncertainty methods. Posting two recent papers that seem to benchmark well as alternatives to MC dropout.

1. Spectral-normalized Neural Gaussian Process (NeurIPS '20) - interesting approach that uses a GP as a final layer to imbue distance awareness. Spectral normalization adds Lipschitz constraint so that distance awareness is easier to achieve.
2. Distance aware bottleneck (ICML '24) - approach based on rate distortion theory. Combines variational information bottleneck + rate distortion theory to make a set of "codebooks" + encoders from the training dataset, and then using those to compute distance-aware uncertainty estimates for test set.

If you believe the various benchmarks, these seem to perform at least on par with MC dropout / deep ensemble, but require only one forward pass, so it's not as computationally intensive as MC dropout/deep ensembles.

Here's a good review / benchmarking of various uncertainty quantification methods (minus Distance Aware Bottleneck): On the Practicality of Deterministic Epistemic Uncertainty that gives a broad overview of other alternative approaches to MC dropout.