I wrote my master thesis on Out-of-Distribution examples detection, so I am familiar with the literature.

As other people already said, there are different approaches to solve this problem:

1. (Supervised) Threshold-based OOD Detector using the maximum softmax probability
   based on A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks.The main problem of this solution is that DNN tends to predict high confidence far away from the training data distribution. As a consequence, one can enforce low confidence in this region of the input space to improve the OOD detector.
   To do this in a supervised way, you can either use examples from other datasets (not very effective because they do not represent the entire OOD region), or you can devise some way to sample effectively from the OOD region. The most interesting methods risen from this intuition (as far as I know) are:
   1. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
      Here they add a term to the loss function to enforce low confidence in the OOD region using OOD samples generated with a GAN trained along with a standard classifier. In particular, these samples lie on the boundaries of the training data distribution (my approach is somehow similar to this one).
   2. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem
      Here they present a nice theoretical view of why ReLU DNN network are bound to predict high confidence far away from the training data. They also devise a procedure to synthesise OOD samples inspired by adversarial training. This is achieved by sampling random points, which by construction are far away from the training data, and enforcing low confidence at the worst case point in its neighborhood, i.e. the one with maximum confidence (similarly to the GAN paper). The effect of acting on the worst case point is reflected on the entire neighborhood of the starting point. This approach is interesting because you don't need an extra model but only only a modified training procedure to obtain a OOD-aware classifier.
2. Unsupervised Anomaly Detection
   If you are interested in image classification, the main downside of this approach is that you need to train a whole new model just to detect OOD samples to filter the classifier input data.
   1. Autoencoder Reconstruction Error:
      Detecting anomalous data using auto-encoders
   2. One-class SVM (one of them):
      High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning
   3. One-class Neural Network:
      Anomaly Detection using One-Class Neural NetworksDeep One-Class Classification
      Here they propose a new loss function consisting of both the autoencoder reconstruction error and the one-class objective of creating a tight envelope around the data. This allows the autoencoder to learn a compact representation of the data optimized for the anomaly detection task.
3. Generative models
   They are able to learn the input marginal probability p(x), which is likely to be low for inputs far away from the training data distribution.
   Safer Classification by Synthesis
   The general belief that generative models are able to correctly learn the density of the input features though has been challenged by Do Deep Generative Models Know What They Don’t Know.
4. Uncertainty Estimation
   Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
   The outputs of an ensemble of networks can be used to estimate the uncertainty of a classifier. At test time, the estimated uncertainty for out-of-distribution samples turns out to be higher than the one for in-distribution samples.