Monte Carlo dropout is approximate Bayesian inference

I demonstrate how keeping dropout active during testing enables uncertainty estimation and improves prediction accuracy in neural networks.

The idea

Standard neural networks turn dropout off at test time to produce a single deterministic output, leaving us with no measure of prediction uncertainty. By forcing dropout to remain active during inference and passing the exact same input through the network multiple times, each pass drops a random set of neurons and produces a slightly different prediction. Taking multiple forward passes allows us to sample from an approximate posterior distribution, where the spread of predictions reveals model uncertainty and their average yields a more accurate prediction than a standard network.

The mechanism

Dropout at test time samples a random weight configuration $\hat{W}_t$ on each pass. Averaging over $T$ passes approximates the Bayesian predictive distribution:

p(y \mid x, \mathcal{D}) \approx \frac{1}{T} \sum_{t=1}^{T} p(y \mid x, \hat{W}_t)

The mean of the passes is the prediction. The variance is the model's uncertainty. Gal and Ghahramani showed this is equivalent to variational inference with a Bernoulli approximating distribution over the weights, which is what makes it "Bayesian" rather than just an ensemble trick.

Worth knowing

The uncertainty you get is only as good as the approximation: because it uses a Bernoulli variational family, the estimates are typically overconfident and do not represent calibrated uncertainty.
Monte Carlo dropout provides the cheapest way to extract uncertainty from a network you have already trained without changing its underlying architecture.

Code

In TensorFlow/Keras, implement Monte Carlo dropout by inheriting from keras.layers.Dropout and overriding call to force training=True during inference. Pass the same input through the network multiple times and stack the outputs to compute prediction mean and variance.

class MCDropout(keras.layers.Dropout):
    def call(self, inputs):
        return super().call(inputs, training=True)

Use it when / don't use it when

Use it when

You need uncertainty estimation in neural networks without modifying your existing network architecture.
You want to improve overall prediction accuracy by ensembling predictions from multiple forward passes.
Your neural network already uses dropout as a regularization technique.

Don't use it when

You require real-time or ultra-low latency inference, as running multiple forward passes per sample adds computational overhead.
Your model architecture does not utilize dropout layers.