The proof is in the pudding

Looking for 𝛂 is never a smooth ride

Two flavours of uncertainty

What the two uncertainties look like

The good and the bad (as in naive)

A proper Bayesian neural network treats the weights \(w\) as random:

\[p(y \mid x, \mathcal{D}) = \int p(y \mid x, w)\, p(w \mid \mathcal{D})\, dw\]

• \(p(w \mid \mathcal{D})\) lives in a space with \(10^6\) to \(10^9\) dimensions.
• HMC is the gold standard and completely infeasible here.
• Mean-field variational inference is tractable and pathologically overconfident.
• The posterior is multimodal. Most approximations happily collapse to one mode and pretend the others don’t exist.

It’s not always easy to navigate

\(\mu(x_1, x_2)\) and \(\log \sigma^2(x_1, x_2)\) from a heteroskedastic head with data \(y = \sin(x_1)\cos(x_2)\).

The landscape the optimizer sees

\(\mu = \sin(w_1 x)\), \(\sigma^2 = \sigma_0^2 + w_2^2 x^2\)

Deep ensembles

Train \(N\) independent networks from different random seeds.

\[p(y \mid x) \approx \frac{1}{N} \sum_{i=1}^{N} p_i(y \mid x, w_i)\]

• Epistemic uncertainty shows up as disagreement between members.
• Empirically beats most “principled” Bayesian methods on calibration and out-of-distribution detection (Lakshminarayanan et al. 2017; Ovadia et al. 2019).
• Embarrassingly parallel. One of the few things in deep learning that actually scales linearly.
• The cost is \(N\times\) training. In practice \(N = 5\) to \(10\) is plenty.
• Pairs naturally with a heteroscedastic head for the aleatoric part.

One training run, many models

Ok so what if \(N\times\) training is too expensive? Can we wring a posterior out of a single run?

Last-layer Laplace, the predictive

# 3. Predictive distribution at a fresh x
function predict(x)
    Φ = ϕ(x)
    μ  = θ ⋅ Φ

    # epistemic + aleatoric
    v  = Φ' ⋅ Σ ⋅ Φ + σ² 
    return μ, v
end

One forward pass for the mean, one quadratic form for the variance. That’s the whole inference cost.

MC Dropout, simple but bad taste

Keep dropout switched on at inference, take \(T\) forward passes, and call the variance your uncertainty (Gal and Ghahramani 2016).

• Trivially easy to implement. No retraining.
• But the uncertainty is tied to the dropout rate, which is a hyperparameter, not a posterior.
• As the network gets wider, the uncertainty collapses.
• Under distribution shift it systematically under-estimates.

Feeding this straight into position sizing is asking for trouble.

Conformal prediction as a post hoc checks

Everything above is a model of uncertainty. Conformal prediction sits on top and gives you a coverage guarantee, whatever model you wrapped.

• Wraps any point predictor to emit intervals with exact marginal coverage.
• Distribution-free. It only assumes exchangeability of calibration and test data.
• Breaks in financial time series.
• Adaptive variants (Gibbs and Candès 2021) try to fix this for streaming data.
• I use it to audit my Bayesian approximations, not to replace them.

So what actually ends up in the stack?

It really depends on what exactly you need but

• A heteroscedastic head for the aleatoric part. Essentially free.
• A deep ensemble with \(N \approx 5\) to \(10\) for epistemic uncertainty.
• Laplace when that ensemble is too expensive.
• A conformal wrapper on top, to check calibration.

Thank you