Last updated: 2017-03-06

Code version: c7339fc

# Pre-requisites

Be familiar with basic probability and Bayesian calculations.

# Overview

The goal here is simply to point out that everything you want to compute in Bayesian calculations is an integral.

# Examples

Consider inference for a parameter $$\theta$$ from data $$D$$.

The posterior distribution of $$\theta$$ is given by Bayes Theorem

$p(\theta | D) = p(\theta) p(D | \theta)/ p(D)$

First note that the denominator $$p(D)$$ is an integral: $p(D) = \int p(D | \theta) p(\theta) d\theta$.

Now suppose we want to estimate $$\theta$$ by its posterior mean. This is $E(\theta | D) = \int \theta p(\theta |D ) d\theta.$

And if we want to find a 90% posterior credible interval for $$\theta$$ then we want to find $$A$$ and $$B$$ such that $$\Pr(\theta \in [A,B] | D) = 0.9$$. Note that the LHS of this is $\Pr(\theta \in [A,B]|D) = \int I(\theta \in [A,B]) p(\theta | D) d\theta,$ where $$I(E)$$ denotes the indicator function for the event $$E$$, which takes the value 1 if $$E$$ is true and $$0$$ otherwise.

# Examples: discrete

Of course, if $$\theta$$ is discrete then the integrals above all become sums.

For example $E(\theta | D) = \sum_n \theta_n \Pr(\theta=\theta_n | D)$ where $$\theta_1,\theta_2,\dots$$ are the possible values for $$\theta$$.

# Summary

Pretty much all the things you want to compute when doing Bayesian inference are integrals (or sums) of one kind or another…

If you are computing 1-dimensional integrals then numerical methods are often useful. For example, Simpsons Rule, Gaussian Quadrature. These can also work in 2-dimensions, and maybe even 3 or 4.

Other simple methods that can work for low dimensions: naive Monte Carlo, and Importance Sampling. Also Laplace approximation.

For higher dimensions we usually resort to Markov Chain Monte Carlo.

## Session information

sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.5 LTS

locale:
[1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8       LC_NAME=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] workflowr_0.4.0    rmarkdown_1.3.9004

loaded via a namespace (and not attached):
[1] backports_1.0.5 magrittr_1.5    rprojroot_1.2   htmltools_0.3.5
[5] tools_3.3.2     yaml_2.1.14     Rcpp_0.12.9     stringi_1.1.2
[9] knitr_1.15.1    git2r_0.18.0    stringr_1.2.0   digest_0.6.12
[13] gtools_3.5.0    evaluate_0.10  

This site was created with R Markdown