Last updated: 2017-02-20
Code version: 92d7daa
The purpose of this vignette is to introduce the Dirichlet distribution. You should be familiar with the Beta distribution since the Dirichlet can be thought of as a generalization of the Beta distribution.
If you want more details you could look at Wikipedia.
You can think of the \(J\)-dimensional Dirichlet distribution as a distribution on probability vectors, \(q=(q_1,\dots,q_J)\), whose elements are non-negative and sum to 1. It is perhaps the most commonly-used distribution for probability vectors, and plays a central role in Bayesian inference from multinomial data.
The Dirichlet distribution has \(J\) parameters, \(\alpha_1,\dots,\alpha_J\) that control the mean and variance of the distribution. If \(q \sim \text{Dirichlet}(\alpha_1,\dots,\alpha_J)\) then:
The expectation of \(q_j\) is \(\alpha_j/(\alpha_1 + \dots + \alpha_J)\).
The variance of \(q_j\) becomes smaller as the sum \(\sum_j \alpha_j\) increases.
The 2-dimensional Dirichlet distribution is essentially the Beta distribution. Specifically, let \(q=(q_1,q_2)\). Then \(q \sim Dirichlet(\alpha_1,\alpha_2)\) implies that \[q_1 \sim \text{Beta}(\alpha_1,\alpha_2)\] and \(q_2 = 1-q_1\).
More generally, the marginals of the Dirichlet distribution are also beta distributions.
That is, if \(q \sim \text{Dirichlet}(\alpha_1, \dots,\alpha_J)\) then \(q_j \sim \text{Beta}(\alpha_j,\sum_{j' \neq j} \alpha_{j'})\).
The density of the Dirichlet distribution is most conveniently written as \[p(q | \alpha) = \frac{\Gamma(\alpha_1+\dots+\alpha_J)}{\Gamma(\alpha_1)\dots \Gamma(\alpha_J)}\prod_{j=1}^J q_j^{\alpha_j-1} \qquad (q_j \geq 0; \quad \sum_j q_j =1).\] where \(Gamma\) here denotes the gamma function.
Actually when writing the density this way, a little care needs to be taken to make things formally correct. Specifically, if you perform standard (Lebesgue) integration of this “density” over the \(J\) dimensional space \(q_1,\dots, q_J\) it integrates to 0, and not 1 as a density should. This problem is caused by the constraint that the \(q\)s must sum to 1, which means that the Dirichlet distribution is effectively a \(J-1\)-dimensional distribution and not a \(J\) dimensional distribution.
The simplest resolution to this is to think of the \(J\) dimensional Dirichlet distribution as a distribution on the \(J-1\) numbers \((q_1, \dots, q_{J-1})\), satisfying \(\sum_{j=1}^{J-1} q_j \leq 1\), and then define \(q_J := (1-q_1-q_2-\dots - q_{J-1})\). Then, if we integrate the density \[p(q_1,\dots,q_{J-1} | \alpha) = \frac{\Gamma(\alpha_1+\dots+\alpha_J)}{\Gamma(\alpha_1)\dots \Gamma(\alpha_J)} \prod_{j=1}^{J-1} q_j^{\alpha_j-1} (1-q_1-\dots - q_{J-1})^{\alpha_J} \qquad (q_j \geq 0; \quad \sum_{j=1}^{J-1} q_j \leq 1). \] over \((q_1,\dots,q_{J-1})\), it integrates to 1 as a density should.
sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X El Capitan 10.11.6
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] backports_1.0.5 magrittr_1.5 rprojroot_1.2 htmltools_0.3.5
[5] tools_3.3.2 rstudioapi_0.6 yaml_2.1.14 Rcpp_0.12.9
[9] stringi_1.1.2 rmarkdown_1.3 knitr_1.15.1 git2r_0.18.0
[13] stringr_1.1.0 digest_0.6.12 workflowr_0.3.0 gtools_3.5.0
[17] evaluate_0.10
This site was created with R Markdown