Overview

The purpose of this vignette is to introduce the Dirichlet distribution. You should be familiar with the Beta distribution since the Dirichlet can be thought of as a generalization of the Beta distribution.

If you want more details you could look at Wikipedia.

The Dirichlet Distribution

You can think of the $J$-dimensional Dirichlet distribution as a distribution on probability vectors, $q=(q_1,\dots,q_J)$, whose elements are non-negative and sum to 1. It is perhaps the most commonly-used distribution for probability vectors, and plays a central role in Bayesian inference from multinomial data.

The Dirichlet distribution has $J$ parameters, $\alpha_1,\dots,\alpha_J$ that control the mean and variance of the distribution. If $q \sim \text{Dirichlet}(\alpha_1,\dots,\alpha_J)$ then:

• The expectation of $q_j$ is $\alpha_j/(\alpha_1 + \dots + \alpha_J)$.

• The variance of $q_j$ becomes smaller as the sum $\sum_j \alpha_j$ increases.

Density

The density of the Dirichlet distribution is most conveniently written as $$p(q | \alpha) = \frac{\Gamma(\alpha_1+\dots+\alpha_J)}{\Gamma(\alpha_1)\dots \Gamma(\alpha_J)}\prod_{j=1}^J q_j^{\alpha_j-1} \qquad (q_j \geq 0; \quad \sum_j q_j =1).$$ where $Gamma$ here denotes the gamma function.

Actually when writing the density this way, a little care needs to be taken to make things formally correct. Specifically, if you perform standard (Lebesgue) integration of this “density” over the $J$ dimensional space $q_1,\dots, q_J$ it integrates to 0, and not 1 as a density should. This problem is caused by the constraint that the $q$s must sum to 1, which means that the Dirichlet distribution is effectively a $J-1$-dimensional distribution and not a $J$ dimensional distribution.

The simplest resolution to this is to think of the $J$ dimensional Dirichlet distribution as a distribution on the $J-1$ numbers $(q_1, \dots, q_{J-1})$, satisfying $\sum_{j=1}^{J-1} q_j \leq 1$, and then define $q_J := (1-q_1-q_2-\dots - q_{J-1})$. Then, if we integrate the density $$p(q_1,\dots,q_{J-1} | \alpha) = \frac{\Gamma(\alpha_1+\dots+\alpha_J)}{\Gamma(\alpha_1)\dots \Gamma(\alpha_J)} \prod_{j=1}^{J-1} q_j^{\alpha_j-1} (1-q_1-\dots - q_{J-1})^{\alpha_J} \qquad (q_j \geq 0; \quad \sum_{j=1}^{J-1} q_j \leq 1).$$ over $(q_1,\dots,q_{J-1})$, it integrates to 1 as a density should.

Examples

sessionInfo()

R version 3.3.0 (2016-05-03)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.5 (Yosemite)

locale:
[1] en_NZ.UTF-8/en_NZ.UTF-8/en_NZ.UTF-8/C/en_NZ.UTF-8/en_NZ.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] tidyr_0.4.1     dplyr_0.5.0     ggplot2_2.1.0   knitr_1.15.1   
 [5] MASS_7.3-45     expm_0.999-0    Matrix_1.2-6    viridis_0.3.4  
 [9] workflowr_0.3.0 rmarkdown_1.3  

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.5      git2r_0.18.0     plyr_1.8.4       tools_3.3.0     
 [5] digest_0.6.9     evaluate_0.10    tibble_1.1       gtable_0.2.0    
 [9] lattice_0.20-33  shiny_0.13.2     DBI_0.4-1        yaml_2.1.14     
[13] gridExtra_2.2.1  stringr_1.2.0    gtools_3.5.0     rprojroot_1.2   
[17] grid_3.3.0       R6_2.1.2         reshape2_1.4.1   magrittr_1.5    
[21] backports_1.0.5  scales_0.4.0     htmltools_0.3.5  assertthat_0.1  
[25] mime_0.5         colorspace_1.2-6 xtable_1.8-2     httpuv_1.3.3    
[29] labeling_0.3     stringi_1.1.2    munsell_0.4.3