# User:Timothee Flutre/Notebook/Postdoc/2012/08/16

< User:Timothee Flutre‎ | Notebook‎ | Postdoc‎ | 2012‎ | 08
Project name <html><img src="/images/9/94/Report.png" border="0" /></html> Main project page
<html><img src="/images/c/c3/Resultset_previous.png" border="0" /></html>Previous entry<html>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</html>Next entry<html><img src="/images/5/5c/Resultset_next.png" border="0" /></html>

## Variational Bayes approach for the mixture of Normals

• Motivation: I have described on another page the basics of mixture models and the EM algorithm in a frequentist context. It is worth reading before continuing. Here I am interested in the Bayesian approach as well as in a specific variational method (nicknamed "Variational Bayes").

• Data: we have N univariate observations, $\displaystyle y_1, \ldots, y_N$ , gathered into the vector $\displaystyle \mathbf{y}$ .

• Assumptions: we assume the observations to be exchangeable and distributed according to a mixture of K Normal distributions. The parameters of this model are the mixture weights ($\displaystyle w_k$ ), the means ($\displaystyle \mu_k$ ) and the precisions ($\displaystyle \tau_k$ ) of each mixture components, all gathered into $\displaystyle \Theta = \{w_1,\ldots,w_K,\mu_1,\ldots,\mu_K,\tau_1,\ldots,\tau_K\}$ . There are two constraints: $\displaystyle \sum_{k=1}^K w_k = 1$ and $\displaystyle \forall k \; w_k > 0$ .

• Observed likelihood: $\displaystyle p(\mathbf{y} | \Theta, K) = \prod_{n=1}^N p(y_n|\Theta,K) = \prod_{n=1}^N \sum_{k=1}^K w_k Normal(y_n;\mu_k,\tau_k)$

• Latent variables: let's introduce N latent variables, $\displaystyle z_1,\ldots,z_N$ , gathered into the vector $\displaystyle \mathbf{z}$ . Each $\displaystyle z_n$ is a vector of length K with a single 1 indicating the component to which the $\displaystyle n^{th}$ observation belongs, and K-1 zeroes.

• Augmented likelihood: $\displaystyle p(\mathbf{y},\mathbf{z}|\Theta,K) = \prod_{n=1}^N p(y_n,z_n|\Theta,K) = \prod_{n=1}^N p(z_n|\Theta,K) p(y_n|z_n,\Theta,K) = \prod_{n=1}^N \prod_{k=1}^K w_k^{z_{nk}} Normal(y_n;\mu_k,\tau_k)^{z_{nk}}$

• Priors: we choose conjuguate ones
• for the parameters: $\displaystyle \forall k \; \mu_k \sim Normal(\mu_0,\tau_0)$ and $\displaystyle \forall k \; \tau_k \sim Gamma(\alpha,\beta)$
• for the latent variables: $\displaystyle \forall n \; z_n \sim Multinomial_K(1,\mathbf{w})$ and $\displaystyle \mathbf{w} \sim Dirichlet(\gamma)$