# Difference between revisions of "User:Timothee Flutre/Notebook/Postdoc/2012/08/16"

Project name <html><img src="/images/9/94/Report.png" border="0" /></html> Main project page
<html><img src="/images/c/c3/Resultset_previous.png" border="0" /></html>Previous entry<html>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</html>Next entry<html><img src="/images/5/5c/Resultset_next.png" border="0" /></html>

## Variational Bayes approach for the mixture of Normals

• Motivation: I have described on another page the basics of mixture models and the EM algorithm in a frequentist context. It is worth reading before continuing. Here I am interested in the Bayesian approach as well as in a specific variational method (nicknamed "Variational Bayes").

• Data: we have N univariate observations, ${\displaystyle y_{1},\ldots ,y_{N}}$, gathered into the vector ${\displaystyle \mathbf {y} }$.

• Assumptions: we assume the observations to be exchangeable and distributed according to a mixture of K Normal distributions. The parameters of this model are the mixture weights (${\displaystyle w_{k}}$), the means (${\displaystyle \mu _{k}}$) and the precisions (${\displaystyle \tau _{k}}$) of each mixture components, all gathered into ${\displaystyle \Theta =\{w_{1},\ldots ,w_{K},\mu _{1},\ldots ,\mu _{K},\tau _{1},\ldots ,\tau _{K}\}}$. There are two constraints: ${\displaystyle \sum _{k=1}^{K}w_{k}=1}$ and ${\displaystyle \forall k\;w_{k}>0}$.

• Observed likelihood: ${\displaystyle p(\mathbf {y} |\Theta ,K)=\prod _{n=1}^{N}p(y_{n}|\Theta ,K)=\prod _{n=1}^{N}\sum _{k=1}^{K}w_{k}Normal(y_{n};\mu _{k},\tau _{k}^{-1})}$

• Latent variables: let's introduce N latent variables, ${\displaystyle z_{1},\ldots ,z_{N}}$, gathered into the vector ${\displaystyle \mathbf {z} }$. Each ${\displaystyle z_{n}}$ is a vector of length K with a single 1 indicating the component to which the ${\displaystyle n^{th}}$ observation belongs, and K-1 zeroes.

• Augmented likelihood: ${\displaystyle p(\mathbf {y} ,\mathbf {z} |\Theta ,K)=\prod _{n=1}^{N}p(y_{n},z_{n}|\Theta ,K)=\prod _{n=1}^{N}p(z_{n}|\Theta ,K)p(y_{n}|z_{n},\Theta ,K)=\prod _{n=1}^{N}\prod _{k=1}^{K}w_{k}^{z_{nk}}Normal(y_{n};\mu _{k},\tau _{k}^{-1})^{z_{nk}}}$

• Priors: we choose conjuguate ones
• for the parameters: ${\displaystyle \forall k\;\mu _{k}|\tau _{k}\sim Normal(\mu _{0},(\tau _{0}\tau _{k})^{-1})}$ and ${\displaystyle \forall k\;\tau _{k}\sim Gamma(\alpha ,\beta )}$
• for the latent variables: ${\displaystyle \forall n\;z_{n}\sim Multinomial_{K}(1,\mathbf {w} )}$ and ${\displaystyle \mathbf {w} \sim Dirichlet(\gamma )}$