User:Timothee Flutre/Notebook/Postdoc/2012/08/16
From OpenWetWare
m (→Variational Bayes approach for the mixture of Normals) |
(→Variational Bayes approach for the mixture of Normals: finish update on q_z) |
||
Line 26: | Line 26: | ||
* '''Latent variables''': N hidden variables, <math>z_1,\ldots,z_N</math>, each being a vector of length K with a single 1 indicating the component to which the <math>n^{th}</math> observation belongs, and K-1 zeroes. | * '''Latent variables''': N hidden variables, <math>z_1,\ldots,z_N</math>, each being a vector of length K with a single 1 indicating the component to which the <math>n^{th}</math> observation belongs, and K-1 zeroes. | ||
+ | |||
+ | <math>p(\mathbf{z}|\mathbf{w},K) = \prod_{n=1}^N p(z_n|\mathbf{w},K) = \prod_{n=1}^N \prod_{k=1}^K w_k^{z_{nk}}</math> | ||
* '''Augmented likelihood''': | * '''Augmented likelihood''': | ||
Line 91: | Line 93: | ||
Then we set this functional derivative to zero. We also make use of a frequent assumption, namely that the variational distribution fully factorizes over each individual latent variables ([http://en.wikipedia.org/wiki/Mean_field_theory mean-field assumption]): | Then we set this functional derivative to zero. We also make use of a frequent assumption, namely that the variational distribution fully factorizes over each individual latent variables ([http://en.wikipedia.org/wiki/Mean_field_theory mean-field assumption]): | ||
- | <math>\frac{\partial \mathcal{F}_K}{\partial q_{\mathbf{z}}} \bigg|_{q_{\mathbf{z}}^{(t+1)}} = 0 \Longleftrightarrow \forall \, n \; \mathrm{ln} \, q_{z_n}^{(t+1)}(z_n) = \int_\Theta \; q_\Theta(\Theta) \; \mathrm{ln} \, p( | + | <math>\frac{\partial \mathcal{F}_K}{\partial q_{\mathbf{z}}} \bigg|_{q_{\mathbf{z}}^{(t+1)}} = 0 \Longleftrightarrow \forall \, n \; \mathrm{ln} \, q_{z_n}^{(t+1)}(z_n) = \int_\Theta \; q_\Theta(\Theta) \; \mathrm{ln} \, p(y_n,z_n|\Theta,K) \, \mathrm{d}\Theta \; - \; 1 \; + \; C_{z_n}</math> |
Recognizing the expectation and factorizing <math>q_\Theta(\Theta)</math> into <math>q_\mathbf{w}(\mathbf{w})q_\mathbf{\mu,\tau}(\mathbf{\mu,\tau})</math>, we get: | Recognizing the expectation and factorizing <math>q_\Theta(\Theta)</math> into <math>q_\mathbf{w}(\mathbf{w})q_\mathbf{\mu,\tau}(\mathbf{\mu,\tau})</math>, we get: | ||
- | <math>\mathrm{ln} \, q_{z_n}^{(t+1)}(z_n) = \mathbb{E}_\mathbf{w}[\mathrm{ln} \, p( | + | <math>\mathrm{ln} \, q_{z_n}^{(t+1)}(z_n) = \mathbb{E}_\mathbf{w}[\mathrm{ln} \, p(z_n|\mathbf{w},K)] + \mathbb{E}_{\mathbf{\mu,\tau}}[\mathrm{ln} \, p(y_n|z_n,\mathbf{\mu},\mathbf{\tau},K)] \; + \; \text{constant}</math> |
+ | |||
+ | <math>\mathrm{ln} \, q_{z_n}^{(t+1)}(z_n) = \sum_{k=1}^K ( z_{nk} \; \mathrm{ln} \, \rho_{nk} ) \; + \; \text{constant}</math> where <math>\mathrm{ln} \, \rho_{nk} = \mathbb{E}[\mathrm{ln} \, w_k] + \frac{1}{2} \mathbb{E}[\mathrm{ln} \, \tau_k] - \frac{1}{2} \mathrm{ln} \, 2\pi - \frac{1}{2} \mathbb{E}[\tau_k (y_n-\mu_k)^2]</math> | ||
+ | |||
+ | Taking the exponential: <math>q_{z_n}^{(t+1)}(z_n) \propto \prod_k \rho^{z_{nk}}</math> | ||
- | + | As this should be a distribution, it should sum to one, and therefore: | |
- | <math> | + | <math>q_{z_n}^{(t+1)}(z_n) = \prod_k r_{nk}^{z_{nk}}</math> where <math>r_{nk} = \frac{\rho_{nk}}{\sum_{k'=1}^K \rho_{nk'}}</math> |
- | + | Interestingly, even though we haven't specified anything yet about <math>q_{z_n}</math>, we can see that it is of the same form as the prior on <math>z_n</math>, a Multinomial distribution. | |
Revision as of 19:34, 25 May 2013
Project name | Main project page Previous entry Next entry |
Variational Bayes approach for the mixture of Normals
The latent variables induce dependencies between all the parameters of the model. This makes it difficult to find the parameters that maximize the likelihood. An elegant solution is to introduce a variational distribution of parameters and latent variables, which leads to a re-formulation of the classical EM algorithm. But let's show it directly in the Bayesian paradigm.
We can now introduce a distribution :
The constant is here to remind us that has the constraint of being a distribution, ie. of summing to 1, which can be enforced by a Lagrange multiplier. We can then use the concavity of the logarithm (Jensen's inequality) to derive a lower bound of the marginal log-likelihood:
Let's call this lower bound as it is a functional, ie. a function of functions. To gain some intuition about the impact of introducing q, let's expand :
From this, it is clear that (ie. a lower-bound of the marginal log-likelihood) is the conditional log-likelihood minus the Kullback-Leibler divergence between the variational distribution q and the joint posterior of latent variables and parameters. As a side note, minimizing D_{KL}(p | | q) is used in the expectation-propagation technique. In practice, we have to make the following crucial assumption of independence on in order for the calculations to be analytically tractable:
This means that approximates the joint posterior, and therefore the lower-bound will be tight if and only if this approximation is exact and the KL divergence is zero. As we ultimately aim at inferring the parameters and latent variables that maximize the marginal log-likelihood, we will use the calculus of variations to find the functions and q_{Θ} that maximize the functional .
This naturally leads to a procedure very similar to the EM algorithm where, at the E step, we calculate the expectations of the parameters with respect to the variational distributions and q_{Θ}, and, at the M step, we recompute the variational distributions over the parameters.
We start by writing the functional derivative of with respect to :
Then we set this functional derivative to zero. We also make use of a frequent assumption, namely that the variational distribution fully factorizes over each individual latent variables (mean-field assumption):
Recognizing the expectation and factorizing q_{Θ}(Θ) into , we get:
where Taking the exponential: As this should be a distribution, it should sum to one, and therefore: where Interestingly, even though we haven't specified anything yet about , we can see that it is of the same form as the prior on z_{n}, a Multinomial distribution.
TODO
TODO |