User:Timothee Flutre/Notebook/Postdoc/2011/12/14
From OpenWetWare
(Difference between revisions)
(→Learn about mixture models and the EM algorithm: add latent variables + algebra for mu_k and sigma_k) 
m (→Learn about mixture models and the EM algorithm: improve motivation) 

Line 8:  Line 8:  
==Learn about mixture models and the EM algorithm==  ==Learn about mixture models and the EM algorithm==  
  * '''Motivation and examples''': it  +  * '''Motivation and examples''': a large part of any scientific activity is about measuring things, in other words collecting data, and it is not unfrequent to collect ''heterogeneous'' data. For instance, we measure the height of individuals without recording their gender, we measure the levels of expression of a gene in several individuals without recording which ones are healthy and which ones are sick, etc. It seems therefore natural to say that the samples come from a mixture of clusters. The aim is then to recover from the data, ie. to infer, (i) the values of the parameters of the probability distribution of each cluster, and (ii) from which cluster each sample comes from. 
* '''Data''': we have N observations, noted <math>X = (x_1, x_2, ..., x_N)</math>. For the moment, we suppose that each observation <math>x_i</math> is univariate, ie. each corresponds to only one number.  * '''Data''': we have N observations, noted <math>X = (x_1, x_2, ..., x_N)</math>. For the moment, we suppose that each observation <math>x_i</math> is univariate, ie. each corresponds to only one number.  
  * '''Hypotheses and aim''': let's assume that the data are heterogeneous and that they can be partitioned into <math>K</math> clusters (see examples above). This means that a subset of the observations come from cluster <math>k=1</math>, another subset come from cluster <math>k=2</math>, and so on.  +  * '''Hypotheses and aim''': let's assume that the data are heterogeneous and that they can be partitioned into <math>K</math> clusters (see examples above). This means that we expect a subset of the observations to come from cluster <math>k=1</math>, another subset to come from cluster <math>k=2</math>, and so on. 
  * '''Model''': technically, we say that the observations were generated  +  * '''Model''': technically, we say that the observations were generated according to a [http://en.wikipedia.org/wiki/Probability_density_function density function] <math>f</math>. More precisely, this density is itself a mixture of densities, one per cluster. In our case, we will assume that each cluster <math>k</math> corresponds to a Normal distribution, here noted <math>g</math>, of mean <math>\mu_k</math> and standard deviation <math>\sigma_k</math>. Moreover, as we don't know for sure from which cluster a given observation comes from, we define the mixture probability <math>w_k</math> to be the probability that any given observation comes from cluster <math>k</math>. As a result, we have the following list of parameters: <math>\theta=(w_1,...,w_K,\mu_1,...\mu_K,\sigma_1,...,\sigma_K)</math>. Finally, for a given observation <math>x_i</math>, we can write the model <math>f(x_i/\theta) = \sum_{k=1}^{K} w_k g(x_i/\mu_k,\sigma_k)</math> , with <math>g(x_i/\mu_k,\sigma_k) = \frac{1}{\sqrt{2\pi} \sigma_k} \exp^{\frac{1}{2}(\frac{x_i  \mu_k}{\sigma_k})^2}</math>. 
* '''Likelihood''': this corresponds to the probability of obtaining the data given the parameters: <math>L(\theta) = P(X/\theta)</math>. We assume that the observations are independent, ie. they were generated independently, whether they are from the same cluster or not. Therefore we can write: <math>L(\theta) = \prod_{i=1}^N f(x_i/\theta)</math>.  * '''Likelihood''': this corresponds to the probability of obtaining the data given the parameters: <math>L(\theta) = P(X/\theta)</math>. We assume that the observations are independent, ie. they were generated independently, whether they are from the same cluster or not. Therefore we can write: <math>L(\theta) = \prod_{i=1}^N f(x_i/\theta)</math>. 
Revision as of 06:49, 22 December 2011
Project name  Main project page Previous entry Next entry 
Learn about mixture models and the EM algorithm
As we derive with respect to μ_{k}, all the others means μ_{l} with are constant, and thus disappear:
And finally:
Once we put all together, we end up with:
By convention, we note the maximumlikelihood estimate of μ_{k}:
Therefore, we finally obtain:
By doing the same kind of algebra, we also obtain the ML estimates for the standard deviation of each cluster:
