# Kullback-Leibler divergence

In contrastive divergence the Kullback-Leibler divergence (KL-divergence) between the data distribution and the model distribution is minimized (here we assume $x$ to be discrete):

Here $P_0(x)$ is the observed data distribution, $P(x\mid W)$ is the model distribution and $W$ are the model parameters. A divergence (wikipedia) is a fancy term for something that resembles a metric distance. It is not an actual metric because the divergence of $x$ given $y$ can be different (and often is different) from the divergence of $y$ given $x$. The Kullback-Leibler divergence $D_{KL}(P \mid \mid Q)$ exists only if $Q(\cdot) = 0$ implies $P(\cdot) = 0$.

The model distribution can be written in the form of a normalized energy function:

The partition function can be written as the sum over all states:

## Gradients

With gradient descent we use the gradient negatively:

With gradient ascend we use the gradient positively:

In both cases $\lambda$ is a predefined parameter. It can be constant, but in learning methods this can also be a function called the learning rate. The parameter $\lambda$ might depend on time $t$.

For both gradient descent and gradient ascent $W_{t+1} - W_t = 0$ means that $\nabla f(W_t) = 0$. Descending a slope up to a zero gradient leads to a minimum if there is one. Ascending a slope up to a zero gradients leads to a maximum if there is one. The extremum found does not necessarily need to be unique, except if the function is concave, respectively convex.

## Gradient descent of the KL-divergence

Below you will find a step-by-step derivation of a description of gradient descent for the KL-divergence. It needs to be minimization so we will indeed need gradient descent (not ascent). Formally, we have to calculate:

### KL-divergence parts that depend on $W$

We are gonna rewrite this equation is a way relevant to taking a derivative: (1) reorganize the equation such that the terms not involving $W$ are separate terms, (2) using log identities to write it as a sum of terms, and (3) removing the terms not involving $W$.

Hence, first, let us rewrite the divergence to obtain separate terms that do and do not involve $W$. Herefore we substitute $P(x\mid W)$ on the fourth line:

Second, use the following identity $\log a + \log b = \log a b$ to reach a sum of terms:

Third, get rid of the first term that does not depend on $W$. Now the part relevant to our derivative is:

In “On Contrastive Divergence Learning” by Carreira-Perpinan and Hinton (proceedings AISTATS 2015) this is written as the log-likelihood objective:

Note, that there is a negative sign here. The maximum log-likelihood is identical to the minimum KL divergence.

### The gradient of the KL-divergence

Taking the gradient with respect to $W$ (we can then safely omit the term that does not depend on $W$):

Recall the derivative of a logarithm:

Take derivative of logarithm:

The derivative of the partition function:

Recall the derivative of an exponential function:

Use this for the partition function derivative:

Hence:

Using $P(x \mid W)$:

Again, the gradient of the divergence was:

Hence:

Compare with Hinton:

Gradient descent:

Thus,

We arrived at the formulation of minimization of KL-divergence that allows comparing it with Contrastive divergence.

# Constrastive divergence

Contrastive divergence uses a different (empirical) distribution to get rid of $P(x \mid W)$:

## Comments

The comments are not automatically shown, so the page loads faster and you only partake in the disqus network if you click. However, this does not mean that your comments are not appreciated, to the contrary!