Measure theoretic description of KL divergence

In this writing, I consider the KL divergence with measure theoretic approach.

 

Suppose $P$ and $Q$ are probability measures on a measurable space $(\mathrm{\Omega },\mathcal{F})$, and $P\ll Q$. That is, $P$ is absolutely continuous with respect to $Q$. (Or, equivalently, $Q$ dominates $P$)

 

KL divergence is defined as below:

$$\begin{array}{rl}{\mathcal{D}}_{KL}(P||Q)& :=E_P[\log \frac{dP}{dQ}]\\ & ={\int }_{\mathrm{\Omega }}\log \frac{dP}{dQ}dP\end{array}$$

If we have density functions for $P$ and $Q$ with respect to Lebesgue measure:

$$\begin{array}{rl}\frac{dP}{d\lambda }& =f\\ \frac{dQ}{d\lambda }& =g\end{array}$$

then the KL divergence can be expressed in the familiar form.

$$\begin{array}{rl}{\mathcal{D}}_{KL}(P||Q)& ={\int }_{\mathrm{\Omega }}\log \frac{dP}{dQ}dP\\ & ={\int }_{\mathrm{\Omega }}\log \frac{\frac{dP}{d\lambda }}{\frac{dQ}{d\lambda }}\frac{dP}{d\lambda }d\lambda \\ & ={\int }_{\mathrm{\Omega }}(\log \frac{f}{g})fd\lambda \end{array}$$

Just to be clear, $dP=P(d\omega )$, $dQ=Q(d\omega )$, and $d\lambda =\lambda (d\omega )$.

Since the Lebesgue measure is defined over a real interval, for the last expression, $\mathrm{\Omega }=\mathbb{R}\cap [\text{some interval}]$.