## Bhattacharyya distance versus Kullback-Leibler divergence

Another question I picked on Cross Validated during the Yule break is about the connection between the Bhattacharyya distance and the Kullback-Leibler divergence, i.e.,

$d_B(p,q)=-\log\left\{\int\sqrt{p(x)q(x)}\,\text{d}x\right\}$

and

$d_{KL}(p\|q)=\int\log\left\{{q(x)}\big/{p(x)}\right\}\,p(x)\,\text{d}x$

Although this Bhattacharyya distance sounds close to the Hellinger distance,

$d_H(p,q)=\left\{1-\int\sqrt{p(x)q(x)}\,\text{d}x\right\}^{1/2}$

the ordering I got by a simple Jensen inequality is

$d_{KL}(p\|q)\ge2d_B(p,q)\ge2d_H(p,q)^2\,.$

and I wonder how useful this ordering could be…

### One Response to “Bhattacharyya distance versus Kullback-Leibler divergence”

1. They’re all very similar. The wiki page for Bhattacharyya distance gives the precise (equality) relationship between d_B and d_H (d_B is not a metric). So I guess that the Bhattacharyya distance and the Hellinger distance define equivalent topologies.

The ordering is that KLD is (topologically) stronger than Hellinger which is stronger than total variation. And convergence in the Hellinger metric implies that L2 funcitonals converge.