Bhattacharyya distance versus Kullback-Leibler divergence

Another question I picked on Cross Validated during the Yule break is about the connection between the Bhattacharyya distance and the Kullback-Leibler divergence, i.e.,

d_B(p,q)=-\log\left\{\int\sqrt{p(x)q(x)}\,\text{d}x\right\}

and

d_{KL}(p\|q)=\int\log\left\{{q(x)}\big/{p(x)}\right\}\,p(x)\,\text{d}x

Although this Bhattacharyya distance sounds close to the Hellinger distance,

d_H(p,q)=\left\{1-\int\sqrt{p(x)q(x)}\,\text{d}x\right\}^{1/2}

the ordering I got by a simple Jensen inequality is

d_{KL}(p\|q)\ge2d_B(p,q)\ge2d_H(p,q)^2\,.

and I wonder how useful this ordering could be…

One Response to “Bhattacharyya distance versus Kullback-Leibler divergence”

  1. They’re all very similar. The wiki page for Bhattacharyya distance gives the precise (equality) relationship between d_B and d_H (d_B is not a metric). So I guess that the Bhattacharyya distance and the Hellinger distance define equivalent topologies.

    The ordering is that KLD is (topologically) stronger than Hellinger which is stronger than total variation. And convergence in the Hellinger metric implies that L2 funcitonals converge.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s