Given, \(\begin{flalign} p,q &= p(x), q(y) \\ Hp &= \text{Entropy} \\ Hp,q &= \text{Cross-entropy} \\ Dp,q &= \text{Relative-Entropy/KL-Divergence} \\ \end{flalign}\) \(\begin{flalign} H_p &= -\sum_p{ 1 / lnp } \\ &= E_p[lnp] \\ \\ H_{p,q} &= -\sum_p{ 1 / lnq } \\ &= E_p[lnq] \\ \\ D_{p|q} &= H(p,q) - H(p) \\ &= E_p[lnp] - E_p[lnq] \\ &= E_p[ln(p / q)] \\ \end{flalign}\) So, - \(Hp\) is the average of \(ln(p)\), weighted by \(p\). - \(Hpq\) is the average of \(ln(q)\), weighted by \(p\). - \(Dpq\) is the average of the log odds ratio \(ln(p/q)\), assuming \(q=1-p\).