Questions tagged [perplexity]

A few papers I have come across say that BLEU is not an appropriate evaluation metric for chatbots, so they use the perplexity. First of all, what is perplexity? How to calculate it? And why is perplexity a good evaluation metric for chatbots?

The perplexity of the $i^{th}$ token in the $k^{th}$ sequence is $$ P_{ki} = \frac{1}{p(t_{ki})} $$ The perplexity aggregated for the $k^{th}$ sequence is then $$ P_{k} = \left(\prod_{i=1}^N P_{ki}\right)^{1/N} \\ = \left(\prod_{i=1}^N…

I know word2vec is not enough to achieve high quality text prediction just by itself. But has any scientific work tried to do it anyway, just to know what the baseline is that you have to beat with more sophisticated ideas? It doesn't have to be…