саша саша - 2 months ago 14
Python Question

score_cbow_pair in word2vec (gensim)

I wanted to output the log-probability during learning of the word and doc vectors in gensim. I have taken a look at the implementation of the score function in the "slow plain numpy" version.

def score_cbow_pair(model, word, word2_indices, l1):
l2a = model.syn1[word.point] # 2d matrix, codelen x layer1_size
sgn = (-1.0)**word.code # ch function, 0-> 1, 1 -> -1
lprob = -log(1.0 + exp(-sgn*dot(l1, l2a.T)))
return sum(lprob)


The score function should make use of the parameters learned during hierarchical softmax training. But in the calculation of the log-probability there is supposed to be a sigmoid function( word2vec Parameter Learning Explained equation (45)).
So does gensim really calculate the log-probability in
lprob
or is it just a score for comparison purposes.

I would have calculated the log-probability as follows:
-log(1.0/(1.0+exp(-sgn*dot(l1, l2a.T))))


Is this equation not used because it explodes for values close to zero or is it in general wrong?

Answer

I've overlooked that the logarithm of the sigmoid function can be rewritten: log(1.0/(1.0+exp(-sgn*dot(l1, l2a.T)))) = log(1)-log(1.0+exp(-sgn*dot(l1, l2a.T))) = -log(1.0+exp(-sgn*dot(l1, l2a.T)))

So the code does compute the log-likelihood.

Comments