Amir Amir - 1 month ago 11
Python Question

How train word2vec with gensim for Arabic/Persian language

I follow this tutorial and everything is fine, I preprocess and train my model But when I want to find similarity with following code:

model = gensim.models.Word2Vec.load("wiki.fa.word2vec.model")
print model.most_similar(U'ماه')

Vectors are not in proper format:

[(u'\u0631\u0648\u0632', 0.6399222612380981), (u'\u0647\u0641\u062a\u0647', 0.5578583478927612), (u'\u0645\u0627\u0647\u0647\u0627\u06cc', 0.5577661991119385), (u'\u062f\u0631\u0645\u0627\u0647', 0.5260834097862244), (u'\u0634\u0627\u0645\u06af\u0627\u0647', 0.5142802596092224), (u'\u06cc\u06a9\u0645\u0627\u0647', 0.48211610317230225), (u'\u0642\u062f\u06cc\u0631\u0641', 0.4799095690250397), (u'\u06cc\u06a9\u0633\u0627\u0644', 0.47623544931411743), (u'\u0645\u0627\u0647\u0647', 0.46996498107910156), (u'\u062d\u0648\u062a', 0.4551585912704468)]

Does anyone knows how to fix this issue?

Answer Source

Use a for loop to display utf-8 words in Intellij:

similar = most_similar(U'ماه')
for word in similar: