Aegean T. Wu Aegean T. Wu - 9 months ago 65
Python Question

How to extract word frequency from document-term matrix?

I am doing LDA analysis with Python. And I used the following code to create a document-term matrix

corpus = [dictionary.doc2bow(text) for text in texts].

Is there any easy ways to count the word frequency over the whole corpus. Since I do have the dictionary which is a term-id list, I think I can match the word frequency with term-id.


You can use nltk in order to count word frequency in string texts

from nltk import FreqDist
import nltk
words = nltk.tokenize.word_tokenize(texts)
fdist = FreqDist(words)

fdist will give you word frequency of given string texts. Another way will be to use CountVectorizer from scikit-learn for list of strings.

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
freq = np.ravel(X.sum(axis=0)) # sum each columns to get total counts for each word

this freq will correspond to value in dictionary vectorizer.vocabulary_

import operator
# get vocabulary keys, sorted by value
vocab = [v[0] for v in sorted(vectorizer.vocabulary_.items(), key=operator.itemgetter(1))]
fdist = dict(zip(vocab, freq)) # return same format as nltk