Bhavika Bhavika - 5 months ago 257
Python Question

TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document

I'm using TfidfVectorizer from scikit-learn to do some feature extraction from text data. I have a CSV file with a Score (can be +1 or -1) and a Review (text). I pulled this data into a DataFrame so I can run the Vectorizer.

This is my code:

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

df = pd.read_csv("train_new.csv",
names = ['Score', 'Review'], sep=',')

# x = df['Review'] == np.nan
# print x.to_csv(path='FindNaN.csv', sep=',', na_rep = 'string', index=True)
# print df.isnull().values.any()

v = TfidfVectorizer(decode_error='replace', encoding='utf-8')
x = v.fit_transform(df['Review'])

This is the traceback for the error I get:

Traceback (most recent call last):
File "/home/PycharmProjects/Review/src/", line 16, in <module>
x = v.fit_transform(df['Review'])
File "/home/b/hw1/local/lib/python2.7/site- packages/sklearn/feature_extraction/", line 1305, in fit_transform
X = super(TfidfVectorizer, self).fit_transform(raw_documents)
File "/home/b/work/local/lib/python2.7/site-packages/sklearn/feature_extraction/", line 817, in fit_transform
File "/home/b/work/local/lib/python2.7/site- packages/sklearn/feature_extraction/", line 752, in _count_vocab
for feature in analyze(doc):
File "/home/b/work/local/lib/python2.7/site-packages/sklearn/feature_extraction/", line 238, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)
File "/home/b/work/local/lib/python2.7/site-packages/sklearn/feature_extraction/", line 118, in decode
raise ValueError("np.nan is an invalid document, expected byte or "
ValueError: np.nan is an invalid document, expected byte or unicode string.

I checked the CSV file and DataFrame for anything that's being read as NaN but I can't find anything. There are 18000 rows, none of which return
as True.

This is what
looks like:

0 This book is such a life saver. It has been s...
1 I bought this a few times for my older son and...
2 This is great for basics, but I wish the space...
3 This book is perfect! I'm a first time new mo...
4 During your postpartum stay at the hospital th...
Name: Review, dtype: object


You need to convert the dtype object to unicode string as is clearly mentioned in the traceback.

x = v.fit_transform(df['Review'].values.astype('U'))  ## Even astype(str) would work

From the Doc page of TFIDF Vectorizer:

fit_transform(raw_documents, y=None)

Parameters: raw_documents : iterable
an iterable which yields either str, unicode or file objects