user3813234 user3813234 - 24 days ago 16
Python Question

sklearn pipeline fit: AttributeError: lower not found

I would like to use a pipeline in sklearn, like this:

corpus = load_files('corpus/train')

stop_words = [x for x in open('stopwords.txt', 'r').read().split('\n')] # Uppercase!

countvec = CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))

X_train, X_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.9,
random_state=0)
x_train_counts = countvec.fit_transform(X_train)
x_test_counts = countvec.transform(X_test)

k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])

pipeline = Pipeline([
('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
('classifier', MultinomialNB()) ])

for train_indices, test_indices in k_fold:

pipeline.fit(x_train_counts, y_train)
predictions = pipeline.predict(x_test_counts)


However, I am getting this error:

AttributeError: lower not found


I have looked at this post:

AttributeError: lower not found; using a Pipeline with a CountVectorizer in scikit-learn

but I am passing a list of bytes to the vectorizer, so that shouldnt be the problem.

EDIT

corpus = load_files('corpus')

stop_words = [x for x in open('stopwords.txt', 'r').read().split('\n')]

X_train, X_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.5,
random_state=0)

k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])

pipeline = Pipeline([
('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
('classifier', MultinomialNB())])

for train_indices, test_indices in k_fold:
pipeline.fit(X_train[train_indices], y_train[train_indices])
predictions = pipeline.predict(X_test[test_indices])


Now I get the error:

TypeError: only integer arrays with one element can be converted to an index


2ND EDIT

corpus = load_files('corpus')

stop_words = [y for x in open('stopwords.txt', 'r').read().split('\n') for y in (x, x.title())]

k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])

pipeline = Pipeline([
('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
('classifier', MultinomialNB())])

for train_indices, test_indices in k_fold:
pipeline.fit(corpus.data, corpus.target)

Ale Ale
Answer

You are not using correctly the pipeline. You don't need to pass the data vectorized, the idea is that the pipeline vectorizes the data.

# This is done by the pipeline
# x_train_counts = countvec.fit_transform(X_train)
# x_test_counts = countvec.transform(X_test)

k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])

pipeline = Pipeline([
    ('vectorizer',  CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
    ('classifier',  MultinomialNB()) ])

# also you are not using the indices...
for train_indices, test_indices in k_fold:

    pipeline.fit(corpus.data[train_indices], corpus.target[train_indices])
    predictions = pipeline.predict(corpus.data[test_indices])
Comments