Mithril Mithril - 1 year ago 989
Python Question

sklearn classifier get ValueError: bad input shape

I have a csv, struct is

, CAT1, CAT2, TITLE ,CONTENT are in chinese.

I want train
with X(TITLE) and feature(CAT1,CAT2), both get this error. below is my code:

PS: I write below code through this example scikit-learn text_analytics

import numpy as np
import csv
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline

label_list = []

def label_map_target(label):
''' map chinese feature name to integer '''
idx = label_list.index(label)
except ValueError:
idx = len(label_list)

return idx

c1_list = []
c2_list = []
title_list = []
with open(csv_file, 'r') as f:
# row_from_csv is for shorting this example
for row in row_from_csv(f):

data = np.array(title_list)
target = np.array([c1_list, c2_list])
print target.shape
# (2, 4405)
target = target.reshape(4405,2)
print target.shape
# (4405, 2)

docs_train, docs_test, y_train, y_test = train_test_split(
data, target, test_size=0.25, random_state=None)

# vect = TfidfVectorizer(tokenizer=jieba_tokenizer, min_df=3, max_df=0.95)
# use custom chinese tokenizer get same error
vect = TfidfVectorizer(min_df=3, max_df=0.95)
docs_train= vect.fit_transform(docs_train)

clf = LinearSVC(), y_train)


ValueError Traceback (most recent call last)
<ipython-input-24-904eb9af02cd> in <module>()
1 clf = LinearSVC()
----> 2, y_train)

C:\Python27\lib\site-packages\sklearn\svm\classes.pyc in fit(self, X, y)
199 X, y = check_X_y(X, y, accept_sparse='csr',
--> 200 dtype=np.float64, order="C")
201 self.classes_ = np.unique(y)

C:\Python27\lib\site-packages\sklearn\utils\validation.pyc in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric)
447 dtype=None)
448 else:
--> 449 y = column_or_1d(y, warn=True)
450 _assert_all_finite(y)
451 if y_numeric and y.dtype.kind == 'O':

C:\Python27\lib\site-packages\sklearn\utils\validation.pyc in column_or_1d(y, warn)
483 return np.ravel(y)
--> 485 raise ValueError("bad input shape {0}".format(shape))

ValueError: bad input shape (3303, 2)

Answer Source

Thanks to @meelo, I solved this probelm. As he say, in my code, data is feature vector, target is target value. I mixed up two things.

I learned TfidfVectorizer process data to [data, feature] , and each data should map to just one target (C1 or C2) if I want to predict two type targets,

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download