Ajay H - 7 months ago 32
Python Question

# sklearn pipeline model predicting same results for all input

In the program, I am scanning a number of brain samples taken in a time series of 40 x 64 x 64 images every 2.5 seconds. The number of 'voxels' (3D pixels) in each image is thus ~ 168,000 ish (40 * 64 * 64), each of which is a 'feature' for an image sample.

I thought of using Recursive Feature Elimination (RFE). Then follow this up with Principle Component Analysis (PCA) because of the rediculously high n to perform dimensionality reduction.

There are 9 classes to predict. Thus a multi class classification problem. Starting with RFE:

``````estimator = SVC(kernel='linear')
rfe = RFE(estimator,n_features_to_select= 20000, step=0.05)
rfe = rfe.fit(X_train,y_train)
X_best = rfe.transform(X_train)
``````

Now perform PCA :

``````X_best = scale(X_best)

def get_optimal_number_of_components():
cov = np.dot(X_best,X_best.transpose())/float(X_best.shape[0])
U,s,v = svd(cov)
print 'Shape of S = ',s.shape

S_nn = sum(s)

for num_components in range(0,s.shape[0]):
temp_s = s[0:num_components]
S_ii = sum(temp_s)
if (1 - S_ii/float(S_nn)) <= 0.01:
return num_components

return s.shape[0]

n_comp = get_optimal_number_of_components()
print 'optimal number of components = ', n_comp

pca = PCA(n_components = n_comp)
pca = pca.fit(X_best)
X_pca_reduced = pca.transform(X_best)
``````

Train the reduced component dataset with SVM

``````svm = SVC(kernel='linear',C=1,gamma=0.0001)
svm = svm.fit(X_pca_reduced,y_train)
``````

Now transform the training set to RFE-PCA reduced and make the predictions

``````X_rfe = rfe.transform(X_test)
X_pca = pca.transform(X_rfe)

predictions = svm.predict(X_pca)

print 'predictions = ',predictions
print 'actual = ',y_test
``````

But I always get the same values for prediction for every sample!

``````predictions =  [2 2 2 2 2 2 2 2 2 2 2 2 2] #Why is it the same?!!
actual = actual =  [[0]
[0]
[6]
[8]
[4]
[5]
[0]
[6]
[2]
[3]
[0]
[5]
[6]]
``````

I made sure to use linear kernel. I also considered modifying C,gamma with values like (1,0.001) (1,0.0001) (10, 0.001), (10, 0.0001) ... but I get the same output.

Is there something I'm missing here?

EDIT 1

I executed

``````print svm.decision_function(X_pca)
``````

and the output was :

``````[[ -4.79479982e+02  -8.01563453e+02  -9.91453849e+02  -1.34641884e+02
-1.02315530e+03  -2.88297991e+02  -8.41843812e+02   4.79807826e+02
-3.50485820e+02  -2.31081776e+02  -1.42555136e+02  -4.79034448e+02
-6.93029988e+01  -7.34288793e+02  -5.49271317e+01   1.98108304e+02
4.80257991e+02   2.46835670e+02   2.90045437e+02   1.53261114e+02
9.15742824e+02  -1.28387833e+01  -3.05045240e+02  -2.19268988e+01
-2.24896384e+02   1.44501465e+02  -9.17438352e+01  -6.96148972e+01
-1.15785658e+02   9.53878230e+01   1.79823987e+01   8.05242433e+01
8.33917960e+02  -1.69686889e+01   9.85949158e+01   2.68935397e+02]
[ -4.62804973e+02  -8.26454112e+02  -9.83214857e+02  -1.43367127e+02
-1.03538041e+03  -2.86397664e+02  -8.47539241e+02   4.63709033e+02
-3.52018380e+02  -2.49936725e+02  -1.43734219e+02  -4.79498907e+02
-6.93338619e+01  -7.51141272e+02  -5.30999658e+01   1.95687050e+02
4.69206888e+02   2.46530774e+02   2.92047409e+02   1.47934614e+02
9.27901865e+02  -1.21801344e+01  -2.99530129e+02  -2.03238750e+01
-2.26862390e+02   1.47692745e+02  -8.81396485e+01  -6.41692405e+01
-1.14247569e+02   1.01567350e+02   1.87874038e+01   6.90549126e+01
8.41984280e+02  -2.04488188e+01   1.00839951e+02   2.75459577e+02]
[ -4.49255185e+02  -7.89243063e+02  -9.78820180e+02  -1.29050171e+02
-1.01784356e+03  -3.23431625e+02  -7.98795796e+02   4.93058279e+02
-3.64674793e+02  -2.46545700e+02  -1.66933546e+02  -4.84571326e+02
-9.93316258e+01  -7.36182373e+02  -6.23110881e+01   2.08061873e+02
4.28119725e+02   2.75927668e+02   2.36425246e+02   1.69950273e+02
9.50488041e+02  -3.17986619e+01  -3.03656967e+02  -4.78710028e+01
-2.20752797e+02   1.36973850e+02  -5.31583763e+01  -1.08205173e+02
-7.94698530e+01   1.37320498e+02  -2.31183352e+01   8.41399154e+01
8.26408412e+02   1.30471236e+01   1.48266050e+02   2.55914495e+02]
[ -4.80424764e+02  -8.07660826e+02  -9.91911478e+02  -1.35981428e+02
-1.02923114e+03  -2.93372818e+02  -8.47420541e+02   4.60149182e+02
-3.48333176e+02  -2.37654055e+02  -1.39277819e+02  -4.78486235e+02
-6.83571401e+01  -7.34632739e+02  -5.73953318e+01   1.95508198e+02
4.80569807e+02   2.37500896e+02   2.89038289e+02   1.49855773e+02
9.09217973e+02  -1.04236971e+01  -3.02128880e+02  -2.16485093e+01
-2.23313869e+02   1.43686084e+02  -9.74071814e+01  -7.22417410e+01
-1.19091495e+02   8.94390723e+01   1.97000084e+01   8.08496457e+01
8.39105553e+02  -1.82282013e+01   9.82685256e+01   2.67791421e+02]
[ -4.74406707e+02  -8.18308535e+02  -9.76126419e+02  -1.74849565e+02
-1.02784293e+03  -2.96842934e+02  -8.42749406e+02   4.83769137e+02
-3.59483221e+02  -2.24264385e+02  -1.61995143e+02  -4.78030614e+02
-8.02309023e+01  -7.54316452e+02  -5.43436450e+01   2.05876768e+02
4.33470519e+02   2.67598191e+02   2.75764466e+02   1.53323191e+02
9.45967383e+02  -2.93192233e+01  -3.04615693e+02  -3.20731950e+01
-2.42783848e+02   1.40891844e+02  -6.13739832e+01  -6.15060481e+01
-9.51924850e+01   1.35666499e+02   2.41364468e+00   6.39635318e+01
8.37881867e+02  -1.03313421e+01   1.19234038e+02   2.76305651e+02]
[ -4.84321668e+02  -8.07444080e+02  -1.01507160e+03  -1.28529685e+02
-1.05601843e+03  -2.99493242e+02  -8.41745493e+02   4.75608122e+02
-3.37295601e+02  -2.49242183e+02  -1.30463265e+02  -4.74284269e+02
-6.05670230e+01  -7.34447396e+02  -4.01117838e+01   1.80948824e+02
4.80450158e+02   2.19859113e+02   2.94798893e+02   1.35958067e+02
9.13259527e+02  -3.52105914e-01  -2.92301811e+02  -1.24432589e+01
-2.13204265e+02   1.64167920e+02  -1.02951065e+02  -7.04800774e+01
-1.31293866e+02   9.12032854e+01   2.67291593e+01   7.78485633e+01
8.74745197e+02  -2.50250734e+01   9.69993408e+01   2.83018293e+02]
[ -4.68184798e+02  -7.85221871e+02  -9.98980941e+02  -1.08799100e+02
-1.02080996e+03  -2.87470373e+02  -8.29552725e+02   4.99360929e+02
-3.31724034e+02  -2.56603688e+02  -1.24320652e+02  -4.60348857e+02
-6.21852802e+01  -7.31782526e+02  -2.56669989e+01   1.74050279e+02
4.74370392e+02   2.26812613e+02   2.78945379e+02   1.29667612e+02
9.21512986e+02   3.74936721e+00  -2.77509203e+02  -1.34603952e+01
-2.12032693e+02   1.72842580e+02  -9.71967056e+01  -8.19354011e+01
-1.32985460e+02   9.55148610e+01   1.66381043e+01   5.88073445e+01
8.62770538e+02  -2.37682031e+01   1.06714435e+02   2.94158166e+02]
[ -4.63681347e+02  -8.22291452e+02  -9.98021515e+02  -1.54810425e+02
-1.03372001e+03  -3.34322759e+02  -8.34407336e+02   4.71050572e+02
-3.69327864e+02  -2.40580250e+02  -1.65003310e+02  -4.88818830e+02
-9.73775374e+01  -7.51246204e+02  -6.69606962e+01   2.13573607e+02
4.49817824e+02   2.79532473e+02   2.41873397e+02   1.69963589e+02
9.53153717e+02  -2.88140674e+01  -3.13030733e+02  -4.54555034e+01
-2.32589565e+02   1.36869994e+02  -6.33773098e+01  -1.06164181e+02
-8.91557438e+01   1.24881490e+02  -1.94528381e+01   7.98035685e+01
8.22835959e+02   8.75642083e+00   1.43002335e+02   2.61562868e+02]
[ -4.77620825e+02  -8.40698094e+02  -1.01067455e+03  -1.56851274e+02
-1.05031578e+03  -3.14666532e+02  -8.46541414e+02   4.61714738e+02
-3.60822150e+02  -2.44485564e+02  -1.53420660e+02  -4.85710648e+02
-7.77752216e+01  -7.55747678e+02  -5.87745617e+01   2.04601581e+02
4.68781099e+02   2.63234873e+02   2.86306284e+02   1.58817281e+02
9.43249321e+02  -1.87631625e+01  -3.06321663e+02  -2.78828679e+01
-2.27554363e+02   1.46508283e+02  -7.88844807e+01  -7.41051812e+01
-1.05094485e+02   1.12231546e+02   7.97692231e+00   7.67304852e+01
8.43518403e+02  -1.12844915e+01   1.13370158e+02   2.70797472e+02]
[ -4.91420429e+02  -7.90722180e+02  -1.05615447e+03  -1.20351520e+02
-1.04098604e+03  -2.92426682e+02  -8.45105853e+02   4.78228854e+02
-3.10412377e+02  -2.77543578e+02  -1.09733119e+02  -4.40834428e+02
-4.35168704e+01  -7.29088994e+02  -6.64581241e+00   1.48560861e+02
4.74565890e+02   2.07485677e+02   2.99817382e+02   1.09936148e+02
9.03346951e+02   2.26102442e+01  -2.45854761e+02   8.31279855e+00
-1.92441568e+02   2.03079787e+02  -1.05267244e+02  -6.41835912e+01
-1.49582656e+02   8.73008441e+01   3.36913246e+01   5.11061286e+01
8.79159912e+02  -3.85152954e+01   9.08938445e+01   3.04037825e+02]
[ -4.85998114e+02  -7.83944995e+02  -9.68132304e+02  -1.54631678e+02
-1.01186983e+03  -2.80419560e+02  -8.72211797e+02   4.97352635e+02
-3.56256101e+02  -2.23204297e+02  -1.55355470e+02  -4.80882457e+02
-7.86287112e+01  -7.58318471e+02  -5.10727433e+01   2.08265151e+02
4.49457388e+02   2.65764723e+02   2.72435473e+02   1.53296624e+02
9.44654406e+02  -2.50922419e+01  -3.17539501e+02  -3.16241295e+01
-2.51387679e+02   1.38109115e+02  -6.97122491e+01  -6.59836763e+01
-1.03441764e+02   1.19472073e+02   3.60256872e+00   6.22040523e+01
8.19929661e+02  -1.26581261e+01   1.12555974e+02   2.80480600e+02]
[ -4.70876215e+02  -7.87431621e+02  -9.96007256e+02  -1.30872700e+02
-1.03175439e+03  -2.94238915e+02  -8.36753617e+02   4.77420371e+02
-3.38091939e+02  -2.44272006e+02  -1.35130348e+02  -4.72973924e+02
-6.19636207e+01  -7.37123284e+02  -4.28620473e+01   1.80929974e+02
4.67912162e+02   2.22731582e+02   2.93578369e+02   1.34101279e+02
9.04139841e+02  -3.91744880e+00  -2.88182153e+02  -1.22493089e+01
-2.15621705e+02   1.59580065e+02  -9.57584381e+01  -6.41773592e+01
-1.28168370e+02   9.42107498e+01   2.61332125e+01   7.00130475e+01
8.58092989e+02  -2.62818439e+01   9.40455319e+01   2.82505159e+02]
[ -4.70908104e+02  -8.29375323e+02  -9.93882131e+02  -1.47050049e+02
-1.03443155e+03  -3.28570789e+02  -8.31014742e+02   4.92865993e+02
-3.70050739e+02  -2.35488125e+02  -1.63833070e+02  -4.86930191e+02
-9.74429858e+01  -7.48852374e+02  -6.17719584e+01   2.13942179e+02
4.52542022e+02   2.83202323e+02   2.43990105e+02   1.72094231e+02
9.65225890e+02  -2.92801036e+01  -3.13220814e+02  -4.60705452e+01
-2.32787033e+02   1.38783264e+02  -6.23061347e+01  -1.05977672e+02
-8.75333469e+01   1.31424380e+02  -1.99414766e+01   7.97712157e+01
8.30620576e+02   9.19139268e+00   1.44727040e+02   2.65196706e+02]]
``````

So, the values differ (though slightly) for every sample. I assume the model is doing something. I just dont know whats wrong.