Lin K Lin K - 9 months ago 29
Python Question

Python linear least squares function not working

Ok, so I'm writing a function for linear least squares in python and it's pretty much just one equation. Yet for some reason, I'm getting a ValueError. My best guess is it has something to do with the

.reshape
function, since in this question I had a very similar problem and reshaping was the solution. I've read up on it and from what I gather, w in my function is in format (n,) and the result would be in (n,1) as in my previously mentioned question. I tried reshaping
x_train
and
y_train
but I only got an error that I can't change the size of the array. I guess my parameters were set wrong. Right now I'm lost, and I have many more functions like these to go through -- I wish I could understand what am I missing in my code. The equation seems to be in order, so I suppose there's something I should be adding everytime - possibly the
reshape
function cause I'm still using the same models as in the last situation. I hope it's the right place to post this question, I don't know what else to do but I really want to understand so I won't have these problems in the future, thank you.

Code (np. stands for numpy):

def least_squares(x_train, y_train, M):
'''
:param x_train: training input vector Nx1
:param y_train: training output vector Nx1
:param M: polynomial degree
:return: tuple (w,err), where w are model parameters and err mean squared error of fitted polynomial
'''
w = np.linalg.inv(design_matrix(x_train, M). * design_matrix(x_train, M)) * design_matrix(x_train, M).T * y_train
err = mean_squared_error(x_train, y_train, w)
return (w, err)


design_matrix
and
mean_squared_error
are working just fine.
Traceback:

ERROR: test_least_squares_err (test.TestLeastSquares)
----------------------------------------------------------------------
Traceback (most recent call last):
File "\content.py", line 48, in least_squares
w = np.linalg.inv(design_matrix(x_train, M).T * design_matrix(x_train, M)) * design_matrix(x_train, M).T * y_train
ValueError: operands could not be broadcast together with shapes (7,20) (20,7)

Answer Source

Assuming that design_matrix returns a matrix, this code

design_matrix(x_train, M).T * design_matrix(x_train, M)

most likely does not do what is intended since * is performing element-wise multiplication (Hadamard product of two matrices). Because your matrices are not square, it thus complains about incompatible shape.

To obtain matrix-matrix product, one might do (assuming numpy was imported as import numpy as np):

np.dot(design_matrix(x_train, M).T, design_matrix(x_train, M))

Similar reasoning then applies to the rest of the statement * design_matrix(x_train, M).T * y_train...

Also, you might want to evaluate design_matrix only once, e.g., to put something like

mat = design_matrix(x_train, M)

before the line calculating w and then use merely mat.