Makaroniiii - 3 years ago 94

Python Question

`import numpy as np`

np.random.seed(0)

a = np.random.randint(1,100, size= 1000).reshape(1000,1)

b = np.random.randint(0,2, size=1000).reshape(1000,1)

y = np.where(b==0,a*2, a*3)

X = np.hstack((a,b))

y = y

from sklearn.preprocessing import StandardScaler

sx = StandardScaler()

X = sx.fit_transform(X)

sy = StandardScaler()

y = sy.fit_transform(y)

w0 = np.random.normal(size=(2,1), scale=0.1)

for i in range(100):

input_layer = X

output_layer = X.dot(w0)

error = y - output_layer

square_error = np.sqrt(np.mean(error**2))

print(square_error)

w0+= input_layer.T.dot(error)

If i understand correctly, linear activation function is always f(x) = x.

If you check this code, you'll see square error is growing and growing, I have no idea how to solve this simple linear problem with NN. I am aware there are other models and libraries, however I am trying to do it this way.

Recommended for you: Get network issues from **WhatsUp Gold**. **Not end users.**

Answer Source

You did not incorporate learning rate (see here and a more formal discussion here) into your model. When you train your network, you need to choose a learning rate parameter as well, and it has a big impact on whether your loss will decrease and how fast it converges.

By setting

```
w0+= input_layer.T.dot(error)
```

you chose the learning rate to be 1, which turned out to be too large. If instead you set

```
w0+= 0.0005*input_layer.T.dot(error)
```

(that is, choose learning rate 0.0005) the loss will decrease:

```
1.0017425183
0.521060951473
0.303777564629
0.21993949808
0.193933601196
0.18700323975
0.185262617455
0.184832603515
0.184726763539
.
.
.
```

It won't converge to 0 though, as your model is not linear. In the end the weight `w0`

that you get is

```
array([[ 0.92486712],
[ 0.318241 ]])
```

Recommended from our users: **Dynamic Network Monitoring from WhatsUp Gold from IPSwitch**. ** Free Download**