Fcoder Fcoder - 14 days ago 10
Python Question

Sigmoid function for large numbers in Python

Im doing a Neural network task with

Sigmoid
activation function. My networks input is image(
MNIST
dataset) and because dimension of each image is
28*28
, When I have to convert them to a vector, i will have
N*784
matrix. Multiplication of this large matrix with weight matrix produce large positive and negative numbers for weights and i have to pass them to a
Sigmoid
function. I use
expit()
as sigmoid function and my problem is that:

numbers until 30 resulting near 1 in
expit()
. for example
expit(28)
results
0.99999999
and
expit(29)
results
1.0
and upper
29
also gets
1
. But my new wights are upper
30
and because of that some of them gets
1
and some 0 in the first cycle of learning and in fact there is no any learning at all.

What i have to do?
Sigmoid's
upper bound is
29
? i cant change it? I have to change my image dimension to overcome this?

Answer

As discussed in comments section, the real problem turned out to be using sigmoid itself, which is not suited for such cases. In any finite-precision calculations one will face the described problem, one system with 29, on other with 38.

One way to tackle the problem would be use softmax activation function, which is less susceptible to such issues. Mind that with cost function you might encounter similar challenges.

Slightly off-topic, ut you might want to check how the problem is resolved with e.g. tensorflow. It has some nice tutorials for beginners.