JohnTitor JohnTitor - 7 months ago 11
Python Question

Creating a huge amount of objects(neuron) and connecting randomly using dictionaries

I'm experimentally trying to create a new kind of neural network with these criterias:


  • Each neuron must be a separate object.

  • Each neuron should have its own thread.

  • Network must be connected partially and randomly (at startup).

  • Neurons have to run asynchronously for calculating its output, updating its weights etc.



These are my implementation attempts in Julia and Python:

Python

import random
import itertools
import time
import signal
from threading import Thread
from multiprocessing import Pool
import multiprocessing

POTENTIAL_RANGE = 110000 # Resting potential: -70 mV Membrane potential range: +40 mV to -70 mV --- Difference: 110 mV = 110000 microVolt --- https://en.wikipedia.org/wiki/Membrane_potential
ACTION_POTENTIAL = 15000 # Resting potential: -70 mV Action potential: -55 mV --- Difference: 15mV = 15000 microVolt --- https://faculty.washington.edu/chudler/ap.html
AVERAGE_SYNAPSES_PER_NEURON = 8200 # The average number of synapses per neuron: 8,200 --- http://www.ncbi.nlm.nih.gov/pubmed/2778101

# https://en.wikipedia.org/wiki/Neuron

class Neuron():

neurons = []

def __init__(self):
self.connections = {}
self.potential = 0.0
self.error = 0.0
#self.create_connections()
#self.create_axon_terminals()
Neuron.neurons.append(self)
self.thread = Thread(target = self.activate)
#self.thread.start()
#self.process = multiprocessing.Process(target=self.activate)

def fully_connect(self):
for neuron in Neuron.neurons[len(self.connections):]:
if id(neuron) != id(self):
self.connections[id(neuron)] = round(random.uniform(0.1, 1.0), 2)

def partially_connect(self):
if len(self.connections) == 0:
neuron_count = len(Neuron.neurons)
for neuron in Neuron.neurons[len(self.connections):]:
if id(neuron) != id(self):
if random.randint(1,neuron_count/100) == 1:
self.connections[id(neuron)] = round(random.uniform(0.1, 1.0), 2)
print "Neuron ID: " + str(id(self))
print " Potential: " + str(self.potential)
print " Error: " + str(self.error)
print " Connections: " + str(len(self.connections))

def activate(self):
while True:
'''
for dendritic_spine in self.connections:
if dendritic_spine.axon_terminal is not None:
dendritic_spine.potential = dendritic_spine.axon_terminal.potential
print dendritic_spine.potential
self.neuron_potential += dendritic_spine.potential * dendritic_spine.excitement
terminal_potential = self.neuron_potential / len(self.axon_terminals)
for axon_terminal in self.axon_terminals:
axon_terminal.potential = terminal_potential
'''
#if len(self.connections) == 0:
# self.partially_connect()
#else:
self.partially_connect()
pass

'''
if abs(len(Neuron.neurons) - len(self.connections) + 1) > 0:
self.create_connections()

if abs(len(Neuron.neurons) - len(self.axon_terminals) + 1) > 0:
self.create_axon_terminals()
'''

class Supercluster():

def __init__(self,size):
for i in range(size):
Neuron()
print str(size) + " neurons created."
self.n = 0
self.build_connections()
#pool = Pool(4, self.init_worker)
#pool.apply_async(self.build_connections(), arguments)
#map(lambda x: x.partially_connect(),Neuron.neurons)
#map(lambda x: x.create_connections(),Neuron.neurons)
#map(lambda x: x.create_axon_terminals(),Neuron.neurons)

def build_connections(self):
for neuron in Neuron.neurons:
self.n += 1
#neuron.thread.start()
neuron.partially_connect()
print "Counter: " + str(self.n)

Supercluster(10000)


Julia

global neurons = []

type Neuron
connections::Dict{UInt64,Float16}
potential::Float16
error::Float16

function Neuron(arg1,arg2,arg3)
self = new(arg1,arg2,arg3)
push!(neurons, self)
end

end

function fully_connect(self)
for neuron in neurons
if object_id(neuron) != object_id(self)
self.connections[object_id(neuron)] = rand(1:100)/100
#push!(self.connections, rand(1:100)/100)
end
end
end

function partially_connect(self)
if isempty(self.connections)
neuron_count = length(neurons)
for neuron in neurons
if object_id(neuron) != object_id(self)
if rand(1:neuron_count/100) == 1
self.connections[object_id(neuron)] = rand(1:100)/100
#push!(self.connections, rand(1:100)/100)
end
end
end
println("Neuron ID: ",object_id(self))
println(" Potential: ",self.potential)
println(" Error: ",self.error)
println(" Connections: ",length(self.connections))
end
end

function Build()
for i = 1:10000
Neuron(Dict(),0.0,0.0)
end
println(length(neurons), " neurons created.")
n = 0
@parallel for neuron in neurons
n += 1
partially_connect(neuron)
println("Counter: ",n)
end
end

Build()


Firstly, these parts that are making connections between each neuron partially and randomly, taking too much time. How can I speed up this process/part?

Python

def build_connections(self):
for neuron in Neuron.neurons:
self.n += 1
#neuron.thread.start()
neuron.partially_connect()
print "Counter: " + str(self.n)


Julia

n = 0
@parallel for neuron in neurons
n += 1
partially_connect(neuron)
println("Counter: ",n)


Secondly, is that a good idea to give each neuron, its own thread when my goal is creating at least a million neuron? It means it will be like a million thread.

What I'm trying to do here is imitating the biological neural networks in the strict sense, instead of using matrix calculations.

ADDITION:

New version of
partially_connect
function according to answer:

def partially_connect(self):
if len(self.connections) == 0:
neuron_count = len(Neuron.neurons)
#for neuron in Neuron.neurons:
elected = random.sample(Neuron.neurons,100)
for neuron in elected:
if id(neuron) != id(self):
#if random.randint(1,neuron_count/100) == 1:
self.connections[id(neuron)] = round(random.uniform(0.1, 1.0), 2)
print "Neuron ID: " + str(id(self))
print " Potential: " + str(self.potential)
print " Error: " + str(self.error)
print " Connections: " + str(len(self.connections))


Performance dramatically increased.

Answer

Just looking at this code:

def partially_connect(self):
    if len(self.connections) == 0:
        neuron_count = len(Neuron.neurons)
        for neuron in Neuron.neurons[len(self.connections):]:
            if id(neuron) != id(self):
                if random.randint(1,neuron_count/100) == 1:
                    self.connections[id(neuron)] = round(random.uniform(0.1, 1.0), 2)

And based on your reply to my comment on the OP, here's a couple of things:

  1. You are making a copy of the lists when you use syntax like L[0:]. The slice syntax is making a shallow copy of the Neuron.neurons array for each call to your function. That's an O(n) operation, and since you call partially_connect once for each neuron in your build_connections function, that makes it O(n²). (Yikes!)

  2. You are doing work in Python that can and should be done in the library (in C, we hope!). Have a look at e.g. the random.paretovariate() and random.sample() functions. You could easily compute num_connections = random.paretovariate(1.0) * 100 and then say connected_nodes = random.sample(neurons, num_connections). Filter out self from the connected_nodes and you're done.

I think you can get a big performance boost by eliminating n² behavior and by using the built-in library routines.

ADDITION

Responding to your addition, consider this:

def partially_connect(self):
    if len(self.connections) == 0:
        elected = random.sample(Neuron.neurons,100)
        try:
            elected.remove(self)
        except ValueError:
            pass

        for neuron in elected:
            self.connections[id(neuron)] = round(random.uniform(0.1, 1.0), 2)

(I'm ignoring the prints for now.)

I don't know how you would communicate from a neuron to its connected neurons, without iterating all the neurons looking for a match of id() values. I'd suggest you store a reference to the connect objects as the key, and use the weight as the value:

self.connections = [n:round(random.uniform(0.1, 1.0), 2) for n in elected]

This assumes you need to traverse the links from source to target, of course.

As for threading solutions, I don't have a good suggestion. A little googling leads me to some old email threads (heh!) that mention numbers like 405 and 254 as being thread creation limits. I haven't seen any documents saying "Python threading is now UNLIMITED!" or whatever, so I suspect you're going to have to alter the way you implement your solution.

Comments