We now have skeletal classes for neurons and synapses. There is one item we need to add to the synapse class. I will pretend like I intentionally left it out even though the reality is that I just forgot to add it.
Since the synapse "belongs to" the neuron it is a member of, that neuron knows where to find it. But, it "comes from" some other neuron. I forgot to add a reference to the "comes from" neuron. How should we go about it?
So, a human brain has around a hundred billion neurons and 100 trillion synapses. I can't afford a computer big enough to store that many. If we figure that each neuron has about a thousand synapses, it is clear that the number of synapses will dominate the storage requirements. As the synapse class is currently defined it takes 8 bytes. An average PC with 8GB of RAM should be able to hold somewhere around a billion synapses, or a million neurons. We will see later that neurons are often arranged in nice, neat layers, but there are some connections into and out of those layers to faraway places. It is probably best if we just assume than, in general, any neuron may connect to any other. To reference any other neuron, we will need numbers up to about a million. A sixteen bit integer won't get it. A 32 bit int holds up to 4 billion +, which is overkill, but it's a standard data type. We will use that for now. We can change it later if we need to. So, in another brilliant demonstration of my variable-naming skills, let's call this one "comesFrom."
Our neuron still won't do anything. Let's fix that. A synapse is only of interest to the neuron it belongs to. Our interface to the outside world will be through the neuron. If we need access to the synapse (we will) from the outside world, we will achieve it by going through the neuron. Let's add some functions to the neuron class.
For now, the default constructor is all we need. If we are making tens of thousands of neurons, it doesn't make much sense to hardcode the values that go into each one as a line of code. We will end up creating a config file that sets all the values and connections. So let's add some functions that set our values.
Each neuron has a threshold value that needs to be configured. We'll add a function to set it, "setThreshold(float thresh);". Since a neuron can't do anything useful without synapses, we need to be able to add them. So let's add a function called "addSynapse(float weight, uint32_t connection);".
Of course, the neuron is useless if it can't output some results to the world. To find out if the neuron is firing we add "bool isFiring();" to the neuron class.
That should cover teh neuron for now. We will want to add some features later, though. Now what do we need to add to the Synapse class?
When we add a synapse to a neuron, we need to construct a new synapse using the passed-in weight and connection. So we need a constructor: "Synapse(float wt, uint32_t conn);". The neuron it belongs to will need to find out the current weighted value, so we need a function to get that: "float getWeightedValue();" When the neuron fires, it needs to be able to set the input values of all synapses to zero: "void clearValue()" should do. Again, we will probably want to add to this later, but that should be enough to get us started.
Now that we have neurons and synapses to connect them together, we can create a neural network. But we still have to add the functionality to make it work. We will get to that in just a moment, but first we need to look at how the neuron actually "calculates."
Remember we said the signals coming into a neuron are pulses. In a biological neuron a single pulse lasts around one millisecond, but the rate they arrive can vary from 0 to around 500 per second. When a pulse arrives at a synapse, it "charges" a storage element in the neuron. The charge will remain after the pulse goes away, but it will decay over time. If you are familiar with electronics, it is much like charging a capacitor which will then discharge over time. Because the pulse is stored for a short time, pulses arriving on separate synapses at different times can still act together inside the neuron. Neurons are often compared to logic gates, but this is one example where they are very different.
Inside the neuron, the stored value is mulitplied by the weight associated with that synapse. The stored value is always positive, but the weight can be positive or negative. All of the multiplied values are summed together. If the total sum is greater than the neuron's threshold value, the neuron will fire (send out a single pulse) and the stored values will all be reset back to zero.
Let's write some code to make this happen. An actual brain and nervous system is a continuous-time device, but our computer is discrete-time. We have to make some accommodations for that. We already added two output fields to our neuron class for that reason. Our approach will be a two-step process. First, we will go through all the neurons and compare all the current inputs to the threshold and update the output value into the "newOutput" field. All those inputs will come from the "oldOutput" field. Once we have updated all the neurons, we will go back through the list and move all the "newOutput" values to the "oldOutput" field. In effect we are calculating all the updated values with the current inputs while we update those inputs for the next round, but we store them separately so that the entire process "appears" to happen simultaneously. In computer graphics this process is known as "double buffering." It allows us to take time to update the outputs but have it appear to happen instantaneously.
We will add two functions to the neuron class: "calculate()" and "update()". calculate() will run first, calculating the new output value based on all the old output values as inputs. After all the new values are calculated, update() will copy the new value to the oldOutput variable;.
The update function is really simple; just assign newOutput to oldOutput. The calculate function is more involved. The function needs to step through all the synapses, multiplying their input values by the synapse weight, then summing all those together. If the sum is equal to or exceeds the neuron's threshold value, the neuron fires (the newOutput field is set to true) and all the synapse values are set back to zero. Keep in mind that the input value varies depending on whether the input neuron is currently firing or has fired in the recent past. So, the first thing we do is check if the input is currently firing. If it is, we set the input value to 1.0 and use that. If it is not currently firing, we "decay" the stored value (since a single time increment has passed since the last time we checked it) and use the new "decayed" value.
There are two things I am uncertain of mentioned in the last paragraph. First, when an input neuron fires, does the value go directly to 1.0, or does it take some number of pulses to "charge" it, much like the time to discharge. I think it charges fully on a single pulse, and that is how we will write it. But if we find out later it works differently, we may need to change it. The other uncertainty is the discharge rate. I suspect it is an exponential function based on e but it could be something different. I don't think that will make much difference, as long as we are in the ballpark. For now, I will use an exponential based on 2, which should be close enough. I chose two because we only need to halve the value each iteration, which is quick and easy.
There is another minor issue with our neuron. A biological neuron won't fire continuously; there will be a gap of varyint time between output pulses. However, that gap can be very short if the inputs to the neuron are strongly activated. I don't know what the minimum gap is. But because of the discrete nature of our software neuron, it will fire for a length of time equal to one pass through the update cycle and the gap between firings must be at least one cycle long as well. That will limit the firing rate and the limit may very well be lower than the max rate of a biological neuron. I don't think it will cause any problems, though.
Enough jabbering, let's get on with it! Here are the steps we need to take:
SUM = 0 // Set the neuron value to 0
if currently firing // If it fired last pass, can't fire again
clear output // So reset everything
for all synapses // Including all the stored values
set value to 0
else // If it didn't fire, see if it does now
For all synapses // step through all the synapses
If input is firing // Check if input is firing now
value = 1.0 // If so, set value to 1.0
Else
value = value / 2.0 // Decaly old value
SUM = SUM + value * weight // add in this weighted value
if SUM > threshold and not currently firing
set new output true
else
set new output false
And that is how our neurons and synapses will work. I think that is
enough for now; my brain hurts. So let's pause here. Later, after I
get all this code written I will post an Eclipse project we can use.
| Previous | Contents | Next |
