BIFURCATION AND NONLINEAR
DYNAMICS IN NEURAL NETWORK
By
MOAYED DANESHYARI
Bachelor of Science in Electrical Engineering
Sharif University of Technology
Tehran, Iran
1995
Submitted to the Faculty of the
Graduate College of the
Oklahoma State University
in partial fulfillment of
the requirements for
the Degree of
MASTER OF SCIENCE
May, 2008
ii
BIFURCATION AND NONLINEAR
DYNAMICS IN NEURAL NETWORK
Thesis Approved:
Dr. Paul A. Westhaus
Thesis Adviser
Dr. John W. Mintmire
Committee Member
Dr. John P. Chandler
Committee Member
Dr. A. Gordon Emslie
Dean of the Graduate College
iii
TABLE OF CONTENTS
Chapter Page
I. INTRODUCTION ......................................................................................................1
1.1 Dynamical Systems ............................................................................................1
1.1.1 Equilibrium Points .................................................................................2
1.1.2 Periodic Solutions .................................................................................2
1.1.3 Quasiperiodic Solutions .......................................................................2
1.2 Lyapunov Exponents .........................................................................................2
1.3 Dimension ..........................................................................................................4
1.3.1 Capacity Dimension (fractal dimension) ...............................................4
1.3.2 Information Dimension .........................................................................4
1.3.3 Correlation Dimension ..........................................................................5
1.3.4 Lyapunoy Dimension ............................................................................6
II. REVIEW OF LITERATURE
2.1 Neuron’s Model .................................................................................................7
2.2 Chaos in a Biological Neuron ..........................................................................10
2.3 Network with Chaotic Elements ......................................................................11
2.4 Transient Chaotic Neural Network ..................................................................14
2.4.1 Transient chaotic dynamics in a single neuron ...................................15
2.4.2 Discussion ...........................................................................................16
2.5 Associative Dynamics in chaotic Neural Network ..........................................18
2.5.1 Associative neural network with chaotic neurons ...............................18
2.5.2 Dependence of recognition process on refractory k ...................................19
2.5.3 Dynamic dependence on initial conditions .........................................20
2.6 Discussion ........................................................................................................24
III. SINUSOIDAL CHAOTIC NEURAL NETWORK
3.1 Introduction ......................................................................................................25
3.2 Neural Network with Logistic Chaotic Elements ............................................26
3.3 Cluster Attractors .............................................................................................27
3.4 Characteristics of Neural Network with Chaotic Logistic Map .......................28
iv
Chapter Page
3.5 Application on Memory and Pattern Recognition .................................................29
3.5.1 Associative memory by controlling a ...............................................29
3.5.2 Associative memory by controlling e ................................................31
3.6 Proposed Network: Developing a Neural Network with Sinusoidal Chaotic
Elements .................................................................................................................35
3.7 Performance of the Sinusoidal Chaotic Neural Network in Pattern
Recognition ...........................................................................................................41
3.8 Memory Capacity of Sinusoidal Chaotic Neural Network ...............................45
3.9 Discussion .........................................................................................................47
IV. NONLIPSCHITZIAN NEURAL NETWORK ....................................................48
4.1 Introduction ......................................................................................................48
4.2 NonLipschitzian Dynamics ...........................................................................49
4.3 NonLipschitzian Neural Networks .................................................................54
4.3.1 Investigating performance of nonLipschitzian neural network for
pattern recognition ........................................................................................57
4.3.2 Discussion ...........................................................................................62
4.4 Proposed Network: NonLipschitzian Neural Network with Repellers and
Attractors................................................................................................................62
4.4.1 Modification of the nonLipschitzian Neural Network .......................63
4.4.2 Investigating on pattern Recognition and Memory .............................64
4.5 NonLipschitzian Learning ..............................................................................65
4.5.1 NonLipschitzian dynamics in learning process .................................65
4.5.2 Dynamic sigmoid steepness during learning process ..........................69
4.5.3 Performance of nonLipschitzian learning process .............................69
4.6 Discussion ........................................................................................................72
V. CONCLUSION ......................................................................................................74
REFERENCES ............................................................................................................79
v
LIST OF TABLES
Table Page
Table 1.1 Lyapunov exponents for different types of steady state ............................3
Table 1.2 Dimensions for different types of steady state ............................................5
Table 2.1 Number of passing from each stored pattern in the network ...................21
Table 2.2 Investigating dependence of dynamics on initial conditions ...................22
Table 2.3 Retrieve frequency with different initial condition ...................................23
Table 3.1 Comparison results for memory capacity .................................................46
Table 4.1 Results of recognition of noisy patterns P1, P2, P3, P4, and P5 ...............64
Table 4.2 Steady state error for convergence of different patterns in 10 classes ......68
Table 4.3 Percentage for recognition of the patterns from different class in (431) .68
Table 4.4 Percentage for recognition of the patterns from different class in (429) .68
vi
LIST OF FIGURES
Figure Page
Fig2.1 Response characteristics vs. bifurcation parameter a when y(0)=0.1 .............9
Fig 2.2 Response characteristics vs. bifurcation parameter a when y(0)=0.5 ...........12
Fig 2.3 A neuron model in chaotic neural network ...................................................14
Fig 2.4 Time series of the output, x, of the chaotic neuron, and time series of the
bifurcation parameter, z, for a single neuron ...............................................................17
Fig 2.5 Four pattern stored in the network ..................................................................20
Fig 2.6 Recognition of disturbed pattern C when .......................................................22
Fig 2.7 Retrieving patterns ..........................................................................................24
Fig 2.8 Disturbed patterns. ..........................................................................................25
Fig 3.1 (a) Asymmetric Logistic function, (b) Symmetric Logistic function .............27
Fig 3.2 Bifurcation diagram: (a) secondorder map, (b) Thirdorder map..................28
Fig 3.3 Fourcluster attractor: (a) For t=1,2,3…, (b) For t=2,4,6 ................................29
Fig 3.4 Five 100bit binary sample pattern ..................................................................31
Fig 3.5 Pattern recognition process with 40% noise in the SGCM model
controlled by ................................................................................................................33
Fig 3.6 SGCM model controlled by ..........................................................................34
Fig 3.7 Pattern recognition process with 40% noise in the SGCM model
controlled by ................................................................................................................35
Fig 3.8 SGCM model controlled ...............................................................................36
Fig 3.9 Sinusoidal chaotic map ....................................................................................37
vii
Figure Page
Fig 3.10 Pattern recognition process with 40% noise in sinusoidal chaotic net ........40
Fig 3.11 Sinusoidal chaotic neural network .................................................................41
Fig 3.12 Fifteen training pattern each consisting 100 binary bits ................................42
Fig 3.13 Comparison of pattern recognition in 4 different system ..............................44
Fig 3.14 Three sets of 20 bit pattern in (a), (b) and (c) ................................................47
Fig 4.1: (a) Unlimited convergence time in Lipschitzian dynamics, (b) Limited
convergence time in nonLipschitzian dynamics .........................................................52
Fig 4.2 Asymptotic stability of
for
around , (a) For , and , (b) and .........54
Fig 4.3 Schema of a system with several attractors and repellers ...............................55
Fig 4.4 Trajectory of the attraction of different patterns to the equilibrium points .....60
Fig 4.5 Change of basin of attraction of the equilibrium point with the change
of
: (a) First neuron, (b) Second neuron .................................................................62
4.6 Moving of the bifurcation surface of the network with two neurons with
patterns P1 and P2. .......................................................................................................63
Fig 4.7 Convergence of the network of equation (431) due to pattern
recognition of samples of different class .....................................................................69
Fig 4.8 Increasing learning rate by increasing parameter from
to
and
.........................................................................................................................73
Fig 4.9 Time series of the network with : (a) Energy function, E, (b) ,
(c) Energy function with constraints,
, (d)Network’s connection coefficient, .....74
4.10 Time series of the network with : (a) Energy function, E, (b) ,
(c) Energy function with constraints,
, (d)Network’s connection coefficient, .....75
1
CHAPTER 1
INTRODUCTION
Nonlinear systems have played an important role in the study of natural phenomena. Unlike
linear systems where knowledge of the eigenvalues and eigenvectors allows one to write a
closedform solution, few nonlinear systems possess closedform solutions and, therefore,
numerical simulations play a crucial role in the process of finding and analyzing nonlinear
phenomena. One of the phenomena in the context of nonlinear system is chaos. Most of
deterministic systems are predictable: given the initial condition and the equations describing
system, the behavior of the system can be predicted for all time. Chaotic system is a
deterministic system that exhibits unpredictable behavior.
1.1 Dynamical Systems
Let’s consider an nthorder dynamical system:
!
, !"
" (11)
The system is linear if f is linear with respect to x. Dynamical systems are classified in terms of
their steadystate solutions and limit sets. Steadystate refers to the asymptotic behavior as
! # . It is required that the steady state be bounded. The difference between the solution and its
steady state is called the transient. A point y is a limit point of x if, for every neighborhood U of
y, the solution repeatedly enters U as ! # . The set of all limit points of x is called the limit set
2
L(x) of x. Limit sets are useful for categorizing the classical types of steadystate behavior (e.g.,
equilibrium points, limit cycles)
There are four different types of steadystate behavior. Each steady state can be described from
three different points of view: in the time domain, in the frequency domain, and as a limit set
(state space domain).
1.1.1 Equilibrium Points: An equilibrium point is a constant solution of equation (11), for all t.
At an equilibrium point, $ %
& implies that % is an equilibrium point. The limit set for an
equilibrium point is just the equilibrium point itself.
1.1.2 Periodic Solutions: % !
is a periodic solution of (11) if % ! '
% !
for all t and
some period T. A periodic solution has a Fourier transform consisting of a fundamental
component at $ (
)' and evenly spaced harmonics at * ', * + , . The amplitude of some
of these spectral components may be zero.
1.1.3 Quasiperiodic Solutions: A quasiperiodic solution is the sum of periodic waveforms each
of whose frequency is one of the various sums and differences of a finite set of base frequencies.
A periodic solution is a quasiperiodic solution with one base frequency [3]. To understand the
concept of chaos, let’s review two more measure, Lyapunov exponent and dimension.
1.2 Lyapunov Exponents
Lyapunov exponents are a generalization of the eigenvalues of the Jacobean matrix at an
equilibrium point. They are used to determine the stability any type of steadystate behavior
including quasiperiodic and chaotic solutions. Lyapunov exponents are defined in terms of the
3
solution of the equation (11) as follows. Let ./01 !
23145
6 be the eigenvalues of the Jacobean
matrix of (11). The Lyapunov exponents are defined by:
71 89:;#
5
; 8<301 !
3 (12)
if the limit exists[1, 4].
Lyapunov exponents of an equilibrium point are equal to the real parts of the eigenvalues at the
equilibrium point and indicate the rate of contraction (71 = &) or expansion (71 > &) near the
equilibrium point. Lyapunov exponents are convenient for categorizing steadystate behavior.
For an attractor, contraction must outweigh expansion so
? @ = & A 4 (13)
One feature of chaos is sensitive dependence on initial conditions. Sensitive dependence occurs
in an expanding flow therefore what distinguishes a chaotic attractor from the other types of
attractors is the existence of at least one positive Lyapunov exponent [5]. Table 1.1 shows the
Lyapunov exponents for different types of attractors.
Table 1.1 Lyapunov exponents for different type of steady state
Steady State Lyapunov Exponents
Equilibrium Points & > 75 B 7C B D B 76
Periodic: subharmonic 75 &, & > 7C B D B 76
Periodic: Twoperiodic 75 7C &, & > 7E B D B 76
Periodic: kperiodic 75 7C D 7F &, & > 7FG5 B D B 76
Chaotic system 75 > &, ?1 71 = &
4
1.3 Dimension
Attractors can also be classified using the concept of dimension. An attractor could be defined to
be ndimensional if in a neighborhood of every point, it looks like an open subset of H6. For
instance, a limit cycle is one dimensional since it looks locally like an interval. A torus is twodimensional
since it resembles an open subset of HC. An equilibrium point is considered to have
zero dimension. The neighborhood of any point of a chaotic attractor does not resemble any
Euclidean space therefore do not have integer dimension. There are several ways to generalize
dimension to the fractional/chaotic case including capacity, information, correlation, and
Lyapunov dimensions [4].
1.3.1 Capacity Dimension (fractal dimension): Let’s cover an attractor A with volume elements
(spheres, cubes, etc.), each with diameterIJ. Let K J
be the number of volume elements needed
to cover A. As J gets smaller, the sum of the volume elements approaches the volume of A. If A
is a Ddimension (D integer), then for small J, number of volume elements needed to cover A is
inversely proportional to JL, that is, K J
M JNL . Therefore:
OPQR 89:S#"
TUV S
TU 5 S
(14)
1.3.2 Information Dimension: It is a probabilistic type of dimension, defined in terms of the
relative frequency of visitation of a trajectory [4, 5]. The setting is the same as for capacity
dimension:
OW 89:S#"
TUX S
TU 5 S
(15)
where Y J
is entropy  the amount of information needed to specify the state of the system to an
accuracy of J if the state is known to be on the attractor. It is defined as Y J
? Z1 8< Z1
V S
145 .
5
Z1 is the relative frequency with which a typical trajectory enters the ith volume element of the
covering[2, 4].
1.3.3 Correlation Dimension: Suppose N points of a trajectory have been gathered either through
simulation or from measurements [4]. Let’s define the correlation as:
[ J
89:V#
5
V\ ]K^0_`aIb$I%1Icd`a`Ie%1 %fe = Jg (16)
Then:
OP 89:S#"
TU h S
TU S
(17)
[ J
can be simplified assuming that Ii1I is the number of points lying in the ith volume
element, so Z1 89:V# i1 K. Since the volume element has diameter J, all the i1 points lie
within J of each other forming i1
C pairs of points. Therefore:
[ J
89:
V#
(
KCji1
C
V S
145
89:
V#
j
i1
C
KC
V S
145
jk 89:
V#
i1
C
KCl
V S
145
jZ1
V S
145
Thus:
OP 89:S#"
TU? mn
o p
nqr
TU S
(18)
Table 1.2 Dimensions for different types of steady state
Steady State Dimension
Equilibrium Points 0
Periodic: subharmonic 1
Periodic: Twoperiodic 2
Periodic: kperiodic K
Chaotic system Noninteger
6
1.3.4 Lyapunov Dimension: Let 75 B 7C B D B 76Ibe the Lyapunov exponents of a dynamical
system. Let j be the largest integer such that ? 71
f1
45 B & [4]. Lyapunov dimension is defined as:
Os t ? un
vn
qr
wuvxrw
(19)
In a 3D chaotic system with Lyapunov exponents 7G, 0, 7N Lyapunov dimension is:
Os + ux
3uy3.
For an attractor, 7G 7N = &, therefore + = Os = ,. Table 1.2 shows the dimensions for
different types of attractor.
7
CHAPTER 2
REVIEW OF LITERATURE
Artificial neural network is made of simple elements which models biological neurons. One
simple neuron is a simple element with input/output function that poses threshold which adds
weighted inputs from other neurons and concludes as single output. But neurobiological research
showed that neurons have much more sophisticated behavior than this simple behavior [14]. one
characteristics of biological neurons is their chaotic behavior which can be experimentally seen
in neurons’ behavior. Not only in different experiments with axons but also with numerical
analysis of HodgkinHuxley equation, it is now known that membrane response to periodic
stimulation is not always periodic and it does have non periodic and indeed chaotic response. In
this chapter, we first review a single chaotic neuron and then a network based on those chaotic
elements.
2.1 Neuron’s Model
Dynamical modeling of biological neurons goes back to McCullochPitts and Caianiello. It is
defined as following [14]:
%1 ! (
^z? ? '1f
{
%f ! a
1
;
{4"
}f
45 f~1 (21)
%1 is neuron’s state and is 1 (active neuron) or 0 (nonactive neuron). %1 ! (
is the ith
neuron’s output at time t+1, u is the step function, M is the number of neurons, and '1f
{
is the
strength between ith neuron and jth neuron. '11
{
is the memory coefficient of relative
refractoriness with which the firing of the ith neuron retains influence on itself after the r+ 1
timeunits; and 1 is the threshold for the allornone firing of the ith neuron. The discontinuous
output function u in the above equation is a mathematical representation of the socalled allor
8
none law that the output of a neuron has the alternatives of presence and absence of a full size of
action potential depending upon whether the strength of stimulation is more than the threshold or
not. It is assumed that the influence of the refractoriness due to a past firing decreases
exponentially with time '11
{
!
*;, where * takes a value between 0 and 1 and is a
positive parameter.
% ! (
^ !
?; *{% ! a

{4"
(22)
where !
is the strength of the input at the discrete time t, and k is the damping factor of the
refractoriness. By defining a new variable ! (
corresponding to the internal state of the
neuron as follows,
! (
!
?; *{% ! a

{4" (23)
Therefore we can have a dynamic system as:
! (
* !
^ !
!
(24)
% ! (
^ ! (
(25)
where:
!
!
* ! (
 ( *
(26)
While the dynamical behavior of the neuron is calculated with eq. (24) on the internal state y,
the value of the output is obtained by transforming the internal state y into the output x with eq.
(25). In particular, when the input stimulation is composed of periodic pulses with the constant
amplitude A as frequently used in electrophysiological experiments,
a(t) of eq. (26) is temporally constant as follows,

( *
(27)
9
Fig 1.1 shows the response characteristics as the bifurcation parameters change. In (a) a
bifurcation diagram is shown with changing value of the bifurcation parameter a. In (b),
Lyapunov exponent is depicted and (c) shows the average firing rate, , defined as follows:
89:6#
5
6
?6N5 % !
;4" (28)
Since negative values of the Lyapunov exponent characterize periodic solutions, (b) concludes
that almost all the responses are periodic.
Fig 2.1 Response characteristics vs. bifurcation parameter a when k=0.5, a=l.0 and y(0)=0.1: (a) neuron’s
output, (b) the Lyapunov exponent and (c) the average firing rate .
10
2.2 Chaos in a Biological Neuron
As it was mentioned in previous sections, however response for equations (24) and (25) is
periodic, but the chaotic response can also be discovered. This incompatibility between theory
and experiment requires change into equations (21) and (22). It should be also mentioned that
action potential of neurons due to stimulated pulse does not obey the general law of allornone.
It should be noted that the generation of action potentials by current stimulation of a single pulse
[15]. In particular, experimentally with squid giant axons and numerically with the Hodgkin
Huxley equations it can be shown that the stimulusresponse curve is not discontinuously allornone
but continuously graded for the ease of spatial uniformity; namely the stimulusresponse
property of the nerve membrane is described not by an allornone step function such as the
function u in equations (21) and (22) but by a continuously increasing function [14, 15]..
Therefore, we assume that the output function of artificial neurons is a continuously increasing
function f and replace the unit step function ^ in (22) by a continuous function $Ias follows:
% ! (
$ !
?; *; % ! a

{4"
(29)
where % ! (
is the output of the neuron, or a graded action potential generated, a continuous
value between 0 and 1; f can be the logistic function as a continuous one as follows:
$ 5
5G y p
(210)
Therefore the equations can be rewritten as:
! (
* !
z$ !
(211)
% ! (
$ ! (
(212)
11
The following figure is related to the response of the above mentioned system. Average firing
rate is defined as:
89:6#
5
6
?6N5 i % !
&
;4" (213)
Fig 2.2 shows the response characteristics as the bifurcation parameters change. In (a) a
bifurcation diagram is shown with changing value of the bifurcation parameter a. In (b),
Lyapunov exponent is depicted and (c) shows the average firing rate, . This figure shows the
chaotic behavior of the system.
2.3 Network with Chaotic Elements
The neuron described in previous section can be used as a base element of a network. Neural
dynamics of the ith chaotic neuron in the network is defined as:
1 ! (
? 1f I? *{df %f ! a
? 1f I?; *{ f ! a
{4"
Vf
45
;
{4"
}f
45 ? *{I f %1 ! a
1
;
{4" (2
14)
where ( 1) i y t + is the internal state of the ith chaotic neuron at time t+1, i f is continuous output
function , M is the total number of neurons in the network, ij W synaptic weight between ith and
jth neurons, ij V is the synaptic weight between jth input to ith chaotic neuron. In equation (2
14) it is assumed that ( ) . r r
ij ij W k = T where k is the damping coefficient. The neuron model in here
has the following characteristics:
1. Continuous output function with graded action potential
2. Relative refractoriness with exponential damping coefficient
12
3. Spatial temporal sum over feedback inputs and external inputs
Fig 2.2 Response characteristics vs. bifurcation parameter a when k=0.5, a=l.0 and y(0)=0.5: (a) neuron’s
output, (b) the Lyapunov exponent and (c) the average firing rate p.
Equation (214) can be simplified as following:
13
1 ! (
* 1 !
? 1fdf
}f
45 $f f !
? 1f f
Vf
45 !
1 $1 1 !
1 ( *
(215)
%1 ! (
$1 1 ! (
) (216)
where ( 1) i y t + is the internal state of the ith chaotic neuron and is defined as following [14]:
1 ! (
I? 1f
}f
45 ? *{df
;
{4" z%f ! a
? 1f
Vf
45 ? *{ f
;
{4" ! a
? *{ 1
;
{4" %1 ! a
1
(217)
equations (215), (216) and (217) are a description of a chaotic neural network which can be
simplified as the following form:
%1 !
5
5G y
n
p
(218)
1 ! (
* 1 !
1%1 !
? 1f
Vf
45 %f !
1
(219)
Where 1 is a selffeedback connection weight or refractory strength (positive constant) and the
following assumption also is made:
hj (x j (t − r)) = x j (t − r)
( ( )) ( ) j j j g x t − r = x t − r
1 1
( 1) { ( ( )) ( ) }
M N
i i ij j j ij j i
j j
x t f W h x t V I t q
= =
+ = + −
( 1) ( ( 1)) i i i x t + = f y t +
14
Fig 2.3 A neuron model in chaotic neural network
2.4 Transient Chaotic Neural Network
Some of the problems in combinatorial optimization area have high computational expenses. We
can name Travelling Salesman Problem (TSP), or Knapsack Problem. Classical neural network
such as Hopfield can be used to solve these problems. But it may reach to the wrong local
optimum. Transient chaotic neural nets have been introduced to overcome this weakness which
in turn can reach to the true optimum point. This network can be described as following:
%1 !
5
5G y
n
p
(i=1,2,…..,N) (220)
1 ! (
(
1 !
(221)
1 ! (
* 1 !
? 1f
Vf
45 %f !
1 1 !
%1 !
"
(222)
15
where Wij =Wji , 0 ii W = , ( ) i z t is timedependent feedback connect weight which also is the
bifurcation parameter. b is the timedependent damping coefficient (between 0 and 1) and 0 I is
a positive parameter with a constant value.
This network possess transient chaotic dynamics, i.e. after several successive bifurcation, it will
finally converge to a fixed equilibrium point. The parameter ( ) i z t plays a controlling role. In
order to compare equations (220) to (222) with the Hopfield network, we can rewrite this
network as following:
%1 !
5
5G y
n
p
( +  K
(223)
n
;
= *
1 ? 1f
Vf
45 %f 1 1 %1 "
(224)
n
;
=
1 ( +  K
(225)
The difference between Hopfield network and this one is the term 0 ( )( ( ) ) i i −z t x t − I . Since the
controlling parameter of ( ) (0) t
i i z t z e −b ¢ = converges to zero, so the above mentioned equations
converge to continuous Hopfield network with 0 ii W = .
2.4.1 Transient chaotic dynamics in a single neuron: Here the nonlinear dynamics of a single
neuron with transient chaos is introduced. Equations (218) and (219) are rewritten:
% !
5
5G y
p
(226)
! (
(
!
(227)
! (
* !
!
% !
"
(228)
where g =a.I . Chaotic dynamics of x(t) is reduced by decreasing of z(t) . Damping parameter
for z(t) is b , which is between 0 and 1. In Fig 2.4, behavior of neuron’s model as a function of
16
time for different values of b is shown. In this figure, z(0) = 0.08, 0 I = 0.65 , e = 0.004 , and
k = 0.9 .
This model can also behave as an equilibrium point, periodic, and quasiperiodic steadystate
besides chaotic dynamics. This change of state of neuron can be controlled with bifurcation
parameter. Important property of this model is that chaotic behavior is temporarily used to search
and then slowly settles down as the bifurcation parameter changes [15, 37].
Assessing the parameter b in network dynamics plays an important role. If b is so large,
network will quickly loose its chaotic characteristics and converges to a wrong solution. On the
other hand of b is so small, the network will not loose its chaotic behavior to converge to the
solution in our desired time. As we can see in (a), the chosen value for b is smaller, therefore
neuron keeps the chaotic behavior longer. In (b) the variation of z(t) for this smaller b is
shown In (c), b is large thus the neuron’s behavior quickly is changed from chaotic and
converges to equilibrium point. Graph in (d) shows the variation of z(t) vs. time.
2.4.2 Discussion: Classical artificial neural networks like Hopfield, guarantees reaching an
equilibrium point, but its problem is that it may converge to local optimum. Chaotic neural
network with fractallike structure adds the possibility of searching entire state space chaotically.
However chaotic dynamics are used for application like information processing and
optimization, but it is not known when is a good time to end chaotic dynamics and converge to a
stable equilibrium point. Classical neural networks have steady or neartoequilibrium behavior
while chaotic neural networks have farfromequilibrium dynamics. These networks can also
behave as equilibrium point, periodic, and quasiperiodic steady state. The transient chaotic
neural network uses get the best out of classical and chaotic neural network. With the controlling
parameter, the chaotic dynamics can be adjusted based on our needs. This keeps the network
17
from falling into wrong local optimum. It also shows that transient chaotic neural network has
more flexibility than classical neural network.
Fig 2.4 Time series of the output, x, of the chaotic neuron, and time series of the bifurcation parameter, z, for
a single neuron. (a) and (b): , (c) and (d):
18
2.5 Associative Dynamics in Chaotic Neural Network
In this section, we review more details of chaotic neural networks and nonperiodic associative
dynamics and their recognition process. Chaotic neural network have specific associative
behavior in such a way that the network recognize the stored memory in the transient states and
then goes away from them, while in classical associative networks recognition process is mainly
done by noting stability of equilibrium state, due to trained pattern. Recognition characteristics
such as frequency of recognition is explained in order to realize the ability of memory search
process.
2.5.1 Associative neural network with chaotic neurons: The neural net mentioned in (214) can
be expressed as:
%1 ! (
$ j 1f Ij * QPF
{ %f ! a
j 1f Ij * ; ¡Q¢
{ f ! a
;
{4"
V
f45
;
{4"
}
f45
? *{ {QP;£{
{ I %1 ! a
1
;
{4" I (229)
where each of the following variables can contribute to the state of the neuron:
¤1 ! (
? 1fI%f !
* QPF¤1 !
Vf
45 (230)
¥1 ! (
? 1fI f !
* ; ¡Q¢¥1 !
}f
45 (231)
¦1 ! (
%1 !
*{ {QP;£{ ¦1 !
1 ( *{ {QP;£{
(232)
Therefore:
%1 ! (
${¤1 ! (
¥1 ! a
¦1 ! (
(233)
In here, i h
is related to external input to neuron, i x
is internal state of neuron, and i j
is feedback
input to neuron. In order to investigate associative memory of the network in (233), let’s
consider a net of 100 neurons. For just simplicity let’s also assume that external input is constant:
%1I ! (
$ ¤1I ! (
¥1I ! a
(234)
19
¤1 ! (
* QPF¤1 !
? 1fI%f 5"" !
f45 (235)
¦1 ! (
*{ {QP;£{ ¦1 !
%1 !
1 (236)
where i a is due to sum of the threshold of each neuron and constant external input and is
considered as 2. Four pattern consisting of 100 binary digits (1 and 0) shown in Fig 2.5 will be
trained into this network using the following training rule:
1f 5
§
? +%1
§ R
(
R45 +%f
R
(
(237)
where ( p)
i x is the ith element of the pth pattern.
2.5.2 Dependence of recognition process on refractory k
In the network of (234), when 0 refractory feedback k = k =a = , it behaves like an associative memory
and stores all patterns. If refractory k is increased from zero, associative process nonperiodically
passes through patterns. If refractory k is close to zero, network has a classical behavior. In Fig 2.6,
recognition process of the noisy pattern C, (see Fig 2.5) is shown when 0.5 refractory k = ,
0.15 feedback k = , and a = 8 . As we can see, after a short transient time, the network will go to the
pattern C and stays there. If refractory k is close to 1, then network has a very different behavior. In
Fig 2.7, time process of neuron’s output is shown when pattern C itself is given to the network.
In here, 0.9 refractory k = , 0.2 feedback k = , anda = 10 . As it can be seen, the network will not stay in
any equilibrium points, but passes by the learned patterns. For instance, Pattern A is seen at time
t = 22 , pattern B is seen at time t = 87 , pattern C at time t = 1, and pattern D is passed at time
t = 45,55,65 . The retrieve frequency of the patterns for large refractory k is shown in Table 2.1.
20
Fig 2.5 Four pattern stored in the network
Table 2.1 Number of passing from each stored pattern in the network
refractory k Pattern A Pattern B Pattern C Pattern D All patterns
0.85 3 2610 8 4 2625
0.9 158 195 103 83 539
0.95 36 53 39 41 169
With large values for refractory k , (see Fig 2.7) the network will not stay in any of the equilibrium
states, but nonperiodically passes through the patterns.
2.5.3 Dynamic dependence on initial conditions: In Fig 2.8, some patterns are shown that have
disturbed and have the Hamming distance1 of 4 (part a), or 8 (part b) with the original patterns.
The system retrieves the stored patterns in the recognition process. In Table 2.2, the retrieve
frequency of patterns with these disturbed patterns is shown. We can see that type of retrieving
does depend on the initial conditions. In Table 2.3, the retrieve frequency of patterns for different
initial condition is shown.
1 Hamming distance between two strings of equal length is the number of positions for which the corresponding
symbols are different.
21
Fig 2.6 Recognition of disturbed pattern C when ¨©ª¨
«¬¨® .
Table 2.2 Investigating dependence of dynamics on initial conditions
Hamming Distance 0 4 8
Pattern A Periodic, (T=9378) Nonperiodic Periodic, (T=20)
Pattern B Periodic, (T=9378) Periodic, (T=20) Periodic, (T=20)
Pattern C Periodic, (T=20) Periodic, (T=9378) Periodic, (T=20)
Pattern D Nonperiodic Periodic, (T=9378) Periodic, (T=20)
22
Table 2.3 Retrieve frequency with different initial condition ( 0.2, 0.9, 10, 2 feedback k = krefractory = a = ai =
with 5000 time step)
Initial Conditions A B C D All patterns
C 27 48 31 21 127
B1 63 68 21 13 165
B2 64 61 25 11 161
C2 14 66 19 28 127
In summary, associative dynamics in chaotic neural network varies with change of initial
condition and external stimulation. With a constant external stimulation, after the transient stage,
the network gives the periodic solution. If refractory k is close to 1, dynamics of transient stage will
move nonperiodically around stored patterns. In another word, the process of memory search in
chaotic neural network is more dynamic than that of classical networks. We should also notice
that, this process can be controlled by parameters refractory k and feedback k . Characteristic of
associative dynamics of chaotic neural network is similar to behavior of biological neural
system. This similarity can be seen in the way that trained patterns are retrieved or even in the
way that new patterns are learned [7,8].
23
Fig 2.7 Retrieving patterns when ¨©ª¨
«¬¨® .
24
2.6 Discussion
Neurobiological research has shown that biological neurons have complex and chaotic behavior
[610]. In fact, the chaotic behavior of neurons can be observed experimentally due to different
biological experiments. Existence of such behavior in biological neurons have become the
motivation to develop chaotic neural network. Chaotic neural network prepares the ability of the
search of the whole search space by chaotic movements. Researchers have shown that the
chaotic dynamics can be used as a new tool for optimization and pattern recognition [14, 16, 17,
23, 24, 27, 28, 33, 36, 37, 38] Main characteristics of these model is that chaos can help them to
search the space and then settle down into the fixed equilibrium point [31].
Fig 2.8 Disturbed patterns: (a) Hamming distance of 4, (b) Hamming distance of 8.
25
CHAPTER 3
SINUSOIDAL CHAOTIC NEURAL NETWORK
3.1 Introduction
There has been extensive research work on nonequilibrium dynamics [32, 33, 34, 35, 36, 37].
Based on the biological research the olfactory system of rabbits has limit cycle attractor for the
learned odors and chaotic attractors for new odors [42]. Spatiotemporal complexity of this type
of dynamics are said to be related to asymmetric connection in the network [20, 21]. From
technical point of view, simulation of nonequilibrium information processor is important in
associative memory, pattern recognition, optimization and retrieving databases. Hopfield in 1984
introduced a neural network useful in associative memory which minimized Lyapunov function.
In this network, equilibrium dynamics and Hebb learning rule 2 is used. After his work, there has
been a lot of research regarding associative memory systems based on nonequilibrium dynamics
which used Hebb’s covariance learning rule. For example, Nara in 1993 introduced associative
dynamics based on asymmetric neural network. On the other hand, it has been shown that paired
nonlinear oscillators can produce complex spatiotemporal patterns. Kaneko introduced models
based on paired chaotic oscillators [24]. In his model, each element has Logistic map and his
network, Global Coupling Map (GCM) is defined as:
%1 ! (
( J
$¯%1 !
° S
V
? $¯%f !
° Vf
45 (31)
$ %
( %C % ± ¯ ( (° (32)
where ( ) i x t is the state of ith neuron at time t, and K is the number of neurons. Each neuron’s
dynamics is based on Logistic map. The last term in (31) is the average feedback from other
2 It states change in the weight connection from neuron to t by c²³ I I ^1^fI where is the learning rate and ^1
and ^f represent the activations of neuron and t respectively.
26
neurons. GCM mapping has two parameters a ande . With the specific values of these
parameters, the neurons will be clustered into some clusters and neurons within one cluster will
go through the same cycle.
3.2 Neural Network with Logistic Chaotic Elements
In Kaneko’s model, second order Logistic map is used for modeling chaotic behavior of each
neuron, while [24] has used thirdorder Logistic map:
%1 ! (
( J
$¯%1 !
° S
V
? $¯%f !
° Vf
45 (33)
$ %
%E % % % ± ¯ ( (° (34)
Since this function is symmetric, this new model is called SGCM. In Fig 3.1, secondorder and
thirdorder Logistic map are compared. In (a), behavior of secondorder Logistic map for
a =1.4,2 . As it can be seen, this function has an extreme point. In (b), the thirdorder Logistic
map is shown for a = 3,4 which has two extreme points.
Fig 3.1 (a) Asymmetric Logistic function with ´ , (b) Symmetric Logistic function with ´.
Therefore symmetric Logistic map has maximum two periodic attractor, while secondorder map
has at most one attractor. Additionally the thirdorder map is symmetric which can be used as
binary representation (these two attractor represent 1, 1) [24]. Fig 3.2 shows the bifurcation
27
diagram of secondorder map (a) and thirdorder map (b) with respect to bifurcation parameter a .
As it can be seen, the attractors in (a) is asymmetric while in (b) symmetric.
Fig 3.2 Bifurcation diagram: (a) secondorder map, (b) Thirdorder map.
3.3 Cluster Attractors
SCGM model has two parameters a ,e . With the specific values of these parameters, system
falls ino cluster attractors. Neurons belonging to one cluster will go through the same orbit. This
can be a twocycle, 4cycle, 8cycle, … or chaotic [24]. Fig 3.3 shows the time series of all
neurons. Initial value of neurons are set randomly between 1 and 1. Parameters are also set as
a = 3.4,e = 0.1. (a) is depicted for all the time while (b) is depicted only for even times
(t=2,4,…). As we can see all neurons are clustered into 4 clusters and neurons inside one cluster
has the same orbit which in this case is a twocycle orbit.
28
Fig 3.3 Fourcluster attractor: (a) For t=1, 2, 3…, (b) For t=2, 4, 6, ….
3.4 Characteristics of Neural Network with Chaotic Logistic Map
Spatiotemporal characteristics of attractors in SGCM is altered by its parameters. Parameter a
is the bifurcation parameter and demonstrates strength of chaos in each neuron while e
demonstrates the strength of connection between neurons. If a is too large, then system behaves
chaotically while a large value of e implies stability. With different values of a ,e the system’s
behavior can be classified as following:
Coherent state: e large, a small; all neurons stay in one orbit.
Ordered state: System stays in cluster attractors. By increasing a or decreasing e , the number
of clusters will increase as 2,4,8,16,….
Partial ordered state: In this case, depending on the initial condition of the system, attractors in
some cases stay in several clusters and in some other cases, only a few clusters.
29
Turbulent state: e small, a large; each neuron follows its chaotic orbit.
In the last two case, turbulent state and partial ordered state, the largest Lyapunov exponent is
positive, i.e. that system in the last two case behaves chaotically.
3.5 Application on Memory and Pattern Recognition
3.5.1 Associative memory by controlling a
Assume { } 1 2 , ,..., { 1,1} m k N c c c c Î − are a set of learning samples where i
k c is the ith element
of the kth pattern. Let’s assume that the learning rule is the covariance learning rule [24]:
c1f 5
V
? µ1
F
¡F
45 µf
F
(35)
The following neural network is an associative memory which is developed by SGCM
(equations 33 and 34):
%1 ! (
( J
$1¯%1 !
° S
V
? $f¯%f !
° Vf
45 (36)
$1¯%1 !
° 1 !
¯%1 !
°E 1 !
%1 !
%1 !
(37)
This network differs with (33) and (34) because the strength of chaos in each neuron decreases
as following:
1 ! (¶
1 !
¯ 1 !
¡16° ·¸<¹¯ º1 !
° (38)
º1 !
%1 !
? c1f%f !
Vf
45 (39)
where min a , b , and e are constant parameters, and º1 !
is the energy function of the ith
neuron. In here, min a = 3.4,b = 2,e = 0.1. Dynamics of each i a is controlled between min a and
max a = 4 in every 16 time units. The applied sigmoid function in this model is only used to
control parameter i a (bifurcation parameter) which then indirectly affects neurons behavior.
30
Fig 3.4 Five 100bit binary sample patterns.
In Fig 3.4 five patterns (alphabetic letters of A, J, P, T, and S) each consisting 100 digits are
shown. They are used as learning samples to the system using the learning rule of (35). The
parameters are again set as amin = 3.4,b = 2,e = 0.1. Each pattern then will be noisy by 40%
(40% of the bits are randomly chosen and then flipped) and then are given to the network. The
system can recognize the pattern after a few time steps. Fig 3.5 shows the recognition process of
learned pattern S in different time steps.
In Fig 3.6, time series of the neurons outputs, energy of each neuron, and parameter a for each
neuron due to recognition of pattern A (see Fig 3.5) are depicted. In (a), the chaotic behavior of
the system can be seen in the early period of time series, while after a while this behavior calms
down and finally it settles down into a 4cluster attractor. In (b), the energy of all neurons are
depicted. Each neuron searches in the early period of time series to find the minimum energy
needed to settle down and finally energy of each neuron gets minimized. In (c), time series of
parameter i a
of each neuron shows that it finally settles down to its minimum value min a = 3.4 ,
while in the early stage larger values for i a
allows neuron to search in the space more
thoroughly.
31
3.5.2 Associative memory by controlling e
Now let’s assume { } 1 2 , ,..., { 1,1} m k N c c c c Î − is a set of learning samples where i
k c is the ith
element of the kth pattern. This time an associative memory is shown that can be controlled by
parameter e based on [24]. The covariance learning rule will again be used:
c1f 5
V
? µ1
F
¡F
45 µf
F
(310)
In this case, the system’s behavior is described as:
%1 ! (
( J1 !
$ %1 !
Sn ;
V
? $ %f !
Vf
45 (311)
$¯%1 !
° ¯%1 !
°E %1 !
%1 !
(312)
This system differs from original SGCM, because connection strength of each neuron, i e
, varies
by the following dynamics:
J1 ! (¶
J1 !
¯J1 !
J¡Q ° ·¸<¹¯ º1 !
° (313)
where:
º1 !
%1 !
? c1f%f !
Vf
45 (314)
Dynamics of i e
is updated every 16 time units between min e = 0 and max e = 0.2 . Indeed, i e
controls connection strength between ith neuron and other neurons. Similar to last section, we
use the same five patterns of alphabetic letters A, J, S, T, and P using the learning rule of (310)
to train this system. In here, a = 3.65,b = 2 . Each pattern then will be noisy up to level of 40%
and is used as the input of the network. System can recognize the pattern after some time steps.
Fig 3.7 shows the time process of pattern recognition of these samples. In Fig 3.8, time series of
the pattern recognition for pattern A is shown. The chaotic behavior of the system in the early
stages can easily be seen. Since a = 3.65 , system will finally be attracted into a 4cluster
attractors. In (b), the partial energy for each neuron gets negative, therefore the total energy of
32
the system is minimized. Finally in (c), time series of i e
for each neuron shows that it finally
reaches its maximum value when the system is settled down to 4 clusters.
Fig 3.5 Pattern recognition process with 40% noise in the SGCM model controlled by .
33
Fig 3.6 SGCM model controlled by : (a) Time series of the output values of neurons, (b) Time series of
energy function, (c) Time series of
34
Fig 3.7 Pattern recognition process with 40% noise in the SGCM model controlled by ».
35
Fig 3.8 SGCM model controlled by »: (a) Time series of the output values of neurons, (b) Time series of
energy function, (c) Time series of »
3.6 Proposed Network: Sinusoidal Chaotic Networks
In this section, we develop a neural network that is based on chaotic dynamics. This network has
the basic elements as Hopfiled network which consists of neurons with sinusoidal chaotic map.
Sinusoidal chaotic map is defined as the following [3]:
$ %
¼9< ½%
1<x<1 (315)
36
Fig 3.9 Sinusoidal chaotic map: (a) Time series of x with I(b) Bifurcation diagram of x, (c)
Bifurcation diagram for all positive x, (d) Bifurcation diagram for all negative x, (e) State space of the secondorder
Logistic map, (f) State space of the sinusoidal chaotic map.
37
In Fig 3.9, Sinusoidal chaotic map is shown. In (a), time series of x(t) for a = 0.92 is depicted.
In (b), bifurcation diagram is shown for the bifurcation parameter of a . In (c) and (d),
bifurcation diagram for the positive and negative values of x(t) are shown. In (e), state space of
the secondorder Logistic map and in (f) state space of sinusoidal chaotic map are shown. As it
can be seen, the variation of the variable x(t) in this map is symmetric. Another important point
is that sinusoidal chaotic map has two separate attractors. The Logistic map does not have this
characteristics. We will show that based on this fact, the neural network based on sinusoidal
chaotic elements will have considerably better memory than Hopfield networks and neural
networks with Logistic chaotic maps. The chaotic dynamics of the ith neuron in the neural
network is defined as following:
%1 ! (
( J1 !
$ %1 !
Sn ;
V
? $ %f !
Vf
45 (316)
$ %1 !
¼9< %1 !
(317)
where ( ) i x t is the output of ith neuron at time t and N is the number of neurons. Parameter i e
shows the strength of connection between ith neuron and other neurons. a is the bifurcation
parameter of the sinusoidal chaotic map. Important point in this network compared to the
previous ones, is that the bifurcation parameter is constant, while i e
varies. This model can
express equilibrium, periodic, and quasiperiod behaviors besides the chaotic behavior which is
controlled by parameter i e
. Now let’s see how we can use this new dynamics in information
processing. Assume that { } 1 2 , ,..., { 1,1} m k N c c c c Î − is a set of learning samples where i
k c is
the ith element of the kth pattern. Associative process with parameter i e
will be done as
following:
J1 ! ¾!
J1 !
¯J1 !
J¡Q ° ·¸<¯ º1 !
° (318)
38
where b = 2,amin = 0.2 and:
º1 !
%1 !
? c1f%f !
Vf
45 (319)
Learning in the network is done by covariance learning rule:
c1f 5
V
? %1
F
¡F
45 %f
F
(320)
Associative process starts with small values of i e
( min e = 0 ) and will then be updated based on
equation (318). When neuron’s energy is small, neuron’s connection will become larger
(neurons behave to maintain information), and while the neurons’ energy is large, the connection
will become smaller (neurons behave randomlike behavior). In order to investigate associative
process in this chaotic neural network, we have used five sample patterns each consisting 100
digits (the same patterns A, J, S, T, and P). After training the network by these patterns using
covariance learning rule, each pattern is then disturbed and is given as input to the network. The
network retrieve the correct pattern after a few time steps. Fig 3.10 shows the recognition
process of the sinusoidal chaotic neural network with 40% noisy patterns. In this investigation,
min a = 0.8,b = 2,e = 0 . In Fig 3.11, time series of the recognition process for pattern T is shown.
In (a), time series of all neuron’s output are depicted. We can see the chaotic behavior of the
system in the early stage of the process. Since a = 0.8, the system finally settles down into 4
cycle attractor. Graph in (b) shows the energy of each neuron, partial energy i E . This shows that
i
i
E = E gets finally minimized. Indeed, the network minimizes E by making each neuron’s i E
negative.
39
Fig 3.10 Pattern recognition process with 40% noise in sinusoidal chaotic neural network.
40
Fig 3.11 Sinusoidal chaotic neural network: (a) Time series of the output values of neurons, (b) Time series of
energy function, (c) Time series of
If i E is large and positive, i e
will be reduced and the connection between neurons become
weak, and neurons behave more chaotically. When i E is small enough and negative, i e
will be
its maximum value and the connection among neurons become strong. In (c), time series of each
neuron’s i e
is shown. As we expect, in early stage, i e
is small, but at the end it reaches its
maximum value and that is when the system settles down into 4cluster attractor.
41
3.7 Performance of the Sinusoidal Chaotic Neural Network in Pattern
Recognition
We investigate the application of the sinusoidal chaotic neural network in pattern recognition,
and compare its performance with Hopfield network and SGCM. For this purpose, 15 pattern
each consisting 100 binary digits are used for training. For updating Hopfield neurons, the
following is used:
Fig 3.12 Fifteen training pattern each consisting 100 binary bits.
^1 ! (
^1 !
? c1f
Vf
45 ¼9¿< ^f !
(321)
%f ! (
¼9¿< ^f ! (
(322)
c1f k
&
? µ1
F
}F
45 µf
F
.
t
À t (323)
42
where i u is the neuron’s state (continuous), while i x is the neuron’s output (binary value ( or
(). And i
k c is the ith element of the kth pattern.
After learning process, noisy pattern will be given to the network for recognition. Initial overlap
between the input pattern I (disturbed pattern) and desired pattern k c is defined as following:
i ! ÁIÂÃ`aÁ Ä 5
5""
? 1
5""
145 µ1
F
(324)
Whenever the initial overlap is close to 1, input pattern is close to the desired pattern. After
updating neurons and network’s convergence, success rate will be calculated as following:
Y^ÅÅ` IÆ !` 5
5""
? %1
5""
145 Ç i Á
µ1
F
(325)
The comparison results for all 15 patterns are then shown in Fig 3.13. It can be seen that
proposed network shows the best performance compared to Hopfield (equations 321 to 323),
controlled SGCM (equations 36 to 39), and J controlled SGCM (equations 311 to 314). In
particular, whenever that initial overlap is 100%, our network shows a full recognition for all
patterns while the other three networks show poor performance. The performance of controlled
SGCM and J controlled SGCM are similar, but J controlled SGCM performs a little better
than controlled SGCM. In summary, all three networks perform better than Hopfield
network. The average recognition rate for Hopfield is 42.6%, for of controlled SGCM 73.5%,
and for J controlled SGCM is 75.3%.
Fig 3.13 Comparison of pattern recognition in 4 different sy
(d) Pattern T, (e) Pattern S, (f) Pattern B, (g) Pa
43
system: (a) Pattern A, (b) Pattern J, (c) Pattern P,
Pattern C, (h) Pattern H,
stem: ttern 44
Fig 3.13 (cont.) (i) Pattern U, (j) Pattern N, (k) Pattern O, (l) Pattern Z, (m) Pattern D, (n) Pattern È, (o)
Pattern *.
45
3.8 Memory Capacity of Sinusoidal Chaotic Neural Network
In this section, memory capacities of these four models are compared. For this purpose, we
consider a network with 20 neurons, and 50 binary (each containing 20 bits) sample training
patterns among which 10 learning sets are selected. Each set consists of 20 patterns that is
randomly chosen among 50 patterns. Fig 3.14 shows three sets of these patterns. In order to find
memory capacity we use the following:
É`0ba I[ Ä Å ! I É[
5
}
? mÊ
F V (326)
Table 3.1 Comparison results for memory capacity
Memory Capacity
Number of
Neurons
Hopfield
controlled
SGCM
» controlled
SGCM
Proposed
Network
20 0.15 0.15 0.20 1
50 0.12 0.18 0.18 1
100 0.12 0.18 0.18 1
Average
capacity
0.13 0.17 0.187 1
where k p is the number of recognizable patterns with less than 5% error for the set k. N is the
number of neurons, and M is the number of sets. The same algorithm is applied for memory
capacity for the network with total number of neurons as 50 and 100 neurons. In the case of 50
neurons, among 60 patterns (each 50 bits) 10 learning sets (each set consisting 50 patterns) is
selected. In the case of 100 neurons, among 120 pat
consisting 100 patterns) is selected. The comparison results are shown in
As we can see, the controlled
memory capacity and both have about 50% better perf
proposed method shows a dramatically better performance than Hopfie
Fig 3.14 Three sets of 20 bit patterns in (a), (b)
46
patterns (100 bits), 10 learning sets (each set
) Table 3.1.
SGCM, and the controlled SGCM have almost the same
performance compared to Hopfield. The
Hopfield and both S
and (c).
terns ormance ld SGCM’s.
47
3.9 Discussion
In this chapter, the sinusoidal chaotic neural network was developed. In this model, clustered
attractors could be seen which are important for retrieving information. We can summarize
characteristics of the model as following:
• Sinusoidal chaotic neural network falls into one of the basin of attraction of cluster
attractors. This is important for the information processing application, and helps us to
use the model as memory.
• Information may be stored or disturbed (periodic case vs. chaotic case) depends on the
parameters of network.
• Number of clusters is limited which makes information processing limited that may not
be a good thing though depends on the application.
We also reviewed some application like associative memory. Although these systems have the
covariance learning method, but their associative ability is better than Hopfield network. It
means that dynamics presented in this chapter is better than dynamics in Hopfield network. As
we have seen, our network performed better than chaotic neural network with Logistic maps and
also Hopfield network. We have seen that the new dynamics reached the memory capacity to
100% of the number of neurons, which is improvement of 7.69 times Hopfield network.
Therefore initial chaotic stage of neurons enables the network to assign the neurons in the right
region so that the recognizing of the memory when the network is settled down into attractors
works better than the other two networks.
48
CHAPTER 4
NONLIPSCHITZIAN NEURAL NETWORK
4.1 Introduction
Artificial neural network have been using as a powerful tool for information processing,
pattern recognition, modeling, and systems control. However there are some limitation on
these network among which are the following:
Performance of the network is predefined by initial condition and the network cannot forget
its behavior without external input, while biological neural systems can easily forget about
their past and adapt themselves with the new environment. Another limitation of the
network is the very long convergence time to limit sets of these network. Low memory
capacity is also a limitation factor of these models. Researchers have introduced methods
such as increasing asymptotic stability around equilibrium points to increase the stability of
the equilibrium states [26]. NonLipschitzian dynamics is a way to overcome this limitation
of neural network [25, 26]. In this chapter we show how we investigate this dynamics in the
performance of neural network specifically memory capacity and pattern recognition. We
will see how the violation of the Lipschitz condition causes convergence of all points around
equilibrium points into these points, and indeed makes these points asymptotically stable,
therefore memory of the network increases dramatically. These types of system will not
converge to false steady state [27], thus very good pattern recognizers.
In this chapter, we first review the nonLipschitzian dynamics, and see how non
Lipschitzian dynamics can increase the equilibrium states of neurons. We see how the basin
49
of attraction of neural network can be changed. We then review another type of non
Lipschitzian neural network with several attractors and repellers and investigate the
limitations of this network. We then propose a paradigm for these limitations by
changing the structure of the network, and finally compare the performance of this
network in pattern recognition and memory capacity. We also review nonLipschitzian
dynamics used as learning paradigm, and investigate this type of learning method.
4.2 NonLipschitzian Dynamics
Let’s consider the equilibrium point of the following dynamical system:
Ën
; $1 ^5 ^C  ^6
( +  i
(41)
Using Jacobean matrix, we can determine if these points are attractors or repellers. If the real
part of all eigenvalues of the matrix is negative, the equilibrium point will be asymptotically
stable (attractor). If the real part of all eigenvalues of the Jacobean matrix is positive, then
equilibrium points will be repeller. A nonlinear system can have several attractors and
repellers, while the repellers can act as the separators of the basin of attractors. In order to
have a unique solution, the Lipschitz condition must be satisfied for all elements of the
Jacobean matrix:
ÌÍ n
ÍËv
Ì = (42)
This criteria guarantees unique solution for every initial condition. Conversely, the violation
of Lipschitz condition in an equilibrium point causes not to have unique solution. Let’s
consider the following dynamics:
^ ^
r
Î (43)
50
u can be considered as the state of the neuron. This neuron has the equilibrium point at
u = 0 and the Lipschitz condition is violated at this point because:
Ë
Ë . 5
E ^N\
ÎÏ
Ë#"
# Ð (44)
Since the real parts of eigenvalues of Jacobean matrix is negative, Re{l}®−¥ < 0 ,
equilibrium point is an infinitely stable attractor. The convergence time for a solution with
initial condition of 0 u is limited and equal to:
!" Ñ Ë
Ë
rÎ
Ë#"
ËÒ
E
C ^"
\
Î = Ð (45)
Therefore this attractor is a nonLipschitzian attractor. State space of this system
demonstrates a singular solution which attracts all transient solutions [25]. Fig 4.1 shows a
time series of both Lipschitzian (a) and nonLipschitzian dynamics (b) for different initial
conditions. As it can be seen, in nonLipschitzian dynamics, convergence time to converge
to equilibrium point of u = 0 is limited while in Lipschitzian one unlimited and all transient
solutions end up to a singular solution in nonLipschitzian dynamics.
Fig 4.1: (a) Unlimited convergence time in Lipschitzian dynamics, (b) Limited convergence time in non
Lipschitzian dynamics.
Now let’s consider the following dynamics:
51
^ ^
r
Î (46)
Equilibrium point u = 0 is a nonLipschitzian repeller, because
2
3
0
1
3
u
du
u
du
−
®
= ®¥
&
and
Re{l}®¥ > 0. If the initial condition is so close to this repeller, the transient response will
go away from it in a limited time !" Ñ Ë
Ë
rÎ
ËÒ
Ë#" E
C ^"
\
Î = Ð, while for the Lipschitzian
repeller, divergence time will be unlimited.
As a more general case, let’s consider the following dynamics:
^ Ó ^
F * > & (47)
a is a positive constant. u can be considered as the state of the neuron. The equilibrium
point of this neuron is at u =a . Convergence time (if attractor), or divergence time (if
repeller) is equal to:
!" k
# Ð
ËÒNÔ
ryÊ
5NF
.
$I* B (
$I* = ( (48)
This shows that convergence (or divergence) time in nonLipschitzian dynamics is limited,
while in Lipschitzian dynamics is unlimited. Limited convergence time in nonLipschitzian
dynamics is an important characteristics of this type of dynamics [25,26].
Now let’s consider the following equation:
^ ^ ' ^
Ô
R ^
r
Õ u>0 (49)
where g(.) is the sigmoid function and
( )
, ,
g a
a p T
a
= are constant. By investigating
Lipschitz condition in this system,
1
1 . ( ) ( )
p
p du
T g u u
du
a a
−
= − + ¢ − −
&
at equilibrium point of
52
u =a for p >1 will be unlimited. Therefore for all transient states in the state space, neuron
will converge to equilibrium state in a limited time, thus equilibrium state of u =a will be
asymptotically stable.
Fig 4.2 shows the asymptotic stability of neuron around u = 5 , in the state space of the
neuron of equation (49) for a = 5 . In (a), the parameters are p = 3,5,7,9 and a = 10 . It can
be seen that for greater values of p , asymptotical stability will increase around equilibrium
point. Therefore parameter p can control stability of this system at equilibrium point. In (b),
parameters are a = 0.01,0.1,1,10 and p = 3 . It can be seen that by increasing a , asymptotic
stability increases around equilibrium point. In fact, the first term in the right side of (49),
i.e. u& + u = T.g(u) can be considered as single neuron Hopfield model. The second term
does not change the location of the equilibrium point, it only dramatically changes its
stability.
Fig 4.2 Asymptotic stability of
for
around , (a) For
, and , (b) and .
Let’s now consider a system with several attractor and repeller:
^ Ö ^
r
Î (410)
53
where Ö ^
^ ^5
^ ^C
 ^ ^6
× ^ ^F
6F
45 ^F = ^FG5
This system has n equilibrium points with half of them ( 2k 1 u u − = ) attractor and the second
half repeller ( 2k u = u ) [26]. In Fig 4.3, a schema of this system can be seen. In fact, in the
small neighborhood of basin of attractor (of basin of repeller) k u , f (u) can be linearized:
Ö ^
F ^ ^F
(411)
where F ^F ^5
 ^F ^FN5
^F ^FG5
 ^F ^6
×1~F ^F ^1
and 1/3 u& =[f (u)] will be converted to simple dynamics of (47).
In summary, three important factors of nonLipschitzian dynamics are stability of
equilibrium points, limited convergence time, and ability to control stability and
convergence time.
Fig 4.3 Schema of a system with several attractors and repellers.
Unpredictability in Dynamical System: Unpredictability in deterministic classical dynamics
is due to chaos in nonlinear systems. This is caused by Lyapunov instability due to
exponential decreasing of dependence of the solution on initial conditions. This means that
inability on predicting the behavior of these systems increase exponentially. If there are two
trajectory in the state space very close to each other, they diverge exponentially:
54
J J"`u;, & = 7 = Ð (412)
Therefore for a infinitesimale 0 ®0 , e (t) will be limited only at t ®¥ . That’s why
Lyapunov exponents are defined at t ®¥ as following:
Ø 89:;# z5
; 8< S
SÒ
(413)
Let’s analyze transient escape from nonLipschitzian repeller:
% %
r
Î %" % &
# & (414)
Solution to this equation is 3/2 x(t) = ±t , x ¹ 0. Therefore there would be two different
solutions for almost identical initial conditions. Thus solutions’ divergence will be
determined by infinite Lyapunov exponent:
Ø 89:;#;Ò
.Ù5
; 8< C;
Î\
C3 Ò3
ÚÛ
Ò#"
# Ð (415)
where 0 t is an arbitrary small positive value. All in all, a nonLipschitzian repeller is
strongly unpredictable.
4.3 NonLipschitzian Neural Network
In this section, we review a neural network based upon violation of Lipschitz condition. We
then apply this network to investigate pattern recognition and memory capacity. Each
pattern will be considered as an equilibrium point of the system, therefore each equilibrium
point will have basin of attraction. We will show how to change the basin of attraction, thus
controlling the convergence by network parameters. We will show that this new dynamics
will increase the memory of the network using several examples.
55
Hopfield network has some limitation when it is used to content addressable memory
(CAM). One limitation is the limited number of patterns that can be recognized correctly
and if more new patterns are stored, the network may converge into a wrong pattern. Correct
recognition happens only when number of patterns to store is less than 15% of the total
number of neurons of the network [11, 12, 13]. Thus the synaptic connection will increase
dramatically, because we will then need too many neurons.
Let’s consider a Hopfield network with n neurons:
^ 1 ^1 ? c1f
Vf
45 ^1
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII(416)I
where i u is the state of ith neuron,
is sigmoid function, c1fIis synaptic connection
between jth neuron to ith neuron. If i u vectors are linear independent, then equilibrium
points of the system are defined as:
^1
F
? c1fI Vf
45 ^f
F
(417)
where (k )
i u are vectors that must be stored. If (k )
i u are linear independent, then c1fI’s can be
found using the above mentioned equation. We use localized nonLipschitzian dynamics
around each equilibrium point for its stability, say like ( ) 1/ 3 ( ) k
i i u − u , in order to
asymptotically stabilize this equilibrium point. Therefore:
^ 1 ^1 ? c1fI Vf
45 z^f
F
? 1
F
VF
45 ^1 ^1
F
r
Î`NÜn
Ê ËnNËn
Ê
\
(418)
Important point is that nonLipschitzian dynamics causes convergence to equilibrium point,
thus contrary to Hopfield network, which needs symmetric synaptic coefficient matrix to
converge, there is no condition on this matrix. Exponential term is to localize the effect of
56
nonLipschitzian dynamics around equilibrium point (k )
i u . With the proper selection of (k )
i g
as:
Ï 1
F ^1
F
^1
¢
Ï Ý (, $I* À Á (419)
It can be concluded that:
`NÜn
Ê ËnNËn
Ê
\ Þ(&
.
$I^1 ß ^1
F
$^1 À ^1
F
(420)
Basin of attraction for different equilibrium point can be changed by positive parameter (k )
i a
and (k )
i g . Now all the equilibrium points of (42) are asymptotically stable. To prove that,
system (43) can be linearized with respect to points ( ) ˆ k
i u which are close enough to
equilibrium points (k )
i u , i.e. ( ) ( ) ˆ 0 k k
i i u − u =e ® :
^ 1 ? à1f
F
Vf
45 ^á1
F
^1
F
$I* ( +  i (421)
where:
à1f
F
â
ãnv
äåæ\ Ëv Ê
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
( ãnv
äåæ\ Ëv Ê
1
F
5
E JN\
Î +^1
F
J
ç
Î
`NÜn
ÊS\
.
$I À t
$I t (422)
Therefore stability of the equilibrium points (k )
i u depends on eigenvalues of matrix èà1f
F
é.
To find these eigenvalues, let’s define àê1f
F
à1f
F
JC E, thus:
.àê1f
F
Ï
Ë#"
# k
&
5
E 1
F
.
$I À t
$I t (423)
Now to calculate eigenvalues:
ëF àê1fJN\
Î .5
E à1
F
JN\
ÎÏ
S#"
# Ð (424)
57
Therefore all eigenvalues of matrix èà1f
F
é at equilibrium points (k )
i u are negative and
unlimited, i.e. they are nonLipschitzian equilibrium points.
System (43) has some other equilibrium points that are repellers. Indeed, these repellers are
the border between the above mentioned attractors. By incorporating nonLipschitzian
dynamics into neural network, n vectors can be stored in the equilibrium points of the
network with no wrong convergence. Furthermore the synaptic connection matrix need not
to be symmetric.
4.3.1 Investigating performance of nonLipschitzian neural network for pattern recognition
Let’s consider a nonLipschitzian neural network in its simplest form having two neurons.
Let’s also store two patterns Z( ¯& + & ,° Z+ ¯& ¶ & ì°. We select parameters (k )
i a
identical and equal to 0.1, and parameter (k )
i g is chosen somehow that satisfies
( ) ( ) ( ) .min( ) 100 k k l
i i i g u −u = . (In this case, (1) (2)
1 1 g =g = 250 and (1) (2)
2 2 g =g = 167 . ij T is
calculated based on (42). Fig 4.4 shows how different initial pattern can converge through
different trajectories to the trained patters P1 and P2. We can see that transient solution near
equilibrium points are converged to a singular solution, which is the trained pattern. Patterns
which are attracted towards P1 are:
[0.13,0.23], [0.15,0.25], [0.17,0.27]
[0.22,0.32], [0.24,34], [0.26,0.36]
g h i
j k l
= = =
= = =
Patterns attracted towards P2 are:
[0.4,0.6], [0.5,0.8], [0.7,1.0]
[0.8,1.1], [0.9,1.2], [1.0,1.3]
a b c
d e f
= = =
= = =
Fig 4.4 Trajectory of the attraction of different p
b=[0.5 0.8], c=[0.7 1.0], d=[0.8 1.1]
(a) Change of basin of attraction by network parame
Lipschitzian neural network with two neurons that h
setting parameter ( ) 0.001 k
i a
=
respectively:
Now if we change parameter
Additionally, border between the basin of attractio
58
patterns to the equilibrium points:
[1.1], e=[0.9 1.2], f=[1.0 1.3], g=[0.13 0.23], h=[0.15 0.25]
j=[0.22 .32], k=[0.24 0.34], l=[0.26 0.36].
parameters: Let’s again consider the non
has stored two patterns P1 and P2. By
, we can see that X1 and X2 will be attracted to P1
X1 [0.227,0.327] 0.001 P1 a = = ¾¾¾®
0.001 X 2 [0.228,0.328] P2 a = = ¾¾¾®
(k )
i a
to ( ) 0.01 k
i a
= , pattern X2 will also be attracted to P1.
attraction of two patterns will change in this way:
0.01 X3 [0.2695,0.3695] P1 a = = ¾¾¾®
atterns a=[0.4 0.6],
0.25], i=[0.17 0.27],
nonas
and P2
n 59
0.01 X 4 [0.2696,0.3696] P2 a = = ¾¾¾®
Now if we change parameter (k )
i a
to ( ) 0.1 k
i a
= , pattern X2 and X4 both will be attracted to
P1 and the border between the basin of attraction of two patterns will change in this way:
0.1 X5 [0.300,0.400] P1 a = = ¾¾¾®
0.1 X6 [0.301,0.401] P2 a = = ¾¾¾®
Fig 4.5 shows the change in the border between basin of attraction of two equilibrium points
of first and second neuron for three different values of (k )
i a
. As it can be seen, by increasing
(k )
i a
(identically for all i and k), from 0.001 to 0.01 and 0.1, basin of attraction for the
second pattern is decreased and that of first pattern increased.
(b) Improving signal to noise by change of network parameters: Let’s consider the same
network and the same stored pattern with fixed ( ) 0.001 k
i a
= . If we change the stored pattern
by noise less than 0.01, the network will recognize the output correctly:
1 1
2 2
P noise P
P noise P
+ ®
+ ®
But if we increase the noise, network will converge to P2 (stronger pattern), i.e. signal to
noise ratio in this case to recognize correct pattern is 100. Now if we increase (k )
i a
due to
first pattern for first neuron (i.e. (1)
1 a ), from 0.001 to 1, maximum noise that can be applied
to the patterns to converge to the correct pattern will be 0.6 (noise<0.6). Therefore the signal
to noise ratio to recognize correctly will be 17.
60
Fig 4.5 Change of basin of attraction of the equilibrium point with the change of
: (a) First neuron,
(b) Second neuron.
(c) Change of bifurcation surface by network parameters: Bifurcation surface for our
same network with ( ) 0.001 k
i a
= is depicted in Fig 4.6 (part a). If (k )
i a
is increased to 1 (for
all i and k), then the bifurcation surface between two patterns change and gets closer to the
second pattern. In (b), we can see how the change of parameter (k )
i a
from 0.1 to 10 for two
other patterns can change the bifurcation surface.
(d) NonLipschitzian neural network with five neurons: NonLipschitzian neural network
with five neurons and the following five patterns are designed:
We set all ( ) 1 k
i a
= and choose
done using (417). In the recognition process, each pattern gets
then see the networks output. The result of the recognition process is ga
As we can see in this table, the average recognitio
the network always converges to one of the stored p
recognition does not happen, it will attract to the str
Fig 4.6 Moving of the bifurcation surface of the ne
Table 4.1 Results of recognition of noisy patterns
Input S0 S1 S2
P1+N P1 P2 P1
P2+N P2 P4 P2
P3+N P2 P3 P3
P4+N P4 P4 P2
P5+N P5 P5 P5
61
1 [7.2,9.9,8.8,2.3,3.0]
2 [3.5,5.1,5.9,8.4,4.1]
3 [8.4,2.6,4.1,5.3,4.6]
4 [2.8,1.7,1.5,5.7,8.0]
5 [0.3,5.3,4.9,9.5,7.4]
P
P
P
P
P
=
=
=
=
=
(k )
i g somehow that ( ) ( ) ( ) .min( ) 1000 k k l
i i i g u − u =
noisy by less than 0.5 and
gathered in Table 4.1.
recognition rate for patterns is 72%, and furthermore
patterns of P1 to P5, and if the
gnition strong pattern of P2.
network with two neurons with patterns P1 and P2.
P1, P2, P3, P4, and P5.
S3 S4 S5 S6 S7 S8
P2 P1 P1 P1 P1 P1
P2 P2 P2 P2 P2 P2
P3 P2 P2 P3 P3 P3
P2 P4 P4 P4 P4 P4
P2 P5 P2 P5 P5 P5
Mean Recognition Rate
. Training is
thered n atterns twork S9
Correct
Recognition
P2 70%
P2 100%
P3 70%
P2 60%
P2 60%
72%
62
4.3.2 Discussion: In this section we investigated how the nonLipschitzian dynamics can
asymptotically stabilize the equilibrium points of the neural network, which in turn increase
the network’s memory. For example by using this dynamic, the capacity of Hopfield
network that is 15% of the number of neurons, will be increased to number of neurons.
Additionally there is no limitation on having symmetric synaptic connection, which indeed
makes the neural network more similar to biological neural systems [8, 18, 19]. Another
property of the network is that since all equilibrium points are stable, there are no wrong
attractor in the network, so it can be used as a base for pattern classification [26]. In the
developed neural network, we also could control basin of attraction of the network by
controlling parameters of the network.
4.4 Proposed Network: NonLipschitzian Neural Network with Repellers
and Attractors
In this section, we investigate a type of neural network that is based on violation of Lipschitz
condition. This nneuron network can store n/2 patterns. We improve the performance of
this network thus it is able to store n patterns in the network. Let’s start with a simple single
neuron system:
^ ^ c ^
í ^ ^á 5
5 RI Ä > ( (425)
where
is sigmoid function, u(t) is the state of neuron, (1) uˆ an equilibrium with infinite
stability. If constant value of cI is selected as:
c ^á 5
¯ ^á 5
°IN5 (426)
63
It can be shown that the second term in (425) has no effect on location of equilibrium points
but only on increasing the stability of these points [27]. Now let’s consider an network with
N neurons with a structure similar to that of Hopfield network [13]:
^ 1 ^1 ? c1f
Vf
45 ^f
(427)
If i u vectors are linear independent, the equilibrium points will be defined as:
^1
F
=? c1f
Vf
45 ^f
F
(428)
where (k )
i u are the vectors that must be stored. c1f will be found from (428) if (k )
i u are
linear independent. Incorporating nonLipschitzian dynamics with this network gives us:
^ 1 ^1 ? c1f
Vf
45 ^f
¯Ö ^1
°I
r
Õ (429)
Ö ^1
^1 ^5
F
^1 ^C
F
 ^1 ^6
F
× ^1 ^f
F
Vf
45 (430)
It can be seen that equilibrium points of the initial network (427) lie among equilibrium
points of the nonLipschitzian dynamics of (429). NonLipschitzian dynamics of 1/ [ ( )] p
i f u
does not affect location of equilibrium points of (427), but we can control the stability of
the equilibrium points of the network (attractors and repellers) [26]. This network can not
store half of the patterns. To overcome this limitation, we introduce a new structure for non
Lipschitzian neural net.
4.4.1 Modification of the nonLipschitzian Neural Network
The network in (429) has N equilibrium points. But due to nonLipschitzian dynamics,
1/ [ ( )] p
i f u , points 2 1
k
k u u − = are attractor and 2
k
k u = u are repeller. This means that the network
consisting n neurons can store n/2 patterns, because half of the patterns will be located in
64
repellers. Therefore the network can not retrieve half of the patterns. We have improved this
characteristics by introducing the following:
^ 1 ^1 ? c1f
Vf
45 ^f
¯î ^1
°I
r
Õ (431)
!d !ïIî ^1
^1 "
F
^1 ^5
F
^1 5
F
^1 ^C
F
 ^1 ^6
F
^1 6
F
nonLipschitzian dynamics of 1/ [ ( )] p
Y ui has 2n equilibrium points which half of them are
attractors and the other half are repeller. Repeller points are located between every two
attractors and can be found using:
f
F
Ëv
Ê
GËvxr
Ê
C
t ( + i (
(432)
4.4.2 Investigating on pattern Recognition and Memory
In order to investigate the performance of the network (431), Gaussian random patterns are
generated:
ð F
^5 F
^C F
 ^6 F
ñ K à F
Ø F
where ð F
^5 F
^C F
 ^6 F
is random pattern, with Gaussian distribution with mean
à F
and variance Ø F
of the kth class. We have considered 10 patterns in 10 class.
Network synaptic coefficient c1f is calculated using 10 random pattern (one pattern from
each class). These patterns are used as equilibrium points of the network, then recognition
process is made using different patterns in each class. In Table 4.2, steady state convergence
error which is normalized mean squared error as defined below, for recognizing different
patterns from each class is shown.
65
KÉYºF 5
V
?
Ëv
Ê
N n
òv
Ê
CI Vf
45
r
\
(433)
where k
i u are vectors that must be stored, 1 2 ( , ,..., ) n X = x x x is a vector in which network is
converged to. We should note that 1 2 ( , ,..., ) n X = x x x belongs to a class that has the least
steady state convergence error:
ó ± ÅÁ µ
$ïIKÉYºô = KÉYºF * ( + iIõIµ À *
(434)
If NMSE for recognizing an unknown pattern is too large, we don’t assign it into any class.
In Table 4.2, this inability is shown as “false”. In Table 4.3, percentage for correct
recognition of different patterns in each class is shown. In order to compare, in Table 4.4,
the same percentage for the network of (429) is shown. As it can be seen, pattern
recognition is improved from 40% to 86%. In Fig 4.7, convergence graph for different
patterns in each class for the improved network (431) is shown.
4.5 NonLipschitzian Learning
In the previous networks, ij T are found adhoc. If the number of neurons is too many, this
does not seem to be a good method for computational expenses. In this section a learning
method based on the nonLipschitzian dynamics is reviewed and then we modify it to
control the parameter of learning process.
4.5.1 NonLipschitzian dynamics in learning process
Let’s define the energy function for synaptic connections as following [30]:
66
Table 4.2 Steady state error for convergence of different patterns in 10 different classes (Sample pattern
A from Class 1, B from class 2, C from class 3, D from class 4, E from class 5, F from class 6, G from
class 7, H from class 8, I from class 9, J from class)
Pattern NMSE1 NMSE2 NMSE3 NMSE4 NMSE5 NMSE6 NMSE7 NMSE8 NMSE9 NMSE10 Result
A 0.572 4.410 13.53 17.93 28.43 36.59 47.15 54.43 65.32 70.16 True
B 8.309 0.027 8.556 17.11 24.83 33.89 41.64 50.00 58.45 66.18 True
C 25.07 16.99 2.777 .002 8.102 16.83 24.85 32.98 41.45 49.39 False
D 0.436 4.568 12.87 18.74 27.69 35.87 47.66 53.70 64.76 69.76 False
E 0.213 5.428 10.12 16.34 24.83 33.92 44.61 52.37 61.54 65.83 False
F 29.45 23.56 14.45 9.657 4.452 0.005 5.905 13.42 19.22 24.34 True
G 38.40 34.47 26.64 19.57 12.487 5.537 0.007 6.673 14.35 21.63 True
H 45.90 39.46 31.37 23.26 15.47 6.870 3.530 0.014 7.352 13.67 True
I 53.88 44.45 35.32 28.32 20.51 15.35 9.675 4.563 0.060 5.342 True
J 69.70 61.46 49.67 42.66 38.76 25.46 19.74 13.69 5.069 1.091 True
Table 4.3 Percentage for recognition of the patterns from different class for (431)
Class 1 2 3 4 5 6 7 8 9 10 Average
Percentage 100 90 80 40 70 80 90 100 100 100 86%
Table 4.4 Percentage for recognition of the patterns from different class for (429)
Class 1 2 3 4 5 6 7 8 9 10 Average
Percentage 100 100 0 0 0 0 0 0 100 100 40%
.
Fig 4.7 Convergence of the network of equation (4
class, (a) Class 1, (b) Class 2, (c) Class 3, (d) C
67
431) due to pattern recognition of samples of differ
Class 4, (e) Class 5, (f) Class 6, (g) Class 7, (h) Class 8,
different
68
Fig 4.7 (cont.) Convergence of the network of equation (431) due to pattern recognition of samples of
different class, (i) Class 9, (j) Class 10.
(435)
This energy is a function of connection coefficients which in turn can be found using:
= (436)
The above energy function must be minimized. Each new pattern for learning constrains
new criteria on . In the case of the content addressable memory (CAM), these limitations
can be expressed as following:
= (437)
By enforcing the Lagrange Multiplier techniques, the modified energy function can be
written as:
(438)
where ik
l are constants that need to be found. Since energy function of ˆE
is a secondorder
function of and ik
l , its gradient, ˆÑE , will be a linear function of these variables,
therefore:
69
Íö÷
Íønv t
ICc1f+? 71F
}F
45 z^ê1
F
(439)
Íö÷
ÍunÊ ? c1f
Vf
45 z^êf
F
^ê1
F
(450)
By linearization of these two equations around equilibrium point [27], we then have the
following dynamics governing the learning process:
c 1f C¯ t
ICc1f+? 71F
}F
45 z^êf
F
° I t ( + K (441)
7 1F C[? c1f
Vf
45 z^êf
F
^ê1
F
° * I( + É ù K (442)
C. 3? ? ¯ t
ICc1f
Vf
45
V1
45 ? 71F
}F
45 z^êf
F
°C ? ? ¯? c1f
Vf
45 z^êf
F
}F
45
V1
45
^ê1
F
°C 3ú\
(443)
4.5.2 Dynamic sigmoid steepness during learning process
Let’s again see the energy function of (438). The sigmoid function in the last section had a
constant steepness, while we make this steepness different for every neuron in the network,
namely:
^1
·¸<¹¯ 1 !
^1° ( +  i (444)
This means that steepness of each sigmoid function varies; therefore along with
minimization of the energy function we should also satisfy the following:
Íö÷
Íûn & ( +  i (445)
4.5.3 Performance of nonLipschitzian learning process
In this section, we simulate network of (441) and (442) with five neurons and parameter
2
3
g = − . The following five continuous sample patterns are to be stored:
70
(446)
The sigmoid function is tanh(.) with b = 5 . After applying nonLipschitzian learning
process of (441) and (442) the network converges and the following synaptic connection is
found:
1.04 0.04 0.00 0.03 0.03
0.02 1.04 0.01 0.01 0.02
0.01 0.02 1.07 0.01 0.03
0.01 0.00 0.01 1.09 0.05
0.00 0.01 0.01 0.01 1.06
T
−
− −
= − −
− −
− − −
(447)
where . This network behaves like an associative memory. The learning speed can
be controlled in this network.
(a) Learning Rate: In Fig 4.8, variation of energy function for different g is depicted. As it
can be seen, by increasing parameter g , learning rate of the network increases.
Fig 4.8 Increasing learning rate by increasing parameter from to and .
71
(b) Effect of convergence coefficient in learning: In this section, we substitute the varying
coefficient of 2
a with 2 μ.a where μ is positive constant fraction number. In Fig 4.9 Fig
4.10, the variation of the energy function for μ = 0.007 and μ = 0.02 are shown. The
dynamics of the parameters during the learning process are also shown. In (a), energy
function E, in (b) time series of ik g (equation 447), in (c) the energy function with
constraints, ˆE
, and finally (d) shows times series of connection coefficients of the network,
c1f. It can be seen that increasing μ , increases the learning rate.
Fig 4.9 Time series of the network with : (a) Energy function, E, (b) , (c) Energy function
with constraints,
, (d)Network’s connection coefficient, .
72
4.6 Discussion
In this chapter, we reviewed and modified nonLipschitzian dynamics used for neural
network. Three important factors about this dynamics is stability of equilibrium point,
limited convergence time, and ability to control stability and convergence time of the
dynamics. Using these properties, different type of nonLipschitzian neural networks have
been reviewed and modified. As a result, we did increase the memory capacity of the
network, using this dynamics dramatically. In one of the networks that we modified, we
could change the basin of attraction of the network.
Fig 4.10 Time series of the network with : (a) Energy function, E, (b) , (c) Energy function
with constraints, , (d)Network’s connection coefficient, .
A network with some repellers and attractors was reviewed. This network with n neurons is
able to store n/2 patterns, because half of the patterns are located in the repellers of the
73
network, therefore half of the patterns will not be stored in the network. To overcome this
weakness, we have introduced a new structure of the nonLipschitzian neural network.
This modified network will be able to store n patterns, because all patterns are located
in the attractors. We also reviewed the performance of this new network in pattern
recognition and memory capacity and observed that recognition rate of correct pattern is
86% while the original nonLipschitzian network has the recognition rate of 40%. We also
reviewed learning methods based on nonLipschitzian dynamics. Learning method is based
on minimization of the energy of the synaptic connection. It implies a dynamics in the
coefficient space and the learning trajectory will converge to a nonLipschitzian dynamics.
74
CHAPTER 5
CONCLUSION
In this thesis, we first reviewed the chaos theory and some of the measures that can be used
to show a chaotic behavior. We have seen that we can divide the dynamical system due to
their asymptotic behavior into four different category: equilibrium points, periodic (stable
limit cycle), quasiperiodic, and chaotic. Chaotic system have very sensitive behavior to
initial condition. Frequency spectrum of the chaotic system similar to a noise includes all
frequencies. State space of the chaotic system has a nonEuclidean and fractal shape,
therefore the trajectories in the state space can diverge and then converge. To distinguish
among different types of systems and recognizing the chaotic behavior, measure of
Lyapunov exponent and dimension was reviewed. In fact, a chaotic system is a system that
has at least one positive Lyapunov exponent. Another measure that was introduced was
dimension that is a fraction number for chaotic systems.
Neurobiological research has shown that behavior of the biological neurons are complex and
chaotic [610]. In fact, chaotic behavior of neurons can be observed experimentally.
Existence of this chaotic behavior is evident in different levels of the neurons and is a
driving motivation for developing chaotic neural network. In this thesis after reviewing
chaotic neurons, we investigated chaotic neural network. We have seen that the chaotic
neural network with chaotic movement in the search space, searches the whole state space.
Contrary to steady behavior, or neartoequilibrium in classical neural network, chaotic
neural network has a behavior far away from equilibrium points. Researchers have shown
that chaotic dynamics can be used for optimization and pattern recognition [14, 16, 17, 23,
75
24, 27, 28, 33, 36]. Transient chaotic neural network has also been used for solving
optimization problems such as Travelling Salesman Problems (TSP). Another type of
chaotic neural network is a network with chaotic Logistic map.
We introduced a sinusoidal chaotic neural network. In this model, we have seen the cluster
attractors that could be used for retrieving information. Important characteristics of this
model are:
• Sinusoidal chaotic neural network will fall into one of the cluster attractors. This is
important for the information processing application.
• The information can be stored or disturbed (equilibrium case vs. chaotic case)
depending the values of the network parameters.
We have seen that Logistic map based chaotic neural network compared to Hopfield
network has the improvement of 50% of performance, but our sinusoidal chaotic neural
network increases the memory capacity to 100% of the number of the neurons of the
network. Therefore chaotic dynamics is able to assign the neurons in the proper location in
the state space that improves the information processing. In these networks, chaotic
movements help the neuron to escape from local optima, thus false pattern. This
improvement in pattern recognition is done by the help of chaotic dynamics where has
enough space between attractors as it could be seen in difference between sinusoidal chaotic
neural network and chaotic neural network based on Logistic map.
We have also reviewed and developed the nonLipschitzian neural network It has been
shown that violation of the Lipschitz condition, causes convergence of the points around
equilibrium point to itself and asymptotically stabilizes these points. This characteristics has
76
been used for pattern recognition and content addressable memory (CAM) of the neural
network. In fact there are three important factors about this dynamics: stability of the
equilibrium points, limited convergence time, and being able to control stability and
convergence time. We have seen that the shape of basin of attraction can be changed by
using the parameters of the neural network. Structure of the nonLipschitzian neural network
is very similar to the Hopfield network, while it does not have the limitation of having
symmetric synaptic coefficient matrix. This property, having asymmetric synaptic
coefficient matrix is more compatible with the biological neural systems [8, 18, 19].
Hopfield network when used as a content addressable memory has a main limitation:
number of patterns that can be stored and retrieved correctly is very limited and if the
number of stored pattern is increased, then the network may converge into a false pattern
[13]. When Hopfield network is used as a pattern classifier, it can retrieve correctly only if
the number of stored randomgenerated patterns is less than 15% of the number of neurons
of the network [11, 12, 13]. Therefore the number of coefficients of the synaptic connection
will be increased dramatically. One of the other properties of the nonLipschitzian neural
network is that there is no false attractor in this network since all the equilibrium points are
asymptotically stable. This fact can be used for pattern classification [26]. The basin of
attraction in this network is very sensitive to network parameters and their adjustment is a
difficult issue. We have also found that however the memory capacity had been increased
but pattern recognition depends on bifurcation surface, which in turn, its adjustment depends
on the network parameters.
77
By the help of nonLipschitzian dynamics, we can produce repellers and attractors in the
network. Attractors and repeller equilibrium points will increase pattern recognition ability
of the network. NonLipschitzian neural network with these attractors and repellers have
limitation on number of stored patterns. A network with n neurons is able to store n/2
patterns, because half of the patterns are located on repellers. To overcome this limitation,
we introduced new structure for nonLipschitzian neural network. The modified network is
able to store n patterns, because locates all patterns in the attractors. We have also
investigated the performance of this modified network on pattern recognition and memory
capacity. We have seen that the correct recognition rate is 86%. Additionally we can
increase the recognition rate by selecting a proper degree of the nonLipschitzian dynamics.
We have also reviewed the learning methods based on nonLipschitzian dynamics. This
process is based on minimization of the energy of the synaptic connection. It concludes a
dynamics in the coefficient space and the learning trajectory will converge to a non
Lipschitzian dynamics.
As future work of this research, chaotic dynamics can be applied in other types of classical
neural networks and compare their performance with the classical neural networks. For
example we may consider the parameters of a neural network having chaotic dynamics, and
they could reach their optimal value after passing the transient chaotic behavior.
Sinusoidal chaotic neural network can be used for optimization problems like Travelling
Salesman Problem (TSP). Instead of sinusoidal chaotic map, we can use other types of
chaotic function to investigate the behavior of new dynamics on the neural network.
78
NonLipschitzian dynamics can be used for other types of neural networks that are based on
the dynamical systems and guarantee the convergence of the network into the equilibrium
points. We can also use this dynamics in gradient based neural network.
From application point of view, we can apply chaotic neural network on pattern recognition,
pattern classification, modeling and optimization.
79
REFERENCES
[1] P. Grassberger, and I. Procaccia, “Characterization of strange attractors,” Physical Rev.
Lett., vol. 50, pp. 346349,1983.
[2] P. Grassberger, and I. Procaccia, “Measuring the strangeness of strange attractors,”
Physica, vol. 9D, pp. 189208, 1983.
[3] F.C. Moon, “Chaotic and fractal dynamics: An introduction for applied scientists and
engineers,” John Wiley & Sons. Inc., 1992.
[4] S. H. Strogatz, “Nonlinear dynamics and chaos with application to physics, biology
chemistry, and engineering,” AddisonWesley Publishing Inc., 1994.
[5] T. S. Parker, and L. O. Chua, “Chaos: A tutorial for engineers”, Proc. of the IEEE,
Special Issue on Chaotic Systems, pp. 9821008, 1987.
[6] T. R. Chay, and Y. S. Fan, “Bursting, spiking, chaos, fractals and universality in
biological rhythms,” Int. J. of Bifurcation and Chaos, vol. 5, pp. 595635, 1995.
[7] Y. Yao, and W.J. Freeman, “Model of biological pattern recognition with spatially
chaotic dynamics,” Neural Networks, vol. 3, pp. 153170, 1990.
[8] W.J. Freeman, “Tutorial in neurobiology: From single neurons to brain chaos,” Int. J. of
Bifurcation and Chaos, Vol.2, pp. 451482, 1992.
[9] R Kozma, and WJ Freeman, “Chaotic resonance: Methods and applications for robust
classification of noisy and variable patterns,” Int. J. of Bifurcation and Chaos, vol.
10, pp. 23072322, 2001
[10] C.S. Skarda, and W.J. Freeman, “How brain make chaos in order to sense of the
world,” Brian Behavioral Sci., vol. 10, pp. 161195, 1987.
[11] J.J. Hopfield, “Neural networks and physical systems with emergent collective
computational abilities,” Proc. Natl. Acad. Sci. USA, vol. 79, pp. 25542558, 1982.
[12] S.V.B. Ayier, M. Niaranjan, and F. fallside, “A theoretical investigation into the
performance of the Hopfield model,” IEEE Trans. on Neural Network, vol. 1, No. 2,
pp. 204215, 1990.
[13] R.P. Lippmann, “An introduction to computing with neural nets,” IEEE ASSP Mag., pp.
422, 1987.
80
[14] M. Adachi, and K. Aihara, “Associative dynamics in a chaotic neural network,” Neural
Networks, vol. 10, No. 1, pp. 8398, 1997.
[15] K. Aihara, T. Takabe, and M. Toyoda, “Chaotic neural network,” Phys. Lett. A, vol.
144, pp. 333340, 1990.
[16] L. Chen, and K. Aihara, “Chaotic simulated annealing by a neural network model with
transient chaos,” Neural Networks, vol. 8, pp. 915930, 1995.
[17] M. Adachi, K. Aihara, and M. Kotani, “An analysis of associative dynamics in a
chaotic neural network with external stimulation,” Proc. of Int. Joint Conf. on Neural
Networks, pp. 409412, 1993.
[18] Z. Xu, G. Hu, and, C. Kwong, “Asymmetric Hopfieldtype networks: theory and
application”, Neural Networks, vol.9, pp. 483501, 1996.
[19] T. Fukai, and M. Shinno, “Asymmetric neural networks incorporating the Dale
hypothesis and noisedriven chaos”, Physical. Rev. Lett., vol. 64, pp. 14651468,
1990.
[20] M. Inoue, and A. Nagayoshi, “A chaos neurocomputer”, Phys. Lett. A, vol.158, pp.
373376, 1991.
[21] H. Sompolinsky, and A. Crisanti, “Chaos in random neural networks,” Phys. Rev. Lett.,
vol. 61, pp. 1720, 1988.
[22] J.C. Principe, and P.C. Lo, “Chaotic dynamics of timedelay neural networks,” Proc. of
Int. Joint Conf. on Neural Networks, vol. 2, pp. 403409, 1990.
[23] J.M. Bertille, and J.C. Perez, “A spatiotemporal novelty using fractal chaos model,”
Proc. of Int. Joint Conf. on Neural Networks, vol. 2, pp. 361364, 1990.
[24] S. Ishii, K. Fukumizu, and S. Watanabe, “A network of chaotic elements for
information processing,” Neural Networks, vol. 9, pp. 2540, 1996.
[25] M. Zak, “Introduction to terminal dynamics,” Complex Systems, vol. 7, pp. 5987,
1993.
[26] M. Zak, “Terminal attractors in neural networks,” Neural Networks, vol. 2, pp. 259
274, 1989.
81
[27] M. Zak, “Terminal attractor for addressable memory in neural networks,” Phys. Lett. A,
vol. 133, pp. 1822, 1988.
[28] M. Zak, “Terminal chaos for information processing in neurodynamics,” Biol. Cybern,
vol. 64, pp. 343351, 1991.
[29] M. Zak, “An unpredictabledynamics approach to neural intelligence,” IEEE Expert,
pp. 410, August 1991.
[30] M. Zak, “The least constraint principle for learning in neurodynamics,” Phys. Lett. A,
vol. 135, pp. 2528, 1989.
[31] I. Tsuda, “Dynamic link of memorychaotic map in nonequilibrium neural networks,”
Neural Networks, vol. 5, pp.313326, 1992.
[32] T. Deguchi, and N. Ishii, “Search of associative memories based on feature matching in
chaotic neural network,” Proc. of the 3rd Int. Conf. on Fuzzy Logic, Neural Nets and
Soft Computing, pp. 423426, 1994.
[33] F. ChapeauBlondeau, “Analysis of neural networks with chaotic dynamics,” Chaos,
Solitons and Fractals, vol. 3, pp. 133139, 1993.
[34] C.M. Thomas, W.G. Gibson, and J. Robinson, “Stability and bifurcations in an
associative memory model,” Neural Networks, vol. 9, pp.5366, 1996.
[35] S. Nara, P. Davis, and H. Totsuji, “Memory search using complex dynamics in a
recurrent neural network model,” Neural networks, vol. 6, pp. 963973, 1993.
[36] M. Pandin, and G. Didone, “Chaos in information processing: simulation of a biological
learning process by time evolution in an unsupervised neural network,” Int. J. of
bifurcation and Chaos, vol. 6, pp. 203210, 1996.
[37] X.S. Xu, Z. Tang, and J.H. Wang, “A method to improve the transiently chaotic neural
network,” Neurocomputing, vol. 67, pp. 456–463, 2005.
[38] Y. Song, Z.Q. Chen, and Z.Z. Yuan, “New chaotic PSObased neural network
predictive control for nonlinear process,” IEEE Trans. on Neural Networks, vol. 18,
pp. 595600, 2007.
ABSTRACT
Nonlinear dynamics play an important role on biological systems, including
biological neural systems. Researchers have shown that nonlinear dynamics and chaos
exist in several levels of neural systems from micro to macro levels. This is a strong
motivation for the addition of nonlinear dynamics and chaos into neural networks. In this
thesis, a new type of neural network is introduced based on the nonlinear dynamics and
chaos. Bifurcation behavior of the neural network shows that the state of the neurons will
move chaotically among the stored states of the network but finally, by decreasing the
degree of chaos, settles down into a proper state. Therefore this new type of neural
network with chaotic dynamics is able to search the state space and seeks the points far
away from equilibrium points. We also find that increasing the memory capacity of the
neural network is one of the consequences of adding the nonlinearity into it. This network
can also be used for pattern recognition, classification, and optimization.
VITA
Moayed Daneshyari
Candidate for the Degree of
Master of Science
Thesis: BIFURCATION AND NONLINEAR DYNAMICS IN NEURAL NETWORK
Major Field: Physics
Biographical:
Education:
o B.S. in Electrical Engineering with emphasis in Automation and Control,
Sharif University of Technology, Tehran, Iran, 19911995, Graduation
Date: September 1995.
o M.S. in Biomedical Engineering with emphasis in Artificial Intelligence,
Iran University of Science and Technology, Tehran, Iran, 19951998,
Graduation Date: October 1998.
o M.S. in Physics, Oklahoma State University, Stillwater, Oklahoma,
December, 2007.
o Ph.D. Student in Electrical and Computer Engineering with emphasis
in Computational Intelligence, Oklahoma State University, Stillwater,
Oklahoma, 2003present,
Professional Memberships: Institute of Electrical and Electronics Engineering
(IEEE), Computational Intelligence Society, Systems Man and
Cybernetics Society, Engineering in Medicine and Biology Society
ADVISER’S APPROVAL: Dr. Paul A. Westhaus
Name: Moayed Daneshyari Date of Degree: May, 2008
Institution: Oklahoma State University Location: Stillwater, Oklahoma, USA
Title of Study: BIFURCATION AND NONLINEAR DYNAMICS IN NEURAL
NETWORK
Pages in Study: 81 Candidate for the Degree of Master of Science
Major Field: Physics
Scope and Method of Study:
Developing the nonlinear neural network was done using simulation with Matlab.
Conclusions:
We introduced a nonlinear network by incorporating nonlinear dynamics into neural
network which showed a great performance in pattern recognition compared to
Hopfield and the other types of nonlinear networks. Nonlinear dynamics is able to
assign the neurons in the network in the proper region during the state space search.
Chaotic movements also help the neuron to escape from local optima, which
guarantees the stability of equilibrium points of the network. As a result the memory
capacity of the network is increased.