IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 The Effect of Number of Training Samples for Artificial Neural Network L. N. M.Tawfiq , M. N. M.Tawfiq Department of Mathematics, College of Education-Ibn Al-Haitham, University of Baghdad . Abstract In this paper we study the effect of the number of training samples for Artificial neural networks ( ANN ) which is necessary for training process of feed forward neural network .Also we design 5 Ann's and train 41 Ann's which illustrate how good the training samples that represent the actual function for Ann's. 1. Introduction Artificial Neural Networks ( ANN ) are relatively crude electronic models based on the neural structure of the brain. The brain basically learns from experience. It is a natural proof that some problems that are beyond the scope of current computers are indeed solvable by small energy efficient packages. This brain modeling also promises a less technical way to develop machine solutions. This new approach to computing also provides a more graceful degradation during system overload than its more traditional counterparts. The, advances in biological research promise an initial understanding of the natural thinking mechanism. This research shows that brains store information as patterns. Some of these patterns are very complicated and allow us the ability to recognize individual faces from many different angles. This process of storing information as patterns, utilizing those patterns, and then solving problems encompasses a new field in computing. This field, as mentioned before, does not utilize traditional programming but involves the creation of massively parallel networks and the training of those networks to solve specific problems. This field also utilizes words very different from traditional computing, words like behave, react, self- organize, learn, generalize, and forget. [1] 2. Artificial Neurons and Their Work The fundamental processing element of artificial neural network (ANN) is a neuron. A neuron is an information-processing unit that is fundamental to the operation of a neural networks, Figure (1) shows the model of a neuron, which forms the basis for Ann's. The artificial neurons we use to build our neural networks are truly primitive in comparison to those found in the brain. An artificial neuron has several inputs but only one output. Here we identify three basic elements of the neuronal model [2], [3] : 1. Synapses or connecting links, each is characterized by a weight or strength of its own. a Specifically, a signal xj at input of synapse j connected to a neuron is multiplied by the synaptic weight wj. 2. An activation function for limiting the amplitude of the output of a neuron. IHJPAS IBN AL- HAITHAM J. FOR PURE & APPL. SCI. VOL.23 (3) 2010 3. The model of a neuron also includes an external bias, denoted by b, which has the effect of increasing or lowering the net input of the activation function. In mathematical terms, we may describe a neuron by : y = φ (  N j jjxw 1 + b) ……… ( 1 ) Where x1,…xn are the input signals w1,…,wn are the synaptic weights of the neuron, b is the bias, φ is the activation function and y is the output signal of the neuron. 3. Training of Ann's Basically, training is the process of determining the weights which are the key elements of an ANN. The knowledge learned by a network is stored in the nodes in the form of weights and node biases. In this paper network training that the desired response of the network (target value) for each input pattern (example) is always available. The training input data is in the form of vectors of input variables or training patterns. Corresponding to each element in an input vector is an input node in the network input layer. Hence the number of input nodes is equal to the dimension of input vectors. The total available data is usually divided into a training set (in-sample data) and a test set (out-of-sample). The training set is used for estimating the weights while the test set is used for measuring the generalization ability of the network. The training process is usually as follows. First, examples of the training set are entered into the input nodes. The activation values of the input nodes are weighted and accumulated at each node in the first hidden layer. The total is then transformed by an activation function into the node’s activation value. It in turn becomes an input into the nodes in the next layer, until eventually the output activation values are found. The training algorithm is used to find the weights that minimize some overall error measure such as the sum of squared errors (SSE) or mean squared errors (MSE). Hence the network training is actually an unconstrained nonlinear minimization problem [4] . 4. Feed forward Neural Network Feed-Forward Neural Network (FFNN) has a layered structure. Each layer consists of units which receive their input from units from a layer directly below and send their output to units in a layer directly above the unit. There are no connections within a layer. The Ni inputs are fed into the first hidden layer of Nh,1 hidden units. The input units are merely 'fan-out' units; no processing takes p lace in these units. The activation of a hidden unit is a function Fi of the weighted inputs p lus a bias, as given in equation ( 1 ). The output of the hidden units is distributed over the next layer of Nh,2 hidden units, until the last layer of hidden units, of which the outputs are fed into a layer of No output units. Although back-propagation can be applied to networks with any number of layers. In most applications a feed-forward network with a single layer of hidden units is used with a sigmoid activation function (bounded, monotonically increasing and differentiable) , where these sigmoid satisfy the boundary condition: x x lim (x) 1, lim (x) 0       ) [ 5 ], [ 6 ] for the units. This activation function also called transfer function or mathematically basis function. In this paper we depend the results in [ 7 ] for choosing most transfer function, which is 'tansig'. IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 5. An example A feed-forward network can be used to approximate a function from data. Suppose we have a system (for example a chemical process or a financial market) of which we want to know the characteristics. The input of the system is given by the two-dimensional vector (x, y)and the output is given by the one-dimensional vector z. We want to estimate the relationship z = f(x, y) from 80 data {(x, y), z} as depicted in figure (2) (top left). A ANN was programmed with two input units, 10 hidden units with 'logsig' activation function and an output unit with a linear activation function. The network weights are initialized to small values and the network is trained for 5000 training iterations with the back-propagation (gradient descent) training rule. The relationship between (x, y) and z as represented by the network is shown in figure (2) (top right), while the function which generated the training samples is given in figure (2) (bottom left). The approximation error is depicted in figure (2) (bottom right). We see that the error is higher at the edges of the region within which the training samples were generated. The network is considerably better at interpolation than extrapolation. 6. How good are multi-layer feed-forward networks? From the example shown in figure (2) it is clear that the approximation of the network is not perfect. The resulting approximation error is influenced by : 1. The training algorithm and number of iterations: This determines how good the error on the training set is minimized. 2. The number of training samples: This determines how good the training samples represent the actual function. 3. The number of hidden units: This determines the "expressive power" of the network. For "smooth" functions only a few number of hidden units are needed (2N+1) [7], for wildly fluctuating functions more hidden units will be needed. In this paper , we particularly address the effect of the number of training samples. We first have to define an adequate error measure. All neural network training algorithms try to minimize the error of the set of training samples which are available for training the network. The average error per training sample is defined as the training error rate: Elearning = (1/Plearning ) ∑ learningP 1=p pE in which E p is the difference between the desired output value and the actual network output for the training samples : E p = (1/2) 2p o n 1=o p o )yd(∑ This is the error which is measurable during the training process. It is obvious that the actual error of the network will differ from the error at the locations of the training samples. The difference between the desired output value and the actual network output should be integrated over the entire input domain to give a more realistic error measure. This integral can be estimated if we have a large set of samples: the test set. We now define the test error rate as the average error of the test set: Etest = (1/Ptest) ∑ testP 1=p pE . IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 In the following section we will see how these error measures depend on training set size and number of hidden units. 7. The effect of the number of training samples S problems are used as example: a function yi = fi(x), i=1,2,3,4, has to be approximated with a ANN. A neural network is created with an input, 3 hidden units with 'tansig' activation function and a linear output unit in first three problems and two input unit, 5 hidden units with 'tansig' activation function and a linear output unit in fourth problem. Suppose we have L number of training samples variant from 1 to 100. The networks are trained with these samples. Training is stopped when the error does not decrease anymore and the number of testing sample is 100 – L. The training samples and the testing sample of the network are shown in the same figure. We see that in the interval [1, 20] and [90, 100] Etraining is small ( the network output goes perfectly through the training samples ) but Etest is large: the test error of the network is large. This experiment was carried out with other training set sizes, where for each training set size the experiment was repeated 10 times. The average training and test error rates as a function of the training and test set size are given in figure (3). Note that the training error increases with an increasing training set size, and the test error decreases with increasing training set size. A low training error on the (small) training set is no guarantee for a good network performance! With increasing number of training samples the two error rates converge to the same value. This value depends on the representational power of the network: given the optimal weights, how good is the approximation. This error depends on the number of hidden units [8] and the activation function. If the training error rate does not converge to the test error rate the training procedure has not found a global minimum. References 1. Jaya,G.P. Prakash and TRBstaff Representative,(1999),"Use of Artificial Neural Networks In Geomechanical And Pavement System", Transportation Research Circular, Number E-co12 ,December. 2. Csáji, B.C. (2001), "Approximation with Artificial Neural Networks", MSC thesis, Faculty of Sciences, Eötvös Loránd University , Hungary . 3. Alldrin,N; Smith, A. and D.Turnbull , (2001)," A Three – Unit Networks is All You Need to Discover Females " , D.Simonl Nerocomputing , 4. Pinkus, A.,(1999), "Approximation theory of the MLP model in neural networks" , Acta Numerica ,.143-195. 5. Zhang, G; Eddy B. patuwoand Y.Hu, (1998), " Forecasting with Artificial Neural Networks: The State of the Art ", International Journal of Fore casting.14, pp.35 - 62 . 6. Dijkstra, E.W;( 2001), " Approximation with Artificial Neural Networks ", MSc thesis, Faculty of Mathematics and Computing Science Eindhoven University of Technology, The Netherlands . 7.Hatim, Q; (2007), " Comparison of Ridge and Radial Basis Functions Neural Networks for Interpolation Problems " , Msc Thesis, Baghdad University , College of Education - Ibn Al-Haitham . 8. Tawfiq, L.N.M. and Eqhaar, Q.H.( 2009), " On Multilayer Neural Networks And Its Application For Approximation Problem ", 3 rd scientific conference of the College of Science , University of Baghdad. 24 to 26 March. IHJPAS IBN AL- HAITHAM J. FOR PURE & APPL. SCI. VOL.23 (3) 2010 Fig 1: Nonlinear model of a neuron Fig 2: Example of functi on approximation wi th a ANN. Top left: The original training samples; Top right: The approximation wi th the network; Bottom left: The function which generated the training samples; Bottom righ t: The error in the approximation . Y1 = f1(x) = sin(x) IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 Y2 = f2(x) = 2x5 + 6x3 – 9x2 + 1 Y3 = f3(x) = 3x (x- 0.6 ) (x + 1.17) Y4 = f4(x) = ( x1- x2 ) 3 18 x1 (x2 – x1)7x2 Fig 3: Effect of the training set size on the error rate. The average error rate and the average test error rate as a function of the number of training samples with L training samples and 100 – L testing samples for 4 problems IHJPAS 2010) 3( 23مجلة ابن الھیثم للعلوم الصرفة والتطبیقیة المجلد ـةتأثیر عدد عـینات التدریب في الشبكات العـصبیة الصناعی ناجي محمد توفیق محمد،لمى ناجي محمد توفیق جامعة بغداد ،أبن الھیثم –كلیة التربیة ،قسم الریاضیات الخالصة دالت�ي تع�� الص�ناعیةف�ي ھ�ذا البح�ث درس�نا ت��أثیر ع�دد عین�ات الت�دریب ف�ي الش��بكات العص�بیة أیض�اً صناعیةش�بكات عص�بیة 5ضروریة في عملیة التدریب لھذا النوع من الشبكات صممنا ی��ؤدي إل��ى دریب ش��بكة عص��بیة توض��ح كی��ف أن االختی��ار الجی��د لع��دد عین��ات الت�� 41دربن��ا ، .تقریب أو تمثیل جید للدالة الحقیقیة IHJPAS