Microsoft Word - issue3.docx 37 CoopRA Algorithm for Universal Characterization of the Experimental Evaluation Results of Cooperative Multiagent Systems Laszlo Barna Iantovics Petru Maior University Str. Nicolae Iorga 1, Târgu Mureș 540088, Romania Phone: 0265 262 275 ibarna@science.upm.ro Muaz A. Niazi* COMSATS Institute of Information Technology Park Road, Tarlai Kalan, Islamabad 45550, Pakistan Phone: +92-51-9247000-9247002 Corresponding author: muaz.niazi@ieee.org Adrian Gligor Petru Maior University Str. Nicolae Iorga 1, Târgu Mureș 540088, Romania Phone: 0265 262 275 adrian.gligor@ing.upm.ro Sándor Miklós Szilágyi Petru Maior University Str. Nicolae Iorga 1, Târgu Mureș 540088, Romania Phone: 0265 262 275 szilagyi@science.upm.ro Matthias Dehmer University of Applied Sciences Upper Austria, Steyr Campus, Steyr Franz-Fritsch-Straße 11 4600 Wels Austria Phone: +43 5 0804 10 matthias.dehmer@umit.at Frank Emmert-Streib Tampere University of Technology, Tampere, Finland Korkeakoulunkatu 10, 33720 Tampere, Finland Phone: +358 3 311 511 frank.emmert-streib@tut.fi Dániel Tokody Óbuda University Budapest, Bécsi út 96b, 1034 Hungary Phone: +36 1 666 5603 tokody.daniel@dosz.hu Abstract Experimental evaluation of the cooperative multiagent systems (CMASs) provides an assessment way that should be analysed. In this paper, we propose an algorithm with acronym CoopRA that can make a deep performance characterization, based on different indicators, of the experimental evaluation results of a CMAS. This could lead to the formulation of helpful information in some decisions related to the performance of the studied CMASs. In order to validate the proposed algorithm, we performed a case study on a CMAS composed of simple reactive agents that operate by mimicking the problem/task solving of natural ants. We chose this type of cooperative multiagent system architecture, based on the fact that even in case of the cooperative BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Issue 3 (September, 2018) ISSN 2067-3957 38 multiagent systems composed of simple efficiently and flexibly cooperating agents could emerges an increased problem solving intelligence at the system’s level. The evaluation was performed for the Travelling Salesman Problem (TSP) solving that is a well-known NP-hard problem, having many real-life applications. Keywords: intelligent system, cooperative multiagent system, NP-hard problem, travelling salesman problem, nature inspired computing, ant colony optimization, digital image processing, medical imaging. 1. Introduction Cooperative multiagent systems (CMASs) represent a subclass of the agent-based systems. By cooperating efficiently and flexible, even in CMASs composed of simple agents could emerge an increased problem-solving performance that could result even in intelligence. Frequently, developed cooperative multiagent systems are considered intelligent based on the extremely performant problem-solving ability (Iantovics & Zamfirescu, 2013; Yang et al., 2003). Many real- life difficult problem-solving is based on CMASs (Beni & Wang, 1993; Mir, Merghem-Boulahia & Gaïti 2009; Hao et al., 2017; Zamfirescu & Filip, 2010; Bouzouita, Chaari & Tagina, 2017; Khalil et al., 2015; Pătruţ, 2014; Arif et al., 2015; Filip & Leiviskä, 2009). There are different types of computational problems, which could be described by data, signals or images. Different kind of difficulties related to the image processing are treated in the papers (Kountchev & Kountcheva, 2017; Georgieva, Kountchev & Draganov, 2014; Georgieva & Draganov, 2016). There are many approaches to image processing based on intelligent agent-based systems such as different medical image segmentation (see for example Bensag, Youssfi & Bouattane, 2015). Agent-based systems require often the treating of different security-related aspects. The papers and books (Iantovics, 2015; Kerti & Nyikes, 2015; Nyikes, Németh & Kerti, 2016; Albini & Rajnai, 2018; Peng et al., 2018; Flammini et al. 2009; Flammini, 2018) treats a variety of aspects and propose some solutions related to the security in different systems including agent-based systems. For the analysis of cooperative multiagent systems experimental evaluation results, usually, there are performed some traditional calculus like, for example, the average problem-solving time; in case of the TSP, the average length of the tour found in more consecutive running of the algorithm. There are several papers that propose some specific approaches related to different aspects of performance evaluation. Gordillo and Giret (2014) studied some specific CMASs applied in manufacturing that use algorithms for task allocation. The main contribution consists in the proposal of a mechanism to measure the performance of agent-based scheduling approaches for manufacturing systems. Ajitha and collaborators (Ajitha et al., 2012) performed an analysis of the performance of software systems. Software performance engineering is important in order to describe the performance of systems at the development stage. It is proposed a methodology to predict the performance of CMASs based on an approach that considers the importance of cooperative behaviour of the agents. In the proposal, a designed mathematical model and the Unified Modeling Language diagrams are used to give a quantitative measure to the cooperation of the agents. Dimou and collaborators (Dimou et al., 2015) outlined the lack of generalized methodologies for assessing the performance of agent-based systems. The authors consider that existing methods do not adequately address the complex nature of many systems. It is proposed a generic methodology for evaluating the performance of agent-based systems. We consider that on obtained experimental evaluations results can be performed some specific analysis that allows the formulation of different kind of useful conclusions related to the performance of studied CMAS operation. As examples, we mention: the verification of the experimental evaluation results normality, the verification of the experimental evaluation results homogeneity/heterogeneity and the spreading of the experimental evaluation results across the mean. In this paper, we propose a more complete analysis and characterization of the cooperative L. B. Iantovics, M. A. Niazi, A. Gligor, S. M. Szilágyi, M. Dehmer, F. Emmert-Streib, D. Tokody - CoopRA Algorithm for Universal Characterization of the Experimental Evaluation Results of Cooperative Multiagent Systems 39 multiagent systems experimental evaluation based on an algorithm called Characterization of the Experimental Evaluation Results (CoopRA). For the validation of the proposed algorithm, we present a case study in which a CMAS composed of simple reactive agents that operates like a colony of natural ants solves an NP-hard problem. It was selected this type of CMAS based on the fact that even very simple efficiently and flexible cooperating agents could have at the group/coalition or multiagent systems level an increased intelligence. It was selected a specific type of problem the well-known Travelling Salesman Problem (TSP) for the case study based on the consideration that is an NP-hard problem, which computationally is extremely difficult. A very large effort is put on the TSP solving. It have numerous real-life applications. Our proposed CoopRA algorithm is universal, it is not restricted to CMASs by the type of operation presented in the case study (is not dependent on the CMASs architecture, and the composing agents architecture) and is not restricted to CMASs that solve the type of problem that is solved in the case study. The upcoming part of the paper is organized as follows: in Section 2 is presented the proposed CoopRA algorithm for characterization of the experimental evaluation results of a CMAS; Section 3 presents the performed case study, in Subsection 3.1 is presented the solved NP-hard problem, Section 3.2 presents the general operation of CMASs that operates like colonies of natural ants, in Subsection 3.3 the operation of the studied CMAS is presented, there are presented and discussed the obtained experimental evaluation results and Section 4 presents the conclusions of the paper. 2. CoopRA proposed algorithm for experimental evaluation results analysis We denote with IC a cooperative multiagent system composed of a set of agents denoted Ag1, Ag2,…,Agn; IC={Ag1, Ag2,…,Agn}. |IC|, |IC|=n represents the number of agents that compose IC. We consider the experimental evaluation of the IC system on a problem set denoted Probl={Prl1, Prl2,…, Prlk}. |Probl|, |Probl|=k denotes the number of problems used in the experimental evaluation. The obtained experimental evaluation results (solving of the problems Probl) are denoted as Exp={Exp1, Exp2,…, Expk}. Where: Exp1 denotes the obtained solution by solving Prl1; Exp2 denotes the obtained solution by solving Prl2;…; Expk denotes the obtained solution by solving Prlk. Figure 1 presents the main processing steps performed by the CoopRA algorithm. This is followed by the presentation of the CoopRA algorithm in details. Prl={Prl1, Prl2,…, Prlk}  Problems transmission for solving Ag1, Ag2,…,Agn  Problems solving Exp={Exp1, Exp2,…, Expk}  Performing an initial characterization of Exp Mean, SEM, LCI, UCI, Median, Mode, SD, Variance  Analysis of Exp homogeneity/heterogeneity CV/Degree of homogeneity  Calculates the Kurtosis and Skewness of Exp Kurtosis and Skewness; Visual representation/Histogram Verification of the Exp data normality Decision on Exp data normality Figure 1. The processing performed by the CoopRA algorithm BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Issue 3 (September, 2018) ISSN 2067-3957 40 In the following, we discuss all the proposed indicators of the experimental evaluation results presented in the CoopRA algorithm and explain their meaning. K represents the number of solved problems used in the experimental evaluation. Mean represents the mean of the experimental evaluation results. The Standard Error (SE) of a parameter is the standard deviation of its sampling distribution. If the parameter or the statistic is the mean, it is called the standard error of the mean. SEM denotes the Standard Error of the Mean. CL denotes the Confidence Level of the Mean, we recommend the use of 95% in most of the cases. LCI denotes the Lower Confidence Interval of the Mean. UCI denotes the Upper Confidence Interval of the Mean. Both LCI and UCI are calculated at the established CL level. Median represents the median of the experimental evaluation results. SD denotes the Standard Deviation of the experimental evaluation results. SD value expresses the quantity by how much the members of a group differ from the mean of the group. SD quantifies the amount of variation or dispersion of a set of data values (Bland & Altman, 1996). Variance denotes the variance. The variance measures how far a set of numbers are spread out from their average value. Min denotes the smallest value. Max denotes the largest value. Range is calculated as the difference between Max and Min; Range=Max-Min. Mode represents the most frequent experimental evaluation result. CoopRA: Characterization of the Experimental Evaluation Results Algorithm IN:IC={Ag1, Ag2,…,Agn}; Probl={Prl1, Prl2,…, Prlk}. Out: //Indicators of the characterization Mean, SEM, LCI, UCI, Median, Mode, SD, CV, Variance. Step 1: Obtaining of the experimental evaluation results. Exp:={Exp1, Exp2,…, Expk}.//Results of the Probl solving. Step 2: Performing an initial characterization. K:=|Exp|. Mode:=Mode(Exp1, Exp2,…, Expk); Median:=Median(Exp1, Exp2,…, Expk); Mean:=Mean(Exp1, Exp2,…, Expk); SEM:= SEM(Mean); CL:=95%;//Set the CL value, we recommend the 95%. @Calculates LCI, UCI. SD:=SD(Exp1, Exp2,…, Expk); Variance:=SD 2; Min:=Min(Exp1, Exp2,…, Expk); Max:=Max(Exp1, Exp2,…, Expk); Range:= Max-Min; Step 3: Analysis of homogeneity/relative homogeneity/heterogeneity. CV:=100×(SD/Mean); If (CV[0,10)) Then “Exp is homogeneous;” ElseIf (CV[10, 30)) Then “Exp is relative homogeneous.” Else //CV30 “Exp is heterogeneous.” EndIf Step 4: Calculates the Kurtosis and Skewness. Skewness:=Skewness(Exp1, Exp2,…, Expk); Kurtosis:= Kurtosis(Exp1, Exp2,…, Expk); @Construct the histogram for the visual interpretation of Kurtosis and Skewness; Step 5: Verification of the Exp data normality. :=0.05; //Set the significance level of the normality test. //Formulates the hypothesis of the normality test. //H0 the null hypothesis and H1 the alternative hypothesis. @Formulates H0 and H1; @Verify the Data Normality using the KS-test; //Let Pks the obtained KS-test result. If (Pks>) Then @Accept H0; “Exp is normally distributed”. Else @Accept H1; “Exp is NOT normally distributed.” Endif EndCoopRA L. B. Iantovics, M. A. Niazi, A. Gligor, S. M. Szilágyi, M. Dehmer, F. Emmert-Streib, D. Tokody - CoopRA Algorithm for Universal Characterization of the Experimental Evaluation Results of Cooperative Multiagent Systems 41 Skewness (Joanes & Gill, 1998) is a measure of lack of symmetry. A dataset is symmetric if it looks the same to the left and right of the center point. Figure 2 ilustrates the graphical representation of Skewness, with Figure 2(a) illustrating the negative Skewness, and Figure 2(b) illustrating the positive Skewness. Figure 2. Graphical representation of the Skewness: (a) Negative skewness and (b) Positive skewness. Kurtosis (Joanes & Gill, 1998) can be defined as the measure of whether the data are light- tailed or heavy-tailed relative to a normal distribution. Figure 3 graphically presents the significance of Kurtosis. Data sets with high kurtosis tend to have heavy tails (outliers). Data sets with low kurtosis tend to have light tails (lack of outliers). Figure 3. Graphical representation of the kurtosis. It is useful to be considered the histogram as a very useful graphical technique for visual interpretation (Pearson, 1895). The humans and computing systems have different strengths and weakness comparatively with each other. From human point of view, easier versus the computing systems is the taking of some decisions based on visual interpretation. Among others, a histogram is useful for showing the skewness and kurtosis of a studied experimental evaluation data set. The Coefficient of Variation (CV) (Everitt, 1998) if is expressed in percentage (%) is calculated as CV=100×(SD/Mean). The coefficient of variation is appropriate for the analysing the homogeneity/relative-homogeneity/heterogeneity of the experimental evaluation results. It can be considered the characterization of homogeneity/heterogeneity based on the CV value as follows: if CV[0%, 10%) we call the experimental evaluation data homogeneous; if CV[10%, 30%) we call the experimental evaluation data relative homogeneous; if CV30% we call the experimental evaluation data heterogeneous. In step 5 of the CoopRA algorithm, is described the verification of the Exp data normality. More specifically, it is verified if the Exp is sampled from a Gaussian population. This information allows the formulation of some conclusions. For this verification, we used the One-Sample Kolmogorov-Smirnov test (KS-test) (Massey, 1951; Miller, 1956; Marsaglia, Tsang & Wang, 2003) that is one of the most frequently used statistical tests in the verification of the normality. The KS- test should be applied at an established significance level that we denote with . In statistical hypothesis testing, a type I error is the incorrect rejection of a true null hypothesis. In other words, this could be called as a "false positive" finding. Concretely  denotes the probability to make a type one error. We recommend in most of the cases the application of the KS-test at the =0.05 significance level. We denote with H0 the Null Hypothesis, which confirm that the Exp dataset is normally distributed. We denote with H1 the Alternative Hypothesis, which confirms that the Exp dataset is NOT normally distributed. The P-value of the KS-test is denoted in the algorithm with Pks. If Pks> than can be concluded that H0 can be accepted, having the significance that the normality BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Issue 3 (September, 2018) ISSN 2067-3957 42 test passed. Elsewhere if Pks, H0 must be rejected and H1 should be accepted, the passing of the normality assumption being failed. 3. The performed case study 3.1. Travelling Salesman Problem definition Travelling Salesman Problem (TSP) was formulated in the 1800s by William Hamilton and Thomas Kirkman. TSP can be enounced as follows (Dorigo, 1997; Bernardino & Paias, 2018; Bao, Liu, Yu, & Li, 2017): given M nodes (cities) that form a directed graph, a salesman starts from a given node, he/she must visit each node exactly once and then return to the starting position (node). The salesman would like to choose the route that minimizes the total travelled distance. TSP is one of the most well-known NP-hard problems. Given n the number of cities to be visited, the total number of possible routes covering all cities can be given as a set of feasible solutions of the TSP calculated as (n-1)!/2. The TSP has several real-life applications, like: drilling of printed circuit boards (Grötschel, Jünger & Reinelt, 1991), overhauling gas turbine engines (Plante, Lowe, & Chandrasekaran, 1987), X-Ray crystallography (Bland & Shallcross, 1989), computer wiring (Lenstra, & Rinnooy Kan, 1974), and vehicle routing (Ratliff & Rosenthal, 1983). 3.2. CMASs that operate like natural ants Dorigo (Dorigo, Maniezzo, & Colorni, 1991; Colorni, Dorigo, & Maniezzo, 1991; Dorigo, 1992) proposed first the problem-solving based on simple computing agents that mimic the behavior of natural ants in searching for food. In an Ant System (AS), initially, each agent (artificial ant) is placed on some randomly chosen node of the graph. A node represents a city in case of the TSP. An agent k currently at node i chooses to move to node j by applying the following probabilistic transition rule: (1) After each agent completes its tour, the pheromone amount on each path will be adjusted as follows: (2) (3) (4) , α, and β are adjustable parameters.  is the evaporation factor. In different implementations,  value (0,1). α and β control the relative weights of the heuristic visibility and the pheromone trail. Q denotes an arbitrary constant. Very frequently Q value is set to 1. dr,z represents the distance between the nodes r and z. rz (rz=1/dr,z) stands for the heuristic visibility of the edge (r,z). The number of agents is denoted by m. Lw stands for the length of the tour performed by the agent w (w[1,m]). The agents’ members of such a studied system have a reactive architecture. They operate in an environment represented by a graph of connected nodes. They are able to move in the environment from node to node during a problem-solving. Many of the multiagent systems that operate by mimicking the natural ants are considered intelligent in the scientific literature. They are considered to have what can be called as Swarm Intelligence (SI). The expression of SI was introduced by Gerardo Beni and Jing Wang (Beni & Wang, 1993). There are many studies (see for ( ) [ ( )] [ ] if ( ) [ ( )] [ ]( ) 0 otherwise k ij ij k k il ilij l J i t j J i tp t                    ( 1) (1 ) ( ) ( )ij ij ijt t t         1 ( ) ( ) k m k ij ij k t t       if (i,j) _ _ _ _ ( ) 0 otherwise k kij Q tour performed by agent k Lt       L. B. Iantovics, M. A. Niazi, A. Gligor, S. M. Szilágyi, M. Dehmer, F. Emmert-Streib, D. Tokody - CoopRA Algorithm for Universal Characterization of the Experimental Evaluation Results of Cooperative Multiagent Systems 43 example Chatterjee et al., 2017) focused on problem-solving using different types of swarm systems. The intelligence of cooperative multiagent systems that mimic by their operation the natural ants, frequently, is considered based on the analogy to the intelligence level at the colony of natural ants and the ability of very difficult problem solving (for example, there is solved an NP- hard problem). Figure 4 illustrates an intelligent task solving by some ants. Figure 4. Intelligent cooperative natural ants [http://sciencenordic.com/ants-make-medicine-out-tree-sap- and-fungi accessed on 27.03.2018] 3.3. Operation of the studied CMAS The first modified version of the AS consisted in the Ant Colony System (ACS). The Ant Colony System was introduced by Dorigo and Gambardella (1997). Min-max Ant System (MMAS) was proposed by Stützle and Hoos (2000). MMAS was applied for different real-life problems solving (Stützle & Hoos 2000; Prakasam & Savarimuthu, 2016). There are developed different applications of MMAS for different real-life problems solving (Stützle & Hoos 2000; Prakasam & Savarimuthu, 2016). IC used in this case study operated similarly as a MMAS. MMAS differs from the conventional AS based on different points of view. An MMAS give dynamically evolving bounds on the pheromone trail intensities. The pheromone intensity on all the paths is always within a specified limit of the path with the greatest pheromone intensity. All the possible paths have permanently a non-trivial probability of being selected. This approach allows a wider exploration of the search space. There are used lower and upper pheromone bounds to ensure that all of the pheromone intensities are between these two bounds. The solution construction is according to (1). There are minimal and maximal pheromone limits to the quantity of pheromone on the paths between nodes, denoted as τmin and τmax. The evaporation is expressed as (5). Equation (6) denotes the pheromone update based on the selected agent's round trip. min( ) max((1 ) ( ), )ij ijt t      (5) max( 1) min( ( ) ( ), ) bs ij ij ijt t t       (6) There are used the following notations. ij bs(t)=Q/Lsel if the path ijTsel, Tsel is the selected best to date agent's round trip, Lsel is the length of the performed trip. In the performed experiments we have initialized τ0=1/NumberOfCities. As another possibility for τ0 initialization that could be applied we mention τ0=τmax. The most appropriate approach for initialization could not be calculated theoretically, it must be established experimentally. 3.4. The performed experimental evaluation There were performed 18 experimental evaluations, Probl={Prl1, Prl2,…, Prl18}, using a computing system with I7-4720HQ processor and 8 GB Ram memory. It was considered the TSP BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Issue 3 (September, 2018) ISSN 2067-3957 44 solving with NumberOfCities=90. The parameters values were established experimentally, as follows: MaxTests(NumberEpochs)=50, α=1.6, β=1.5, =0.28, m=10. Figure 5 illustrates graphically the obtained experimental evaluation results. Table 1 depicts the obtained experimental evaluation results. Figure 6 shows graphically the epochs in case of each problem-solving when is obtained the global-best solution during the search for the problem- solution. Figure 5. The obtained experimental evaluation data Figure 6. Number of epochs for the obtaining of the solution Table 1. The obtained experimental evaluation data BestToDate Epoch BestToDate Epoch 1257 15 1241 28 1286 13 1317 20 1134 11 1342 15 1159 18 1260 16 1456 27 1309 36 1211 35 1291 17 1330 25 1088 14 1218 15 1228 11 1268 15 1298 14 Table 2 presents the results obtained by applying the proposed CoopRA algorithm. The mode value has not been obtained based on the fact that each experimental evaluation value appeared a single time. Figure 7 represents the histogram created based on the best-to-date-data. Among others, it was created in order to make a visual appreciation of the Kurtosis and Skewness. Table 2. Results obtained by applying CoopRA Indicator Value Median/Mode 1264/NA SD/Variance 84.03/7061.5 Mean/SEM 1260.72/19.8 CL, [LCI, UCI] 95%, [1218.9, 1302.5] CV/CV interpretation 6.67/Homogeneose Kurtosis/Skewness 1.1/0.02 Min/Max/Range/Count 1088/1456/368/18 1000 1100 1200 1300 1400 1500 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ProblemNumber B e s tT o d a te 10 15 20 25 30 35 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 NoEpochs P ro b le m N u m b e r L. B. Iantovics, M. A. Niazi, A. Gligor, S. M. Szilágyi, M. Dehmer, F. Emmert-Streib, D. Tokody - CoopRA Algorithm for Universal Characterization of the Experimental Evaluation Results of Cooperative Multiagent Systems 45 Figure 7. Histogram of the BestToDate For the appreciation of the data normality, we considered the visual interpretation of Figure 7 and the interpretation of the KS-test result. It was considered the application of the KS-test at the =0.05 significance level. For the KS-test result, it was obtained the value of KS=0.1112 and the Pks>0.1 (P value of the KS-test). According to step 5 of the algorithm, based on the fact that Pks>, can be concluded that H0 can be accepted. This has the meaning that Exp is normally distributed. The obtained CV value was 6.67, according to the Step 3 of the algorithm, CV<10 indicating a homogeneous experimental evaluation data. 4. Conclusions Cooperative multiagent systems, in many cases, outperform other systems, like the agents that operate individually, in different computational hard problems solving. Based on this fact they can be successfully applied for a large variety of real-life problems solving. Difficulties in the computing problem solving could appear based on fact that they are NP-hard, solving encounter different types of challenges such as: incomplete description, the description contains erroneous data, etc. In case of experimental evaluation of many cooperative multiagent systems, there are missed some calculus that could allow the formulation of different useful conclusions related to the performance. Based on this motivation, we propose an algorithm called Characterization of the Experimental Evaluation Results (CoopRA). Our proposal is useful for a deeper analysis of the cooperative multiagent systems experimental evaluation results than other approaches. This analysis could lead to the possibility of a formulation of more accurate decisions related to the problem- solving performance mostly in case of CMAS that have a heuristic problem-solving behaviour. For example, we mention a swarm of mobile robotic agents specialized in exploring an unknown place of environment. Different runnings on the same problem solving could lead to different experimental evaluation results. For the validation of the CoopRA algorithm, we performed an illustrative experimental case study. It was considered the Travelling Salesman Problem solving by a cooperative multiagent system composed of simple reactive agents that mimic the operation of natural ants in search of food. TSP is one of the most intensely studied NP-hard problems, which has applications for many real-life problems solving. The proposed algorithm for characterization of a CMAS is universal. It is not restricted to a specific type of cooperating multiagent system, or a specific type of problem-solving. As examples of possible applications we mention: cooperative robotic agents specialized in collecting objects in the environment or cooperative swarms of agent-based drones specialized in delivering goods to clients. BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Issue 3 (September, 2018) ISSN 2067-3957 46 The future works will consist in the study if this characterization could be extended in order to make a deeper characterization of the experimental evaluation results such that it allows performing more precise characterization of a CMAS performance. One of the studied direction will consists in the analysing of the possibility to make a characterization of the central performance tendency. For this purpose, we intend to design an algorithm that will be based on some specific calculus. Acknowledgment This research has been supported by the COROFLOW project PN-III-P2- 2.1-BG-2016-0343, contract 114BG/2016. References Ajitha, S., Suresh Kumar, T.V., Geetha, D.E., & Rajanikanth, K. (2012). Early Performance Prediction of Co-Operative Multi-Agent Systems, Procedia Engineering, 38, 3037-3048. Albini, A., & Rajnai, Z. (2018). General Architecture of Cloud. Procedia Manufacturing, 22, 485- 490. Arif, M., Illahi, M., Karim, A., Shamshirband, S., Alam, K.A., Farid, S., Iqbal, S., Buang, Z., Balas, V.E. (2015). An architecture of agent-based multi-layer interactive e-learning and e-testing platform, Quality & Quantity, 49(6), 2435-2458. Bao, X., Liu, Z., Yu, W., & Li, G. (2017). A note on approximation algorithms of the clustered traveling salesman problem, Information Processing Letters, 127, 54-57. Beni, G., & Wang, J. (1993). Swarm Intelligence in Cellular Robotic Systems. In: Dario P., Sandini G., Aebischer P. (eds) Robots and Biological Systems: Towards a New Bionics?. NATO ASI Series (Series F: Computer and Systems Sciences), vol 102. Springer, Berlin, Heidelberg. Bensag, H., Youssfi, M., Bouattane, O. (2015). Embedded agent for medical image segmentation, 27th Int. Conf. on Microelectronics (ICM), 20-23 Dec. 2015, IEEE Society Press, 190-193. Bernardino, R., & Paias, A. (2018). Solving the family traveling salesman problem, European Journal of Operational Research, 267(2), 453-466. Bland, J.M., & Altman, D.G. (1996). Statistics notes: measurement error. BMJ. 312 (7047): 1654. Bland, R.E., & Shallcross, D.E (1989). Large traveling salesman problem arising from experiments in X-ray crystallography: a preliminary report on computation. Operations Research Letters, 8(3), 125-128. Bouzouita, K., Chaari, W.L., & Tagina, M. (2017). Assessing Organizational Effectiveness of Cooperative Agents, Procedia Computer Science, 112, 917-926. Chatterjee, S., Sarkar, S., Hore, S., Dey, N., Ashour, A.S., Balas, V.E. (2017). Particle swarm optimization trained neural network for structural failure prediction of multistoried RC buildings, Neural Computing and Applications 28 (8), 2005-2016. Colorni, A., Dorigo, M., & Maniezzo, V. (1991). Distributed optimization by ant colonies. In Actes de la première conférence Européenne sur la vie artificielle, (pp. 134–142). Dimou C., Tzima F., Symeonidis A.L., Mitkas P.A. (2015). Performance Evaluation of Agents and Multi-agent Systems Using Formal Specifications in Z Notation. In: Cao L. et al. (eds) Agents and Data Mining Interaction. ADMI 2014. LNCS, vol 9145. Springer, 64-78. Dorigo, M. (1992). Optimization, Learning and Natural Algorithms (in Italian). PhD thesis, Dipartimento di Elettronica, Politecnico di Milano, Milan, Italy. Dorigo, M. (1997). Ant Colonies for the Traveling Salesman Problem. IEEE Transactions on Evolutionary Computation, 1(1):53-66. Dorigo, M., & Gambardella, L.M. (1997). Ant Colony System: A cooperative learning approach to the traveling salesman problem. IEEE Transactions on Evolutionary Computation, 1(1),53- 66. Dorigo, M., Maniezzo, V., & Colorni, A. (1991). Positive feedback as a search strategy. Technical Report 91-016, Dipartimento di Elettronica, Politecnico di Milano, Milan, Italy. L. B. Iantovics, M. A. Niazi, A. Gligor, S. M. Szilágyi, M. Dehmer, F. Emmert-Streib, D. Tokody - CoopRA Algorithm for Universal Characterization of the Experimental Evaluation Results of Cooperative Multiagent Systems 47 Everitt, B. (1998). The Cambridge Dictionary of Statistics. Cambridge, UK New York: Cambridge University Press. Filip, F.G., & Leiviskä, K. (2009). Large-Scale Complex Systems, In: Nof S. (eds), Springer Handbook of Automation pp 619-638, Springer, Berlin, Heidelberg Flammini, F. (Ed.) (2018). Resilience of Cyber-Physical Systems: From Risk Modeling to Threat Counteraction, Series: Advanced Sciences and Technologies for Security Applications, Springer. Flammini, F., Marrone, S., Mazzocca, N., & Vittorini, V. (2009). A new modeling approach to the safety evaluation of N-modular redundant computer systems in presence of imperfect maintenance, Reliability Engineering & System Safety 94 (9), 1422-1432. Georgieva, V. Draganov, I. (2016). Multistage Approach for Simple Kidney Cysts Segmentation in CT Images. In: New Approaches in Intelligent Image Analysis. Springer (pp. 223-251). Georgieva, V., Kountchev, R., Draganov, I. (2014). An adaptive approach for noise reduction in sequences of CT images. In: Advanced Intelligent Computational Technologies and Decision Support Systems. Springer, 43-52. Grötschel, M., Jünger, M., & Reinelt, G. (1991). Optimal Control of Plotting and Drilling Machines: A Case Study. Mathematical Methods of Operations Research, 35(1), 61-84. Gordillo, A., & Giret, A. (2014). Performance Evaluation of Bidding-Based Multi-Agent Scheduling Algorithms for Manufacturing Systems, Machines, 2, 233-254. Hao, J., Huang, D., Cai, Y., & Leung, H.F. (2017). The dynamics of reinforcement social learning in networked cooperative multiagent systems, Engineering Applications of Artificial Intelligence, 58, 111-122. Iantovics, L.B. (2015). A Knowledge-based Security Approach for Intrusion Detection, JDCTA: International Journal of Digital Content Technology and its Applications, 9(5), 11-20. Iantovics, L.B., & Zamfirescu, C.B. (2013). ERMS:An Evolutionary Reorganizing Multiagent System, Innovative Computing, Information and Control, 9(3), 1171-1188. Joanes, D.N., & Gill, C.A. (1998). Comparing measures of sample skewness and kurtosis. Journal of the Royal Statistical Society (Series D), the Statistician 47 (1), 183-189. Kerti, A, Nyikes, Z., (2015). Overview of the Information Security Standardization, Acta Technica Corvininesis - Bulletin of Engineering, 8(3), 109-116. Khalil, K.M., Abdel-Aziz, M., Nazmy, T.T., & Salem, A.B.M. (2015). MLIMAS: A Framework for Machine Learning in Interactive Multi-agent Systems, Procedia Computer Science, 65, 827- 835. Kountchev, R., Kountcheva, R. (2017). Processing of Correlated Images through Double PCA- based Transform in Color and Time Domains, Int. J. of Mathematics and Computers in Simulation, 11, 135-142. Lenstra, J.K., & Rinnooy Kan, A.H.G. (1974). Some Simple Applications of the Travelling Salesman Problem. BW 38/74, Stichting Mathematisch Centrum, Amsterdam. Massey, F.J. (1951). The Kolmogorov-Smirnov Test for Goodness of Fit. Journal of the American Statistical Association, 46(253), 68-78. Marsaglia, G., Tsang, W., & Wang, J. (2003). Evaluating Kolmogorov’s Distribution. Journal of Statistical Software 8(18). Miller, L.H. (1956). Table of Percentage Points of Kolmogorov Statistics. Journal of the American Statistical Association, 51(273), 111-121. Mir, U., Merghem-Boulahia, L., & Gaïti, D. (2009). Utilization of a Cooperative Multiagent System in the Context of Cognitive Radio Networks. In: Strassner J.C., Ghamri-Doudane Y.M. (eds) Modelling Autonomic Communications Environments. LNCS 5844. Springer, 100-104. Nyikes, Z., Németh, Zs., & Kerti, A. (2016). The electronic information security aspects of the administration system, In A. Szakal (Ed.) 2016 IEEE 11th Int. Symp. on Applied Computational Intelligence and Informatics (SACI), IEEE Computer Society Press 327-331. BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Issue 3 (September, 2018) ISSN 2067-3957 48 Pătruţ, B. (2014). Designing a Multi-Agent System for Improving the Accounting E-Learning, Artificial Intelligence Applications in Distance Education, IGI, 47-71. Pearson, K. (1895). Contributions to the Mathematical Theory of Evolution. II. Skew Variation in Homogeneous Material. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 186. 343–414. Peng, S.L., Wang, S.J., Balas, V.E., Zhao, M. (Eds), (2018). Security with Intelligent Computing and Big-data Services, Advances in Intelligent Systems and Computing, 733, Springer. Plante, R.D., Lowe, T.J., & Chandrasekaran, R. (1987). The Product Matrix Traveling Salesman Problem: An Application and Solution Heuristics. Operations Research, 35, 772-783. Prakasam, A., & Savarimuthu, N. (2016). Metaheuristic algorithms and probabilistic behaviour: a comprehensive analysis of Ant Colony Optimization and its variants. Artificial Intelligence Review, 45(1), 97-130. Ratliff, H.D., & Rosenthal, A.S. (1983). Order-Picking in a Rectangular Warehouse: A Solvable Case for the Travelling Salesman Problem. Operations Research, 31, 507-521. Stützle, T., & Hoos, H.H., (2000). Max-min ant system. Future Generation Computer Systems, 16, 889-914. Yang, K., Galis, A., Guo, X., & Liu, D. (2003), Rule-Driven Mobile Intelligent Agents for Real- Time Configuration of IP Networks, Knowledge-Based Intelligent Information and Engineering Systems, Lecture Notes in Computer Science, 2773, 921-928. Zamfirescu, C.B., Filip, F.G. (2010). Swarming Models for Facilitating Collaborative Decisions, Int. J. of Computers, Communications & Control, V(1), 125-137. Laszlo Barna Iantovics received the B.Sc and M.Sc degree in Mathematics and Informatics at the Transylvania University of Brasov; a Ph.D. in Artificial Intelligence at Babes-Bolyai Univ. of Cluj-Napoca and finished a Postdoctoral Study in Artificial Intelligence at Alexandru Ioan Cuza University of Iasi. Dr. Iantovics recently is Associate Professor at Petru Maior University, teaching also at University of Medicine and Pharmacy from Tirgu Mures. He is the Director of the Research Center “Advanced Computational Technologies” from Petru Maior University. His principal research interests include the Intelligent Systems, Measuring the Machine Intelligence, Computational Intelligence, Biostatistics, and Artificial Intelligence applied for difficult problems solving in healthcare, topics on that he published dozens of papers and book chapters and contributed to research projects as project director or researcher. Dr. Muaz A. Niazi is Chief Scientific Officer (Professor) at the COMSATS Institute of IT, Islamabad, Pakistan. He has an undergraduate degree in Electrical Engineering, an MS, and a PhD in Computer Sciences from Boston University, MA, USA, and the University of Stirling, Scotland, UK respectively. His areas of research interest include Modeling, Simulation, and Engineering of Complex Adaptive Systems (CAS) using various techniques such as agent-based and complex network-based approaches. Dr. Niazi is the Founding Editor-in-Chief of Springer Nature Complex Adaptive Systems Modeling and IGI Global’s International Journal of Privacy and Health Information Management. Adrian Gligor is Associate Professor at the "Petru Maior University" of Tîrgu- Mureș at the Electrical Engineering and Computers Department. He has received his B.Sc in Automation and Industrial Informatics (1997), M.Sc in Advanced Systems for Energetic and Industrial Process Control (1999) from "Petru Maior University" of Tîrgu-Mureș and PhD in Civil Engineering (2007) from Technical University of Cluj Napoca. His areas of research interest include real-time and embedded systems, process control, system modeling, high-performance computing and artificial L. B. Iantovics, M. A. Niazi, A. Gligor, S. M. Szilágyi, M. Dehmer, F. Emmert-Streib, D. Tokody - CoopRA Algorithm for Universal Characterization of the Experimental Evaluation Results of Cooperative Multiagent Systems 49 intelligence. He has (co-) authored 6 books or book chapters, more than 75 scientific papers published in journals or conferences proceedings, member in International Program Committee of 3 conferences and workshops. Dr. Sándor Miklós Szilágyi, professor at the "Petru Maior” University of Tîrgu- Mureș has received his BSc in Automation and Industrial Informatics (1996) from "Petru Maior” University of Tîrgu-Mureș, PhD in Computer Science (2008) from Budapest University of Technology and Economics and Habilitation in Informatics (2016) from "Babeș Bolyai” University of Cluj Napoca. His areas of research interest include artificial intelligence, biological system modeling, parallel and distributed computing, real-time systems and bioinformatics. He has (co-) authored 8 books or book chapters, more than 100 scientific papers, more than 25 conferences participation, member in International Program Committee of 3 conferences and workshops. Dr. Matthias Dehmer is a professor at University of Applied Sciences Upper Austria, Campus Steyr and University of Applied Sciences Upper Austria and UMIT - The Health and Life Sciences University. He also holds a guest professorship at Nankai University. His research interests are in graph theory, complex networks, complexity, machine big data, analytics, and information theory. In particular, he is also working on machine learning-based methods to design new data analysis methods for solving problems in manufacturing and production. Dr. Frank Emmert-Streib is an Associate Professor at the Tampere University of Technology in Data Science. Before he came to Tampere he was a Senior Lecturer (Associate Professor) at the Queen’s University Belfast at the Center for Cancer Research and Cell Biology. He studied Physics and Mathematics and obtained his Ph.D. in Theoretical Physics from the University of Bremen (Germany) and a Diploma (with distinction) as well as a Bachelor in Theoretical Physics from the University of Siegen (Germany). His research interests are in the area of data science, machine learning and digital society. Daniel Tokody is a doctoral candidate at Doctoral School of Safety and Security Sciences, Óbuda University, Budapest, Hungary. He is an Electrical Engineer (BSc and MSc). Daniel Tokody is the general board member of Association of Hungarian PhD and DLA Students and member of IEEE SMC Technical Committee on Homeland Security. Daniel does research in Safety Engineering, Electrical Engineering and Railway Engineering. His areas of research interest include intelligent systems, intelligent cooperative systems development, and safety-critical system.