Format Template Volume 1 | No. 2 | July – December 2017 SJCMS | P-ISSN: 2520-0755 | E-ISSN: 2522-3003 © 2017 Sukkur IBA University – All Rights Reserved 102 Comparative Study of Load Testing Tools: Apache JMeter, HP LoadRunner, Microsoft Visual Studio (TFS), Siege Rabiya Abbas1, Zainab Sultan1, Shahid Nazir Bhatti1, Farrukh Latif Butt Abstract: Software testing is the process of verifying and validating the user’s requirements. Testing is ongoing process during whole software development. Software testing is characterized into three main types. That is, in Black box testing, user doesn’t know domestic knowledge, internal logics and design of system. In white box testing, Tester knows the domestic logic of code. In Grey box testing, Tester has little bit knowledge about the internal structure and working of the system. It is commonly used in case of Integration testing.Load testing helps us to analyze the performance of the system under heavy load or under Zero load. This is achieved with the help of a Load Testing Tool. The intention for writing this research is to carry out a comparison of four load testing tools i.e. Apache JMeter, LoadRunner, Microsoft Visual Studio (TFS), Siege based on certain criteria i.e. test scripts generation , result reports, application support, plug-in supports, and cost . The main focus is to study these load testing tools and identify which tool is better and more efficient. We assume this comparison can help in selecting the most appropriate tool and motivates the use of open source load testing tools. Keywords: Testing, manual testing, automated testing, testing tools, load testing, stress test. 1. Introduction The objective of software testing is to find defects, errors and bugs in a software, system or product. Software testing is characterized into manual testing and automation testing. Figure 1: Software testing Manual testing is executed by the testers. First of all a written test plan is created, and followed by testers that provides a guideline through different steps. But there are a lot of problems faced by testers like it is very time taking and consuming, no reusability, has no scripting feature, much human effort required, and 1 Department of Software Engineering, Bahria University Islamabad, Pakistan  Corresponding Author: zaini.1984@gmail.com still major or minor bugs remain unexplored. Therefore to cover all types of errors and bugs automation testing has introduced that explores all the issues exist in manual testing [11]. All Automation testing tools test software in less time, produce more reliable, repeatable, and reusable final product. Load testing is activated when we steadily raise the load upon a system until it reaches a target load. Usually this is the maximum load, average or even zero load. The goal of a load testing is to discover functional and performance issues of a system under load. Load testing is appropriate for testing the performance of web site, and its framework[18]. The intention for writing this research is to carry out a comparison of four load testing tools i.e. Apache JMeter, LoadRunner, Microsoft Visual Studio (TFS), Siege based on some parameters. This research paper is divided into different sections. Section 1 consists of introduction. mailto:zaini.1984@gmail.com Rubiya Abbas (et al.) Comparative Study of Load Testing Tools: Apache JMeter, HP LoadRunner, Microsoft Visual Studio (TFS), Siege (pp. 102 - 108) Sukkur IBA Journal of Computing and Mathematical Sciences - SJCMS | Volume 1 No. 2 July – December 2017 © Sukkur IBA University 103 Literature review is discussed in Section 2. Research methodology is presented in section 3 .In section 4, we present evaluation study. In section 5, and comparison table of automated testing tools is presented. In section 6, results and analysis of study is presented and in section 7, on the basis of research, conclusion is presented. 2. Literature review This section presents the literature review of research topic. Farmeena Khan Et.al in their paper describes and compares three main testing techniques: “White box, Black box and Grey box testing”. Authors’ presents the future changes in software testing industry due to new technologies like SOA (Service Oriented Architecture) & mobile technologies etc[3]. Niranjanamurthy et .al in their paper discusses testing terminologies used in testing, levels of testing, analysis of automated and manual techniques, comparison of Selenium and QTP Tools, comparison of “White box, Black box and Grey box testing techniques” [2]. Taraq Hussain et al in their research mentions that testing can never completely diagnoses all errors of a software but it provides evaluation techniques which helps the tester to diagnose a problem. After comparison they show that the White Box Testing is best suitable for software reliability[4]. “The Growth of Software Testing” is written by “David Gelperin and Bill Hetzel”. In this paper, authors describes the evolution of different software testing process models , their merits and demerits due to which some of these are failed . From 1956 to present, different software testing models are discussed; Changes in these models are evaluated[14]. Harpareet Kaur and Dr.Gagan Gupta in their paper discussed the two ways of testing: Manual testing and Automation testing. In this research paper, they discuss the parameters of “Selenium 2.0.0, Quick Test Professional 10.0, and TestComplete 9.0”. These three tools comparison is based on different specification and parameters. After analysis, researchers concluded that anyone can choose the testing tool on the basis of budget and nature of software that has to be tested. Researchers found that QTP is more suitable among all that three tools[6]. Neha Bhatia in her paper discussed manual testing and automation testing. If the requirements are continuously changing and regression testing is needed to perform repeatedly, then automated testing is more suitable in that environment. In this paper, researcher discussed different automation tools[7]. Neha Dubey et al in their paper compare and study concepts and characteristics of two “software automated tools Ranorex and the Automated QA TestComplete” that are based on some criteria. After comparison they concluded that “Ranorex” is the best tool for web based applications [10].Monika Sharma, Vaishnavi S. Iyer, Sugandhi Subramanian, and Abhinandhan Shetty in their paper focuses on comparing load testing tools- Apache JMeter, HP LoadRunner, WebLOAD, and The Grinder on the basis of different parameters[11]. 3. Related Work Comparison between load testing tools has been done by many authors. Vandana E. [13] [20], have done comparative study of testing tools which are jmeter and load runner. They described advantages and disadvantages of both tools and recommended that Jmeter is much better than Load Runner because it has clean UI that offers much simplicity. Bhatti Et.al [19], described number of load testing tools for test web applications. The testing tools they discussed are Load Runner, NeoLoad, WAPT, Soasta Cloud Test, LoadStorm, Loadster, Apache, JMeter, HTTPERF, LoadUI, and LoadImpact. They analyzed that among all tools to test a web application, NEOLOAD is best for load testing due to its visual programming and its script less design. Rina [21] analyzed the NeoLoad, WAPT and Loadster tools on different browsers and compared the results of their performance. One site has tested on above three tools for performance. The comparison they done provides a better understanding for selecting the best tool according to requirements and possibilities, however they concluded that It is difficult to compare tools because many parameters values are not considers in all tools. Upadhyay [22]compared some specific performance testing tools for their usability and effectiveness. WAPT and RANOXEX performance testing tools inferences, implications and results have been presented and discussed. Different attributes, their ability to compare the Rubiya Abbas (et al.) Comparative Study of Load Testing Tools: Apache JMeter, HP LoadRunner, Microsoft Visual Studio (TFS), Siege (pp. 102 - 108) Sukkur IBA Journal of Computing and Mathematical Sciences - SJCMS | Volume 1 No. 2 July – December 2017 © Sukkur IBA University 104 results, test cases documentation ability and regression testing performance ability have been compared. Dart Et.al compared software web tools in terms of their dynamic test generation ability [18]. A survey has been presented on static and dynamic testing analysis. Sufiani [23] compared different performance testing tools response time and justified these differences include architecture and simulation mechanism. 4. Research Methodology Testing is an important and critical part of the SDLC. In recent times different automated software testing tools are available in market. Several studies are available in which comparisons of different testing tools are done. According to our observations, there is no comparative analysis on the load testing tools, such as “JMeter, Siege, LoadRunner, and Microsoft Visual Studio (TFS)”. In this paper, we compare these load testing tools on the basis of different parameters. 4.1 Automated software testing tools A brief explanation and comprehensive account of automated software testing tools is taken here in this section. 4.1.1 Apache JMeter Apache JMeter is an “open-source testing tool” developed by “Apache Software Foundation (ASF)”. JMeter’s main function is to load test client/server. Moreover, JMeter is used in regression testing by generating test scripts [12]. JMeter provides offline reporting of test results. JMeter test reports are shown in fig 2. Figure 2: JMeter test reports 4.1.2 LoadRunner HPE LoadRunner is an “automation testing tool” from “Hewlett Packard” enterprise [15]. HP LoadRunner software testing tool helps you to detect and prevent from software performance issues by identifying bottlenecks [16]. HP LoadRunner Scripting and test report summary is shown in Fig 3. Figure 3: HP LoadRunner scripting & test summary reports 4.1.3 Siege “Siege” was developed and implemented by “Jeffrey Fulmer” as Webmaster for Armstrong World Industries. It is a software load testing tool which is very productive in detecting the performance of system when load exists [13]. Siege executing commands and Test summary reports is shown in fig 4. Figure 4: Siege executing commands &test report summary 4.1.4 Microsoft visual studio (TFS) “Team Foundation Server (TFS)” is a load testing tool which facilitate with source code management, Project management, Requirement management, reporting, testing capabilities, automated builds, lab management, [17] TFS test Summary reports are shown in fig 5. Rubiya Abbas (et al.) Comparative Study of Load Testing Tools: Apache JMeter, HP LoadRunner, Microsoft Visual Studio (TFS), Siege (pp. 102 - 108) Sukkur IBA Journal of Computing and Mathematical Sciences - SJCMS | Volume 1 No. 2 July – December 2017 © Sukkur IBA University 105 Figure 5: TFS test summary report 5. Evaluation Study Now a day’s different open source and profitable Load testing tools are available in the market. For this comparative study, we are using the latest and running versions of “Apache JMeter, LoadRunner, Microsoft Visual Studio (TFS), and Siege”. Through these tools we test the Bahria University Islamabad campusWebsite (http://www.buic.edu.pk/) and for Siege, we test telecommunication company website (www.telenor.pk). Comparison between these four tools is made on the basis of list evaluation parameters with the explanation. 6. Result and Analysis of Study For assessment of the parameters, we use 3-point scale in a graph i.e. 3, 2, 1 as Best, Average, and Worst respectively. Different value for different parameters with selected automated tools is verified. The calculated value of parameters is used for conclusion and investigation of this comparative study. The overall comparison based graph for these four automated load testing tools is shown in Fig 6. Figure 6: Comparison graph based on selected parameter results 7. Results and Findings After comparison and analysis, we conclude that anyone can choose the testing tool on the basis of budget and nature of software that has to be tested. Apache JMeter, LoadRunner, Microsoft Visual Studio (TFS), Siege all four are good tools for test automation. We take two different websites Bahria University Islamabad campus Website (http://www.buic.edu.pk/) and for Siege, we test telecommunication company website (www.telenor.pk) because Siege cannot test live sites. Each tool has its own benefits and drawbacks too; a detail analysis in this context is in Table I below. It is to be noted for Telenor System we did have the access code available but for the Bahria University we did not have access to the code. Siege can reduce the cost as it is open source but it has limited options to be used as it is command line tool and it sometimes generate inaccurate result. HP LoadRunner is best for performance checking when load found. It can handle multiple users at same time butit has some configuration or installation problems across firewalls and its licensing cost is high. Microsoft Visual Studio (TFS) is user friendly .It has built-in testing capabilities whether there are 100 parallel users or 1000s, it is easy to test according to user requirements but it can only supports Windows OS and it has high licensing cost. Apache JMeter is best option as it is free of cost (see fig 7 in this). It takes more time on one time installation but it has broad set of options for result analysis and it is good for different tests to be run simultaneously .It has several plugins which raise its testing capabilities. Figure 7: Pie chart showing JMeter as highest use http://www.buic.edu.pk/ http://www.telenor.pk/ http://www.buic.edu.pk/ http://www.telenor.pk/ Rubiya Abbas (et al.) Comparative Study of Load Testing Tools: Apache JMeter, HP LoadRunner, Microsoft Visual Studio (TFS), Siege (pp. 102 - 108) Sukkur IBA Journal of Computing and Mathematical Sciences - SJCMS | Volume 1 No. 2 July – December 2017 © Sukkur IBA University 106 8. Comparison of Automation Testing Tools In this section, we compare the automation testing tools. This comparison is beneficial for the testers/researchers (technical stake holders) to choose the more appropriate load test tool as per requirements. Table II presents comparison of automated testing tools i.e. Apache JMeter, LoadRunner, Microsoft Visual Studio (TFS), and Siege on the basis of different parameters (characteristics). TABLE-I: EVALUATION PARAMETERS WITH EXPLANATION DETAILS Appropriate Parameters Explanation Recording Efficiency For handling the application which is to be tested. Generating scripts capability Generating the interrelated scripts. Test Result Reports generation Efficient investigation of test scripts Cross platform Operating systems on which tool operates on Easy to learn GUI(GRAPHICAL USER INTERFACE) Application support Applications that are supported by testing tool Scripting languages Languages that are used for scripting Plugin support Testing tools either support plugins or not Licensing Cost Reasonable or low cost TABLE-II: COMPARISON OF WEB SERVICE TESTING TOOLS FEATURES JMeter Siege LoadRunner Microsoft Visual Studio(TFS) Cost It is free or no renovation cost for this tool. It is free of cost. License is expensive About $2K licensing cost Application support Static and dynamic resources, Web services and databases. Only used for testing websites. Websites and other applications. Website & Other applications. Scripting language Javascript ,BeanShell Skrit Citrix, ANSI C, .Net and Java PowerShell ,Perl Cross Platforms Supports Windows PC/MAC/UNIX Platforms. Supports UNIX, AIX, BSD, Solaris. Supports Microsoft Windows and LINUX OS. Supports Windows 7, Windows Vista, Windows Server 2008 or later operating systems. Plugin support It has several plugins which raise its testing capabilities. Siege has small no. of plugins. It has several plugins which raise its testing capabilities. It has several plugins which raise its testing capabilities. Interface GUI Command Line Interface GUI GUI Benefits It provides GUI and has many features that can be used while testing. It has faster setup. It is good for quick results. It is best for performance checking where there is actual load. It is simple to use. It has inherent testing capabilities. Rubiya Abbas (et al.) Comparative Study of Load Testing Tools: Apache JMeter, HP LoadRunner, Microsoft Visual Studio (TFS), Siege (pp. 102 - 108) Sukkur IBA Journal of Computing and Mathematical Sciences - SJCMS | Volume 1 No. 2 July – December 2017 © Sukkur IBA University 107 9. Conclusion Improving software quality and performance has become a priority for almost every organization that relies on the software development. Thus the quality issue related to the software’s industry becomes more important, apparent and more technical also considering the user’s requirements in this aspect. Software systems have to ensure consistent and bug free execution at a rapid pace every time they are used especially in web based development. In this work we have performed a thorough and comprehensive comparison and analysis using different tools/ technologies available for testing (load testing as case scenario). After a through analytical review of these different tools mentioned in sections IV and V for Load testing, we summarize here that anyone can choose the testing tool but on the basis of budget, time and nature of software system under consideration that has to be tested. Besides Each tool have its own benefits and drawbacks, and have to keep in queue when performing anyone of the mentioned testing strategies (or any other). Apache JMeter, LoadRunner, Microsoft Visual Studio (TFS), Siege all four are good tools for test automation. But we have shown that JMeter provides better results than any other tested tools (techniques) as it is a ratio scale methodology, and also includes a consistency check. In future work, with the access to code (for web projects) the applications and values (attributes) of these tools can be estimated, especially in case of stress testing while performing Load testing. As stress testing evaluate the system when stressed to its limits over a short period of time and that following testing is especially important for systems that usually operate below maximum capacity but are severely stressed at certain times of peak demand. REFERENCES [1] Rapinder Singh ,Manprit Kaur ,”A review of software testing techniques”,( IJEEE), ISSN 0974- 2174, Volume 7, pp. 463, 2014. [2] Nitesh S N, Niranjanamurthy M, Balaji Sriraman ,Nagesh S N, “Comparative Study of Software Testing Techniques “, IJCSMC, Vol. 3, Issue. 10, October 2014, pg.151 – 158. [3] Farmeena Khan, Mohammad Ehmar Khan ,”A comparative study of white box , black box , grey It has vast set of options for result analysis. It is good for different tests to be run simultaneously. It gives accurate results. It can handle large no. of users at the same time. It can also checks network and server resources for improving performance. It automatically trace client/server performance while testing It uses graphical illustrations in reports. Whether there are 100 parallel users or 1000s, it is easy to test as per requirements. Drawbacks JMeter takes more time to setup as it involves many steps. It has limited options to be used as it is command line tool. It sometimes generate inaccurate result. It has some configuration or installation issues across firewalls. It only supports Windows OS. It has high licensing cost. Report Generation JMeter supports dashboard report generation to get graphical illustrations. Reports total no.of transactions ,server response etc. It allow user to convert performance report into word, excel, pdf etc. In this, reports are generated in SQL Server Reporting Services. Rubiya Abbas (et al.) Comparative Study of Load Testing Tools: Apache JMeter, HP LoadRunner, Microsoft Visual Studio (TFS), Siege (pp. 102 - 108) Sukkur IBA Journal of Computing and Mathematical Sciences - SJCMS | Volume 1 No. 2 July – December 2017 © Sukkur IBA University 108 box testing techniques” , (IJACSA) , Vol. 3, No.6, 2012. [4] Taraq Hussain, Dr.Satyaveer Singh, “A Comparative Study of Software Testing Techniques Viz. White Box Testing Black Box Testing and Grey Box Testing”, ( IJAPRR) , ISSN 2350-1294. [5] Kamna Solanki, Jyoti, , “A Comparative Study of Five Regression Testing Techniques: A Survey “, International Journal of Scientific & Technology Research (IJSTR), Volume 3, Issue 8, August 2014. [6] Neha Bhateja , “A Study on Various Software Automation Testing Tools” ,(IJACSA), Volume 5, Issue 6, June 2015 . [7] Richa Rattan, “Comparative Study Of Automation Testing Tools: Quick Test Professional & Selenium”, IJCSIT, Vol. 3, No. 6 June 2013. [8] Raj Kumar, Manjit Kaur, “Comparative Study of Automated Testing Tools: Test Complete and Quick Testpro”, Intl. Journal of Computer Applications (0975-8887), Volume 24, No. 1, June 2011. [9] Monika Sharma, , Abhinandhan Shetty, Sugandhi Subramanian, Vaishnavi S. Iyer,“A Comparative Study on Load Testing Tools” , Int. Journal of Innovative Research in Computer and Communication Engineering ,Vol. 4, Issue 2, February 2016 [10] Neha Dubey, Mrs. Savita Shiwani , “Studying and Comparing Automated Testing Tools; Ranorex and TestComplete” , (IJECS) , Volume 3, Issue 5, Pp. 5916-5923 [11] Sanjay Tyagi , Pooja Ahlawat , “A Comparative Analysis of Load Testing Tools Using Optimal Response Rate” , (IJARCSSE), Volume 3, Issue 5, May 2013. [12] Dr.K..V..K .K Prasad , Software Testing Tools: Covering WinRunner, SilkTest, LoadRunner, JMeter, TestDirector and QTP with Paperback , 2007 . [13] Ibrahima Kalil Toure, Abdoulaye Diop, Shariq Hussain and Zhaoshun Wang, “Web Service Testing Tools: A Comparative Study”, IJCSI Int. Journal of Computer Science Issues, Vol. 10, 2013. [14] Daniel A. Menasce, “Load Testing of Websites”, http:/computer.org/internet, IEEE Internet Computing, PP.70- 74, 2002. [15] Li Xiao-jie, Zhang Hui-li and Zhang Shu.“Research of Load Testing and Result Application Based on LoadRunner”, National Conference on Information Technology and Computer Science, 2012. [16] Sneha Khoria, Pragati Upadhyay, “Performance Evaluation and Comparison of Testing Tools”, VSRD Int. Journal of Compt. Science & IT, Vol. 2, 2012. [17] Sinha M, and Arora A., “Web Application Testing: A Review on Techniques, Tools and State of Art”, (IJSER), Volume 3, Issue 2, 2012. [18] Vandana Chandel, Shilpa Patial Sonal Guleria, “ComparativeStudy of Testing Tools: Apache JMeter and Load Runner”,IJCCR, VOLUME 3 ISSUE 3 May 2013. [19] Sandeep Bhatti, Raj Kumari, “Comparative Study of LoadTesting Tools”, ijircce, Vol. 3, Issue 3, March 2015. [20] Rina, Sanjay Tyagi, “A Comparative Study of PerformanceTesting Tools”, Volume 3, Issue 5, May 2013. [21] Dr. S. M. Afroz, N. Elezabeth Rani and N. Indira Priyadarshini, “Web Application– A Study on ComparingSoftware Testing Tools”, International Journal of ComputerScience and Telecommunications, Volume 2, Issue 3, June2011. [22] Muhammad Dhiauddin Mohamed Suffiani, Fairul RizalFahrurazi, “Performance Testing: Analyzing Differences ofResponse Time between Performance Testing Tools”, inproceeding of International Conference on Computer &Inf. Science (ICCIS) 2012.