INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL ISSN 1841-9836, 12(3), 347-364, June 2017. System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen Baozhuang Niu, Kanglin Chen, Lei Chen* School of Business Administration, South China University of Technology Guangzhou, 510640, P.R.China bmniubz@scut.edu.cn, kanglin_chen@qq.com *Corresponding author: jayden_business@foxmail.com Huizhong Huang Groupe Danone, Guangzhou, 510620, P.R.China 290092790@qq.com Yuan Li Lingnan College, Sun Yat-sen University Guangzhou, 510275, P.R.China jerry_yuanli@163.com Abstract: Enterprise Resource Planning (ERP) system is an important investment for manufacturing companies that can affect their competitive advantages and op- erational performance. However, the implementation of ERP can be a complicated process, where many strategic decisions have to be made. We focus on two critical de- cisions in ERP implementation: (1) ERP system selection, and (2) ERP operational performance evaluation. For the former, we use Analytic Hierarchy Process (AHP) to design the key performance indicator (KPI) system. For the later, we combine AHP and Fuzzy Integrated Evaluation (FIE) methods to effectively evaluate the implemen- tation of ERP. We use a typical industrial example and data analysis to illustrate our framework. Keywords: ERP system selection, ERP performance evaluation, analytic hierarchy process, fuzzy integrated evaluation, manufacturing companies. 1 Introduction Nowadays, severe market competition has dramatically transformed the business environ- ment. For manufacturing companies, whose competitive advantages are mainly low cost op- erations and quick-response management, the implementation of information systems becomes critical. It is widely accepted that Enterprise Resource Planning (ERP) has the ability to inte- grate the flow of material, finance, and information and to support organizational strategies [10]. However, the implementation of ERP system can be a highly complicated process, especially for those contract manufacturing companies who have multiple businesses such as self-branded business, manufacturing business, and design business [4]. According to an independent research report [3], in 2014, 42% of the surveyed companies consider their ERP projects as a “neutral", or “not clear" , or “failed" project. An important reason for the failure of ERP system implementation is that the ERP systems on the market do not fit the company’s operations properties. For the successful implementa- tion of ERP system, the adjustment of business process, the selection of suitable ERP system and IT tools, and the effective performance evaluation are the most critical decisions [15], [8], although all of them are hard to make. Being aware of these, managers in the manufactur- ing industry turn to consulting companies (e.g., IBM and Accenture) to find ERP solutions. Copyright © 2006-2017 by CCC Publications 348 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen Meanwhile, many IT service companies identify this demand and build online service plat- form to help manufacturing companies to implement ERP systems. For example, TECTEC (http://www.technologyevaluation.com) proposes a ERP selection and assessment approach for its customers, and this approach is proven to be effective in their application cases for manu- facturing companies that produce pharmaceutical and botanical products, industrial machinery products, and electronics and high-tech products [9], [10]. Recently, we have consulting interactions with a multinational manufacturing company which is Austria-headquartered. They turned to us for suggestions to implement ERP system to manage their supply chain. We conducted surveys and found that the standard approach proposed by TECTEC need be detailed. Thus we develop a framework to help the company to select the ERP system based on analytic hierarchy process (AHP) and use fuzzy integrated evaluation (FIE) method to measure the performance of their ERP implementation. Our work is summarized in this paper. All the data that is used to illustrate our framework comes from this consulting project. We combine the objectives of choosing the most appropriate ERP system and vendor with different criteria. We also compare the alternatives based on evaluators’ opinions and identify the most appropriate ERP system. 2 Literature Selecting the suitable ERP system for enterprise can help avoid the failure of ERP system implementation, so it is important to select the appropriate ERP system. There are several common methods to choose appropriate ERP system or the other management information system ( [10], [15], [8]). The scoring method is one of the most popular methods, which is simple and intuitive, but does not guarantee the feasibility of resources. For example, [12] uses 10 criteria to evaluate the ERP system and develop a framework based on nominal group technique (NGT) and analytic hierarchy process (AHP) to select the ERP system. Some other methods are developed to improve the efficiency of ERP system implementation procedures, for example, [13], [14], and [4]. In practice, many companies use some financial indicators to select ERP systems. Since financial indicators are reported by professional institutions, they can be viewed as a trustable data resource, and can be used to index the implementers of ERP system [7]. Useful information includes the market size, the vendors and the overall system performance, etc. Industry and academia also pay attention to the evaluation of ERP system. Companies want to use timely, accurate and objective performance evaluation to continuously adjust and improve the ERP project. Academia also try to identify the factors affecting the performance of ERP system through empirical study, and then construct the evaluation system to evaluate the performance of ERP system implementation ( [5], [15]). [13] points out that it takes companies a long time to see the effect of ERP system on the performance. As a result, in the study on the comparison of performance between companies adopting ERP system and companies without ERP system, researchers couldn’t find a significant difference [5]. When the time window is large enough that can eliminate this effect, there is a significant difference on the performance between companies using ERP system and companies without it [15]. 3 Selection of ERP system 3.1 The criteria for selection Before we discuss the criteria for selection of ERP system, determining the strategic objectives of ERP project is very necessary. Strategic objectives guide the team and indirectly coordinate System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption 349 the interests of different departments inside the company. The implementation of the ERP system including software and vendors. The quality of the system itself decides the influence of ERP system to the company. ERP vendors are responsible for ERP system development, implementation and maintenance services. Without vendors, the companies is unable to successfully implement ERP project. These two aspects are essential to the success of an ERP project. Therefore, we defined two objectives: Selecting the most appropriate ERP system and selecting the most appropriate ERP system vendor. After determining the strategic objectives, we need to find the specific attributes of criteria according to two objectives. The following will discuss the attributes of criteria for ERP system selection and ERP system vendor selection. ERP system Most of the enterprisers have gradually understood the benefits from ERP system. According to the report of Panorama in 2014 [11], the most popular reason enterprisers implement ERP project is to improve the business (15%). The reason followed by is to better integrate the cross-regional and cross-department system (14%), and to get better service to customers (12%). Figure 1: Reasoning for implementing ERP We can analyze the attributes of criteria to select ERP system from the reasons for imple- menting ERP system: Corresponding to the reason to improve business performance, ERP system should have complete functionality and help improve the company’s performance by integrate business process through the complete module and fit function. Meanwhile, user-friendly interface and operations can help the internal and external personnel operate and understand the system, which can also help improve the operation efficiency and improve business performance. Corresponding to the reason to better integrate the cross-regional and cross-department sys- tem, ERP system should have excellent system flexibility, providing the ease of in-house devel- opment and the ease of integration. The compatibility is particularly important to integrating the cross-regional and cross-department system. Corresponding to better customer service, ERP system should have high system reliability, high system stability. Recovery ability can help avoid the loss of customers in the face of mistakes. 350 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen At the same time, long-term maintenance can also improve customers’ satisfaction. In addition, the total cost of the ERP system implementation is a factor that company must take into consideration. The total cost including system purchase price, consultant cost expenses, system maintenance cost and infrastructure cost. According to the report of Panorama in 2014, it shows that more than 54% of project will exceed the budget for unexpected technical or orga- nizational issues. Considering a long-time implementation of ERP system, the implementation time is also an important attributes. According to the report of Panorama, 63% of the ERP system implementation will take more time than expected. ERP vendor According to the report of Panorama in 2014 [11], among the global famous vendors of ERP system, Oracle (34%) is the most popular, followed by Microsoft Dynamics (20%) and SAP (16%). Companies will pay great attention on the reputation of ERP system vendors. The financial condition, scale of vendor and market share will be taken into consideration. Figure 2: Factors to help select ERP systems In addition, the service that vendors provide matters a lot to the companies. In the service that vendors provide, the implementation of ERP is the most common (21%), followed by re- lated training (19%), organizational change management (14%), software selection (11%).The service is very important because the vendors have professional knowledge and the development and maintenance of system largely depends on the vendors. With the service of vendors, the System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption 351 companies can integrate internal resources and external knowledge and play an important role on the development, implementation and maintenance of ERP system project. Therefore, in order to choose suitable system vendors, companies need to consider the vendors’ technical capability, including R&D capability, technical support capability and implementation ability; Also, ongo- ing service needs considering, which includes warranties, consultant service, training service and service speed. Based on the attributes of criteria above for the objective of ERP system and ERP system vendor, we sums up the attributes affecting the selection of ERP system: Choosing the most appropriate ERP system: minimizing total cost (price, maintenance costs, consultant expenses, infrastructure costs), minimizing implementation time, having complete functionality (module completion, function-fitness, security), having user-friendly interface and operations (ease of operation, ease of learning), having excellent system flexibility (upgrade ability, ease of integration, ease of in-house development), having high system reliability(stability, recovery ability). Choosing the most appropriate ERP vendor: having good reputation (financial condition, scale of vendor, market share), providing good technical capability (R&D capability, technical support capability, implementation ability), and supplying ongoing service (warranties, consul- tant service, and training service, service speed). 3.2 AHP-based approach to select ERP system Introduction of AHP Analytic Hierarchy Process (AHP) was developed by Thomas Saaty in 1971, mainly used in decision-making problems with uncertain circumstances and many criteria [16]. The main property of AHP is that it can turn qualitative problem quantitative. It gives a quantitative importance of each level and uses mathematical method to determine the weights of all elements [5]. Basic steps are as follows: (a) Determine the objectives and criteria P attributes u = {u1,u2, ...up}, (b) Pairwise comparison and judgment matrix The pairwise comparison show the importance of one attributes to another. This subjective judgment can be convert to a numerical value using a scale of 1-9. We can draw the judgment matrix from pairwise comparison. (c) Weights calculation and aggregation Calculate the greatest characteristic root and char- acteristic vector of the judgment matrix S. The characteristic vector is the importance of each evaluation attributes or alternatives and also is the distribution of weight coefficient. (d) Check the consistency. We need to check the consistency of the judgment matrix with CI = λmax−n n−1 . If the consistency index CI of judgment matrix is less than 0.10, we believe the results of the analytic hierarchy sort have satisfactory consistency and the weights is reasonable; Otherwise, the pairwise comparison matrix need to adjust and redistribute the weights. An example of company A Company A is a large multinational manufacturing corporation. Company A has its own production workshop and assembly workshop in mainland China. Its product is involved in seven industries and there are thousands of different types of products. After ten years of development, Company A has rapidly expanded business and grown fast. The staff team has 352 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen grown from dozens to more than 1500 and the annual sales expand to millions of dollars from only six hundred million. With the growing of business, the difficulty of company’s management is also appearing, which makes the implement of ERP system become necessary. Within the company, the sales department is only responsible for the order fulfillment. The lack of a standard process makes the sales department low-efficient. Besides, there is contradiction between purchasing department and project department. Project department, as the service department of the purchasing groups, makes the final decision in the procurement process. The purchasing department can only give suggestions and fulfill the order. This mismatching between right and duty during the procurement process in two departments induces many conflicts. Besides, the financial department uses an independent financial system which only manages the cash flow in that department but not the whole company. Outside the company, purchasing department does not collaborate well with the suppliers. Most of the suppliers are small and medium-sized companies and the information construction remains to be improved. The company cannot timely access to the useful information and the low standardization level of business operation process brings lots of troubles to company A. ERP system may help company A to integrate the departments and manage the information within and outside the company. The leaders of company A are thinking whether they should purchase the professional ERP system and form a project team. They have already selected three ERP systems from different vendors, denoted as system 1, system 2, system 3. The required function for ERP system of different industries has a huge difference. Therefore, enterprisers need to know its industry characteristics and function requirements, when choosing the appropriate ERP system. In addition, the enterpriser need consider the scale of company. We ask three leaders as evaluators for a questionnaire survey and propose ERP system se- lection framework as follows. a) Identify the ERP system characteristics For the ERP system selection, we collect the opinions through the purchasing department, project department, finance department, human resources department and marketing department. It is decided that the system selection is considered from two aspects: One is the ERP system itself; the other is the ERP system vendor. b) Organize the hierarchy structure of Objectives, Criteria, and Alternatives. Objectives are the target of the problem. Criteria is to extract the attributes for evaluating ERP systems. Alternatives are the feasible solutions of the problem. In the case of company A, there are two Objectives with different Criteria. The first objective is screening out the most appropriate ERP system. There are six attributes for evaluating the ERP system, including minimizing total cost (C1), minimizing implementation time (C2), having complete functionality (C3), having user-friendly interface and operations (C4), having excellent system flexibility (C5), having high system reliability(C6). There are three alternatives, called as system 1,system 2 and system 3. The second objective is choosing the most appropriate ERP vendor. There are three at- tributes for evaluating the ERP vendor, including having good reputation (D1), providing good technical capability (D2), supplying ongoing service (D3). The alternatives are same, called as system 1, system 2 and system3. c) The comparison of attributes among Criteria (for example) C1:Minimizing total cost; C2:Minimizing implementation time. If the ratio is 3:1, the evaluator think that minimizing total cost is more important than mini- mizing the implementation time. The importance degree of former is 3 compared to the later. If the ratio is 1:5, the evaluator think that minimizing the implementation time is more important. Its importance degree is 5 compared to minimizing total cost. System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption 353 Figure 3: AHP-based ERP system selection framework Figure 4: AHP-based ERP vendor selection framework Table 1: Criteria questionnaire 9:1 8:1 7:1 6:1 5:1 4:1 3:1 2:1 1:1 1:2 1:3 1:4 1:5 1:6 1:7 1:8 1:9 C1 C2 354 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen Table 2: Alternatives questionnaire 9:1 8:1 7:1 6:1 5:1 4:1 3:1 2:1 1:1 1:2 1:3 1:4 1:5 1:6 1:7 1:8 1:9 S1 S2 Table 3: Judgment matrix of attributes for ERP system Evaluator 1 C1 C2 C3 C4 C5 C6 Wi(weight) C1 Minimizing total cost 1 1/3 1/3 1 1/3 3 0.0766 C2 Minimizing implementation time 3 1 1/4 3 1/3 5 0.1500 C3 Having complete functionality 3 4 1 5 3 7 0.3954 C4 Having user-friendly interface and operations 1 1/3 1/5 1 1/5 5 0.0766 C5 Having excellent system flexibility 3 3 1/3 5 1 5 0.2690 C6 Having high system reliability 1/3 1/3 1/7 1/5 1/5 1 0.3240 λmax: 65.222; Consistency: 0.0829 The comparison of alternatives: Known from the AHP selection framework, each attributes correspond three alternatives which need to take account of the project, so decision-makers need to compare the alternatives for each attributes. d) Select ERP system We can obtain the corresponding judgment matrix through the pairwise comparison after we collect the questionnaire of the evaluators. We need to check the consistency of the judgment matrix with CI = λmax−n n−1 . We find that the consistency index CI of judgment matrix is all less than 0.10.The following only show the judgment matrix for ERP system and each alternative of evaluator 1. Comparing the importance of attributes to each evaluator, we can find that three evaluators tend to share the same opinion: For the attributes of ERP system, “having complete functionality " is considered as a very important attributes for three evaluators, of which the relative weight comes to be the 1st for evaluator 1 and evaluator 2, 2nd for evaluator 3. “having excellent system flexibility " is also of great importance, respectively to be the 2nd,2nd and 1st for evaluator1, 2 and 3. While, “having high system reliability "is considered as the least important attribute for all three evaluators. It is necessary to analyze the result with the situation of company. Company A, a large manufacturing enterpriser, is an integrated supplier providing complete sets of production lines, equipment and services. It has its own production workshop and assembly workshop with com- plete functional departments. More specifically, the number of project team is large and the organizational structure is loose. There are too many types of equipment and spare parts to procure. There is too much communication between project teams and different functional departments, so are the purchasing department and suppliers of company A. Moreover, the difficulty of purchasing has worsen the information distortion and then deepen the contradic- tions between project teams and functional departments. Thus, an information system covered multi-department is particularly important, which can reduce the conflicts between different de- Table 4: Judgment matrix of alternatives for C1 and importance of attributes The pairwise comparison of alternatives for C1 System 1 System 2 System 3 Wi(weight) System 1 1 3 5 0.6370 System 2 1/3 1 3 0.2583 System 3 1/5 1/3 1 0.1047 λmax: 3.0385; Consistency: 0.0370 System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption 355 Table 5: Judgment matrix of alternatives for C2 and importance of attributes The pairwise comparison of alternatives for C2 System 1 System 2 System 3 Wi(weight) System 1 1 5 1/3 0.2790 System 2 1/5 1 1/7 0.0719 System 3 3 7 1 0.6491 λmax: 3.0649; Consistency: 0.0624 Table 6: Judgment matrix of alternatives for C3 and importance of attributes The pairwise comparison of alternatives for C3 System 1 System 2 System 3 Wi(weight) System 1 1 7 3 0.6694 System 2 1/7 1 1/3 0.0879 System 3 1/3 3 1 0.2426 λmax: 3.0070; Consistency: 0.0068 Table 7: Judgment matrix of alternatives for C4 and importance of attributes The pairwise comparison of alternatives for C4 System 1 System 2 System 3 Wi(weight) System 1 1 1/3 1/7 0.0879 System 2 3 1 1/3 0.2426 System 3 7 3 1 0.6694 λmax: 3.0070; Consistency: 0.0068 Table 8: Judgment matrix of alternatives for C5 and importance of attributes The pairwise comparison of alternatives for C5 System 1 System 2 System 3 Wi(weight) System 1 1 5 1 0.4806 System 2 1/5 1 1/3 0.1400 System 3 1 3 1 0.4054 λmax: 3.0291; Consistency: 0.0279 Table 9: Judgment matrix of alternatives for C6 and importance of attributes The pairwise comparison of alternatives for C6 System 1 System 2 System 3 Wi(weight) System 1 1 1 1/5 0.1336 System 2 1 1 1/7 0.1194 System 3 5 7 1 0.7471 λmax: 3.0126; Consistency: 0.0121 356 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen Table 10: Judgment of importance of attributes Attributes Evaluator 1 Evaluator 2 Evaluator 3 ERP System minimizing total cost 0.0766(4) 0.1500(3) 0.0378(5) minimizing implementation time 0.1500(3) 0.1500(3) 0.1790(3) having complete functionality 0.3954(1) 0.3910(1) 0.3356(2) having user-friendly interface and operations 0.0766(4) 0.0565(5) 0.0566(4) having excellent system flexibility 0.2690(2) 0.2085(2) 0.3710(1) having high system reliability 0.0324(6) 0.0420(6) 0.0200(6) ERP Vendor having good reputation 0.0719(3) 0.0554(3) 0.0995(3) providing good technical capability 0.2790(2) 0.5990(1) 0.3355(2) supplying ongoing service 0.6491(1) 0.3456(2) 0.5650(1) partments and project teams by sharing the information effectively. To sum up, we can find the great importance of “having complete functionality". At the same time, company A is a foreign multinational enterprisers. It has different branches in 44 countries around the world and five business areas. There are many different brands and products in each business area. While in the implementation of procurement, the boundaries between different departments is clear and they independently do different works in the business. There is no communication and no collaboration which generates a lot of repetitive work and additional costs such as assessment, repeated negotiation, travel cost, quality control cost and so on. It is necessary for company A share information and resources cross areas and departments, so the system flexibility is very valued. For the attributes of ERP vendor, “providing good technical capability" and “supplying on- going service" are of great importance while “having good reputation" is not considered as an important attribute. “Providing good technical capability" includes R&D capability, technical support capability and implementation ability. Besides the initial system development, maintenance and upgrade stage of ERP project also need excellent technical support. In the adjustment stage, ERP system need continuous operation maintenance, and even need a new version or new functions. The system’s maintenance and upgrade require a long-term technical support. “Supplying on-going service" includes the most basic warranty service, consultant service and training services. The users’ feedback is an important criterion to see whether ERP system is running smoothly. It is necessary that users approve and understand the system. The implemen- tation of ERP project requires users to master the complicated operation skills. If the employee do not understand how the system works, it will ultimately affect the entire ERP system. The success of ERP system must be based on reasonable operation. In the ERP system implementation of company A, training objects include the suppliers of A company besides the employees. There are more than 500 suppliers and quite of them are small and medium-sized companies, which adopt the traditional manufacturing management mode and are lack advanced management philosophy. The information cannot be inputted and processed timely and the information management system is incomplete and imperfect. To help the suppliers to adapt the ERP system is necessary and challenging. Therefore, ongoing training service is valued. After three evaluators give a weight to all attributes, we can get the evaluation score of three ERP systems through the two judgment matrix (including criteria judgment matrix and alternative judgment matrix). Considering that we develop two objectives (ERP system and ERP vendor), we give equal weights to them. Finally we get the evaluation score of three ERP systems. Following is the result of evaluator 1: System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption 357 Table 11: Evaluation score of ERP systems (1) Evaluator 1 ERP system ERP vendor final score System 1 0.4957 0.3129 0.4043 System 2 0.1185 0.323 0.22075 System 3 0.3858 0.3641 0.37495 (1) Equal weights to ERP system and ERP vendor Table 12: Evaluation score of ERP systems (2) Evaluator 1 Evaluator 2 Evaluator 3 final score System 1 0.40430(1) 0.37155(2) 0.27362(2) 0.34982(2) System 2 0.22075(3) 0.18145(3) 0.23123(3) 0.21114(3) System 3 0.37495(2) 0.44700(1) 0.49515(1) 0.43904(1) (2) Equal weights to each evaluator e) Get the final result. Evaluator 2 and evaluator 3 prefer to choose system 3 while evaluator 1 prefers to choose system 1. We can find that the score of evaluator 1 to system 1 and system 3 is close. Moreover, system 3 gets the highest final score. The result show that system 3 is the most appropriate ERP system for company A. 4 Evaluation of ERP system Company A started to promote the ERP project after selecting the appropriate ERP system. In order to implement ERP project successfully, company A set up a team to take charge of the entire implement. In the preparatory stage, company A focused on the training and helped employee understand the ERP system. Then, company A started to research and analyze, even the specific operation of each departments, in order to adapt the ERP system to match the company. After this, company A formally set up ERP system. They built a complete system framework taking full consideration of opinions from each departments and vendors. Company A fully combined the original function with the business process. In addition, ERP project team also optimized the mismatch between ERP system and company’s business. Next is to import massive data. Company A successfully imported the internal and external data before changing the system and checked the accuracy of data. In November 2013, company A officially started using ERP system. After cautious consideration and selection, the new system still bring impact to the company on business. With time goes by, employees have been familiar with ERP system and it has run methodically. Reviewing the ERP project of company A, it went well during the implementation, but we need to see whether it brings significant benefits to company A. The evaluation of implementation of ERP system is of great importance, which involved the influence on company’s strategy, the impact on performance of management and the business process. The following will show the evaluation for the performance of the ERP system implementation. 4.1 The framework of evaluation The success of ERP project is far more than that system goes live. How to judge or define the success of ERP project is also different for different companies or different industries. According 358 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen to company A’s business and structure, we listened the opinion of managers and sort out the following performance evaluation structure. Figure 5: Structure of performance evaluation The description of each factors in the structure of evaluation above is as follows. It’s worth mentioning that “functionality" and “implementation result" are positive statements, “matching" and “attitude of users" is reverse. This is to reduce the respondents’ deflection and is also helpful to remove those regardless of content. As a result, we need to adjust correspondingly when scoring. In the following results, “average score" is not adjusted and the “real score" is adjusted. Functionality: ERP system has user-friendly interface and operations; ERP system has high intelligence; ERP system has excellent flexibility and compatibility; ERP system has high reliability. Matching: ERP system does not match the company’s operation process; ERP system does not adapt to the mismatch; The data entry and processing of ERP system does not match with the original model; ERP System does not match company’s organizational structure and strategy. Implementation results: ERP system helps improve the efficiency and communication cross-department; ERP system promotes the collaboration with suppliers; ERP system help the company with demand forecasting and capacity management; ERP system help the company improve the quality of the products and arrange the production reasonably. Attitude of users: The users of ERP system do not get the corresponding training and do not understand ERP system; The management does not know the implementation of ERP system implementation and give no support to it; The users of ERP system think that it does not improve the performance. The performance get even worse than before; ERP system lacks flexibility and makes the company lose advantages. 4.2 The analysis based on FIE 30 questionnaires were distributed within the company, we recycled 30 questionnaires and the 23 of them were valid. The results are as follows: From the data we can see that the mean of real average score is 3.3894 and the total real score is 54.2308, higher than the total real average score 48, under normal distribution assumption. If System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption 359 Table 13: Evaluation score Strongly disagree Disagree Neutral Agree Strongly agree Average score Real score Standard devia- tion Functionality 1 0.0000 0.0769 0.2308 0.6923 0.0000 3.6154 3.6154 0.62 2 0.0000 0.3846 0.3077 0.3077 0.0000 2.9231 2.9231 0.83 3 0.0000 0.3077 0.4615 0.2308 0.0000 2.9231 2.9231 0.73 4 0.0000 0.0769 0.3846 0.4615 0.0769 3.5385 3.5385 0.75 Matching 5 0.0000 0.3077 0.5385 0.1538 0.0000 2.8462 3.1538 0.73 6 0.0000 0.5385 0.3846 0.0769 0.0000 2.5385 3.4615 1.12 7 0.0000 0.3846 0.3077 0.3077 0.0000 2.9231 3.0769 0.84 8 0.1538 0.3846 0.3077 0.1538 0.0000 2.4615 3.5385 1.42 Implementation result 9 0.0000 0.2308 0.3077 0.4615 0.0000 3.2308 3.2308 0.80 10 0.0000 0.0000 0.3077 0.6154 0.0769 3.7692 3.7692 0.58 11 0.0000 0.1538 0.3846 0.4615 0.0000 3.3077 3.3077 0.72 12 0.0000 0.0769 0.0769 0.7692 0.0769 3.8462 3.8462 0.66 Attitude of users 13 0.0769 0.4615 0.4615 0.0000 0.0000 2.3846 3.6154 1.38 14 0.2308 0.3846 0.3846 0.1538 0.0000 2.7692 3.2308 1.26 15 0.2308 0.5385 0.1538 0.0769 0.0000 2.0769 3.9231 2.02 16 0.0000 0.3077 0.4615 0.2308 0.0000 2.9231 3.0769 0.75 Total 54.2308 only judging from this data, we can say that company A thinks the ERP project help improve the performance. There are some shortcomings in the classical statistical analysis. It cannot show the overall attitude intuitively and cannot directly show the proportion of different order of evaluation. Given the order of evaluation in questionnaires is fuzzy, we analyze the data using the fuzzy integrated evaluation (FIE) method. When determining the weights of each sub-factor, we use the AHP method. FIE FIE is a method to evaluate after fuzzy transform according to the criteria and measured values. The process of FIE: Assume the evaluation target as a fuzzy set composed of a number of factors; Then set order of evaluation to these factors and make up a fuzzy set; Next calculate the membership degree of each factors to the order of evaluation; And then according to the weights of factors in the evaluation, calculate the quantitative value [18]. The evaluation of ERP implementation of company A a) The factors set Ut: U = {U1,U2,U3,U4}= {functionality, matching, implementation results, attitude of users} U1 = {U11,U12,U13,U14}= {friendly operation, intelligence, flexibility, reliability} U2 = {U21,U22,U23,U24}= {match up, changeability, data entry, organizational structure} U3 = {U31,U32,U33,U34}= {cross-department communication, external communication, de- mand forecasting, production arrangement} U4 = {U41,U42,U43,U44}= {operational training, management training ,performance, rigid- ity} b) The evaluation set for factors: Evaluation set is a collection of all results of the evaluation by evaluators. V= strongly disagree, disagree, neutral, agree, strongly agree c) The fuzzy relationship matrix R: According to the questionnaire statistics, we can get the proportion of different order of evaluation .The statistical records are as follows: 360 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen Table 14: Factor set U1 Factor set U1 Strongly disagree Disagree Neutral Agree Strongly agree Friendly operation 0.00% 7.69% 23.08% 69.23% 0.00% Intelligence 0.00% 38.46% 30.77% 30.77% 0.00% Flexibility 0.00% 30.77% 46.15% 23.08% 0.00% Reliability 0.00% 7.69% 38.46% 46.15% 7.69% Table 15: Fuzzy relationship matrix R1 =   0.00% 7.69% 23.08% 69.23% 0.00% 0.00% 38.46% 30.77% 30.77% 0.00% 0.00% 30.77% 46.15% 23.08% 0.00% 0.00% 7.69% 38.46% 46.15% 7.69%   R2 =   0.00% 15.38% 53.85% 30.77% 0.00% 0.00% 7.69% 38.46% 53.85% 0.00% 0.00% 30.77% 30.77% 38.46% 0.00% 0.00% 15.38% 30.77% 38.46% 15.38%   R3 =   0.00% 23.08% 30.77% 46.15% 0.00% 0.00% 0.00% 30.77% 61.54% 7.69% 0.00% 15.38% 38.46% 46.15% 0.00% 0.00% 7.69% 7.69% 76.92% 7.69%   R4 =   0.00% 0.00% 46.15% 46.15% 7.69% 0.00% 15.38% 38.46% 38.46% 23.08% 0.00% 7.69% 15.38% 53.85% 23.08% 0.00% 23.08% 46.15% 30.77% 0.00%   We can get the fuzzy relationship matrix R1 from U1. Similarly, we can get fuzzy relationship matrix R2 from U2. While this subset is disjunctive, we should reverse the arrangement. d) The weight of each factor: In the evaluation system, the importance of each factor to realize the goal of system is different. The weight of each factor show the different importance. Set the weights reasonably and appropriately is important for evaluation. Here we use AHP to get the weights. The Supervisors of company score the four factors: functionality, matching, implementation results, and attitude of users. We get the following results: Calculate the greatest characteristic root and characteristic vector of the judgment matrix. The characteristic vector is the importance of each evaluation factors and also is the distribution of weight coefficient. U = [0.2477, 0.1259, 0.5538, 0.0727]. Similarly, calculate the weight of each factor under the four dimensions according to the experts’ scoring: Each weight of factors passes the consistency check. e) Get the evaluation results. B = U ∗ R: B1 = U1 ∗ R1= (0.0000 0.2299 0.3975 0.3571 0.0154); B2 = U2 ∗ R2= (0.0000 0.1679 0.4360 0.3590 0.0370); B3 = U3 ∗ R3= (0.0000 0.0927 0.2966 0.5657 0.0451); B4 = U4 ∗ R4= (0.0000 0.0799 0.3147 0.4684 0.1618). D = U ∗ R= (0 System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption 361 Table 16: Comparison matrix of factors Comparison matrix of factors U1 U2 U3 U4 Wi(weight) U1 functionality 1 3 1/3 3 0.2477 U2 matching 1/3 1 1/5 1/3 0.1259 U3 implementation result 3 3 1 5 0.5538 U4 attitude of users 1/3 5 1/5 1 0.0727 λmax: 4.1975; Consistency: 0.0740 Table 17: Comparison matrix of sub-factors in U1 Comparison matrix of sub-factors in U1 U11 U12 U13 U14 Wi(weight) U11 functionality 1 3 1/3 1/2 0.1612 U12 matching 1/3 1 1/5 1/3 0.0740 U13 implementation result 3 5 1 5 0.5641 U14 attitude of users 2 3 1/5 1 0.2006 λmax: 4.2219; Consistency: 0.0831 Table 18: Comparison matrix of sub-factors in U2 Comparison matrix of sub-factors in U2 U21 U22 U23 U24 Wi(weight) U21 match up 1 5 3 3 0.5244 U22 change ability 1/5 1 1/2 1/2 0.0957 U23 data entry 1/3 2 1 1/3 0.1390 U24 organizational structure 1/3 2 3 1 0.2408 λmax: 4.1575; Consistency: 0.0590 Table 19: Comparison matrix of sub-factors in U3 Comparison matrix of sub-factors in U3 U31 U32 U33 U34 Wi(weight) U31 cross-department communication 1 1/3 4 3 0.2854 U32 external communication 3 1 3 4 0.4944 U33 demand forecasting 1/4 1/3 1 2 0.1290 U34 production arrangement 1/3 1/4 1/2 1 0.0912 λmax: 4.2367; Consistency: 0.0886 Table 20: Comparison matrix of sub-factors in U4 Comparison matrix of sub-factors in U4 U41 U42 U43 U44 Wi(weight) U41 operational training 1 3 1/2 3 0.3089 U42 management training 1/3 1 1/3 3 0.1612 U43 performance 2 3 1 3 0.4369 U44 rigidity 1/3 1/3 1/3 1 0.0930 λmax: 4.2148; Consistency: 0.0805 362 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen 0.1352 0.3405 0.4810 0.0452). f) Analyze the results. The results show 1.64% of evaluators strongly disagree that ERP project bring positive effect; 18.75% of them disagree; 34.05% remain neutral; 42.87% agree the positive effects of ERP project and 2.88% strongly agreed with it. According to the maximum membership degree principle, the conclusion is "agree". Multiply the raw score (1-5) in Likert scale by the number of sub-factors, 16. Then we get the level parameters in evaluation set and the column vector is: p = DE = 016 + 0.135232 + 0.340548 + 0.481064 + 0.045280 = 55.0677. The result is close to the statistical analysis result, 54.2308. It shows that the result based on FIE is consistent with the result based on classical statistical analysis. Company A recognizes ERP project as a beneficial project. Figure 6: 6 S Assessment scale We can see the factor “attitude of users" get the highest score (60.5846). Factor “imple- mentation results" follows (57.0085). The other two factors “matching degree "(52.2384) and “functionality" (50.5253) is not ideal, which are lower than the average score (55.0677). This re- sult is meaningful for company A’s management. They should focus on improving the matching degree and the system’s functionality in the future. Regarding the factor “matching degree", we can see that in most cases, company’s orga- nizational structure and process mismatch the ERP system’s functionality. When there exist mismatches between ERP system and company’s business process, what to do depends on dif- ferent situations. On the one hand, if the operation of ERP system is inefficient, company can ask the vendors to adjust the system to adopt company. On the other hand, if the ERP system can improve more efficient performance, then company can make appropriate changes on the operation process to adapt to ERP system. Regarding the factor “functionality", although company A pays much attention to system’s functionality during the selection, we still find that the respondents is not very satisfactory with the functionality. It reflects that there exists difference between the effects after t implementation and expectations. Company A should fully understand the ERP is a long-term project and it is System Selection and Performance Evaluation for Manufacturing Company’s ERP Adoption 363 ongoing to look for problems and put forward the solution. The project team needs to stay close to the ERP vendors and solve the problems together. 5 Conclusions It is necessary to use the proper process control method during the implementation of a successful ERP system project. This research focuses on the selection and performance evaluation of ERP system. We study the criteria of the ERP system selection and develop a framework to select ERP system based on AHP method. We combine the objective and criteria, then compare the im- portance of attributes among criteria and alternatives, which represents the opinion of different evaluators from different departments. Finally, we select the most appropriate ERP system. The selection based on AHP helps the ERP system match with the strategies of the company. With the help of AHP, we can divide the goal of company into simple ones. This help the goal be put into practice. The selection framework based on AHP could be adjusted according to the development of company and has a high degree of flexibility. After the selection of ERP system, we study the evaluation of ERP implementation in com- pany A. We combine the Likert scale, AHP and FIE methods, from four dimensions (the system’s functionality, the matching degree, implementation results and attitude of users), to evaluate the implementation of ERP project. We find that objective and accurate evaluation of the ERP im- plementation can help company allocate resources. We develop an evaluation framework based on FIE and also use AHP to determine the weights, which reducing the subjectivity of evaluation. Regarding the future work, creating a practical decision support software package could serve. This can be valuable for companies facing similar decision-making problems as company A, which we have extensively studied in this paper. Acknowledgement This paper is supported in part by National Natural Science Foundation of China (No. 71571194, 71301032, 71201175), and Excellent Young Teachers Program of Guangdong Univer- sities and Colleges (YQ2015014). Bibliography [1] Borne P., Popescu D., Filip F. G., Stefanoiu D., Dubuisson B.(2013); Optimization in En- gineering Sciences: Exact Methods, John Wiley & Sons, Inc., Hoboken, NJ, USA, 2013. [2] Elisabeth J. U., Ronald R. H., Umble M. M. (2003); Enterprise resource planning: Imple- mentation procedures and critical success factors, European Journal of Operation Research, 146, 241-257, 2003. [3] Filip, F. G. (2011); Designing and building modern information systems; A series of decisions to be made, Computer Science Journal of Moldova, 19(2), 119-129, 2011. [4] Filip F. G. (2014); A decision-making perspective for designing and building information systems, International Journal of Computers Communications & Control, 7(2), 264-272, 2014. 364 B.Z. Niu, K.L. Chen, H.Z. Huang, Y. Li, L. Chen [5] Lai V.S., Wong B.K., Cheung W. (2002); Group decision making in a multiple criteria environment: A case using the AHP in software selection. European Journal of Operational Research, 137(1), 134-144, 2002. [6] Moriso M., Tsoukias A. (1997); JusWare: A methology for evaluation and selection of software roducts, IEE Proc-Softw. Eng., 144(2), 162-174, 1997. [7] Nicolaou A. I. (2014); Firm performance effects in relation to the implementation and use of enterprise resource planning systems. J Information System, 18, 79-105, 2014. [8] O’Leary D. (2000); Enterprise resource planning system: system, life cycle, electronic com- merce and risk. New York: Cambridge University Press, 2000. [9] Panorama Consulting (2014); The 2014 Manufacturing ERP Report. http://panorama- consulting.com/resource-center/erp-industry-reports/ [10] Panorama Consulting (2014); A Panorama Consulting Solution 204 ERP Research Report. http://Panorama-Consulting.com/resource-center/2014-erp-report/. [11] Poston R., Grabski S. (2001); Financial impacts of enterprise resource planning implemen- tation. Int J Account Inf Syst, 2(4), 271-94, 2001. [12] Saaty T. L. (1979); Application of analytical hierarchies, Mathematics and Computers in Simulation, 21(1), 1-20, 1979. [13] Shang, S., Seddon, P. (2002); Assessing and managing the benefits of enterprise system: the business manager’s perspective. Information System Journal, 20(12), 271-299, 2002. [14] Siriginidi S.R. (2000); Enterprise resource planning in re-engineering business. Business Process Management Journal, 6(5), 376-391, 2000. [15] Teltumbde A. (2000); A framework of evaluating ERP projects. International Journal of Prodcution Research, 27(8), 12-16, 2000. [16] Wang Y., Niu B., Guo P. (2013); On the advantage of quantity leadership when outsourcing production to a competitive contract manufacturer. Production and Operations Manage- ment, 22 (1), 104-119, 2013. [17] http://www.technologyevaluation.com/products-and-services/our-proven-approach/ [18] Zadeh L.A. (1999); Fuzzy logic and the calculi of fuzzy rules, fuzzy graphs, and fuzzy probabilities. Computers & Mathematics with Applications, 37(11-12), 35, 1999.