IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 223 Vol. 5 Issue 2 2013 ISSN 1936-6744 Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process Yasaman Bijan Azad University of Qazvin Faculty of Industrial & Mechanical Engineering, Iran yasi.bijan@gmail.com Abbas Keramati University of Tehran, Tehran, Iran Associate Professor of Industrial Engineering Department Faculty of Engineering Keramati@ut.ac.ir Mona Salehi Lulea University of Technology Industrial Marketing, E-commerce and Logistics Sweden monaasalehi@gmail.com ABSTRACT This study proposes a model for comparing the customer satisfaction indices of two or more ecommerce competitors in order to select the most preferred website in a specific context. The importance of customer satisfaction factors from the user’s point of view were calculated in the specific context of ecommerce. This study takes a new step towards integrating satisfaction literature by proposing a model for ranking the American Customer Satisfaction Index (ACSI) factors based on users' expectations about different online contexts. Hence, the approach provides a new way to compare customer satisfaction among e-business competitors. The suggested model was shaped by merging the Analytic Network Process (ANP) approach with the ACSI for ecommerce. The model tested two Iranian e-recruitment websites through a survey designed and conducted via emails to those who had used both web sites. Subsequently, the relative importance of the factors was determined, and finally e-recruitment websites were compared with each other. As a result, the most preferred website with respect to different ACSI factors was chosen and the relative importance of each ACSI factor considering the influence it had on the use of the e- recruitment website and user satisfaction was identified. Key word: Analytic Network Process (ANP), Multi Criteria decision making (MCDM), Online Satisfaction, Customer Satisfaction Index, Ebusiness mailto:yasi.bijan@gmail.com mailto:Keramati@ut.ac.ir mailto:monaasalehi@gmail.com Rob Typewritten Text http://dx.doi.org/10.13033/ijahp.v5i2.180 IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 224 Vol. 5 Issue 2 2013 ISSN 1936-6744 1. Introduction The World Wide Web is one of the most important Internet services and it has been largely responsible for the phenomenal growth of the Internet in recent years. An increasingly popular and important web-based activity is ecommerce (Graja & McManis, 2001). Ecommerce through the Internet has become an important transaction model in international trade (Liu, Zeng, xu, & Koehl, 2008). When designing ecommerce systems it is important to determine the customer believes is required for satisfactory service. Studies have shown that the financial value of customer relationship to the company has received growing attention among top executives. Managers increasingly tend to see customer satisfaction as a valuable intangible asset and thus an important corporate target (Luo, Wieseke, & Homburg, 2012). Customer satisfaction can not only improve the financial situation of enterprises, but also bring about a unique competitive advantage for ecommerce websites and enhance customer loyalty (Yang & Ding, 2009). Customer satisfaction is critical for establishing long-term client relationships, and has considerable impact on customer loyalty (Schaupp & Bélanger, 2005; Chang & Wang, 2011) which is an important factor for business growth (Kumar, Sharma, Shah, & Rajan, 2013). Organizations which are more efficient in providing value for customers leading to higher customer satisfaction are more likely to survive in a competitive situation (Kujala & Ahola, 2005). Online recruitment services are among the most popular applications on the internet. Since employers are required to pay for the service, their perceptions of the level of service are typically of the most concern to the recruitment websites, and the recruitment-service quality level for the job seeker is typically ignored. Therefore, it is critical to study the e-recruitment customer satisfaction from the job seekers’ perspective for any e-recruitment website to stay profitable in a competitive environment (Keramati & Salehi, 2013). In addition, in order to avoid ineffectiveness during deployment, it is important to clarify the impact factors of the customer satisfaction index (CSI) and develop a suitable assessment method to evaluate its performance. In this research, an Analytic Network Process (ANP)–based assessment model was constructed to assess the effects of factors of CSI in the context of ecommerce, and to find the preferred e- recruitment website with respect to customer satisfaction factors in Iran. In order to achieve this goal, we provide a model that considers factors and the customer satisfaction model. In addition, the model has the ability to compare two or more e- recruitment websites scientifically through measuring and ranking of relative importance factors. The rest of the paper is organized as follows: section 2 reviews the literature on customer satisfaction models, section 3 provides the research framework, section 4 provides an in-depth description of our research methodology, and section 5 presents http://link.springer.com/search?facet-author=%22Hubert+Graja%22 http://link.springer.com/search?facet-author=%22Jennifer+McManis%22 http://www.sciencedirect.com/science/article/pii/S0378475407003576 IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 225 Vol. 5 Issue 2 2013 ISSN 1936-6744 the results of our analysis. The last section (Section 6) presents the discussion and conclusions of the study. 2. Literature review Since Oliver put forward a cognitive model for characterizing antecedents and consequences of satisfaction in 1980, customer satisfaction and customer satisfaction index (CSI) have been widely developed in both theory and applications (Liu et al., 2008). While customer satisfaction has been defined in various ways, the high-level conceptualization that appears to have gained the widest acceptance states that satisfaction is a customer’s post-purchase evaluation of a product or service (Bayraktar, Tatoglu, Turkyilmaz, Delen, & Zaim, 2012). Within the existing literature on customer satisfaction research, various customer satisfaction models have been developed based on a cumulative view of satisfaction (Bayraktar et al., 2012). In 1989, a Swedish researcher (Fornell, 1992) built the first model of CSI – the Swedish Customer Satisfaction Barometer (SCSB). The American customer satisfaction index (ASCI) was set up in 1994 (Wood, Siegel, Feldman, Love, Rodrigues, Malamud, Lagana, & Crafts, 2008). Another well-known CSI, the European Customer Satisfaction Index (ECSI) was built by 11 countries of the European Union in 1999 (Liu et al., 2008). A review of the methods used by the various indices indicates that the ACSI methodology meets or exceeds accepted standards for validity and accuracy of online user surveys .Additionally, the ACSI uses a standard set of core questions across platforms dna its results can be benchmarked to individual and aggregate results. Furthermore, the ACSI survey methodology allows for the inclusion of custom questions geared to each individual client, and thus permits a blend of both standardized and customized queries. Customer satisfaction surveys like the ACSI are tools for listening to “what users say” to determine user perceptions of a website’s usefulness and performance. Perceptions are inherently subjective, but they do help web managers to understand another facet of user opinion (Wood et al., 2008). The American Customer Satisfaction Index (ACSI) was first implemented in 1994 as an offline survey measuring customer satisfaction with businesses, and was adapted to the Internet in 2002. More than two dozen other federal websites began using the survey in 2002 (Wood et al., 2008). The core ACSI methodology was developed by Professor Claes Fornell, Director of the National Quality Research Center, University of Michigan Business School, and it is offered as an online service by ForeSee Results, Inc. of Ann Arbor, Michigan. The ACSI method uses multiple regression analysis to link questions on key elements that drive customer satisfaction with questions on overall customer satisfaction that are in turn linked to questions on future customer behavior. All standardized questions are framed by using a 10-point Likert scale (Wood et al., 2008). IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 226 Vol. 5 Issue 2 2013 ISSN 1936-6744 The ForeSee Results ACSI-based questionnaire for websites is composed of a core set of 14 to 20 questions (Tullis & Albert, 2008). The results for the website are divided into six quality-related elements that drive satisfaction: content, functionality, look and feel, navigation, search, and site performance. The results also assess the overall satisfaction and future behavior of the user (likelihood to return and likelihood to recommend) (Wood et al., 2008).  Content – Accuracy, quality, and freshness of news, information, and content on the website.  Functionality – Usefulness, convenience, and variety of online features.  Look and feel – Visual appeal of the site and its consistency.  Navigation – Organization of the site and ease of navigation.  Search – Utility and effectiveness of the site’s search tool.  Site performance – Speed, consistency, and reliability of loading of pages on the website.  Likelihood to return – Reflects customers’ propensity to return to the site, which provides an opportunity for companies to further engage customers online.  Likelihood to recommend – Shows how likely online visitors are to engage in positive word-of mouth marketing by recommending the website to a friend, family member, or colleague. The intensity of the online competition has resulted in a simple mouse click to select a new provider (Singh, 2002). Disruptive internet technologies force ecommerce to focus on the factors that contribute to their success in the competitive environment. Some research has been conducted comparing different models’ factors that lead to success in a competitive environment. For example, Keramati and Salehi (2013) compared website successes in the context of e-recruitment through an Analytic Network Process (ANP) approach. Different researchers have used different descriptions of measures for customer satisfaction. However, no one has focused on comparing the customer satisfaction indices between two or more e-business competitors in order to select the most preferred website within a context, or on finding the importance of customer satisfaction factors from the users’ points of view in a specific context of ecommerce. Therefore, this research tries to fill that gap in the literature. 3. Research framework The ANP, developed by Thomas L. Saaty, is a comprehensive decision-making technique that has the capability to include all the relevant criteria which have some bearing on a decision (Jharkharia & Shankar, 2007). ANP is an extension of the Analytic Hierarchy Process (AHP) that models a decision-making framework that assumes a unidirectional hierarchical relationship among the decision levels. IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 227 Vol. 5 Issue 2 2013 ISSN 1936-6744 The AHP/ANP approaches offer several advantages over other MCDM methods. Firstly, they are not proportionately complicated. Secondly, they have the supplemental power of being able to mix quantitative and qualitative factors into a decision. Thirdly, they can be fit together with other solution approaches such as optimization, and goal programming. Fourthly, they may use a hierarchical structuring of the factors involved and finally, their judgment elicitations are completed using a decompositional approach, which has been shown in experimental studies to reduce decision-making errors (Taslicali & Ercan, 2006). The ANP method has advantages over AHP for decision making. One of the advantages is that in the ANP rank reversal problem is not an issue, thereby it is more accurate and useful than the AHP as a decision support instrument for intricate situations (Taslicali & Ercan, 2006). The greatest advantage of the ANP model is that it can handle intangible factors based on individual or collective judgment of the situation (Saaty & Vargas, 2006). Therefore, based on these advantages, the ANP was used for modeling and comparing in this study. Technically, the model consists of clusters and elements where dominance or relative importance of influence is the central concept. Judgments are provided based on Saaty’s 1–9 scales (Saaty, 2005). A score of 1 represents indifference between the two components, and a score of 9 represents overwhelming dominance of the component under consideration over the compared component (Keramati & Salehi, 2013). The strength of the ANP lies in its use of ratio scales to capture all kinds of interactions, formulate accurate predictions, and make better decisions (Saaty, 2003). The relationship between different categories of the model complies with the variance or causal model as well as the process model. According to the arrows between the different categories, one category precedes another category because of the process model. Further, any increase or decrease in a category would lead to an increase or decrease in the other interrelated category, for instance, content to overall satisfaction according to the process model. However according to the causal or the variance model, a positive (negative) experience in content will lead to increased (decreased) overall satisfaction. Certainty to recommend and likelihood to return will occur because of certain overall satisfaction. The decision maker will use a series of pairwise comparisons and compare two components at a time with respect to the source or parent criterion to elicit preferences of various components and attributes. All nodes that are to be compared pairwise are always in the same cluster and are compared with respect to their parent (source) element – the node from which they are connected. This results in local priorities of the nodes with respect to the source node. IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 228 Vol. 5 Issue 2 2013 ISSN 1936-6744 In ACSI, model clusters have influence on each other. The ANP is an approach used to determine the relative importance of variables in the ACSI model and to compare the customer satisfaction rate in a similar context of ecommerce. and then to rank alternative websites. ACSI is extended through the ANP in this research article. The proposed framework is shown in Figure 1. IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 229 Vol. 5 Issue 2 2013 ISSN 1936-6744 Figure 1. ACSI is extended through ANP in this research article IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 230 Vol. 5 Issue 2 2013 ISSN 1936-6744 The parent element directs the arrows in the ANP. Different categories in the model are compared pairwise with respect to their influence on the parent element. In the ACSI model, "customer satisfaction" is influenced by "content,” "functionality,” "look and feel,” "navigation,” "search,” and "site performance". Therefore, in the ANP framework, customer satisfaction is the parent element. In ANP, "content,” "functionality,” "look and feel,” "navigation,” "search,” and "site performance" are compared pairwise with respect to their influence on the parent element. Likewise, other categories in the model are compared pairwise with respect to their parent elements. 4. Research methodology 4.1. Data collection and sampling Data for this study were collected from the common online academic union users of the only two Iranian e-recruitment websites, www.irantalent.com and www.agahjobs.com. The sampling group consisted of 100,000 university students who had graduated. The group was chosen randomly from http://www.irexpert.ir which consists of graduated Iranian university experts. Simple random sampling was selected as the most appropriate sampling technique based on probability. The minimum sample size required for the population of 100,000 with a confidence level of 95% and confidence interval of 5 was calculated as 383 (Keramati & Salehi, 2013). Based on this calculation, 383 users were selected randomly and the survey questionnaire (Appendix 1) was sent to them by email. At the beginning of the survey, a question asked the users whether or not they had used both websites. Then, only those users who gave a positive response to this question were allowed to fill in the rest of the survey. 46 of eligible users returned usable surveys, which were answered completely and consistently. 4.2. Measures As mentioned before, Saaty suggested a scale of 1 to 9 when comparing two components. A score of 1 indicates that the two options have equal importance or indifference, where a score of 9 indicates the overwhelming dominance of the component under consideration over the compared component in a pairwise comparison matrix (Saaty, 1980). 4.3. Group Decision Making In order for more than one person to make a decision, group decision-making techniques are used to integrate the judgments of decision-makers (Sadeghi, Rashidzadeh & Soukhakian, 2012). Decision-makers were users of the websites who filled in the survey completely and consistently. In order to achieve consistent judgments, the individual judgments were gathered and individually entered into SuperDecisions in order to find the consistent responses. This is how we gathered 46 judgments. The next step involved integrating the individual decisions and making a group decision based on complete and consistent judgments. As Saaty and Vargas http://www.irexpert.ir/ IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 231 Vol. 5 Issue 2 2013 ISSN 1936-6744 (2006) state, the reciprocal property plays an important role in combining the judgments of several individuals to obtain a judgment for a group. Judgments must be combined so that the reciprocal of the synthesized judgments must be equal to the syntheses of the reciprocals of these judgments. It has been proved that the geometric mean method is the unique way to do that (Satty & Vargas, 2006). Equation 1 In Equation 1, wi is the weight given to each decision-makers’ opinion. In this instance it is 1, which means all the decision-makers opinions were given the same weight of importance. M is the number of decision makers, which is 46. Aij indicates the elements of the decision-makers’ comparison matrices and indicates the elements of the final comparison matrix which were entered in super decisions software. 4.4. Data Analysis The ANP calculations were implemented through the Super Decisions software created by Saaty (2004) to alleviate the mathematical burden. Super Decisions provides results including weights of variables with respect to variables that pairwise compared to priorities of alternatives and sensitivity analysis. The results of the questionnaire were analyzed so that the weights of nodes in each cluster with respect to the parent node with which they are compared to were calculated. Furthermore, alternatives were ranked with respect to each customer satisfaction variable in the ACSI model. According to Saaty (2001), the ANP is comprised of four main steps: Step 1: Model construction and problem structuring: Our objective was to determine the influence of each element of the model of customer satisfaction of an e- recruitment website, and determine whether Irantalent.com or Agahjobs.com is the best e-recruitment site. The clusters of the constructed model are based on ACSI clusters and an alternative cluster and the arrows indicate the direction of impact. The model is shown in Figure 1. Step 2: Formulating the interdependencies and performing paired comparisons between the clusters/elements: This step leads to the creation of a network containing all the decision elements and their inner (within the same cluster) and outer relationships (among elements of different clusters) (Bayazit, 2006).  Cluster comparisons. The clusters themselves must be compared to establish their relative importance, and then the priorities used to weight the IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 232 Vol. 5 Issue 2 2013 ISSN 1936-6744 supermatrix to make it column stochastic (Saaty & Sodenkamp, 2008). Cluster matrix is shown in Table 1. Table 1 Cluster weights matrix  Comparisons of elements and comparisons for alternatives: According to Saaty and Sodenkamp (2008), the unweighted supermatrix is constructed from the priorities derived from the different pairwise comparisons. The nodes, grouped by the clusters that they belong to, are the labels of the rows and columns of the supermatrix. The column of priorities for a node at the top of the supermatrix includes the priorities of the nodes on the left side of the matrix that have been pairwise compared as to their influence with respect to customer satisfaction on that node. The sum of these priorities is equal to one. A reciprocal value is assigned to the inverse comparison (that is, ai j = 1/ai j) where ai j denotes the importance of the ith element compared to the jth element. The unweighted supermatrix of the network of Figure 1 is shown in Table 2. Recommend Likelihood to Return Satisfaction Site performance Search Navigation Look & Feel Functionality Content Alternatives 0.176 0.196 0.044 1 1 1 1 1 1 0 Alternatives 0 0 0.241 0 0 0 0 0 0 0.194 Content 0 0 0.114 0 0 0 0 0 0 0.105 Functionality 0 0 0.037 0 0 0 0 0 0 0.036 Look & Feel 0 0 0.104 0 0 0 0 0 0 0.073 Navigation 0 0 0.123 0 0 0 0 0 0 0.090 Search 0 0 0.337 0 0 0 0 0 0 0.207 Site performance 0.824 0.804 0 0 0 0 0 0 0 0.243 Satisfaction 0 0 0 0 0 0 0 0 0 0.025 Likelihood to Return 0 0 0 0 0 0 0 0 0 0.027 Recommend IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 233 Vol. 5 Issue 2 2013 ISSN 1936-6744 Table 2 Unweighted supermatrix Alternatives Content Functionality Look & Feel Navigation Search Site performance Satisfaction Likelihood to return Recommend A g a h jo b s Ir a n ta le n t In fo rm a ti o n A c c u ra c y In fo rm a ti o n Q u a li ty U se fu ln e ss o f th e in fo rm a ti o n C o n v e n ie n c e o f th e i n fo rm a ti o n E a se o f re a d in g th e s it e C la ri ty o f si te o rg a n iz a ti o n A b il it y t o f in d w a n te d in fo rm a ti o n E a se o f n a v ig a ti o n U se fu ln e ss o f se a rc h r e su lt s P ro v id e s c o m p re h e n si v e se a rc h r e su lt s O rg a n iz a ti o n o f se a rc h r e su lt s S e a rc h f e a tu re s h e lp y o u n a rr o w th e r e su lt s R e li a b il it y o f si te p e rf o rm a n c e o v e ra ll sa ti sf a c ti o n w it h th is s it e m e e t y o u r e x p e c ta ti o n s c o m p a re t o y o u r id e a o f a n i d e a l W e b si te re tu rn t o t h e s it e R e c o m m e n d t h e si te t o s o m e o n e e ls e Alternatives Agahjobs 0 0 0.294 0.240 0.374 0.293 0.449 0.271 0.267 0.289 0.374 0.290 0.237 0.397 0.280 0.311 0.404 0.372 0.387 0.300 Irantalent 0 0 0.706 0.760 0.626 0.707 0.551 0.729 0.733 0.711 0.626 0.710 0.763 0.603 0.720 0.689 0.596 0.628 0.613 0.700 Content Information Accuracy 0.337 0.407 0 0 0 0 0 0 0 0 0 0 0 0 0 0.387 0.257 0.321 0 0 Information Quality 0.663 0.593 0 0 0 0 0 0 0 0 0 0 0 0 0 0.613 0.743 0.679 0 0 Functionality Usefulness of the information 0.453 0.542 0 0 0 0 0 0 0 0 0 0 0 0 0 0.684 0.556 0.562 0 0 Convenience of the information 0.547 0.458 0 0 0 0 0 0 0 0 0 0 0 0 0 0.316 0.444 0.438 0 0 Look & Feel Ease of reading the site 0.528 0.332 0 0 0 0 0 0 0 0 0 0 0 0 0 0.354 0.378 0.399 0 0 Clarity of site organization 0.472 0.668 0 0 0 0 0 0 0 0 0 0 0 0 0 0.646 0.622 0.601 0 0 Navigation Ability to find wanted information 0.431 0.598 0 0 0 0 0 0 0 0 0 0 0 0 0 0.639 0.542 0.612 0 0 Ease of navigation 0.569 0.402 0 0 0 0 0 0 0 0 0 0 0 0 0 0.361 0.458 0.388 0 0 Search Usefulness of search results 0.190 0.156 0 0 0 0 0 0 0 0 0 0 0 0 0 0.440 0.478 0.260 0 0 Provides comprehensive search results 0.263 0.410 0 0 0 0 0 0 0 0 0 0 0 0 0 0.162 0.273 0.242 0 0 Organization of search results 0.222 0.205 0 0 0 0 0 0 0 0 0 0 0 0 0 0.171 0.110 0.262 0 0 Search features help you narrow the results 0.325 0.229 0 0 0 0 0 0 0 0 0 0 0 0 0 0.227 0.139 0.236 0 0 Site performance Reliability of site performance 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 Satisfaction overall satisfaction with this site 0.366 0.155 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.224 0.379 meet your expectations 0.367 0.391 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.227 0.478 compare to your idea of an ideal Website 0.267 0.454 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.549 0.143 Likelihood to Return return to the site 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Recommend Recommend the site to someone else 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 234 Vol. 5 Issue 2 2013 ISSN 1936-6744 Step 3: Constructing the supermatrix: The weights derived from step 2 are used to populate the columns of the supermatrix. Either each column of a supermatrix is a normalized eigenvector with possibly some zero entries or all of its block entries are zero. The unweighted supermatrix, which is illustrated in Table 2, is then multiplied by the priority weights from the clusters (which were determined in the first bullet point in step 2), yielding the weighted supermatrix (second panel of Table 3). This is done because the resulting matrix must be column stochastic, that is, its columns must add to one, for a limit that is not zero in order to exist (Saaty & Vargas, 2006). IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 235 Vol. 5 Issue 2 2013 ISSN 1936-6744 Table 3 Weighted supermatrix Alternatives Content Functionality Look & Feel Navigation Search Site performance Satisfaction Likelihood to return Recommend A g a h jo b s Ir a n ta le n t In fo rm a ti o n A c c u ra c y In fo rm a ti o n Q u a li ty U se fu ln e ss o f th e in fo rm a ti o n C o n v e n ie n c e o f th e i n fo rm a ti o n E a se o f re a d in g th e s it e C la ri ty o f si te o rg a n iz a ti o n A b il it y t o f in d w a n te d in fo rm a ti o n E a se o f n a v ig a ti o n U se fu ln e ss o f se a rc h r e su lt s P ro v id e s c o m p re h e n si v e se a rc h r e su lt s O rg a n iz a ti o n o f se a rc h r e su lt s S e a rc h f e a tu re s h e lp y o u n a rr o w th e r e su lt s R e li a b il it y o f si te p e rf o rm a n c e o v e ra ll sa ti sf a c ti o n w it h th is s it e m e e t y o u r e x p e c ta ti o n s c o m p a re t o y o u r id e a o f a n i d e a l W e b si te re tu rn t o t h e s it e R e c o m m e n d t h e si te t o s o m e o n e e ls e Alternatives Agahjobs 0 0 0.294 0.240 0.374 0.293 0.449 0.271 0.267 0.289 0.374 0.290 0.237 0.397 0.280 0.014 0.018 0.016 0.076 0.053 Irantalent 0 0 0.706 0.760 0.626 0.707 0.551 0.729 0.733 0.711 0.626 0.710 0.763 0.603 0.720 0.030 0.026 0.028 0.120 0.123 Content Information Accuracy 0.065 0.079 0 0 0 0 0 0 0 0 0 0 0 0 0 0.093 0.062 0.077 0 0 Information Quality 0.129 0.115 0 0 0 0 0 0 0 0 0 0 0 0 0 0.148 0.179 0.164 0 0 Functionality Usefulness of the information 0.048 0.057 0 0 0 0 0 0 0 0 0 0 0 0 0 0.078 0.063 0.064 0 0 Convenience of the information 0.057 0.048 0 0 0 0 0 0 0 0 0 0 0 0 0 0.036 0.051 0.050 0 0 Look & Feel Ease of reading the site 0.019 0.012 0 0 0 0 0 0 0 0 0 0 0 0 0 0.013 0.014 0.015 0 0 Clarity of site organization 0.017 0.024 0 0 0 0 0 0 0 0 0 0 0 0 0 0.024 0.023 0.022 0 0 Navigation Ability to find wanted information 0.031 0.044 0 0 0 0 0 0 0 0 0 0 0 0 0 0.066 0.056 0.064 0 0 Ease of navigation 0.042 0.029 0 0 0 0 0 0 0 0 0 0 0 0 0 0.038 0.048 0.040 0 0 Search Usefulness of search results 0.017 0.014 0 0 0 0 0 0 0 0 0 0 0 0 0 0.054 0.059 0.032 0 0 Provides comprehensive search results 0.024 0.037 0 0 0 0 0 0 0 0 0 0 0 0 0 0.020 0.034 0.030 0 0 Organization of search results 0.020 0.018 0 0 0 0 0 0 0 0 0 0 0 0 0 0.021 0.013 0.032 0 0 Search features help you narrow the results 0.029 0.021 0 0 0 0 0 0 0 0 0 0 0 0 0 0.028 0.017 0.029 0 0 Site performance Reliability of site performance 0.207 0.207 0 0 0 0 0 0 0 0 0 0 0 0 0 0.337 0.337 0.337 0 0 Satisfaction overall satisfaction with this site 0.089 0.038 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.180 0.312 meet your expectations 0.089 0.095 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.183 0.394 compare to your idea of an ideal Website 0.065 0.110 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.441 0.118 Likelihood to Return return to the site 0.025 0.025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Recommend Recommend the site to someone else 0.027 0.027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 236 Vol. 5 Issue 2 2013 ISSN 1936-6744 Finally, by increasing a matrix to powers the long-term relative influences of the elements on one another can be obtained. To achieve convergence of the importance weights, the weighted supermatrix is increased to the power of 2k+1, where k is an arbitrarily large number. This new matrix is termed the limit supermatrix. The limit supermatrix possesses the same form as the weighted supermatrix, but all the columns of the limit supermatrix are identical. Normalizing each block of the supermatrix can obtain the final priorities of all the elements (Table 4). Now that the synthesized results from the limit supermatrix have been determined, the limit supermatrix shows the relative e-customer satisfaction of the alternatives Irantalent and Agahjobs as 0.306 and 0.126, respectively. When normalized they are 0.708 and 0.292, respectively (Table 5). IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 237 Vol. 5 Issue 2 2013 ISSN 1936-6744 Table 4 Limit supermatrix Alternatives Content Functionality Look & Feel Navigation Search Site performance Satisfaction Likelihood to return Recommend A g a h jo b s Ir a n ta le n t In fo rm a ti o n A c c u ra c y In fo rm a ti o n Q u a li ty U se fu ln e ss o f th e in fo rm a ti o n C o n v e n ie n c e o f th e i n fo rm a ti o n E a se o f re a d in g th e s it e C la ri ty o f si te o rg a n iz a ti o n A b il it y t o f in d w a n te d in fo rm a ti o n E a se o f n a v ig a ti o n U se fu ln e ss o f se a rc h r e su lt s P ro v id e s c o m p re h e n si v e se a rc h r e su lt s O rg a n iz a ti o n o f se a rc h r e su lt s S e a rc h f e a tu re s h e lp y o u n a rr o w th e r e su lt s R e li a b il it y o f si te p e rf o rm a n c e o v e ra ll sa ti sf a c ti o n w it h th is s it e m e e t y o u r e x p e c ta ti o n s c o m p a re t o y o u r id e a o f a n i d e a l W e b si te re tu rn t o t h e s it e R e c o m m e n d t h e si te t o s o m e o n e e ls e Alternatives Agahjobs 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 0.126 Irantalent 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 0.306 Content Information Accuracy 0.041 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 0.042 Information Quality 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 0.072 Functionality Usefulness of the information 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 0.032 Convenience of the information 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 Look & Feel Ease of reading the site 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 0.008 Clarity of site organization 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 Navigation Ability to find wanted information 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 0.025 Ease of navigation 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 0.019 Search Usefulness of search results 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 Providing comprehensive search results 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 0.018 Organization of search results 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 Search features help you narrow the results 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 0.013 Site performance Reliability of site performance 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 0.131 Satisfaction overall satisfaction with the site 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 0.028 meeting your expectations 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 comparing the site with your ideal Website 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 0.048 Likelihood to Return return to the site 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 0.011 Recommend Recommend the site to someone else 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 0.012 IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 238 Vol. 5 Issue 2 2013 ISSN 1936-6744 Table 5 The synthesized results of the alternatives Alternatives Raw Normals Ideals Agahjobs 0.126 0.292 0.413 Irantalent 0.306 0.708 1 4.5. Sensitivity analysis A sensitivity analysis is conducted in order to determine the stability of the preference ranking among the alternative websites by changing the priority weights of the criteria. If the ranking does not change, the results are said to be robust. In this study, sensitivity is performed by varying the priority of the reliability of site performance criterion, by moving the vertical line and determining the corresponding alternatives priorities. Figures 2 and 3 show a graphical representation of sensitivity for the reliability of site performance criterion. In Figure 2, priority of the reliability of site performance is 0.72, and the priorities of the alternatives are the original overall synthesized priorities as 0.292 and 0.708 respectively for Agahjobs and Irantalent which are shown in the Table 5. In Figure 3, priority of the reliability of site performance is shifted to 0.01, and the ranking between the alternatives has changed. When priority of the reliability of site performance is less than about 0.03, Agahjobs.com will receive a higher rank than Irantalent.com. When a sensitivity analysis was performed on the other criteria the alternatives’ ranking were not affected. Therefore, this analysis shows that Irantalent.com is the better e-recruitment website based on customer satisfaction than Agahjobs.com. IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 239 Vol. 5 Issue 2 2013 ISSN 1936-6744 Figure 2. Sensitivity Graphs for the reliability of site performance when its priority is 0.72 IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 240 Vol. 5 Issue 2 2013 ISSN 1936-6744 Figure 3. Sensitivity Graphs for the reliability of site performance when its priority is 0.01 4.6. Reliability and validity analysis The reliability analysis of a questionnaire determines its ability to yield the same results on different occasions, and validity refers to the measurement of what the questionnaire is supposed to measure (Albadvi, Keramati, & Razmi, 2007). Several precautions were taken in order to assess the reliability of the results. After establishing a questionnaire according to Saaty (2004), the questionnaire was reassessed by independent individuals. Firstly, we invited a recruitment industry expert to format questions to ensure that they were properly phrased. In addition, ANP experts reviewed the questionnaire to make certain that pairwise comparisons were properly established. After gathering the completed questionnaires, the inconsistency ratio for each judgment was checked in order to find any possible errors and eliminate the inconsistent judgments for each group of pairwise comparison for each respondent. The inconsistency measure is useful in identifying possible errors in judgments as well as actual inconsistencies in the judgments themselves (Saaty, 2003). For example, if you were to say that A is more important than B, and B is more important than C, and then say that C is more important than A, you are not being consistent. In general, the inconsistency ratio should be less than 0.1 to be considered reasonably consistent (Saaty, 2003). 5. Results IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 241 Vol. 5 Issue 2 2013 ISSN 1936-6744 According to the limit supermatrix table, the importance and influence of each customer satisfaction factor is achieved with respect to their influence on e-recruitment customer satisfaction. With respect to content category, information quality (0.072) in comparison to information accuracy (0.042) has considerable influence on e-recruitment customer satisfaction. Usefulness of information (0.032) has more influence on an e-recruitment website than the relative convenience of the information factor (0.028) in the functionality factors. Clarity of site organization (0.012) was found to be more beneficial to the customer satisfaction factor in the look and feel category in comparison to ease of reading of the site (0.08). In the navigation category, the ability to find wanted information (0.025) was ranked higher than ease of navigation (0.019) by online users. In the search category, in comparison to other search measures, the factor providing comprehensive search results (0.018) received the most attention from online users, while the factor relative organization of search results (0.011) received the lowest ranking in the limit supermatrix. The highest preference factor in the satisfaction category was comparing the site with your ideal website (0.048), and the factor meeting your expectations with little difference (0.047) in terms of users’ attention. However, the overall satisfaction with the site (0.028) received less attention. Synthesized results (Table 5) support that the Irantalent website was chosen as the priority e-recruitment website with respect to customer satisfaction, and with the highest weighted score of 0.708 when compared to Agahjobs website’s score of 0.292. 6. Discussion and conclusions This paper has studied the framework for investigating factors that influence customer satisfaction index in ecommerce. For this purpose, the study focused on the relative clusters priorities in the customer satisfaction model and also the priorities of factors in each cluster based on users' opinion and compared and ranked the websites satisfaction based on these priorities. This paper puts forward an approach to find the priorities, and compare the customer satisfaction of e-recruitment websites (agahjobs.com and Irantalent.com) based on the priorities. In addition, customer satisfaction is the leading factor that determines online customers’ loyalty. Thus, understanding the factors that influence online customer satisfaction is of great importance to e- businesses (Hsu, 2008). Since the findings suggest that customer satisfaction is crucial for a better financial performance, providing exceptional customer service by exceeding customers’ expectations is likely to offer opportunities for growth (Chi & Gursoy, 2009). The ANP model helped discover the influence of different factors on e-recruitment customer satisfaction. The mathematical procedure of the ANP methodology helps the analyst or the decision maker to include all the factors and elements that users think might influence the alternatives. Therefore, users can gradually choose the less important factors, but focus first on the most important ones. The model provides an e-business analyst with a very flexible analysis tool that is easily understood. According to the results, it is recommended that both the websites pay more attention to site performance, a factor that has the most influence on e-recruitment customer satisfaction. Some service features increase the overall satisfaction linearly when performance improves (Fuller & Matzler, 2008). Such features have IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 242 Vol. 5 Issue 2 2013 ISSN 1936-6744 been labeled as performance factors. Hence, it is important to identify which product or service attributes lead to an increase in satisfaction or customer delight when their performances are improved, and which attributes can lead to dissatisfaction through their absence as the customer expects their presence (Fuller & Matzler, 2008). Site performance is all about what the user actually does in interacting with the site. It includes measuring the degree to which users can successfully accomplish a task or a set of tasks. Many measures related to the performance of these tasks are also important, including the time it takes to perform each of them, the amount of effort used to perform each (such as number of mouse clicks or amount of cognitive effort), the number of errors committed, and the amount of time it takes to become proficient in performing these tasks (learnability) (Hsu, 2012). Results suggest that Agahjobs.com improve all ACSI factors, especially site performance. It scored much weaker than Irantalent.com in that aspect, and this factor has a high influence on e-recruitment customer satisfaction. In addition, it is important that Agahjobs.com focus on the speed, consistency, and reliability of loading pages on the website to increase users' satisfaction. Furthermore, Irantalent.com should pay more attention to the factor meeting users' expectations in the satisfaction cluster as there is the least difference between their site and Agahjobs.com in this category. The proposed model provides a method with an objective, and it is effective in selecting an optimal independent e-recruitment website based on the ACSI, making it highly applicable for academic and commercial purposes. An important limitation of this study lies in the study’s sample size. As mentioned in Section 4.1, we needed a sample size of 383 for 95% confidence. However, 46 of eligible users returned usable surveys, which were answered completely and consistently. This number fell well below what was needed. Although this analysis still provides quality insight, we cannot claim these 46 respondents actually represent the opinions of the 100,000 graduates. In the study, questionnaires which were found inconsistent in judgments were eliminated. In the future research, the respondents with the most inconsistent answers will be contacted and asked for new judgments. This approach will lead to an improvement in the inconsistency in judgments. IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 243 Vol. 5 Issue 2 2013 ISSN 1936-6744 REFERENCES Albadvi, A., Keramati, A., & Razmi, J. (2007). Assessing the impact of information technology on firm performance considering the role of intervening variables: Organizational infrastructures and business processes reengineering. International Journal of Production Research, 45(12), 2697–2734. Bayazit, O. (2006). Use of analytic network process in vendor selection decisions. Benchmarking: an International Journal, 13(5), 566-579. Bayraktar, E., Tatoglu, E., Turkyilmaz, A., Delen, D., & Zaim, S. (2012). Measuring the efficiency of customer satisfaction and loyalty for mobile phone brands with DEA. Expert Systems with Applications, 39(1), 99–106. Chang, H. H., & Wang, H. W. (2011). The moderating effect of customer perceived value on online shopping behavior. Online Information Review, 35(3), 333 – 359. Chi, C. G., & Gursoy, D. (2009). Employee satisfaction, customer satisfaction, and financial performance: An empirical examination. International Journal of Hospitality Management, 28, 245–253. Fornell, C. (1992). A national customer satisfaction barometer: The Swedish experience. Journal of Marketing, 56(1), 6–21. Fuller, J., & Matzler, K. (2008). Customer delight and market segmentation: An application of the three- factor theory of customer satisfaction on life style groups. Tourism Management, 29, 116-126. Graja, H., & McManis, J., (2001). Modelling user interaction with e-commerce servers. Springer Berlin Heidelberg, 2093, 803-810. Hsu, P. F. (2012). Selection model based on ANP and GRA for independent media agencies. Quality & Quantity, 46(1), 1-17. Hsu, S. H. (2008). Developing an index for online customer satisfaction: Adaptation of American customer satisfaction index. Expert Systems with Applications, 34(4), 3033–3042. Jharkharia, S., & Shankar, R. (2007). Selection of logistics service provider: an analytic network process (ANP) approach. Omega, 35(3), 274-289. Keramati, A., & Salehi, M. (2013). Website success comparison in the context of e-recruitment: An analytic network process (ANP) approach. Applied Soft Computing, 13(1), 173–180. Kujala, J., & Ahola, T. (2005). The value of customer satisfaction surveys for project-based organizations: symbolic, technical, or none. International Journal of Project Management, 23(5), 404– 409. http://link.springer.com/search?facet-author=%22Hubert+Graja%22 http://link.springer.com/search?facet-author=%22Jennifer+McManis%22 IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 244 Vol. 5 Issue 2 2013 ISSN 1936-6744 Kumar, V., Sharma, A., Shah, R., & Rajan, B. (2013). Establishing profitable customer loyalty for multinational companies in the emerging economies: A conceptual framework. Journal of International Marketing, 21(1), 57-80. Liu, X., Zeng, X., Xu, Y., & Koehl, L. (2008). A fuzzy model of customer satisfaction index in e- commerce. Mathematics and Computers in Simulation, 77 (5-6), 512–521. Luo, X., Wieseke, J., & Homburg, C. (2012). Incentivizing CEOs to build customer- and employee-firm relations for higher customer satisfaction and firm value. Journal of the Academy of Marketing Science, 40, 745–758. Saaty, R. W. (2003). Decision making in complex environment: The analytic hierarchy process (AHP) for decision making and the analytic network process (ANP) for decision making with dependence and feedback. Pittsburgh: Super Decisions. Saaty, T. L. (1980). The analytic hierarchy process. New York: McGraw-Hill International. Saaty, T. L. (1990). How to make a decision: the analytic hierarchy process. European Journal of Operational Research, 48(1), 9-26. Saaty, T. L. (2004). Super decisions software. Pittsburgh: RWS Publications. Saaty, T. L. (2005). Theory and applications of the analytic network process: Decision making with benefits, opportunities, costs and risks. Pittsburgh: RWS Publications. Saaty, T. L., & Vargas, L. G. (2006). Decision making with the analytic network process: Economic, political, social and technological applications with benefits, opportunities, costs and risks. New York: Springer Science+ Business Media, LLC. Saaty, T. L., & Sodenkamp, M. (2008). Making decisions in hierarchic and network systems. Int. J. Applied Decision Sciences, 1(1), 24-79. Sadeghi, M., Rashidzadeh, M. A., & Soukhakian, M. A. (2012). Using analytic network process in a group decision-making for supplier selection. Informatica, 23(4), 621–643. Schaupp, L. C., & Bélanger, F. (2005). A conjoint analysis of online consumer satisfaction. Journal of Electronic Commerce Research, 6(2), 95-111. Singh, M. (2002). E-services and their role in B2C e-commerce. Journal of Management Service Quality, 12(6), 434-446. http://books.google.com.au/books?hl=en&lr=&id=TegYgX6JRo8C&oi=fnd&pg=PA1&dq=Decision+Making+with+the+Analytic+Network+Process:+Economic,+Political&ots=yDLBKwtgdO&sig=oA2LngaPOvj7gJIWukHsoC2yLaA http://books.google.com.au/books?hl=en&lr=&id=TegYgX6JRo8C&oi=fnd&pg=PA1&dq=Decision+Making+with+the+Analytic+Network+Process:+Economic,+Political&ots=yDLBKwtgdO&sig=oA2LngaPOvj7gJIWukHsoC2yLaA http://books.google.com.au/books?hl=en&lr=&id=TegYgX6JRo8C&oi=fnd&pg=PA1&dq=Decision+Making+with+the+Analytic+Network+Process:+Economic,+Political&ots=yDLBKwtgdO&sig=oA2LngaPOvj7gJIWukHsoC2yLaA https://www.google.com.au/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&sqi=2&ved=0CCsQFjAA&url=http%3A%2F%2Fwww.informatica.si%2F&ei=iQ2cUtGZOMW6lAX_-YHgDQ&usg=AFQjCNG0Hx0qS-GQOiAm2BFiO1WLD9QQtg&sig2=SqeOfbvdI7fO9EKPVdHgpg IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 245 Vol. 5 Issue 2 2013 ISSN 1936-6744 Taslicali, A. K., & Ercan, S. (2006). The analytic hierarchy & the analytic network Processes in multicriteria decision making: A comparative study. Journal of Aeronautics and Space Technologies, 2(4), 55-65. Tullis, T., & Bill, A. (2008). Measuring the user experience: Collecting, analyzing, and presenting usability metrics. Burlington: Morgan Kaufmann Publishers. Wood, F. B., Siegel, E. R., Feldman, S., Love, C. B., Rodrigues, D., Malamud, M., Lagana, M., & Crafts, J. (2008). Web evaluation at the US National Institutes of Health: Use of the American customer satisfaction index online customer survey. Journal of Medical Internet Research, 10, e4. Yang, J., & Ding, Y. (2009). B2B e-commerce website customer satisfaction: a formula and scale. International Asia Symposium on Intelligent Interaction and Affective Computing. IEEE Xplore, Wuhan, 191-197. IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 246 Vol. 5 Issue 2 2013 ISSN 1936-6744 APPENDIX I This appendix includes the survey questionnaire which was sent to the recruitment websites users by email. At the beginning of the survey, a question asked the users whether they have used both websites. Then, only those users who gave a positive answer to this question were allowed to fill in the rest of the survey. A glossary of literature terms is also included in this questionnaire for users’ information. Also, It is mentioned that scale of 1–9 is used for two components, with a score of 1 representing indifference between the two components and 9 being overwhelming dominance of the component under consideration. Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website do you prefer based on information quality? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website do you prefer based on information accuracy? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website do you prefer based on convenience of the information? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website do you prefer based on usefulness of the information? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website has clearer site organization? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website is easier for you to read? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website is easier for you to find information that you are looking for? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website is easier to navigate? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website has higher rate at the reliability of site performance in your point of view? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website does provide a higher level of your overall satisfaction? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website does meet your expectations more? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website is closer to your idea of an ideal website? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website do you visit more regularly? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website do you prefer to recommend to someone else? IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 247 Vol. 5 Issue 2 2013 ISSN 1936-6744 Please compare characteristics of each pair for the websites. Agahjobs Information Accuracy 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Information Quality Usefulness of the information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Convenience of the information Clarity of site organization 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of reading the site Ability to find wanted information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of navigation Meet your expectations 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Overall satisfaction with this site Meet your expectations 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Compare to your idea of an ideal Website Overall satisfaction with this site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Compare to your idea of an ideal Website Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search features help you narrow the results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Comprehensive search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Irantalent Information Accuracy 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Information Quality Usefulness of the information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Convenience of the information Clarity of site organization 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of reading the site Ability to find wanted information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of navigation Meet your expectations 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Overall satisfaction with this site Meet your expectations 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 compare to your idea of an ideal Website Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website has more organized search results? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website has more search features that help you narrow the results? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website does provide more comprehensive search results? Irantalent 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Agahjobs Which website has more useful search results? IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 248 Vol. 5 Issue 2 2013 ISSN 1936-6744 Overall satisfaction with this site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Compare to your idea of an ideal Website Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search features help you narrow the results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Comprehensive search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results In your opinion, which of the following characteristics is more encouraging to recommend the websites to someone else? Recommend the site to someone else Meet your expectations 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Overall satisfaction with this site Meet your expectations 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Compare to your idea of an ideal Website Overall satisfaction with this site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Compare to your idea of an ideal Website In your opinion, which of the following characteristics is more encouraging to return to the websites? Return to the site Meet your expectations 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Overall satisfaction with this site Meet your expectations 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Compare to your idea of an ideal Website Overall satisfaction with this site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Compare to your idea of an ideal Website Which of the following factors help more to meet your expectations? Meet your expectations Information Accuracy 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Information Quality Usefulness of the information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Convenience of the information Clarity of site organization 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of reading the site Ability to find wanted information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of navigation Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search features help you narrow the results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Comprehensive search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 249 Vol. 5 Issue 2 2013 ISSN 1936-6744 Which of the following factors help more to reach your overall satisfaction? Overall satisfaction with a website Information Accuracy 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Information Quality Usefulness of the information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Convenience of the information Clarity of site organization 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of reading the site Ability to find wanted information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of navigation Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search features help you narrow the results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Comprehensive search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Which factor has a better performance compared to your ideal website? Compare to your idea of an ideal Website Information Accuracy 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Information Quality Usefulness of the information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Convenience of the information Clarity of site organization 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of reading the site Ability to find wanted information 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Ease of navigation Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search features help you narrow the results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Organized search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Comprehensive search results Search features help you narrow the results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results Comprehensive search results 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Usefulness of search results In your opinion, which factor is more important to choose a recruitment website? Alternatives Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Functionality Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 loyalty to return Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Look & Feel Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Navigation Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Others recommendations on the site IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 250 Vol. 5 Issue 2 2013 ISSN 1936-6744 Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Satisfaction Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 loyalty to return Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Look & Feel Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Navigation Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Others recommendations on the site Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Satisfaction Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance loyalty to return 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Look & Feel loyalty to return 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Navigation loyalty to return 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 others recommendations on the site loyalty to return 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Satisfaction loyalty to return 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search loyalty to return 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Look & Feel 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Navigation Look & Feel 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 others recommendations on the site Look & Feel 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Satisfaction Look & Feel 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Look & Feel 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Navigation 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 others recommendations on the site Navigation 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Satisfaction Navigation 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Navigation 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Others recommendations on the site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Satisfaction Others recommendations on the site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Others recommendations on the site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Satisfaction 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Satisfaction 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Search 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance What make you more likely to return to your preferred website? Likelihood to Return Compared to the other site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Satisfaction IJAHP: Bijan, Keramati, Salehi/Comparison of User Satisfaction of Ecommerce Websites by the Analytic Network Process International Journal of the Analytic Hierarchy Process 251 Vol. 5 Issue 2 2013 ISSN 1936-6744 What make you more likely to recommend your preferred website? Likelihood to Recommend Compared to the other site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Satisfaction What would make you more satisfied with your preferred website? Satisfaction Compared to the other site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Content Compared to the other site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Functionality Compared to the other site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Look & Feel Compared to the other site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Navigation Compared to the other site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Compared to the other site 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Functionality Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Look & Feel Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Navigation Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Content 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Look & Feel Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Navigation Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Functionality 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Look & Feel 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Navigation Look & Feel 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Look & Feel 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Navigation 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Search Navigation 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance Search 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Site performance