Engineering, Technology & Applied Science Research Vol. 8, No. 4, 2018, 3194-3198 3194 www.etasr.com Daghouri et al.: Enhanced Model For Evaluating Information System Success: Determining … Enhanced Model For Evaluating Information System Success: Determining Critical Criteria Ansar Daghouri ENSET of Mohammedia University Hassan II of Casablanca Casablanca, Morocco dagh.ansar@gmail.com Khalifa Mansouri ENSET of Mohammedia University Hassan II of Casablanca Casablanca, Morocco khmansouri@hotmail.com Mohammed Qbadou ENSET of Mohammedia University Hassan II of Casablanca Casablanca, Morocco qbmedn7@gmail.com Abstract—The purpose of this study is to enhance the DeLone and McLean information system success by adding a SERVQUAL instrument to evaluate the service quality dimension. An integrated model for evaluating the information system has been proposed. One of the most popular multi criteria decision making techniques, namely AHP was used to determine the weights of each criterion and sub-criterion of the model in order to identify the most influential criterion on information system’s evaluation. System quality dimension has a strong influence on information system success (0.39) followed by information quality (0.22). The proposed model could be used to enhance information system performance evaluation. Keywords-Delone & McLean information system success model;SERVQUAl; multicriteria decision making; AHP method; criteria I. INTRODUCTION Many definitions are given to define the information system (IS). In [1], author considers IS as an organized set of resources: hardware, software, personnel, data, procedures etc. to acquire, process, store information (in the form of data, texts, images, sounds, etc.) within and between organizations. There are many frameworks and models to evaluate IS success. The most common issue is the choice of right and appropriate factors and assessing them. In the current work, we seek to propose a new model to evaluate information system success based on the model presented in [2] and the SERVQUAL instrument presented in [3]. An integrated model was generated and its criteria and sub-criteria were chosen from previous studies. Analytical hierarch process (AHP) method [4] was applied to select the most influent criteria and sub-criteria on information system success. A sample of one hundred participants expressed their opinions about the importance of each proposed criterion via an online tool. The results show that participants believe that the system quality dimension has the highest impact on IS success (56.70% strongly agreed and 27.40% agreed). II. INFORMATION SYSTEMS’ EVALUATION MODELS This work is based on two main research models: information system success model [2] which is a reference in the field of IS evaluation and SERVQUAL model [3] widely developed and used in the last years. In this section, we will present an overview of the most used models: A. Information System Success Model Authors in [5] conducted a large review of IS success literature to present a new model that consists of six variables and the interdependencies between them. They proposed a multidimensional model that recognizes the success as a process assessed by: system quality, information quality, system usage, use satisfaction, individual impact and organizational impact. In [2] they made three main changes based on the critics of those who have tested the initial model: the addition of a technical variable “service quality” mentioned in [6], the decomposition of the variable “system use” into two variables “intention to use” and “use” referring to the theory of reasoned action [7], the technology acceptance model [8] and the grouping of individual and organizational impacts under a “net benefits” variable. B. The SERVQUAL Model Authors in [9] developed a model designed to measure service quality by capturing respondents’ expectations and perceptions along with the dimensions of service quality. The SERVQUAL model was made of ten dimensions of service quality when created but later on, these dimensions were reduced to five because some dimensions were overlapping (Table. I) TABLE I. SUMMARY OF SERVQUAL ITEMS [3, 10] Dimensions Definition Reliability The ability to perform the promised service dependably and accurately Assurance The knowledge and courtesy of employees and their ability to convey trust and confidence Tangibles The appearance of physical facilities, equipment, personnel and communication materials Empathy The provision of caring, individualized attention to customers Responsiveness The willingness to help customers and to provide prompt service III. MCDM METHODS Multiple criteria decision making (MCDM) or multiple- criteria decision analysis (MCDA) [11] is a sub-discipline of op cri ma eva fin bas [13 the mo MA AH stu and det suc A. dec hie cri the pai pre ele the kn attr den to Engineerin www.etasr eration researc iteria in decis athematical an aluation of a nite number of se of all MCD 3] for a proble e decision tab∗ is the score Many differe ost popular o ACBETH, M HP method w udied compani d sub-criterio termine the m ccess. The AHP Me AHP is a m composes a c erarchies. The iteria that affe e main criteria irwise comp eference scale ement at each e AHP involve Step 1: Con own as the AH Step 2: Co ribute i with a notes the com attribute j. In t Step 3: Cons = /∑ i=1, 2, 3,..., n Step 4: Cons =∑ / W= … Step 5: Calcu ng, Technology r.com ch that explici sion making. nd computation finite number f performance DM methods em with M cr le will be dra e of alternative TABLE II. … … ent types of M of them are MAUT, MAVT was used to d ies. Subsequen n of the prop most influent ethod multiple criter complex MC e essential com ect the overall a and finally th parison matri e [16] to obta level, as show es the followin nstruct the st HP decision onstruct the p attribute j yiel mparative impo this matrix, A = ⋮ struct normaliz n and j=1,2,3, struct the weig /n,i=1,2,3,..., n ulate eigenvec y & Applied Sci D itly evaluates It is concern nal tools to sup r of decision a criteria and su is a decision riteria (C) and awn as showe e n related to c DECISION TABL … … … … … … … … MCDM metho AHP, ANP T [11, 14]. In determine the ntly, the weigh posed model element on i ria decision m DM problem mponents are l goal, sub-cri he alternatives x was deve ain the degree wn in Table II ng steps [17-19 tructural hiera pairwise com ld a square m ortance of attr =1 when i=j a⋯⋱ ⋮⋯ zed decision m …,n hted normaliz n ctor and row m ience Research aghouri et al.: multiple confl ned with desi upport the subj alternatives un ub-criteria [12 table. Accord N alternative d in Table II criteria m. LE … … ods are referre P, ELECTRE, n this research IS performan hts of each cri were calculat information s making [4, 1 m into a syste the main goa iteria that infl s to the proble eloped using e of importan II. The procedu 9]: archy, this st mparison matr matrix ∗ whe ribute i with re and =1/ matrix (1) ed decision m (2) (3) matrix h V Enhanced Mod licting igning ective nder a ]. The ding to es (A), while ed, the , GP, h, the nce of iterion ted to system 15]. It em of al, the luence em. A g 1-9 nce of dure of tep is rix of ere espect matrix rati whe gen Sc 2 6 A. SER mo wer sys ben serv tang B. mea stud Vol. 8, No. 4, 20 del For Evalua E= rootva Row matrix=∑ Step 6: Calcuλ =Rowma Step 7: Calc io CI= (λ -n) CR= CI/RI ere n and RI nerated consist TABL cale Def 1 Equal i 3 Moderat of one o 5 Essenti imp 7 Ver imp 9 Extreme 2, 4, 6, 8 Interme between jud Proposed Suc In accordanc RVQUAL [9] del for assess re proposed fo tem quality, s nefits. The mo vice quality u gibles, empath Measurement We have ch asure each d dies as follows 018, 3194-3198 ting Informatio lue/∑ root∑ ∗ ulate the maxim atrix/E culate the con ) / (n-1) I denote the tency index re LE III. PAIRW finition importance te importance over another ial or strong portance ry strong portance e importance ediate values two adjacent dgments IV. WORK ccess Model ce with DeL ] models, thi sing IS succes for measuring service quality odel uses the using five dim hy and respons Fig. 1. Propo t of Variables hosen five s dimension (cr s: 8 on System Succ tvalue mum eigenvalu nsistency inde order of mat spectively. WISE COMPARISON Exp Two elements co pr Experience and favor one Experience and favor one An element is str dominance is dem The evidence fa over another is possible ord Comprise is ne jud METHODOLOG Lone & McL is work propo s. Consequent IS success: in y, use, user s SERVQUAL mensions: reli siveness. (Figu osed success mod specific items riteria) adapte 3195 cess: Determini (4) (5) ue, λ (6) ex and consist (7) (8) trix and rand N SCALE planation ntribute equally to roperty d judgment slight over the other d judgment strong e over another rongly favored an monstrated in prac favoring one eleme s one of the highe der of affirmation eeded between tw dgments GY Lean [2, 5] oses an integ tly, six dimen nformation qu atisfaction an model to me iability, assur ure 1) del s (sub-criteria ed from prev ing … tency domly o the tly gly nd its ctice ent est wo and grated nsions uality, nd net easure rance, a) to vious Engineering, Technology & Applied Science Research Vol. 8, No. 4, 2018, 3194-3198 3196 www.etasr.com Daghouri et al.: Enhanced Model For Evaluating Information System Success: Determining … 1) System Quality The success criterion system quality constitutes the desirable characteristics of an IS. The measures (sub-criteria) of this criterion focus on studied system’s performance. The selected system quality elements are: Access [20], ease of learning [21], flexibility [22], reliability [23] and response time [22, 23]. 2) Information Quality The success criterion information quality is an important factor relative to the characteristics of the information system’s output. The selected information quality sub-criteria are: accuracy [24], completeness [22, 24], understandability [20], security [25], and usefulness [26]. 3) Service Quality The success criterion system quality represents the quality of the support that the IS offer to the users. The selected service quality elements are: reliability, assurance, tangibility, empathy and responsiveness [3]. 4) Intention to Use/Use The success criterion Intention to use/Use represents the manner of using an IS. The selected elements are: daily use [27], frequency of use [27], nature of use [2], number of site visits [2] and number of transactions [2]. 5) User Satisfaction The success criterion user satisfaction constitutes the user’s level of satisfaction when using an IS. The selected user satisfaction elements are: adequacy [27], effectiveness [27], efficiency [27], enjoyment [20] and overall satisfaction [27]. 6) Nets Benefits This success criterion constitutes the extent to which IS is contributing to the success of different stakeholders, it subsumes the former separate individual and organizational impacts. The selected net benefits elements are: productivity [8], usefulness [8], competitive advantage [27], cost reduction [27] and overall success [27]. C. Population and Sample A structured questionnaire was used for data collection, based on the proposed success model criteria and sub-criteria. A sample of 150 participants belonging to different sectors was chosen randomly based on their use of IS. Questionnaire development and returns were done by an online Google tool. The participants were asked to indicate their agreement with the influence of each criteria of IS success model. A five point Likert type scale was used with anchors from “Strongly agree” to “Strongly disagree”. A total of 100 questionnaires were returned for a response rate of 67%. Table IV shows the demographic characteristics of the received sample according to gender, age and experience. V. IMPLEMENTATION AND RESULTS A. The Purpose of this Study The main purpose of this study is to present an IS success model guided by the updated DeLone & McLean [2, 5] and SERVQUAL [9] models. The proposed model composes six interrelated constructs of IS success measures: the quality dimension (system, information and service) which affects intention to use/use and user satisfaction. The net benefits dimension is a result of use and user satisfaction and could affect them. For our case, the quality service is measured by the SERVQUAL instrument. This study also aims to determine the critical and most dimension affecting IS success criteria. AHP, one of the most used MCDM techniques, was used to calculate the weights of each criterion and sub-criterion in order to determine the criterion most affecting IS success. The pairwise comparison matrix was developed according to the questionnaire responses. The criteria of our hierarchical model are the different dimensions of the model and the sub-criteria are the measures of each dimension chosen from previous studies (Table V). TABLE IV. SAMPLE DISTRIBUTION Properties Percentage Gender Female 29 Male 71 Age (years) Less than 30 21.4 30-39 25 40-50 46.4 More than 50 7.2 Experience (years) Less than 5 16.1 5-15 19.4 More than 15 64.5 TABLE V. HIERARCHICAL CRITERIA PRESENTATION Main Criteria Sub Criteria System quality (C1) Access (C11), ease of learning (C12), flexibility (C13), reliability (C14) and response time (C15) Information quality (C2) Accuracy (C21), completeness (C22), understandability (C23), security (C24) and usefulness (C25) Service quality (C3) Reliability (C31), assurance (C32), tangibility (C33), empathy (C34) and responsiveness (C35) Use (C4) Dailey use (C41), frequency of use (C42), nature of use (C43), number of site visits (C44) and number of transactions (C45) User satisfaction (C5) Adequacy (C51), effectiveness (C52), efficiency (C53), enjoyment (C54) and overall satisfaction (C55) Net benefits (C6) Productivity (C61), usefulness (C62), competitive advantage (C63), cost reduction (C64) and overall success (C65) TABLE VI. AGGREGATED PAIRWISE COMPARISON MATRIX C1 C2 C3 C4 C5 C6 C1 1 3 5 7 3 5 C2 0.33 1 3 5 3 5 C3 0.2 0.33 1 3 5 7 C4 0.14 0.2 0.33 1 3 5 C5 0.33 0.33 0.2 0.33 1 5 C6 0.2 0.2 0.14 0.2 0.2 1 B. Implementation After developing the proposed model, our objective is to select the most influential criterion on IS success. The hierarchical model contains 6 criteria and 30 sub-criteria as sho cri col com the cal 0.2 pri com sub use con res val C. cri sys the inf ind on qu (0. sho Engineerin www.etasr own in Table iteria are est llection are co mparison matr e weights are lculated using TABL C C1 0. C2 0. C3 0. C4 0. C5 0. C6 0. Then the priow =2.364/6=w =1.012/6=w =0.500/6= The normaliz 22, 0.17, 0.1, iority of 0. mputational w b-criteria pres ed (4), (5), an nsistency rati spectively (for lue of CR (0.0 T Sub-Crit Survey Resul AHP method iterion of the stem quality d e higher value formation syst dicate that sys information s ality (0.22), .10), user satis ows the weigh ng, Technology r.com VII. The weig timated using ollected by an rix of criteria e given. A n (1). LE VII. NORM C1 C2 45 0.59 0 15 0.20 0 09 0.06 0 06 0.04 0 15 0.06 0 09 0.04 0 ority weights a = 0.39w =1.33 = 0.17w =0.57 = 0.08w =0.20 zed weight ve 0.08, 0.03), .39 is C “ way is anticipa sented in Tab nd (6): λ =6 io (CR) are r RI=1.24): C 012) is less tha TABLE VIII. SU teria Weight 0.42 0.26 0.15 0.11 0.06 0.34 0.25 0.19 0.15 0.06 0.37 0.25 0.15 0.13 0.11 lts d was used to proposed mo dimension has e, it means tha tem success i stem quality h system succes service qualit sfaction (0.08) hts of each d y & Applied Sci D ghts of the ma g AHP meth online questio (Table VI) an normalized m MALIZED DECISION C3 C4 0.52 0.42 0.31 0.30 0.10 0.18 0.03 0.06 0.02 0.02 0.01 0.01 are calculated 36/6=0.22 74/6=0.1 06/6=0.03 ector of main c the most val “System Qua ated to determ ble VIII. To c 6.08. Consiste calculated th CI=0.016 and an 0.10, it is ac UB-CRITERIA WEI Sub-Criteria o determine th odel. The res s a weight equ at the most infl is it system q has a strong s ss (0.39) follow ty (0.17), int ) and net bene dimension. The ience Research aghouri et al.: ain criteria and hod and the onnaire. A pa nd the calculat matrix C has N MATRIX C5 C6 0.20 0.18 0.20 0.18 0.33 0.25 0.20 0.18 0.07 0.18 0.01 0.04 using (2): criteria is W= luable criteria ality”. The mine the weig calculate λ ency index (CI hrough (7) an CR= 0.012. A ccepted. IGHTS a Weight 0.42 0.23 0.15 0.12 0.07 0.45 0.24 0.15 0.11 0.04 0.46 0.21 0.19 0.09 0.04 he weights of sults show tha ual to 0.39 wh fluential criteri quality. The r ignificant infl wed by inform tention to use efits (0.03). Fig e results show h V Enhanced Mod d sub- data irwise tion of been =(0.39, a with same ghts of , we I) and nd (8) As the f each at the hich is ion on results luence mation e /use gure 3 w also that spe thes dist dev sho from D Inf In Sa Ne Mc eva sys Thu lear Sys com info dim thei Vol. 8, No. 4, 20 del For Evalua t service qual ecially the de se results, as w tributed and th veloped accor ows the degree m the participa Dimension S tr o n g ly System Quality 56 formation Quality 55 Service Quality 5 ntention to use/Use 48 User atisfaction 40 et Benefits 3 This study pr cLean IS succ aluate service tem quality ha us, system dev rning, flexibi stem develo mpleteness, un ormation. Fin mensions can b ir information 018, 3194-3198 ting Informatio lity exhibited gree of servi we already me he aggregated rding to the q e of influence ants’ point of v TABLE IX. S tr o n g ly a gr ee A g re e 6.7% 27.4% 5.8% 23.4% 0% 21.4% 8.2% 19.8% 0.6% 31.7% 5% 34.2% Fig. 2. Elem VI. CO roposed an IS cess model a quality. The as a strong sig velopers shoul ility, reliabilit opers shoul nderstandabili nally, the pro be used as a t systems. Fig. 3. W 8 on System Succ a stronger eff ice reliability entioned that a pairwise comp questionnaire e of each crite view. SURVEY RESULT A v er a ge 9% 11.5% 5 13.70% 8 13.80% 10 14.4% 1 12.80% ments' importance ONCLUSION model based and the SERV e analysis resu gnificant influe ld focus more ty and system d fully e ity, security oposed succe tool in organi eights of criteria 3197 cess: Determini fect on IS suc (0.37). To o a questionnaire mparison matrix results. Tabl erion on IS su TS D is a g re e S tr o n g ly d is a gr ee 5% 2% 5.3% 4% 8.6% 6.3% 0.40% 7.8% 0.3% 3% 10% 8% e on the Delone VQUAL mod ults indicated ence on IS suc e on access, ea m response exploit accu and usefulnes ss model an zations to eva ing … ccess, obtain e was x was le IX uccess R a n k 1 2 3 4 5 6 e and del to d that ccess. ase of time. uracy, ss of nd its aluate Engineering, Technology & Applied Science Research Vol. 8, No. 4, 2018, 3194-3198 3198 www.etasr.com Daghouri et al.: Enhanced Model For Evaluating Information System Success: Determining … REFERENCES [1] A. Carugati, C. Rossignoli, Emerging Themes in Information Systems and Organization Studies, Psysica Verlag, 2011 [2] W. H. DeLone, E. R. McLean, “The Delone and McLean Model of Information Systems Success: a Ten-Year Update”, Journal of Management Information Systems, Vol. 19, No. 4, pp. 9-30, 2003 [3] A. Parasuraman, V. Ziethaml, L. L. Berry, “SERVQUAL: A Multiple- Item Scale for Measuring Consumer Perceptions of Service Quality”, Journal of Retailing, Vol. 62, No. 1, pp. 12-40, 1988 [4] T. L. Saaty, The analytic hierarchy process, McGraw-Hill, 1980 [5] W. H. DeLone, E. R. McLean, “Information Systems Success: The Quest for the Dependent Variable”, Information Systems Research, Vol. 3, No. 1, pp. 60-95, 1992 [6] L. F. Pitt, R. T. Watson, C. B. Kavan, “Service Quality: a Measure of Information Systems effectiveness”, MIS Quarterly, Vol. 19, No. 2, pp. 173-187, 1995 [7] M. A. Fishbein, I. Ajzen, Belief, Attitude, Intention and Behavior: an Introduction to Theory and Research, Addison Wesley, 1975 [8] F. D. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology”, MIS Quarterly, Vol. 13, No. 3, pp. 319-340, 1989 [9] A. Parasuraman, V. A. Zeithaml, L. L. Berry, “A conceptual model of service quality and its implications for future research”, Journal of Marketing, Vol. 49, No. 4, pp. 41-50, 1985 [10] F. Buttle , “SERVQUAL: review, critique, research agenda”, European Journal of Marketing, Vol. 30, No. 1, pp. 8-32, 1996 [11] N. H. Zardari, K. Ahmed, S. M. Shirazi, Z. B. Yusop, Weighting Methods and their Effects on Multi-Croteria Decision Making Model Outcomes in Water Ressources Management, Springer, 2015 [12] F. A. Lootsma, Multi-criteria decision analysis via ratio and difference judgement, Kluwer Academic Publishers, 1999 [13] J. Fulop, “Introduction to Decision Making Methods”, Working Paper 05-6, Laboratory of Operations Research and Decision Systems, Computer and Automation Institute, Hungarian Academy of Sciences, Budapest, 2005 [14] V. Podvezko, “The Comparative Analysis of MCDA Methods SAW and COPRAS”, Inzinerine Ekonomika-Engineering Economics, Vol. 22, No. 2, pp. 134-146, 2011 [15] T. L. Saaty, K. Penivati, Group decision making: Drawing out and reconciling Differences.: RWS Publications, Pittsburgh, USA, 2008 [16] T. L. Saaty, “Ran6k from Comparisons and From Ratings in the Analytic Hierarchy/Network Process”, European Journal of Operational Research, Vol. 168, No. 2, pp. 557-570, 2009 [17] T. L. Saaty, “Decision making with the analytic hierarchy process”, International Journal of Services Sciences, Vol. 1, No. 1, pp. 83-98, 2008 [18] A. H. I. Lee, W. C. Chen, C. J. Chang, “A fuzzy AHP and BSC approach for evaluating performance of IT department in the manufacturing industry in Taiwan”, Expert Systems with Applications, Vol. 34, No. 1, pp. 96-107, 2008 [19] M. Dagdeviren, S. Yavuz, N. Kilinc, “Weapon selection using the AHP and TOPSIS methods under fuzzy environment”, Expert Systems with Applications, Vol. 36, No. 4, pp. 8143-8151, 2009 [20] G. G. Gable, D. Sedera, T. Chan, “Re-conceptualizing Information System Success: The IS-Impact Measurement Model”, Journal of Association for Information Systems, Vol. 9, No. 7, pp. 377-408, 2008 [21] D. Sedera, G. G. Gable, “A Factor and Structural Equation Analysis of the Enterprise Systems Success Measurement Model”, 25th International Conference on Information Systems, Washington DC, pp. 449-463, December 12-15, 2004 [22] J. Iivari, “An empirical test of the Delone–Mclean model of information system success”, The DATA BASE for Advances in Information Systems, Vol. 36, No. 2, pp. 8-27, 2005 [23] S. Hamilton, N. L. Chervany, “Evaluating information system effectiveness - part I: Comparing evaluation approaches”, MIS Quarterly, Vol. 5, No. 3, pp. 55-69, 1981 [24] J. E. Bailey, S. W. Pearson, “Development of a tool for measuring and analyzing computer user satisfaction”, Management Science, Vol. 29, No. 5, pp. 530-545, 1983 [25] S. Knight, J. Burn, “Developing a Framework for Assessing Information Quality on the World Wide Web”, Informing Science Journal, Vol. 8, pp. 159-172, 2005 [26] V. McKinney, Y. Kanghyun, F. M. Zahedi, “The measurement of web- customer satisfaction: An expectation and disconfirmation approach”, Information Systems Research, Vol. 13, No. 3, pp. 296-315, 2002 [27] H. Almutairi, G. H. Subramanian, “An empirical application of the Delone and Mclean model in the Kuwaiti private sector”, Journal of Computer Information Systems, Vol. 45, No. 3, pp. 113-122, 2005