IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 203 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 USING THE ANALYTIC HIERARCHY PROCESS TO ASSESS THE IMPACT OF INTERNAL CONTROL WEAKNESSES ON FIRM PERFORMANCE Amanda Lyn Wilford Dixie L. Leavitt School of Business Southern Utah University amandawilford@suu.edu Lawrence D. Bodin Robert H. Smith School of Business University of Maryland lbodin@rhsmith.umd.edu Lawrence A. Gordon Robert H. Smith School of Business University of Maryland lgordon@rhsmith.umd.edu ABSTRACT Since the passage of the Sarbanes-Oxley Act (SOX) of 2002, many studies have examined the impact of material weaknesses in internal control systems (MWICS) on firm performance. Overall, these studies indicate that a negative association exists between poor internal control and firm performance. Prior research suggests that the above noted association between internal control and firm performance should be affected by both the actual number and the different types of MWICS. However, this stream of research has focused on using a binary measure for internal control and has not considered the combined impact that the different types of MWICS may have on firm performance. In this study, we create and introduce a new internal control index, derived from the Analytic Hierarchy Process (AHP). We then show that more information regarding the impact of MWICS can be obtained through our AHP index measure as opposed to the binary measure that is commonly used. These findings have important implications for a firm’s stakeholders (e.g., managers, stockholders, creditors, financial analysts, employees, and auditors). Keywords: Analytic Hierarchy Process; internal control; performance measurement; material weaknesses; internal control measures. 1. Introduction The Sarbanes-Oxley Act of 2002 (SOX) was passed in response to the perceived lack of internal controls within the scandal-ridden companies of the early 2000s (e.g., Enron, TYCO, WorldCom). To comply with section 404 of SOX, large public companies are required to fulfill two main internal control requirements. First, the firm’s management must prepare a report that states its responsibility for establishing, maintaining and mailto:amanda.wilford@ie.edu mailto:lbodin@rhsmith.umd.edu mailto:lgordon@rhsmith.umd.edu IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 204 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 assessing its firm’s internal control system. Second, the firm’s auditors must attest to and report on the management’s assessment of the firm’s internal control system. A key aspect of compliance with the SOX requirements is that firms must report their material weaknesses in internal control systems (MWICS) within their 10-K (annual) and 10-Q (quarterly) reports that are filed with the Securities and Exchange Commission (SEC). With the increased attention on internal controls, Audit Analytics has created a database that aggregates the MWICS from all of the 10-K and 10-Q reports that are filed with the Securities and Exchange Commission (SEC). In addition, Audit Analytics classifies the number of SOX-related MWICS reported by firms and auditors into 21 different categories that represent different types of MWICS. This database has been widely used to empirically examine issues surrounding the association between a firm’s internal control system and its performance. Generally speaking, studies have provided strong evidence of a negative association between the presence of MWICS and firm performance (e.g., Beneish, Billings & Hodder, 2008; Gordon & Wilford, 2012; Cheng, Dhaliwal & Zhang, 2013; Feng, Li, McVay & Skaife, 2015; Gao & Jia, 2016; Mao & Ettredge, 2016; D’Mello, Gao & Jia, 2017; Ge, Koester & McVay, 2017; Cheng, Goh & Kim, 2018). A cursory look at the different MWICS reported by firms under SOX and prior research suggests that some types of MWICS can have a greater impact on firm performance than others (Beneish et al., 2008; Gordon & Wilford, 2012; Weiss, 2014; Balsam, Jiang & Lu, 2014; Cheng et al., 2018). Additionally, a few empirical studies found that the number of MWICS can negatively impact firm performance (Gordon & Wilford, 2012; Keane, Elder & Albring, 2012; Darrough, Huang & Zur, 2018). Most research, starting with the seminal MWICS work by Ge and McVay (2005), presents MWICS as a binary variable (BV) that is set equal to one if a firm reports at least one MWICS and/or at least one type of MWICS and set at zero otherwise. Nevertheless, the BV does not capture the fact that reporting multiple MWICS and multiple types of MWICS may impact a firm’s performance differently than just acknowledging the presence of a MWICS. The use of the BV implicitly assumes that firm performance is associated with the mere reporting of MWICS, irrespective of the aggregation of MWICS types that are reported. It is also worth noting that the number of MWICS reported does not always directly correspond to the number of MWICS types. A firm could report only one MWICS in its 10-K report, but that MWICS could be included in multiple MWICS types in the Audit Analytics database. Given that most firms that report MWICS report more than one MWICS, as well as multiple types of MWICS, our objective in this paper is to examine how the Analytic Hierarchy Process (AHP) can be used to aid the stakeholder’s examination of the severity of control issues in their organizations. To do so, we examined the issue through two different, but related, analyses. First, we developed an AHP model to determine weights that could be applied to the different MWICS that are reported by companies. We used these weights to develop an internal control index (ICI) as a measure of MWICS. Our ICI is derived from an experiment with auditors and an experiment with managers. (i.e., what we refer to as ICI-A and ICI-M, respectively). Second, we assessed whether there was a relationship between our ICI and firm performance (where firm performance is measured IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 205 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 as return on assets [ROA]). This second analysis helped us determine the utility of our ICI measures compared to the BV measure that is commonly used. Our findings indicate that auditors and managers assign different levels of significance to different types of weaknesses. These differences support the use of an AHP-based index to analyze MWICS rather than, or as a supplement to, a BV measure. Furthermore, our findings show that using an AHP-derived index to examine the association between firm performance and MWICS is more informative than using a BV measure. The current study contributes to the growing AHP and internal control literature in the following ways. First, this study provides a practical application for the use of the AHP in determining the impact of MWICS on firm performance. Second, using an aggregate measure of internal control, this study provides empirical evidence that different weaknesses will have differing impacts on firm performance. Third, based on a larger and more contemporary sample of firms, this study confirms and expands upon the prior literature that shows a negative association between firm performance and MWICS. This latter issue highlights the incentive that firms have to not report any internal control weaknesses within their 10-Ks that are filed with the SEC (Securities and Exchange Commission). It also helps explain why many firms claim to have effective internal controls when in reality they have MWICS (Rice & Weber, 2012; Rice, Weber & Wu, 2015). Furthermore, only a small percentage of firms actually report MWICS in their 10- K reports and most of them that are reported were detected by the firm’s auditors rather than the firm’s management (Bedard & Graham, 2011). Fourth, the findings from the current study help explain why firms have an incentive to delay the reporting of MWICS until the time of reporting restatements rather than in their original 10-K reports (Rice & Weber, 2012). The remainder of this paper proceeds as follows. In Section 2, we review the relevant literature leading up to the development of the basic hypotheses underlying this study. In Section 3, we discuss the research design that forms the basis for our study. In Section 4, we examine the results related to our hypotheses. Finally, in Section 5, we discuss the implications and conclusions of our analysis. 2. Background and hypotheses 2.1 The use of the AHP in accounting research Not surprisingly, the AHP has been used within accounting research as a tool to examine accounting evaluation and planning decisions. Using the AHP to model auditing procedure preferences, Arrington et al. (1984) provided one of the first applications of the AHP within accounting research. Additionally, Arrington et al. (1984) suggested the AHP as an applicable tool to make “judgments such as materiality, internal control evaluation, opinion qualifications, and strategic planning” (Arrington et al. 1984). Other applications of the AHP in accounting include examining auditors’ evaluations of internal audit functions (Messier & Schneider, 1988; Campbell, 1994), tests of controls (Spires, 1991), audit planning (Bedard et al., 1991), performance evaluation (Emby & Etherington, 1996), accounting treatment (Boyle, 1985), and the evaluation of information security investments (Bodin et al., 2005; 2008). Similarly, Amponsah (2011) used the AHP to determine the critical success factors of Public-Private-Partnerships. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 206 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 And more recently, Chen et al. (2017) applied the AHP to an internal control framework using Chinese firms. Internal controls, by nature, are qualitative and when internal control system issues arise, there is no general process in place to evaluate and compare these issues. First, we sought to determine whether users of financial statements perceive that there are differences within the types of weaknesses that are reported within their organizations. To do so, we used the structure of the AHP rating model to test the following hypothesis, stated in the null form. H1: There is no difference in the perception of severity about the types of reported MWICS between the different users of financial statements (auditors vs. managers). To test hypothesis H1, we developed and ran an AHP ratings model that uses pairwise comparisons determined by an experienced set of auditors from various firms. Further, we developed a second AHP ratings model based on the pairwise comparisons generated by a set of managers from various firms. This process is described in Section 3 below. As will be seen in Section 3, the AHP ratings models developed in this study are variants of the traditional AHP ratings model. In a traditional ratings model, each alternative is given a score between 0 and 1 and the alternatives with the largest scores (or smallest scores) are identified as being the best alternatives. In contrast, the ratings models studied in this paper determine weights to assign to each MWICS or aggregated MWICS (alternative) and these weights are used to set up an Ordinary Least Squares (OLS) regression model similar to Feng et al. (2015). 2.2 Internal control measurement in current accounting research Internal control research in accounting literature has examined the link between the reliability of financial reporting, through MWICS, and firm performance. For example, Cheng et al. (2013) found that firms that report MWICS are less efficient with their capital investments. Using frontier analysis, Cheng et al. (2018) found that operational efficiency is significantly lower when MWICS are reported. Weiss (2016) showed that the negative impact on performance, from reporting MWICS, is more severe for family- owned firms than for non-family-owned firms. Additionally, Feng et al. (2015) found that if a firm reports MWICS, they are more likely to have a lower return on assets. Furthermore, Darrough et al. (2018) showed that acquirers that reported MWICS have a larger negative stock market reaction to acquisitions and lower future performance than acquirers without such MWICS. There are numerous additional empirical studies that document an association between MWICS that are reported under SOX and firm performance. For example, research indicates that there is a negative association between firms reporting MWICS and stock market returns (e.g., Beneish et al., 2008; Hammersley, Myers & Shakespeare, 2008; Rezee, R. Espahbodi, P. Espahbodi & H. Espahbodi, 2012; Chen et al., 2017). This negative relationship also holds between firms that report MWICS and return on assets (Ge & McVay, 2005; Feng et al., 2015). Firms that report MWICS also tend to exhibit lower accrual quality (e.g., see Doyle et al., 2007b; Asbaugh-Skaife et al., 2008; Bedard, R. Hoitash, U. Hoitash & Westermann, 2012), and lower accrual quality has been tied to IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 207 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 lower firm performance (e.g. Dechow & Dichev, 2002; Francis, LaFond, Olsson & Schipper, 2005). As noted earlier, the initial post-SOX internal control research measures MWICS as a binary variable (BV) set equal to one if a firm reports at least one MWICS in their financial reports and set to zero otherwise (Ge & McVay, 2005; Doyle, Ge & McVay, 2007a; Doyle et al., 2007b; Ogneva, Subramanyam & Raghunandan, 2007; Li, Peters, Richardson & Watson, 2012). As research has developed in this area, MWICS have also been classified into account-level types and entity-level types, established as individual BVs that are set equal to one if the MWICS reported was related to one or the other and set to zero otherwise. Using this classification, Bedard et al. (2012) demonstrated that remediation of entity-level MWICS results in significant changes in abnormal accruals (i.e., amounts in the financial statements that have been incurred during the period, but not yet realized through a cash transaction). Additionally, Asare and Wright (2013) found that analysts are more skeptical of audit reports that are associated with an internal control report that lists entity-level MWICS, and Balsam et al. (2014) showed a relationship between equity incentives and the likelihood of reporting entity-level MWICS. More recent internal control research further refines the MWICS classification and examines the Audit Analytics specified types of MWICS, reported in isolation, as BVs that are set equal to one if the firm has an MWICS that corresponds to an MWICS type that is under consideration, and set to zero otherwise. Focusing on the efficiencies in inventory management, Feng et al. (2015) found that inventory management declines when inventory account-related MWICS are reported. Li et al. (2012) indicated that MWICS types related to a firm’s information technology environment influence the accuracy of management decisions differently than other MWICS types. Koester, Lim, and Vigeland (2015) investigated the relationship between tax account-related MWICS and unrecognized tax benefits. In addition to the differences related to the MWICS types, Gordon and Wilford (2012) and Darrough et al. (2018) showed that the number of reported MWICS has an impact on firm performance. In their examination of MWICS and audit fees, Keane et al., (2012) found that audit fees increase in relation to the number of MWICS reported. Despite the suggestion in recent research that different MWICS types can have differing effects on performance, the common practice of measuring MWICS as a single binary variable has ignored the aggregate impact of the number of MWICS reported and the impact of reporting multiple types of MWICS. Accordingly, in this paper we tested the following basic hypothesis (stated in its null form): H2: There is no additional information provided about the relationship between MWICS and firm performance when a firm’s MWICS is measured based on an AHP internal control index (ICI) versus a binary variable (BV). 3. Research design To test the above hypotheses, we first created and analyzed the results from an Analytic Hierarchy Process evaluation of internal control that we call the Internal Control Index IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 208 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 (ICI). The elements ascribed to the creation of the ICI are detailed in Section 3.1. After this initial evaluation, we compared the ICI to the BV to examine whether the ICI would provide more information about the impact of internal control on firm performance than the BV. The details of this analysis are presented in Section 3.2. 3.1 AHP experiment The ICI was developed through the following five major steps. Step 1: Design of the AHP Tree. Our AHP tree has three levels. Level 1 of our AHP tree is the goal node. The overall goal of this AHP is the development of an ICI and the hierarchy was designed around this goal. Level 2 of our AHP tree consists of the five primary criteria that include Personnel Weaknesses (PW), Financing/Accounting Reporting Issues (FRI), Policy Issues (PI), Restatements/Adjustments (R/A), and Regulatory Issues (RI). We developed the primary criteria as aggregate categories of the 21 sub-criteria discussed below to reduce the number of required pairwise comparisons. If each of these categories were to be classified as a primary criterion, decision makers would have to analyze (21*20)/2 = 210 pairwise comparisons. Two accounting professionals, each with over eight years of experience, examined our classification of the 21 MWICS into these five criteria and their review resulted in no significant modifications to the classification. Level 3 of our AHP tree consists of 21 sub-criteria that we derived from the categories used by Audit Analytics to classify firm reported MWICS. A sideways view of our AHP tree is shown in Figure 1. Each criterion and sub- criterion, along with its abbreviation in parentheses is documented in Figure 1. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 209 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 Figure 1 AHP tree for internal control comparisons Step 2: Pairwise comparison experiment. We conducted an experiment on specific users of financial statements, with practicing corporate auditors and managers evaluating the criteria and sub-criteria in the AHP tree hierarchy using pairwise comparisons. These pairwise comparisons are a key component of the AHP and are used to supply the data to carry out the analysis. The pairwise comparisons represent a ratio of the weights assigned to the two criteria that are being compared. The experts selected for this AHP experiment were managers of financial reporting and senior auditors responsible for auditing the financial reports of public companies. Auditors and managers of financial reporting are two similar groups of stakeholders with one important distinguishing characteristic, independence. This independence may influence the auditors’ ability to be more objective Internal Control Weaknesses Personnel Weaknesses (PW) Accounting personnel resources, competency/training (ACT) Ethical compliance issues with personnel (ECI) Senior management competency, tone, reliability issues (SMI) Insufficient or non-existent internal audit function (INI) Segregations of duties/design of control (personnel) (SD) Financing/Accounting Reporting Issues (FRI) Treasury control issues (TC) Journal entry control issues (JEC) Non-routine transaction control issues (NTC) Inadequate disclosure controls (ID) Material and/or numerous auditor/YE adjustments (MA) Policy Issues (PI) Untimely or inadequate account reconciliations (UAR) Accounting documentation, policy and/or procedure (ADP) Information technology, software, security and access issues (IT) Restatements/Adjustment s (R/A) Restatement or nonreliance of company filings (RNF) Restatement of previous 404 disclosures (RPD) SAB 108 adjustments noted (SAB) Regulatory Issues (RI) Ineffective or understaffed audit committee (IUA) Ineffective regulatory compliance issues (IRC) Management/Board/Audit Committee investigations (MI) SEC or other regulatory investigations and/or inquiries (SI) Scope/disclaimer of opinion or other limitations (SDL) IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 210 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 in their assessment of the different types of MWICS. Experts were randomly selected from the LinkedIn social network of one of the authors. An online questionnaire was sent to ten auditors and ten managers. Seven auditors and seven managers responded to our invitation to participate in this study. The auditor experts have achieved at least an audit manager position within a top ten public accounting firm and each have at least seven years of auditing experience. The manager experts are corporate managers at SEC registered firms and large (over 1,000 employees) private firms who oversee their company’s financial reporting function. Even though the managers at private firms are not required to comply with SOX, managers of financial reporting in all organizations must have a strong understanding of internal controls for financial reporting. The experts performed pairwise comparisons of the categories and types of MWICS that are identified in the AHP tree, presented in Figure 1. These pairwise comparisons were used to supply the data to calculate the weights assigned to each of the five categories and each of the 21 MWICS. Step 3: Determining the AHP weights. The expert responses were evaluated to determine the weights of the various criteria and sub-criteria. To determine these weights, all of the pairwise comparisons (i.e., judgments) that were made by the experts were stored in pairwise comparison matrices. Pairwise comparison matrix A = (aij) is a positive reciprocal matrix (i.e. aji = 1/aij for each element (i,j) in a positive reciprocal matrix). Thus, the comparison of criterion i to criterion j will always be the reciprocal of the comparison of criterion j to criterion i. These pairwise comparison matrices were then used to determine the weights assigned to each criterion and sub-criterion. Then, the eigenvector associated with the maximum eigenvalue of a pairwise comparison matrix was used to determine the weights assigned to all of the criteria or sub-criteria associated with the pairwise comparison matrix. To aggregate each of the seven experts’ pairwise comparison matrices, we took the geometric mean to determine the overall weights for the manager group. This same process was performed for the auditor group. Different pairwise comparison matrices were developed for the auditor group and the manager group. Step 4: Computing the overall weight assigned to each MWICS. In the fourth step, we determined the overall weight assigned to each MWICS. In the solution found in Step 3, the sum of the weights computed for each of the criteria values must sum to one. Furthermore, the sum of the weights assigned to a MWICS where the MWICS is assigned to one of the five criteria is equal to the product of the AHP weight for the MWICS, given that the MWICS is assigned to a particular criterion, and multiplied by the weight assigned to the criterion. As a result, the overall weight assigned to each of the 21 MWICS must sum to one. Table 1 presents the tabular solution to the determination of these weights for both the auditors and managers. Step 5: Weight to assign to an entry in the Audit Analytic database. To calculate the ICI for any given entry i in the Audit Analytic database, we applied the weights, determined in the AHP experiment in Step 4 to the data contained for each entry i in the Audit Analytics database. The result of this application is an ICI for each firm-specific observation. The Audit Analytics data does not specifically define which types of IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 211 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 MWICS are associated with which weakness and as such we were unable to construct indices that combined both the number and type of MWICS reported (see Appendix A for detailed definitions of each of the 21 MWICS types). The number of MWICS reported in Audit Analytics is derived directly from firm reported information in SEC documents. Mathematically, the ICI for each firm-year observation i derived from the Audit Analytics database is defined as follows: ICIi = ∑ 21 k=1 (Wk * MWik) where ICIi = internal control index value for firm-year observation i; Wk = AHP weight assigned to MWICS type k, where type k is one of the 21 different types of MWICS categorized by Audit Analytics where ∑ 21k=1 (wk)=1; MWik = variable assigned a value of one if firm-year observation i reports MWICS type k, and set to zero otherwise. The value of each ICIi ranges between 0 and 1. We interpreted the ICIi for firm-year observation i as an indication of the firm’s level of internal control. A high (low) value of the ICIi indicates weaker (stronger) internal control for firm-year observation i. In the procedures outlined above, we calculated two alternative values for the ICI. ICI-A corresponds to the AHP results from the auditors’ responses whereas ICI-M corresponds to the AHP results from the managers’ responses. BV, ICI-A and ICI-M, our MWICS measures, were calculated based on the presence/absence of MWICS. Firms that reported MWICS have positive index values and firms that do not report MWICS have index values of 0. It is unlikely that the weights produced from the AHP will be the optimal set of weights. However, if the ICI provides a more complete explanation of the relationship between firm performance and MWICS than the BV, then we have shown that the ICI is capable of providing additional insights into prior research results. 3.2 Research design for OLS regression analysis of firm performance The establishment of the ICI is shown above through the AHP experiment analysis explanation. Additionally, our examination of firm performance requires that we also use a BV for comparison purposes. The BV, like the ICI, was also calculated for all Audit Analytics observations and becomes a key independent variable in our firm performance analysis. The BV is set equal to one if a firm-year observation from Audit Analytics reports any MWICS and set to zero otherwise. To examine and compare the BV and ICI measures of MWICS to firm performance, we established an Ordinary Least Squares (OLS) model whose specifications are similar to Feng et al. (2015). As noted above, our chosen measure of firm performance for this analysis is ROA. Equation (1) defines the MWICS-ROA model that was used to test the relationship between ROA and MWICS. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 212 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 We utilized two different measures for MWICS: BV and ICI. When the ICI is included in Equation (1), it can be either ICI-A or ICI-M. Also, the coefficients produced through an OLS regression of Equation (1) are identified as  within Equation (1). OLS regression equation for the MWICS-ROA model: ROAi = 0 + 1*MWICSi +2*Sizei + 3*Segi + 4*Foreigni + 5*Growthi + 6*Lossi + 7*Cap_Inti + 8*Vol_Salei + 9*Agei + 10*Aud_Bigi + 11*Pr_ROAi +  (1) The variables in Equation (1) for firm-year observation i are shown below. We used data from the Audit Analytics, CRSP, and Compustat databases to calculate the dependent and independent variables in our analyses (with the raw database used to gather the information included in parentheses): ROAi = measured as earnings before extraordinary items divided by average total assets for firm-year observation i (Compustat); MWICSi = internal control measure for firm-year observation i in year t. This measure is interchangeable as BV, ICI-A, or ICI-M (Audit Analytics); Sizei = log of the market value of equity for firm-year observation i (Compustat); Segi = measured as the number of business segments reported for firm- year observation i (Computat Segments); Foreigni = indication of foreign sales that is set equal to one, zero otherwise, if a firm-year observation i reports a foreign currency translation adjustment (Compustat); Growthi = measured as the sales growth rate for firm-year observation i over the time period t-2 through t-1 (Compustat); Lossi = measured as the proportion of years (t-2 through t) during which a firm reports a loss in earnings (Compustat); Cap_Inti = capital intensity that is measured as the log of PP&E for year t (Compustat); Vol_Salei = standard deviation of sales to average assets from years t-6 through t, at least three years of data are required (Compustat); Agei = log of years firm is included in the CRSP database t (CRSP Header File); Aud_Bigi = dummy variable set to 1 if firm utilizes one of the 6 biggest Auditors (Audit Analytics); IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 213 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 Pr_ROAi = measured as earnings before extraordinary items divided by average total assets in year t-1 (Compustat); The dependent variable in Equation (1) described above is return on assets (ROA). To provide comparability with prior research (Feng et al., 2015), we utilized the same type of OLS model specification and similar dependent and independent variables. We clustered by firm and controlled for year and industry fixed effects. In the analysis described above, we expected that the coefficient values of the independent variables would be similar in sign and significance to those of prior studies. Additionally, we expected the coefficient values of the MWICS measures generated by our analysis, our key independent variables, would be significant and negative (as suggested by previous research). Through the above analyses, we wanted to determine whether the ICI measures could provide significant additional information with respect to the impact of MWICS on performance. More specifically, we tested H2 by comparing the coefficients obtained for our MWICS measures in Equation (1). If the models that utilized the ICI measures produced more information than the information produced through the utilization of the BV measure, we would have evidence to reject H2. As noted above, to calculate the dependent and independent variables in our performance analyses, we used data from the Audit Analytics, CRSP, and Compustat databases. The Audit Analytics data serves as the base for our dataset and includes all yearly firm- specific internal control data for fiscal years 2004 through 2018. To construct a sample for our AHP application/performance analysis, we began by including all observations from Audit Analytics with audit internal control reports in fiscal years 2004 through 2018. The raw initial sample is comprised of 60,544 firm-year observations. Then, we applied two screens to the data to arrive at our final sample that was used in the analyses. First, we eliminated 1,135 duplicate observations (i.e., in cases where internal control reports are restated, we retained the most recent observation). In our second screen, we eliminated an additional 21,484 observations that did not have the necessary data available to calculate the variables required for our analysis. Our final sample consisted of 37,925 observations. This final total included 35,326 observations with no MWICS (what we refer to in the next section as the control group) reported and 2,599 observations with MWICS reported. 4. Results 4.1 Results of AHP analysis In Table 1, we present the results of our AHP analysis that are derived from the responses of both auditors and managers of financial reporting. These results were obtained through our application of the procedures described above. As stated above, we aggregated our expert responses utilizing the geometric mean because of homogeneity in the group structure and there were no signs of conflicts of interest among the experts (Ossadnik, Schinke & Kaspar, 2016). IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 214 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 To examine the consistency of our results, we applied a formula, denoted as Equation (2), developed by Saaty (1980). Equation (2) calculates the internal consistency of the experts’ judgments: (2) where: = Average (Aω/ω); A = Matrix of pairwise comparisons; ω = Vector of weights; C.R. = Consistency ratio, determined by calculating the consistency index from a large sample of purely random judgments. The values associated with this variable are derived from Saaty (1980). For purposes of our analysis, C.R. is set equal to 1.12 for the 5x5 matrices and set equal to 0.58 for the 3x3 matrices; C.I. = consistency index. Saaty (1980) argued that reliable judgments had consistency indices of less than 0.10. After the weights for all of the criteria and sub-criteria were calculated, we applied the above consistency check to each pairwise comparison matrix. The results of this first pass suggested that there was a lack of consistency within some of the expert’s judgments. Consistency indices for all of the individual judgments ranged between 0.00 (completely consistent) and 0.57 (highly inconsistent). Given the online nature of the AHP questionnaire, we examined each of the pairwise judgments and attempted to emulate the results that would have been the product of a group discussion that is generally associated with the AHP. This additional step ensured consistency within each expert’s pairwise comparison matrices (i.e., CI was less than 0.10 for all of the matrices). Inconsistencies among some of the judgments could be due to a couple of different factors. First, the auditors and managers that carried out this analysis had a limited understanding of the AHP environment. If the pairwise comparisons had been made within a controlled group environment (all of the auditors or managers being in the same room at the same time with an experienced AHP moderator), this problem would have been eliminated. Second, limited AHP training can lead to difficulty visualizing the relationship between all of the judgments. To ensure that the integrity of the weights were maintained, any adjustment that was made to a pairwise comparison was based on the judgment made with respect to the first variable judgment and the weights associated with each matrix before and after the judgments were revised. We found that the differences in weights before and after the pairwise comparisons were adjusted were insignificant and the C.I. after the adjustments were made was always less than 0.10 (unadjusted weights are available from the authors upon request). Table 1 and Figures 2 - 7 present a summary of the results of our AHP analysis. The criteria weights in Table 1 and the associated figures indicate the level of impact that each criterion has on the financial statements when compared to the other criteria and sub-criteria. Within the AHP derived weights, we see similarities among the weights derived by both the auditors (Table 1, Panel A) and the managers (Table 1, Panel B). It is C.I. = l max -n n-1 æ è ç ö ø ÷ /C.R. l max IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 215 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 important to note that the PW criterion (personnel weaknesses) has the highest criterion weight among both the auditors and the managers. This result is in line with internal control guidance that states that the control environment is foundational to a strong internal control system (COSO, 2013). These criteria weights indicate that managers feel that the PW criterion has the greatest impact on the financial statements, with the criterion having 51% of the weight. Auditors also feel that the PW criterion has the greatest impact (29.2%) on the financial statements. However, the auditors also feel that the FRI and PI criteria have a significant impact on the financial statements (27.6% and 19.7%, respectively). The sub-criteria weights indicate the level of impact that each of the sub-criteria (types of MWICS) has on the financial statements when compared to the other sub-criteria. The auditor experts indicated that with respect to the PW criterion, sub-criterion SMI has the greatest impact on the financial statements (36.4%). Additionally, they clearly indicated that sub-criteria JEC and MA have the greatest impact on the financial statements (28.4% and 25.0%, respectively), within the FRI criterion category. Auditor experts indicated that within the PI criterion, sub-criterion ADP has the greatest impact on the financial statements (45.4%). They also indicated that sub-criterion RNF has a significantly greater impact (58.3%) on the financial statements than the other types of MWICS in the R/A category. Finally, the auditor experts concluded that within the RI criteria category, sub-criteria SDL, MI, and SI have the greatest impact (27.9%, 27.4%, and 24.5%, respectively) on the financial statements. In contrast to the auditor results, the managers found that with respect to the PW criterion, sub-criteria SMI and ECI have the greatest impact on the financial statements (36.7% and 26.9%, respectively). Managers found that criterion FRI dominating sub- criteria are not as clear-cut and three types of MWICS or sub-criteria have about an equal impact on the financial statements with sub-criterion, MA at 27.9%, sub-criterion NTC at 24.6%, and sub-criterion JEC at 22.4%. Managers view sub-criterion IT as having the greatest financial statement impact (41.4%) within the PI criteria category. The manager experts indicated that sub-criteria RNF and SAB, within the R/A criterion, have an equally significant impact on the financial statements (39.5% and 38.2%, respectively). They also indicated that within the RI criterion category, SI has the greatest impact (31.9%) on the financial statements. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 216 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 Table 1 Weights for the ICI with auditors and managers* Panel A: Weights based on auditor assessments Criteria PW FRI PI R/A RI 0.292 0.276 0.197 0.084 0.150 Sub- criteria ACT ECI SMI INI SD TC JEC NTC ID MA UAR ADP IT RNF RPD SAB IUA IRC MI SI SDL 0.106 0.207 0.364 0.100 0.223 0.182 0.284 0.181 0.103 0.250 0.295 0.454 0.251 0.583 0.188 0.229 0.122 0.080 0.274 0.245 0.279 Weight 0.031 0.061 0.106 0.029 0.065 0.050 0.078 0.050 0.028 0.069 0.058 0.090 0.049 0.049 0.016 0.019 0.018 0.012 0.041 0.037 0.042 Panel B: Weights based on manager assessments Criteria PW FRI PI R/A RI 0.510 0.113 0.122 0.108 0.147 Sub- criteria ACT ECI SMI INI SD TC JEC NTC ID MA UAR ADP IT RNF RPD SAB IUA IRC MI SI SDL 0.165 0.269 0.367 0.064 0.134 0.147 0.224 0.246 0.104 0.279 0.281 0.305 0.414 0.395 0.222 0.382 0.164 0.172 0.119 0.319 0.225 Weight 0.084 0.137 0.187 0.033 0.068 0.017 0.025 0.028 0.012 0.032 0.034 0.037 0.051 0.043 0.024 0.041 0.024 0.025 0.018 0.047 0.033 *Panel A presents the weights from the AHP analysis, based on the auditor sample assessments. Panel B presents the weights from the AHP analysis, based on the manager sample assessments. The abbreviations provided for the criteria represent the following categorizations: PW – Personnel Weaknesses, FRI – Financing/Accounting Reporting Issues, PI – Policy Issues, R/A – Restatements/Adjustments, and RI – Regulatory Issues. The sub-criteria are listed in Figure 1 and are defined and described in Appendix A. The values tied to the criteria and sub-criteria were obtained through an AHP experiment. The weight associated with each sub-criterion, MWICS type, was calculated as the product of the criteria value multiplied by the sub-criteria value. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 217 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 *Figures 2 – 7 (F2-F7) graphically present and provide a comparison of the weights from the AHP analysis, based on the auditor and manager sample assessments. The abbreviations provided for the criteria represent the following categorizations: PW – Personnel Weaknesses, FRI – Financing/Accounting Reporting Issues, PI – Policy Issues, R/A – Restatements/Adjustments, and RI – Regulatory Issues. The sub-criteria are listed in Figure 1 and are defined and described in Appendix A. The values tied to the criteria and sub-criteria were obtained through an AHP experiment. The weight associated with each sub- criterion, MWICS type, was calculated as the product of the criteria value multiplied by the sub-criteria value. 29.2% 27.6% 19.7% 8.4% 15.0% 51.0% 11.3% 12.2% 10.8% 14.7% PW FRI PI R/A RI F2. Criteria Weights by Expert Category Auditor Manager 3.1% 6.1% 10.6% 2.9% 6.5% 8.4% 13.7% 18.7% 3.3% 6.8% ACT ECI SMI INI SD F3. PW: Overall Sub- Criteria Weights by Expert Category Auditor - PW Manager - PW 5.0% 7.8% 5.0% 2.8% 6.9% 1.7% 2.5% 2.8% 1.2% 3.2% TC JEC NTC ID MA F4. FRI: Overall Sub- Criteria Weights by Expert Category Auditor - FRI Manager - FRI 5.8% 9.0% 4.9% 3.4% 3.7% 5.1% UAR ADP IT F5. PI: Overall Sub-Criteria Weights by Expert Category Auditor - PI Manager - PI 4.9% 1.6% 1.9% 4.3% 2.4% 4.1% RNF RPD SAB F6. R/A: Overall Sub- Criteria Weights by Expert Category Auditor - R/A Manager - R/A 1.8% 1.2% 4.1% 3.7% 4.2% 2.4% 2.5% 1.8% 4.7% 3.3% IUA IRC MI SI SDL F7. RI: Overall Sub-Criteria Weights by Expert Category Auditor - RI Manager - RI IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 218 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 As a final analysis, we performed t-tests on each of the criteria to determine if there were significant differences between the perceptions of the auditors and the perceptions of the managers as they relate to MWICS. Our results indicate that there are significant differences between the auditor and manager expert judgements for all of the criteria, with the exception of the RI criterion. More specifically, our two-sided t-test results show differences in the PW criterion that are significant at the 0.01 level, differences in the FRI criterion that are significant at the 0.05 level, differences in the PI criterion that are significant at the 0.10 level, and differences in the R/A criterion that are significant at the 0.10 level. Overall, the results of our AHP analysis indicate that differences do exist in the perceptions of financial statement users and that certain types of MWICS are viewed as more severe by managers and auditors than other types of MWICS. As such, this analysis goes against the null hypothesis presented as H1 and therefore H1 is rejected. 4.2 Results for OLS regression analysis of firm performance To examine our second hypothesis, we ran an OLS regression based on Equation (1) and outlined and discussed in Section 3. The results of this regression are displayed in Table 2. Column 1 presents the regression results when BV was used as the measure for MWICS. Column 2 presents the regression results when ICI-A was used as the measure for MWICS. Finally, Column 3 presents the regression results when ICI-M was used as the measure for MWICS. Using the results in Table 2, we found that the coefficients and signs related to the independent variables align closely with the results in prior research (Feng et al., 2015). Additionally, we found that, as expected, the coefficients associated with the MWICS measures based on BV, ICI-A and ICI-M have a significant and negative relationship with our measure of ROA, indicating that firms that report MWICS experience lower profitability. Further, the MWICS coefficients for each of the models (see Columns 1, 2, and 3 in Table 2) are significant at the 0.01 level, and there is only a marginal difference among the t-values of the different measures. Next, we analyzed the three ROA models, BV, ICI-A and ICI-M. The variables for these models are similar with the exception of the variable associated with the MWICS metric. First, we examined the overall model results of Table 2 through the R-square measure and found that there was essentially no difference between the fit provided by the three ROA models when the MWICS measures were interchanged. To compare the MWICS measures, we compared the t-values (shown in parentheses below the coefficients in Table 2) and found that there was only a marginally significant difference among the t- values of these three measures. These results were expected. Next, we compared the differences between the results generated using the BV model and the results generated using the ICI-A and ICI-M models. First, we focused on the coefficients attached to the measures BV, ICI-A and ICI-M (given in Table 2) and the difference in firm performance when we used the ICI measures versus the BV measure. These results show that using the ICI-A and the ICI-M metrics provide additional information that has not been investigated in prior research. This information was derived from the weights in Table 1 that were generated by our use of the AHP. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 219 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 The raw data for our sample consisted of 2,599 firms that reported at least one MWICS and 35,326 firms that reported no MWICS. The average ROA for our sample is 5.93%. This value indicates that on average every dollar of assets reported by a firm will generate a 5.93% profit. We applied the above information to the data contained in Table 2 and found the following. If a firm reports one or more MWICS and the BV measure is utilized, our regression estimates indicate that the ROA decreases by 2%, since the BV coefficient in Column 1 of Table 2 is equal to -0.02. More specifically, the ROA decreases from 5.93% to 3.93% (the ROAs for the raw initial sample are available from the first author upon request). In other words, since the coefficient that is tied to BV is -0.02, then -0.02 multiplied by the value of 1 for the BV indicates a flat decrease of 2% regardless of the type and/or aggregation of MWICS that are reported. This important observation occurs since the only information that we used to determine the impact on ROA, when using the BV measure, was whether the firm reported an MWICS (value is set to 1) or did not report an MWICS (value is set to 0). The actual MWICS that the firm reports were not used to determine the impact on ROA. Given the base information related to our analysis with the BV, we turned our attention to examining how the results indicate that the ROA will be impacted when we use either the ICI-A metric or the ICI-M metric. The weights that we found for the 21 MWICS and considered in this analysis are displayed in Table 1. As described earlier, the AHP was used to compute these weights. Then, we summed the weights (from Table 1) for all of the MWICS that were reported with the 2,599 observations that report MWICS in our sample and divided this sum by 2,599. The average ICI-A weight for firms in our sample is 0.2379 and the average ICI-M weight for firms in our sample is 0.1816. To examine the economic implications of the ICI-A and the ICI-M, we applied the same practice used in the BV analysis above. However, when we utilized ICI-A as our MWICS metric, we were able to estimate the impact that different combinations of MWICS would have on performance. This is not possible with the BV. The average ICI-A weight for firms in our sample is 0.2379 and the coefficient associated with ICI-A (from Table 2) is -0.082. Applying this to the coefficient in Table 2, Column 2, we see that the impact on the average ROA would be approximately 2% (.2379 multiplied by -0.082 equals - 1.95%). Therefore, the ROA for an average firm that reports weaknesses when the ICI-A is used would be equal to 5.93% - 1.95% = 3.98%, similar to the results obtained using the BV. However, the values of ICI-A do not consist of a sole measure for all firms and the variation provides additional insight regarding the impact of MWICS on the ROA. For example, the maximum value of the ICI-A is 0.876. Applying this value to the coefficient indicates that MWICS firms reporting the maximum value will have an ROA that is 7.2% (.876 multiplied by -0.082) lower than the average. With an average ROA of 5.93%, the ROA of this MWICS firm is estimated to be -1.27% (5.93% less 7.2 %). As discussed above, the impact on the ROA depends on the MWICS that are reported. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 220 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 Table 2 OLS regression results: Return on assets analysis Dependent variable: ROA (1) (2) (3) VARIABLES BV ICI-A ICI-M Intercept -0.002 -0.002 -0.003 (-0.26) (-0.31) (-0.45) BV -0.020*** (-8.10) ICI-A -0.082*** (-8.66) ICI-M -0.093*** (-7.94) Size 0.001 0.001 0.001* (1.61) (1.62) (1.69) Seg 0.001 0.001 0.001 (1.17) (1.20) (1.18) Foreign -0.002 -0.002 -0.002 (-1.39) (-1.38) (-1.41) Growth 0.000 0.000 0.000 (-.86) (-0.83) (-0.83) Loss -0.010 -0.010 -0.010 (-1.60) (-1.59) (-1.61) Cap_Int 0.003*** 0.003*** 0.003*** (3.74) (3.76) (3.75) Vol_Sale -0.002 -0.002 -0.002 (-0.79) (-0.79) (-0.82) Age 0.004*** 0.004*** 0.004*** (5.28) (5.28) (5.30) Aud_Big 0.004 0.004 0.004 (1.26) (1.21) (1.22) Pr_ROA 0.740*** 0.740*** 0.740*** (22.36) (22.37) (22.40) Industry Controls Yes Yes Yes Year Controls Yes Yes Yes Firm Cluster Yes Yes Yes Observations 37,925 37,925 37,925 R-squared 66.07% 66.08% 66.08% *Table 2 reports results for the following OLS regression using the Feng et al. (2015) model with differing MWICS index variables that represent BV, ICI-A and ICI-M, respectively: ROA = 0 + 1*MWICS +2*Size + 3*Seg + 4*Foreign + 5*Growth + 6*Loss + 7*Cap_Int + 8*Vol_Sale + 9*Age + 10*Aud_Big + 11*Pr_ROA +  (1) Column 1 presents the regression results when BV is used as the measure for MWICS. Column 2 presents the regression results when ICI-A is used as the measure for MWICS. Finally, Column 3 presents the regression results when ICI-M is used as the measure for MWICS. Statistical significance at the 0.01, 0.05, and 0.10 level is indicated by *** , ** , and * , respectively. t-statistics are listed in parentheses and based on two-way standard errors that are clustered at the firm level. All independent variables are winsorized at the 1% and 99% levels. All variables are defined above in the Research Design section. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 221 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 To examine the results using our ICI-M measure, we applied the same procedures that were applied to the ICI-A. As shown in Table 1, the AHP weights for the ICI-M differ from those of the ICI-A. The average ICI-M weight for firms in our sample is 0.1816. Applying the average ICI-M value to the coefficient in Table 2, Column 3, we see that the average impact on the ROA would be approximately -1.7% (.1816 multiplied by - 0.093). This percentage of -1.7% is similar to the corresponding values generated by applying the BV and the ICI-A. The maximum value of the ICI-M is 0.857. Applying this value to the coefficient would result in an estimated decrease of 7.97% (0.857 multiplied by -0.093). For a firm that reports the maximum ICI-M, their corresponding ROA would be -2.04% (5.93% less 7.97%). As with the ICI-A, the impact on the ROA is dependent upon the types of MWICS that the firm reports. These three examples illustrate the utility of the AHP approach presented in this paper. Using the AHP to identify and apply weights to the MWICS allows stakeholders to gain more detailed and granular information than can be obtained with a binary measure such as the BV. Therefore, the above analysis leads us to reject hypothesis H2. 5. Conclusions The above analysis generates several important implications for researchers, the stakeholders of firms, and regulators. The most obvious of these implications are as follows. First, the negative association between MWICS and firm performance continues to exist. Therefore, the emphasis on strong internal control systems by management, investors and regulators continues to be well justified. Second, management has a strong incentive to avoid reporting any MWICS, regardless of type, because of the impact on performance. The fact that external auditors identify a much larger share of MWICS than management clearly shows that management is responding to this incentive. Therefore, external auditors need to continue to be vigilant in their efforts to identify MWICS not identified by management. Furthermore, in their role as advisors, external auditors need to emphasize the importance of preventing MWICS, as well as the importance of quickly remediating any MWICS that do arise to their clients. Third, regulators (and in particular the SEC) need to re-evaluate the incentives, or lack thereof, for firms to report MWICS in their initial internal control reports rather than as part of a restatement filing. In this latter regard, it may be that regulators need to strengthen the penalties associated with identifying MWICS at the time of restatements, rather than on the 10-K report. Fourth, using the AHP within the internal control context can provide more granularity for a firm’s management to make decisions on how to proceed with corrective action. More specifically, a firm’s management can use the AHP to determine which types of MWICS are more detrimental to different aspects of a firm’s performance and then remediate the most serious types of weaknesses that the company is reporting. Research examining the association between internal control systems and firm performance has expanded significantly since the passage of SOX in 2002 because of the disclosure requirements related to material weaknesses in internal control systems (MWICS). The research contained in the current study confirms the findings of previous researchers concerning a negative association between MWICS and firm performance. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 222 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 However, based on a large and contemporary sample, it also extends the prior research by showing that the weighted aggregation of MWICS types that are reported will have an impact on performance that is not clearly defined through the use of a binary variable. The fact that a very small percentage of firms report MWICS in their internal control reports filed with the SEC (Rice et al., 2015), coupled with the findings that most MWICS are identified by external auditors (Bedard & Graham, 2011), suggest that management recognizes the fact that disclosing any MWICS is likely to be detrimental to a firm’s performance. The findings from the current study also provide insight into why many firms report MWICS in a restatement filing with the SEC rather than in their initial 10-K (Rice et al., 2015). As with all empirical studies, our study has limitations, of which three seem most notable. First, since all MWICS measures are based on firms that have MWICS, our sample is limited to assessing internal control strength based on firms that report MWICS. Second, we are limited to examining the time period from 2004 forward because of the SOX-related data availability. Third, and as noted in the introduction, internal control systems include more than just controls over financial reporting. They also include controls that promote the efficiency and effectiveness of operations and controls that ensure compliance with laws and regulations (COSO, 2013). Although beyond the scope of the current study, links between operations and financial reporting have been made via measures of financial performance (Cheng et al., 2013; Feng et al., 2015) and future research could examine whether an internal control system index could be developed that includes operational aspects of internal control. Another interesting avenue for future research could examine how the different categories of criteria impact firm performance. While prior research has looked at individual types of MWICS, it has not as yet categorized them as we did within the current study and this type of categorization could add additional depth to the growing research in this area. Additionally, further analysis on differences between auditor and manager understanding of the relationship between controls and performance could provide insights into this area of research. As a final mention, future research could examine how the indices are related to the risk factors identified by companies within their 10-Ks. IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 223 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 REFERENCES Amponsah, C.T. (2011). Application of multi-criteria decision making process to determine critical success factors for procurement of capital projects under public-private partnerships. International Journal of the Analytic Hierarchy Process, 3(2), 107-129. dDoi: https://doi.org/10.13033/ijahp.v3i2.121 Asare, S. K. & Wright A. (2013). The effect of type of internal control report on users’ confidence in the accompanying financial statement report. Contemporary Accounting Review, 29(1), 152–175.dDoi: https://doi.org/10.1111/j.1911-3846.2011.01080.x Ashbaugh-Skaife, H., Collins, D.W. & Kinney, W.R. (2007). The discovery and reporting of internal control deficiencies prior to SOX-mandated audits. Journal of Accounting and Economics, 44(1-2), 166–192. doi: https://doi.org/10.1016/j.jacceco.2006.10.001 Ashbaugh-Skaife, H., Collins, D.W., Kinney, W.R. & LaFond, R. (2008). The effect of SOX 404 internal control deficiencies and their remediation on accrual quality. The Accounting Review, 83(1), 217–250. doi: https://doi.org/10.2308/accr.2008.83.1.217 Balsam, S., Jiang, W. & Lu, B. (2014). Equity incentives and internal control weaknesses. Contemporary Accounting Review, 31(1), 178–201. doi: https://doi.org/10.1111/1911-3846.12018 Bedard, J.C. & Graham L. (2011). Detection and severity classifications of Sarbanes- Oxley section 404 internal control deficiencies. The Accounting Review, 86(3), 825–855. doi: https://doi.org/10.2308/accr.00000036 Bedard, J.C., Hoitash, R., Hoitash, U. & Westermann, K. (2012). Material weakness remediation and earnings quality: a detailed examination by type of control deficiency. Auditing: A Journal of Practice & Theory, 31(1), 57–78. doi: https://doi.org/10.2308/ajpt-10190 Beneish, M.D., Billings, M. & Hodder, L. (2008). Internal control weaknesses and information uncertainty. The Accounting Review, 83(3), 665–703. doi: https://doi.org/10.2308/accr.2008.83.3.665 Carhart, M.M. (1997). On persistence in mutual fund performance. Journal of Finance, 52(1), 57–82. Chen, J., Chan, K.C., Dong, W. & Zhang, F. (2017). Internal control and stock price crash risk: evidence from China. European Accounting Review, 26(1), 125–152. doi: https://doi.org/10.1080/09638180.2015.1117008 Cheng, M., Dhaliwal, D. & Zhang, Y. (2013). Does investment efficiency improve after the disclosure of material weaknesses in internal control over financial reporting. Journal of Accounting and Economics, 56(1), 1–18. doi: https://doi.org/10.1016/j.jacceco.2013.03.001 IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 224 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 Cheng, Q., Goh, B.W., & Kim, J.B. (2018). Internal control and operational efficiency. Contemporary Accounting Research, 35(2), 1102–1139. doi: https://doi.org/10.1111/1911-3846.12409 Committee of Sponsoring Organizations of the Treadway Commission (COSO) (2013). Internal Control – Integrated Framework. American Institute of Certified Public Accountants. Darrough, M., Huang, R. & Zur, E. (2018). Acquirer internal control weaknesses in the market for corporate control. Contemporary Accounting Research, 35(1), 211–244. doi: https://doi.org/10.1111/1911-3846.12366 D’Mello, R., Gao, X. & Jia, Y. (2017). Internal control and internal capital allocation: evidence from internal capital markets of multi-segment firms. Review of Accounting Studies, 22(1), 251–287. doi: https://doi.org/10.1007/s11142-016-9377-8 Daniel, K. & Titman S. (1997). Evidence on the characteristics of cross sectional variation in stock returns. The Journal of Finance, 52(1), 1–33. doi: https://doi.org/10.3386/w5604 Dechow, P.M. & Dichev, I. D. (2002). The quality of accruals and earnings: The role of accrual estimation errors. The Accounting Review, 77(s-1), 35–59. doi: https://doi.org/10.2308/accr.2002.77.s-1.35 Doyle, J. T., Ge, W., & McVay S. E. (2007a). Determinants of weaknesses in internal control over financial reporting. Journal of Accounting and Economics, 44(1-2), 193– 223. doi: https://doi.org/10.1016/j.jacceco.2006.10.003 Doyle, J.T., Ge, W. & McVay S. (2007b). Accruals quality and internal control over financial reporting. The Accounting Review, 82(5), 1141–1170. doi: https://doi.org/10.2308/accr.2007.82.5.1141 Fama, E.F. & French K.R. (1992). The cross-section of expected stock returns. Journal of Finance, 47(2), 427–465. doi: https://doi.org/10.1111/j.1540-6261.1992.tb04398.x Fama, E.F. & French K.R. (2008). Average return, b/m, and share issues. The Journal of Finance, 63(6), 2971–2995. doi: https://doi.org/10.1111/j.1540-6261.2008.01418.x Fama, E.F. & MacBeth J.D. (1973). Risk, return, and equilibrium: empirical tests. The Journal of Political Economy, 81(3), 607–636. doi: https://doi.org/10.1086/260061 Feng, M., Li, C., McVay, S.E. & Skaife, H. (2015). Does ineffective internal control over financial reporting affect a firm’s operations? Evidence form Firms’ Inventory Management. The Accounting Review, 90(2), 529–557. doi: https://doi.org/10.2308/accr- 50909 Francis, J., LaFond, R., Olsson, P. & Schipper, K. (2005). The market pricing of accruals quality. Journal of Accounting and Economics, 39(2), 295–327. doi: https://doi.org/10.1016/j.jacceco.2004.06.003 https://doi.org/10.2308/accr-50909 https://doi.org/10.2308/accr-50909 IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 225 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 Gao, X. & Jia, Y. (2016). Internal control over financial reporting and safeguarding of corporate resources: Evidence from the value of cash holdings. Contemporary Accounting Research, 33(2), 783–814. doi: https://doi.org/10.1111/1911-3846.12164 Ge, W. & McVay, S. (2005). The disclosure of material weaknesses in internal control after the Sarbanes-Oxley act. Accounting Horizons, 19(3), 137–158. doi: https://doi.org/10.2308/acch.2005.19.3.137 Ge, W., Koester, A. & McVay, S. (2017). Benefits and costs of Sarbanes-Oxley Section 404(b) exemption: Evidence from small firms’ internal control disclosures. Journal of Accounting and Economics, 63(2), 358–384. doi: https://doi.org/10.1016/j.jacceco.2017.01.001 Gordon, L.A. & Wilford, A. (2012). An analysis of multiple consecutive years of material weaknesses in internal control. The Accounting Review, 87(6), 2027–2060. doi: https://doi.org/10.2308/accr-50211 Hammersley, J.S., Myers, L.A. & Shakespeare, C. (2008). Market reactions to the disclosure of internal control weaknesses and to the characteristics of those weaknesses under section 302 of the Sarbanes Oxley Act of 2002. Review of Accounting Studies, 13(1), 141–165. doi: https://doi.org/10.1007/s11142-007-9046-z Keane, M.J., Elder, R.J. & Albring, S.M. (2012). The effect of the type and number of internal control weaknesses and their remediation on audit fees. Review of Accounting and Finance, 11(4), 377–399. doi: https://doi.org/10.1108/14757701211279178 Koester, A., Lim, S.C. & Vigeland, R.L. (2015). The effect of tax-related material weakness in internal controls on the market valuation of unrecognized tax benefits. The Journal of the American Taxation Association, 37(1), 129–155. doi: https://doi.org/10.2308/atax-50952 Li, C., Peters, G., Richardson, V.J. & Watson, M.W. (2012). The consequences of information technology control weaknesses on management information system: The case of Sarbanes-Oxley Internal Control Reports. MIS Quarterly, 36(1), 179–204. doi: https://doi.org/10.2307/41410413 Loughran, T. & Wellman, J.W. (2011). New evidence on the relation between the enterprise multiple and average stock returns. Journal of Financial and Quantitative Analysis, 46(6), 1629–1650. doi: https://doi.org/10.1017/s0022109011000445 Mao, J. & Ettredge, M. (2016). Internal control deficiency disclosures among Chinese reverse merger firms. Abacus, 52(3), 441–472. doi: https://doi.org/10.1111/abac.12087 McNichols, M.F. (2002). Discussion of the quality of accruals and earnings: the role of accrual estimation errors. The Accounting Review, 77(s-1), 61–69. doi: https://doi.org/10.2308/accr.2002.77.s-1.61 IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 226 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 Ogneva, M., Subramanyam, K.R. & Raghunandan, K. (2007). Internal control weakness and cost of equity: Evidence form SOX section 404 disclosures. The Accounting Review, 82(5), 1255–1297. doi: https://doi.org/10.2308/accr.2007.82.5.1255 Ossadnik, W., Schinke, S. & Kaspar, R.H. (2016). Group aggregation techniques for Analytic Hierarchy Process and Analytic Network Process: A comparative analysis. Group Decision and Negotiation, 25(2), 421-457. doi: https://doi.org/10.1007/s10726-015-9448-4 Rezee, Z., Espahbodi, R., Espahbodi, P. & Espahbodi, H. (2012). Firm characteristics and stock price reaction to SOX 404 Compliance. Abacus, 48(4), 473–501. doi: https://doi.org/10.1111/j.1467-6281.2012.00377.x Rice, S.C. & Weber, D.P. (2012). How effective is internal control reporting under SOX 404? Determinants of the (non-) disclosure of existing material weaknesses. Journal of Accounting Research, 50(3), 811–843. doi: https://doi.org/10.1111/j.1475- 679x.2011.00434.x Rice, S.C., Weber, D.P. & Wu, B. (2015). Does SOX 404 have teeth? Consequences of the failure to report existing internal control weaknesses. The Accounting Review, 90(3), 1169–1200. doi: https://doi.org/10.2308/accr-50974 Saaty, T.L. (1980). The Analytic Hierarchy Process. New York: McGraw-Hill. Securities and Exchange Commission (SEC) (2003). Final rule: Management's report on internal control over financial reporting and certification of disclosure in exchange act periodic reports, 33-8238. Washington, D.C. http://www.sec.gov/rules/final/33-8238.htm Weiss, D. (2014). Internal controls in family-owned firms. European Accounting Review, 23(3), 463–482. doi: https://doi.org/10.1080/09638180.2013.821814 https://doi.org/10.1007/s10726-015-9448-4 IJAHP Article: Wilford, Bodin, Gordon/Using the Analytic Hierarchy Process to assess the impact of internal control weaknesses on firm performance International Journal of the Analytic Hierarchy Process 227 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.709 Appendix A Internal Control Weakness Classifications from Audit Analytics Accounting personnel resources, competency/training (ACT) Accounting personnel resources, competency and training MWICS that result from a lack of required skills or experience will be included in this category. Ethical or compliance issues with personnel (ECI) This category of MWICS will include any deficiencies related to personnel complying with policies or ethical standards, committing fraudulent acts or intentionally misrepresenting financial reports. Senior management competency, tone, reliability issues (SMI) This category of MWICS is reserved for issues that are related to senior management conduct. Insufficient or non-existent internal audit function (INI) This category of MWICS will include any issues where the company states that the MWICS was related to an inadequate internal or non- existent internal audit function. Segregations of duties/design of controls (personnel) (SD) MWICS issues related to the segregation of duties (i.e., separating duties between different individuals) will be included in this category. Treasury Control Issues (TC) Treasury-related MWICS (i.e., cash receipts and cash disbursements) will be included in this category. Journal entry control issues (JEC) If an MWICS states that the issue is the result of deficiencies in the journal entry process, it will be included in this category. Non-routine transaction control issues (NTC) If the MWICS is tagged as the result of a non-routine process (i.e., acquisition, etc.), then it will be included in this category. Inadequate disclosure controls (ID) If there is an issue related to the disclosure of financial reporting information, the MWICS will be included in this category. Material and/or numerous auditor/YE adjustments (MA) If the MWICS is included because of a high number of auditor/manager proposed adjustments at year end, the MWICS will be included in this category. Untimely or inadequate account reconciliations (UAR) If untimely or inadequate account reconciliations are identified as the reason for the MWICS, it will be included in this category. Accounting documentation, policy and/or procedures (ADP) Any MWICS that is the result of inadequate documentation, policies or procedures should be included in this category. This is a category that will generally be checked whenever an MWICS is reported. Information technology, software, security & access issues (IT) If an MWICS is related to information technology issues associated with accounting and financial reporting, it will be included in this category. Restatement or nonreliance of company filings (RNF) This category is used for MWICS that result in the restatement of financial information. Restatement of previous 404 disclosures (RPD) This category is used for MWICS that result from a restatement of a prior 404 opinion. SAB 108 adjustments noted (SAB) This category is used for MWICS where a SAB 108 is used to correct financial balances related to accounting errors. Ineffective or understaffed audit committee (IUA) This is an MWICS category used when an audit committee does not exist, have the experience, or have the independence required through SOX. Ineffective regulatory compliance issues (IRC) This is an MWICS category that is used when regulatory requirements are not met. Management/Board/Audit Committee investigations (MI) If there is an internal investigation in progress that is related to accounting or financial reporting, this category is used. SEC or other regulatory investigations and/or inquiries (SI) This MWICS category is used when there is an SEC or other regulatory investigation underway. Scope/disclaimer of opinion or other limitations (SDL) This MWICS category is used when a company or auditor indicates that they could not audit the internal controls. *Source: Data Dictionary – Internal Controls, Audit Analytics.