MTABS Analyses of Chi-square Cross-tabulations at ABSEL Page 31 - Developments in Business Simulation and Experiential Learning, Volume 47, 2020 ABSTRACT Cross-tabulation with its accompanying chi-square statistical hypothesis test of independence is among the most commonly applied types of analysis. This popularity holds for research published in the conference proceedings of the Association for Business Simulation and Experiential Learning (ABSEL), Developments in Business Simulation and Experiential Learning (aka Bernie Keys Library, BKL). Recently, Dickinson (2019, 2017, 2016) has developed a procedure–Moves To And Beyond Significance (MTABS)– for measuring the reliability of that statistical test. In light of their popularity at ABSEL, the present study MTABS analyzes several ABSEL cross-tabulations, illustrating a spectrum of results. KEYWORDS: Cross-tabulation, chi-square test of independence, statistical significance, MTABS INTRODUCTION In social sciences research generally, cross-tabulation and the usually accompanying chi-square test of independence are one of the most common types of analysis. In the specific realm of marketing research, the popularity of cross-tabulations is widely attested to: • “A cross-tabulation...is easily the most widely used data analysis technique in marketing research...” (Iacobucci & Churchill 2010, p. 362) • “Cross-tabulation is widely used in commercial marketing research...” (Malhotra 2012, p. 452) • “Among the most common types of bivariate analysis in marketing practice is the cross-tabulation of two nominal variables.” (Feinberg, Kinnear, & Taylor 2013, p. 427) This popularity is the case, too, for research at the Association for Business Simulation and Experiential Learning (ABSEL). Recently, a procedure for measuring the reliability of this common analysis has been introduced: Moves To And Beyond Significance (MTABS, Dickinson 2019, 2017, 2016). Illustrating a range of MTABS results, this study applies that procedure to selected cross-tabulations published in ABSEL’s proceedings, Developments in Business Simulation and Experiential Learning (aka Bernie Keys Library, BKL). As just recognized, MTABS is not peculiar to business simulations or experiential learning, but the (un) reliability of ABSEL-common chi-square cross-tabulations should be of interest. A corollary purpose served is toward the establishment of norms for MTABS analyses. THE CHI-SQUARE TEST AND STATISTICAL SIGNIFICANCE Often accompanying published cross-tabulations is the chi-square (χ2) test of independence of the two variables comprising the cross-tabulation (Malhotra 2012, pp. 454-455; Guenther 1965, pp. 187-193; Siegel 1956, pp. 104-111). Statistical hypothesis tests, including the chi-square test of independence, invariably culminate in a p-value (and, perhaps, relatedly a confidence interval). The p-value is the basis on which the tested null hypothesis of independence is rejected or not rejected. Needless to say, the p-value is of central, vital importance to the widespread practice of statistical hypothesis testing. In cross-tabulations a “statistically significant” test is taken as support that the two variables comprising the rows and columns, respectively, of the table are not independent, i.e., they are related. The most widely endorsed statistical significance level is 0.05 (Everitt 2002, p. 345; Malhotra 2015, p. 313; Sahai & Khurshid 2002, p. 215; Smith 2014, pp. 18-19). However, published studies often report significant p-values as being less than 0.01 and less than 0.10 in addition to being less than 0.05. One of the few conditions for the accuracy of the chi-square statistic is that the expected frequency in each of the cross- tabulation cells be at least five (Aaker et al. 2013, p. 431; Feinberg, Kinnear, & Taylor 2013, p. 429; Yates 1934, p. 217). For all of the results reported here this condition holds. HOW MTABS WORKS MTABS analysis is a complement to the usual chi-square hypothesis test of independence in a cross-tabulation. As a form of sensitivity analysis, its purpose is to provide a measure of the reliability of that statistical test. From the original cross-tabulation, MTABS successively moves single observations between pairs of cells comprising the cross-tabulation. Moving TO significance, MTABS identifies the two cells where moving a single observation from one to the other achieves the greatest decrease in the p- value. With that move made, MTABS repeats the process for the revised cross-tabulation. The progression is the same for moving the p-value BEYOND significance, only the pair of cells identified is the one that achieves the greatest increase in the p-value. MTABS Analyses of Chi-Square Cross-Tabulations at ABSEL John R. Dickinson University of Windsor MExperiences@bell.net Page 32 - Developments in Business Simulation and Experiential Learning, Volume 47, 2020 From the p-value of the original cross-tabulation MTABS determines the number of observations that must be moved to achieve a p-value less than, say, 0.05 and the number of observations that must be moved to achieve a p-value greater than or equal to, say, 0.10. Any levels as specified by the researcher, though, may be used. The prescribed resultant of an MTABS analysis is to report: • the chi-square p-value for the original table • MTABS moves TO α=0.05 % (n) • MTABS moves BEYOND α=0.10 % (n) where n is the number of moved observations to decrease the p-value to less than 0.05 or to increase the p-value to greater than or equal to 0.10 and % is the number of such moved observations as a percent of the total number of observations. INCIDENCE OF CROSS-TABULATIONS AND CHI-SQUARE TESTS IN THE BERNIE KEYS LIBRARY (BKL) The popularity of the chi-square test of independence in the social sciences generally also holds for the ABSEL Proceedings. A September 2019 search of the Bernie Keys Library for “chi-square” yielded 46 works. (A very few of these are other than tests of independence.) A search for “cross tabulation” yielded three works, all three among the 46 found for “chi- square”. Since a given study often reports more than one chi-square statistical test, the number of tests conducted is considerably greater than the number of published works. A SPECTRUM OF MTABS RESULTS Below are illustrated a range of MTABS results across four studies published in the BKL. Nothing to See Here...Or Is There? A cross-tabulation published by Wolfe & Jackson (1989, Table 2, p. 34) may be used to illustrate an MTABS analysis. Students participated in a business policy simulation in which, for the “experimental” participants, the advertising expenditure effect on product demand had been intentionally manipulated to incorrectly have little demand effect. Following the competition, students were asked to describe the features of a mix of algorithms comprising the game. With respect to the advertising algorithm and to the remaining algorithms, students’ responses were classified into perfect (i.e., accurate) perception, describing an algorithmic feature that did not exist (a Type I error), and the statement of the nonexistence of an algorithmic feature that did exist (a Type II error). The original cross-tabulated data are presented in Table 1A and the cross- tabulated data where two observations have been MTABS-moved (both from cell [1,2] to cell [1,3]) are presented in Table 1B. The students’ 380 descriptions revealed no significant difference in their perceptions of the incorrect and correct algorithms: p=0.23808. (Lowest expected frequency is 6.71.) Yet with the MTABS movement of just 2 of the 380 observations (0.5263 percent) among the six cells of the cross-tabulation, the p-value can be reduced to 0.03869, at which a conclusion of statistical significance can be drawn. (The lowest expected frequency in that cross-tabulation is 6.98.) TABLE 1A ORIGINAL DATA TABLE 1B MTABS MOVEMENT OF TWO OBSERVATIONS PERCEPTION PERCEPTION ALGORITHM Perfect Type I Type II Perfect Type I Type II Advertising Algorithm 31 10 10 31 8 12 Remaining Algorithms 199 90 40 199 90 40 Chi-square = 2.8703 p-Value = 0.2381 Min. expected frequency = 6.71 Chi-square = 6.5045 p-Value = 0.03869 Min. expected frequency = 6.98 Source: Wolfe & Jackson (1989) Page 33 - Developments in Business Simulation and Experiential Learning, Volume 47, 2020 Williams (1987, Table 3, p. 236) cross-tabulated 352 professors’ beliefs as to whether there is one best method to teach the capstone business policy course or no one best method with whether or not the professor employed a simulation game. The p-value for that 2x2 cross-tabulation is 0.41114. (Lowest expected frequency of 13.18.) Yet with the MTABS movement of less than two percent of the 352 observations (1.7045 percent or 6 observations), the p-value can be reduced to less than 0.01 (p=0.00378). (The lowest expected frequency in that cross-tabulation is 12.45.) For both of these examples, an original finding of “clear insignificance” can be made to be “clear significance” with the MTABS movement within the cross-tabulation of very few observations. Can’t Get Much More Certain...Well, Not So Fast Wellington, Hutchinson, & Faria (2014) investigated the awareness of ABSEL (aware~unaware) among business faculty across seven specific disciplines within business. 927 faculty members responded to their 2013 e-mail survey. The variables of awareness and discipline comprised a 2x7 cross-tabulation (Table 3, 2013, p. 4). The null hypothesis that awareness is independent of discipline was tested with the usual chi-square test of independence. The calculated chi-square statistic equals 25.3919 with an accompanying p-value of 0.00029. That p-value is less than the lowest criterion for rejection of the null hypothesis in published works: 0.001. That is, the null hypothesis of independence would be rejected by virtually all researchers and managers and rejected with an extremely low probability of that rejection being incorrect. With the MTABS movement of just 6 observations (0.6472 percent of 927 total observations), the resulting p-value of 0.05481 exceeds 0.05. With the movement of 8 observations (0.8630 percent), the resulting p-value of 0.15331 exceeds 0.10. And with the movement of 9 observation (0.9709 percent) the p-value of 0.23282 exceeds 0.20, the probability of a Type I error that virtually no research or manager would deem significant. In sum, with the MTABS movement among the 14 cross-tabulation cells of less than 1 percent of the total observations, the original p-value of 0.00029–virtual certainty–becomes unacceptably uncertain. (The lowest expected frequency in the original cross-tabulation is 9.83 and across all of the MTABS moved observations no cell ever has an expected frequency less than 5.) Can Go Either Way Above are presented examples of ABSEL cross-tabulations originally of “clear” statistical significance that can be made clearly insignificant and vice versa with the movement of very few observations within the table. As noted earlier, commonly reported benchmarks for concluding statistical significance are 0.05 and 0.10. For a 2x2 cross- TABLE 2 MTABS ANALYSIS OF “CLEARLY” SIGNIFICANT (p≺0.05) ABSEL CROSS-TABULATIONS MTABS Source Dimen- sions Total Observations Original p-Value p-Value ≺ 0.05 p-Value ≺ 0.10 [1] 5 x 3 686 0.00113 0.06519 1.31% (9) * 0.11564 1.60% (11) [2] 2 x 3 376 0.02290 0.06985 0.80% (3) 0.13177 1.33% (5) [3] 7 x 3 1076 0.02401 0.05173 0.19% (2) 0.13197 0.46% (5) [1] [2] [3] Chang et al. (2005), Table 4, p. 375 Gentry et al. (2002), Table 3, p. 70 Wellington & Faria (2004), Table 2, p. 328 * It takes the MTABS movement of 9 observations (1.31% of the 686 total observations) to move the p-value from the original p=0.00113 to p=0.06519. Page 34 - Developments in Business Simulation and Experiential Learning, Volume 47, 2020 tabulation Gentry et al. (2002, Table 2, low-high overall grade~low-medium/high confidence level) found an equivocal p-value, i.e., greater than 0.05 and less than 0.10, of 0.07261. With the movement of just two observations (0.5141 percent of 389 total observations), the result may be seen as more conclusively significant (p=0.03639). With the movement of two different observations the result may be seen as more conclusively insignificant (p=0.13405). A SAMPLER OF MTABS ANALYSES AT ABSEL MTABS results of three types have been explained above. Tables 2 and 3 present MTABS analyses of additional cross- tabulations found in the BKL. In Table 2 the original p-values are considerably less than 0.05. In Table 3 the original p-values are considerably greater than 0.10. (See Table 2 on Page 33) DISCUSSION MTABS is a procedure that describes the reliability of the chi-square test of independence for a given cross-tabulation table; it is a form of sensitivity analysis providing complementary information to the test. It may be thought unnecessary. Why not carry out the chi-square test and leave it at that? That position, though, would dismiss the use of the chi-square test itself and statistical inference generally. Once the data are placed into a cross-tabulation, why not make use of cell counts and/or row total counts and/or column total counts or their corresponding percentages and leave it at that? Contrarily, in light of its ubiquity researchers obviously do find the chi-square test informative. Describing the reliability of that test enhances the information of the test. In a different context, suppose two variables are found to have a correlation of, say, 0.16. The research/manager could just leave it at that. Contrarily, what researcher/manager would not want to know whether the p-value for the test of H0:ρ = 0 (Kutner, Nachtsheim, & Neter (2004, p. 84) is, say, p=0.02 or p=0.20? Too, there is no role for rationalizing how the perturbations effected by MTABS might literally take place in the real world. The purpose of sensitivity analysis generally is to determine, “...how the final outcome of an analysis changes as a function of varying one or more input parameters...” and it is “...frequently carried out to assess the impact of different...scenarios on the results of a study.” (Sahai & Khurshid 2002, p. 244) The purpose of sensitivity analysis is to investigate What if. Any explanation of “why” or “how” is simply in the varying of the parameters. ABSEL members who publish cross-tabulations and the accompanying chi-square test might consider complementing those results with an MTABS analysis. TABLE 3 MTABS ANALYSIS OF “CLEARLY” INSIGNIFICANT (p≺0.10) ABSEL CROSS-TABULATIONS MTABS Source Dimen- sions Total Observations Original p-Value p-Value ≺ 0.05 p-Value ≺ 0.10 [1] 2 x 2 232 0.16720 0.03516 1.72% (4) * 0.08124 0.86% (2) [2] 2 x 2 500 0.21086 0.04064 1.00% (5) 0.08409 0.60% (3) [3] 2 x 2 578 0.58234 0.03290 1.38% (8) 0.08185 1.04% (6) [1] [2] [3] Gentry et al. (2002), Table 2, p. 70, low/mod~high by first test Wellington & Faria (2004), Table 4, p. 328, users~former users Wellington, Hutchinson, & Faria (2014), Table 2, p. 3, 2013, users~former users * It takes the MTABS movement of 4 observations (1.72% of the 232 total observations) to move the p-value from the original p=0.16720 to p=0.03516. Page 35 - Developments in Business Simulation and Experiential Learning, Volume 47, 2020 Aaker, David A., Kumar, V., Leone, Robert P., & Day, George S. (2013). Marketing Research, Eleventh Edition. Hoboken, NJ: John Wiley & Sons, Inc. ISBN: 978-1- 118-15663-6 Chang, Jimmy, Choi, Ka-Fai, Moon, Karen Ka-leung, Chan, Priscilla, Chan, Theresa LK, & To, Chester (2005), Teaching practices: a cluster analysis of students in Hong Kong. Developments in Business Simulation and Experiential Learning, Volume32, 373-380. Dickinson, John R. (2019). An obvious basis for MTABS analysis of 2x2 cross-tabulations that doesn’t work. Decision Sciences Institute 50th Annual Conference. Dickinson, John R. (2017). An anomaly in MTABS analysis of cross-tabulations. In Zhao, Xuying (Proceedings Cooordinator, Proceedings, Decision Sciences Institute 48th Annual Conference, 1289154-1-. Dickinson, John R. (2016). Two observations between ‘truth’ and meh: a complement to cross-tabulation chi-square hypothesis testing. Iin Blackhurst, Jennifer (Proceedings Coordinator), Proceedings, 2016 Decision Sciences Institute 47th Annual Conference. Everitt, B. S. (2002). The Cambridge Dictionary of Statistics. Cambridge, UK: Cambridge University Press. ISBN: 0-521-81099 Feinberg, Fred, Kinnear, Thomas C., & Taylor, James R. (2013). Modern Marketing Research, Second Edition. Mason, OH: South-Western, Cengage Learning. ISBN -13: 978-1-133-18896-4; ISBN-10: 1-133-18896-6 Gentry, James W., Burns, Alvin C., Dickinson, John R., Putrevu, Sanjay, Chun, Seungwoo, Hongyan, Yu, Williams, Laura, Bare, Thomas, & Gentry, Ruth Ann (2002). Managing the curiosity gap does matter: what do we need to do about it? In Developments in Business Simulation and Experiential Learning, Vol. 29. 67-73. Guenther, William C. (1965). Concepts of Statistical Inference. New York: McGraw-Hill Book Company. Library of Congress: 64-8276 Iacobucci, Dawn a&Churchill, Gilbert A., Jr. (2010). Marketing Research: Methodological Foundations, Tenth Edition. Mason, OH: South-Western Cengage Learning. Student Edition ISBN 13: 978-0-324-35995 -4; Student Edition ISBN 10: 0-324-35995-0 Kutner, Michael H., Nachtsheim, Christopher J., & Neter, John (2004). Applied Linear Regression Models, Fourth Edition . New York: McGraw-Hill/Irwin). ISBN: 0- 07-301344-7 Malhotra, Naresh K. (2015). Essentials of Marketing Research. Boston: Pearson. ISBN:978-0-13-706673-5 Malhotra, Naresh K. (2012). Basic Marketing Research, Fourth Edition. Upper Saddle River, NJ: Pearson. ISBN: 978 -0-13-25448 Sahai, Hardeo & Khurshid, Anwar (2002). Pocket Dictionary of Statistics. New York: McGraw-Hill. 0-07-251693- 3 Siegel, Sidney (1956). Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill Book Company. ISBN: 07-057348-4 Smith, Gary (2014). Standard Deviations. New York: Overlook Duckworth, Peter Mayer Publications, Inc.. ISBN: 978-1-4683-0920-1 Wellington, William J. & Faria, A. J. (2004). ABSEL awareness among business school faculty. Developments in Business Simulation and Experiential Learning, Volume 31, 325-331. Wellington, William J., Hutchinson, David, & Faria, A. J. (2014). ABSEL awareness amongst business school faculty: a ten year update study. Developments in Business Simulation and Experiential Learning, Volume 41, 1-7. Williams, Edgar L., Jr. (1987). “Business Simulation in the Policy Course: A Survey of American Assembly of Collegiate Schools of Business,” Developments in Business Simulation & Experiential Exercises, Volume 14, 235-238. Wolfe, Joseph & Jackson, Ralph (1989). An investigation of the need for valid business game algorithms. Developments in Business Simulation & Experiential Exercises, Volume 16, 31-36. Yates, F. (1934). Contingency tables involving small numbers and the χ2 test. Supplement to the Journal of the Royal Statistical Society, Vol. 1, No.2, 217-235. REFERENCES