External Reviewers’ Influence on Citations of Articles Published in Pedagogical Business Journals: Holding Acceptance Rate Constant Arshad Alam Prairie View A&M University Reginald L. Bell Prairie View A&M University An analysis of covariance with a 4 x 3 factorial design was used to compare the citation means of 560 articles published in 28 pedagogy journals. The independent variables were four business fields: accounting, economics and finance, general management, and marketing; three levels of external reviewers were used as the other independent variable. The dependent variable was the number of citations of articles. We found that the means differed on the main effects of four levels of business fields and three levels of external reviewers when acceptance rate was a covariate. The number of citations increase as the number of reviewers increases from one to two. INTRODUCTION In recent decades, there has been a proliferation of journals in all business fields. This is both a reflection of the amount of research being conducted as also of the pressure on members of the academia to publish for tenure requirement. Moreover, business schools are continually striving to find adequate ways to maintain their accreditation with the Association for the Advancement of Collegiate Schools of Business (AACSB) by justifying faculty qualifications. Scholarly Academic qualified faculty must publish intellectual contributions pertinent to the mission of the school. In AACSB accredited business schools which have teaching missions as their primary focus there is increased pressure on the faculty to publish a portion of their research in pedagogical journals. This has in recent decades contributed to a growth in the number of pedagogical publications in different business fields. There is a good amount of literature that shows bias in the selection of articles business journals publish and that hierarchies exist based on Carnegie classifications of institutions within top journals in various fields (Bell & Chong, 2010; Chong & Bell, 2012; Oswald, 2008). Expert opinions are an important indicator of how a business journal is perceived and ranked (Mingers & Harzig, 2007). Stochastic models have shown the inevitable obsolescence and decline in the citation rate of an article over time (Mingers & Burrell, 2006). The key driver of articles’ citations can be the management journal itself (Mingers & Xu, 2010). Submissions to good quality journals are always peer reviewed or refereed since peer review is expected to serve as a quality check (Gorman, 2008). In other words it must be evaluated by an editor/an editorial panel or reviewed by a group of peers considered to be knowledgeable about the subject matter. 18 American Journal of Management Vol. 16(3) 2016 Weaknesses in the peer review system have been highlighted by many. External review may not be appropriate in cases for example where the study deals with something new for the reviewer may lack the necessary expertise (Armstrong, 1997). Furthermore, there may be a bias in editorial decision making which Miller (2006, p. 426) describes as “Dissensus.” When there is disagreement among the reviewers evaluating a paper the editor will adjudge the negative review as being more valid than a positive review, and this behavior might be at the center of the peer review problem. This is why the peer review process warrants further scrutiny in an attempt to balance tensions among authors, editors and referees (Bedeian, 2004). It causes delay in an article’s publication and potentially injects bias in the selection process. Reviewers do not necessarily agree with one another (Cicchetti, 1991), and many of them spend less than adequate time and do not do quality reviews (Armstrong, 1997). Peters and Ceci (1982) conducted a fraud study. They submitted 12 already published articles, nine of which were papers they sent to the same highly regarded journals which had previously published that same paper. The papers, however, included fictitious names and affiliations. The paper were rejected by eight editors on the basis of negative reviews from 16 of 18 reviewers (p. 188). Despite the drawback of the review system the review system is an integral part of the acceptance process in all established journals and is in the interest of all parties- journals, reviewers and authors (Phaman, 2014). Authors generally find the reviewers’ suggestions to be reasonable (MacNealy, Speck & Clements, 1994) and it tends to improve quality through feedback; additionally, the competition to ensure acceptance by the reviewers motivates authors to produce their best work (Shatz, 1996). As always in any research, the quality of articles published is of much greater importance than the number of articles published on any subject. Article quality is however, a nebulous and a debatable issue and for which no metric has been developed. Neither the journal in which it is published automatically guarantees quality nor does the quality of an article automatically ensures quality of a journal (Chow et al, 2007, Smith 2004). An article’s citations is an accepted measure of that article’s influence on the field since citations made by others is indicative of its influence on scholars in the field (Podsakoff et al 2005). In his famous February 5, 1676 letter to Robert Hooke, Isaac Newton made famous the words, “if I have seen further it is by standing on the shoulders of Giants”. Citations also enable authors to acknowledge the contributions of others to their own scholarship (Van Noorden, 2010). While “celebrity” authors appear to influence the number of citations of articles published by top journals (Bell, 2013; Fei, Chong, & Bell, 2015; Fei & Bell, 2013) it would be absurd to argue that a paper cited 5,000 times over a 10-year period is not having an influence on the field. To the contrary, it would be more appropriate to argue that a paper with 5000 citations is groundbreaking, perhaps seminal to a field. Recently, Google Scholar has started to indicate an article’s most recent citations, the author’s h- index. There is now broad support from the academic community for the h-index—an h-index of 5 means an author has 5 articles cited at least 5 times each; Google Scholar is an earnest alternative to other types of indexes that measures a journal’s impact (Delgado-López-Cózar & Cabezas-Clavijo, 2013; Harzing, & Van Der Wal, 2009). Google Scholar is now a good source of data for academic institutions to rely on when compiling information on faculty productivity (Harzing, 2015; Harzing, & Van Der Wal, 2008; Harzing, & Van Der Wal, 2009). If the number of citations are being used as a proxy for the quality of a published article and its contribution to a field of study then it is worthwhile to explore the relationship that may exist between citations and other measures of a journal’s acceptance process. Various factors have been identified by Seglen (1997) as positively impacting citations that include: • articles in the English language; • generalist areas rather than specific applied disciplines; • review articles rather than original research; • cutting edge articles with a short lifespan; • longer rather than shorter articles; • and articles regarding established rather than emerging disciplines. American Journal of Management Vol. 16(3) 2016 19 Other studies have also identified factors affecting citations and though there are commonalities among them they tend to differ depending on the field of study. Importantly, the probability of being cited may depend on factors, many of which may be difficult to incorporate in a study. It may depend, for example, on a ‘halo’ effect which suggests that well recognized authors are more likely to be cited because of their name recognition than other less well known authors (Bell, 2013; Peters & Raan, 1994) and mediocre research receives much more attention when published in top journals (Starbuck, 2005); the perceived quality of journal per se in which an article is published may also have a bearing on the likelihood of it being cited. Being difficult to measure with any precision it is difficult to incorporate such factors in a study. Furthermore, the inclusion of factors which cannot be measured with precision is likely to lead to misidentification of the individual effect of well-defined factors such as External Reviews or Acceptance Rate, considering that most of the factors in citation studies are interrelated (Peters & Raan, 1994). Such factors are hence not included in this study. The motivation of the study stems from the fact that, while there exists no theory of the editorial process, editors would surely like to maximize the impact of the articles published in their journals (Laband & Piette, 1994) and identification of factors affecting impact would clearly be relevant. Testing the empirical data (the actual citations of articles across pedagogic business journals) is a better measure than the prevailing circular arguments which dominate the reasoning of many who assert that a top journal is a proxy for the quality of an article because it is published in that top journal, or vice versa (Chow, Haddad, Singh & Wu, 2007; Smith, 2004). Macdonald and Kam (2008, p. 596) described the circular reasoning in journal (article) quality arguments the following way: Once a journal is on one list of quality journals, it is fairly likely to appear on other lists of quality journals. It is a quality journal because it is on a list of quality journals. Conversely, journals not on the lists are likely to remain excluded…One characteristic of quality journals in Management Studies is that authors from top business schools publish in them, but then, which are top business schools is often determined by publication in quality journals. Research Purpose Specifically, this study analyzes data from business journals with a teaching focus, from different disciplines, to analyze the nature of relationships that may exist between citations and other variables of interest such as External Reviews and Acceptance Rate, and to determine whether these relationships vary by discipline. The study was restricted to journals dedicated to pedagogy. Consequently, many of the factors identified by others, such as generalist versus applied disciplines, are not relevant for our study. While there have been numerous citation studies in the last two decades thanks to availability of data, to the best of our knowledge there have been none so far devoted exclusively to pedagogical business journals. We must emphasize here that our study was not motivated to determine the quality of journal per se based on the number of citations of articles published in it but rather to see whether the process by which journals decide to accept or reject an article submitted for publication has any relationship on that article’s citation rate and whether these relationships differ by fields of study. DATA SOURCES AND DESCRIPTIVE STATISTICS A total of 28 publications were considered for analysis. Data was collected on nature of review i.e. review code (blind peer, double blind peer or editorial), field (accounting, economics, management and marketing), Acceptance Rate (as self-reported by journals), number of External Reviews (one, two and three or more reviewers) and the number of citations (Google data) for the period 1989 to 2010. The publication period was categorized in three categories: published in 1989 or before, 1990 to 1999 and 2000 and later. Table 1 provides a list of the publications with relevant citation statistics. Figure1 provides the list of journals and mean of citations by business field. 20 American Journal of Management Vol. 16(3) 2016 TABLE 1 LIST OF PEDAGOGY JOURNALS AND CITATIONS RANKED BY MEANS WITH STANDARD DEVIATIONS Citations Journal Title Mean N Std. Deviation % of Total Sum Academy of Management Learning & Education 374.450 20 422.3069 24.0% Economics of Education Review 352.300 20 137.3433 22.6% Issues in Accounting Education 201.450 20 632.9288 12.9% Journal of Management Education 136.400 20 77.3750 8.8% Journal of Marketing Education 110.150 20 17.9569 7.1% Journal of Accounting Education 93.550 20 39.9124 6.0% Journal of Marketing for Higher Education 53.050 20 26.3348 3.4% Journal of Economic Education, The 47.800 20 22.8924 3.1% Marketing Education Review 31.800 20 15.8500 2.0% Accounting Education: an international journal 30.600 20 9.2872 2.0% Journal of Hospitality & Tourism Education 19.300 20 17.5202 1.2% Journal of Financial Education 16.750 20 12.8468 1.1% Journal of Leadership Education 12.000 20 11.2156 0.8% Advances in Accounting Education 11.100 20 4.3878 0.7% Academy of Educational Leadership Journal 10.500 20 4.4069 0.7% Journal of Entrepreneurship Education 9.900 20 9.7165 0.6% Journal of Strategic Management Education 7.750 20 11.5707 0.5% Global Perspectives on Accounting Education 7.250 20 7.4189 0.5% Journal of Business Ethics Education 6.400 20 6.3528 0.4% Journal of Legal Studies Education 6.400 20 9.5057 0.4% Journal for Advancement of Marketing Education 3.550 20 3.2683 0.2% Australasian Journal of Economics Education 3.500 20 4.6848 0.2% Journal of Economics and Finance Education 3.250 20 4.4233 0.2% Journal of Economics and Economic Education Research 2.500 20 5.2666 0.2% Advances in Financial Education 1.950 20 2.0384 0.1% International Journal of Pluralism and Economics Education 1.900 20 3.1103 0.1% Journal of Human Resources Education 1.250 20 3.5522 0.1% Operations Management Education Review 1.100 20 3.1271 0.1% Total 55.639 560 173.7875 100.0% American Journal of Management Vol. 16(3) 2016 21 FIGURE 1 PEDAGOGY JOURNALS IN BUSINESS FIELDS WITH MEANS ACCOUNTING JOURNALS (Mean = 65.008) Accounting Education: an international journal Advances in Accounting Education Global Perspectives on Accounting Education Issues in Accounting Education Journal of Accounting Education ECONOMICS AND FINANCE JOURNALS (Mean =56.777) Advances in Financial Education Australasian Journal of Economics Education Economics of Education Review International Journal of Pluralism and Economics Education Journal of Economic Education, The Journal of Economics and Economic Education Research Journal of Economics and Finance Education Journal of Financial Education GENERAL MANAGEMENT JOURNALS (Mean = 71.797) Academy of Educational Leadership Journal Academy of Management Learning & Education Journal of Business Ethics Education Journal of Entrepreneurship Education Journal of Human Resources Education Journal of Legal Studies Education Journal of Management Education Journal of Strategic Management Education Operations Management Education Review MARKETING JOURNALS (Mean = 12.133) Journal for Advancement of Marketing Education Journal of Hospitality & Tourism Education Journal of Leadership Education Journal of Marketing Education Journal of Marketing for Higher Education Marketing Education Review Table 2 provides frequency data relating to type of review policy, number of external reviewers and the respective fields of study. TABLE 2 TYPE OF REVIEW POLICY, NUMBER OF EXTERNAL REVIEWERS AND THE RESPECTIVE FIELDS OF STUDY Frequency Percent Cumulative Percent Review Policy Blind 480 85.7 85.7 Blind Peer 20 3.6 89.3 Double Blind Peer 60 10.7 100.0 Total 560 100.0 External Reviewers One 80 14.3 14.3 Two 260 46.4 60.7 Three or More 220 39.3 100.0 Total 560 100.0 Field Accounting 100 17.9 17.9 Economics and Finance 160 28.6 46.4 Management 200 35.7 82.1 Marketing 100 17.9 100.0 Total 560 100.0 PEDAGOGY JOURNALS IN BUSINESS FIELDS 22 American Journal of Management Vol. 16(3) 2016 HYPOTHESES AND DATA ANALYSIS The cross tab analysis of External Reviews and Field (Table 3) indicates a significant difference between the count and expected count values which suggests a lack of independence between the two variables. TABLE 3 EXTERNAL REVIEWERS * FIELD CROSS-TABULATION Fields T ot al A cc ou nt in g Fi na nc e & E co no m ic s M an ag em en t M ar ke tin g External Reviewers One Count 0 40 20 20 80 Expected Count 14.3 22.9 28.6 14.3 80.0 % within External Reviewers 0.0% 50.0% 25.0% 25.0% 100.0% % of Total 0.0% 7.1% 3.6% 3.6% 14.3% Two Count 60 80 80 40 260 Expected Count 46.4 74.3 92.9 46.4 260.0 % within External Reviewers 23.1% 30.8% 30.8% 15.4% 100.0% % of Total 10.7% 14.3% 14.3% 7.1% 46.4% Three or More Count 40 40 100 40 220 Expected Count 39.3 62.9 78.6 39.3 220.0 % within External Reviewers 18.2% 18.2% 45.5% 18.2% 100.0% % of Total 7.1% 7.1% 17.9% 7.1% 39.3% Total Count 100 160 200 100 560 Expected Count 100.0 160.0 200.0 100.0 560.0 % within External Reviewers 17.9% 28.6% 35.7% 17.9% 100.0% % of Total 17.9% 28.6% 35.7% 17.9% 100.0% To test for the independence of External Reviews and Field we propose the following hypothesis. H1: External Reviews and Field are not independent. The appropriate test to examine for significant association between two categorical variables from a single population is the Chi Square test. Applying the Chi Square test for independence to the data resulted in a p-value of 0.0 (Table 4) which indicates that the number of External Reviews is not independent of the field or discipline i.e. the two variables are related. The relative frequency differs mostly between finance and economics (expected count 22.9, observed count 40) and management journals (expected count 78.6, observed count 100). American Journal of Management Vol. 16(3) 2016 23 TABLE 4 CHI-SQUARE TESTS Value df Asymp. Sig. (2-sided) Pearson Chi-Square 53.259a 6 0.000 Likelihood Ratio 65.778 6 0.000 Linear-by-Linear Association 0.098 1 0.755 N of Valid Cases 560 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 14.29. Finance and economics journals account for 50% of the four fields using only one reviewer, as opposed to management journals which accounts for 45.5% of the four business fields that use three or more reviewers in the review process. What is more, notice in Figure 2, that the mean citations for management journals (71.797) is larger than the mean citations for finance and economics journals (56.777). This is an important indication that the number of reviewers in the review process merits further investigation. FIGURE 2 MEAN CITATIONS FOR PEDAGOGY ARTICLES BY BUSINESS FIELD We next propose that Period and Field are not independent. H2: Period and Field are not independent. Results of the cross tab analysis (Table 5) and of the Chi Square test (Table 6) once again confirms the hypothesis and we conclude that the variables are related. The Pearson Chi Square test is significant, 24 American Journal of Management Vol. 16(3) 2016 with p-value of 0. The relative frequency differs mostly between accounting journal (expected count 23.6, observed count 37), finance and economics journals (expected count 37.7, observed count 45) and management journals (expected count 152.9, observed count 172). Therefore, accounting journals account for 28%, econ and finance journals account for 34.1% of articles published 1999 and before, respectively, among the four business fields. On the other hand, management journals account for 40.2% of the articles published 2000 or after, among all four business fields. This is an indication that the accounting and finance journals are older than management journals, and that these fields have had a pedagogic journals longer than the management field. TABLE 5 PERIOD * FIELD CROSS-TABULATION Field T ot al A cc ou nt in g Fi na nc e & E co no m ic s M an ag em en t M ar ke tin g Period 1999 and Before Count 37 45 28 22 132 Expected Count 23.6 37.7 47.1 23.6 132.0 % within Period 28.0% 34.1% 21.2% 16.7% 100.0% % of Total 6.6% 8.0% 5.0% 3.9% 23.6% 2000 and After Count 63 115 172 78 428 Expected Count 76.4 122.3 152.9 76.4 428.0 % within Period 14.7% 26.9% 40.2% 18.2% 100.0% % of Total 11.3% 20.5% 30.7% 13.9% 76.4% Total Count 100 160 200 100 560 Expected Count 100.0 160.0 200.0 100.0 560.0 % within Period 17.9% 28.6% 35.7% 17.9% 100.0% % of Total 17.9% 28.6% 35.7% 17.9% 100.0% TABLE 6 CHI-SQUARE TESTS Value df Asymp. Sig (2-sided) Pearson Chi-Square 22.159a 3 0.000 Likelihood Ratio 22.340 3 0.000 Linear-by-Linear Association 13.105 1 0.000 N of Valid Cases 560 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 23.57. A priori, we believe Acceptance Rate is linked to the quality of the published articles. The Acceptance Rate of a journal is suggestive of how stringent is the review process of the journal. A lower Acceptance Rate, indicating a rigorous quality check, is likely to lead to better quality articles which in turn would result in higher citations. We thus propose the following hypothesis. H3: Acceptance Rate and number of citations are negatively related. Tables 7 and 8 present the results of the ANOVA analysis which indicates that Acceptance Rate indeed has a negative effect on number of citations and is significant at 1 % level. American Journal of Management Vol. 16(3) 2016 25 TABLE 7 ANOVAa Model Sum of Squares df Mean Square F Sig. 1 Regression 917877.877 1 917877.877 32.081 0.000b Residual 15965097.259 558 28611.285 Total 16882975.136 559 a. Dependent Variable: Citations b. Predictors: (Constant), Acceptance Rate TABLE 8 COEFFICIENTSa Model Unstandardized Coefficients Standardized Coefficients t Sig. B Std. Error Beta 1 (Constant) 141.248 16.719 8.448 0.000 Acceptance Rate -337.612 59.607 -.233 -5.664 0.000 a. Dependent Variable: Citations Data on number of citations by Field is reported in the Table 9 below followed by the non-significant Levene’s test (> .05) for homogeneity of variances (Table 10). TABLE 9 CITATIONS BY FIELD N Mean Std. Deviation Std. Error 95% Confidence Interval for Mean Lower Bound Accounting 100 68.790 287.4550 28.7455 11.753 Economics and Finance 160 53.744 124.0124 9.8040 34.381 Management 200 56.615 174.4700 12.3369 32.287 Marketing 100 43.570 41.0900 4.1090 35.417 Total 560 55.639 173.7875 7.3439 41.214 Model Fixed Effects 174.0873 7.3565 41.189 Random Effects 7.3565a 32.228a a. Warning: Between-component variance is negative. It was replaced by 0.0 in computing this random effects measure. TABLE 10 TEST OF HOMOGENEITY OF VARIANCES Levene Statistic df1 df2 Sig. 2.116 3 556 0.097 The groups are independent based on the non-significant Levene’s test, p-value of 0.097. Do the number of citations differ among fields? Considering that the study is limited to pedagogical journals of different field, we do not expect to see a significant difference in the number of citations by field. We therefore propose the following hypothesis. 26 American Journal of Management Vol. 16(3) 2016 H4: Number of citations do not differ by field. The ANOVA results (Table 11) indicates the differences between groups i.e. the different fields is not significant, p-value of 0.783. Post Hoc test results corroborate the ANOVA findings where none of the group differences are significant. However, when we control for Acceptance Rate as a covariate then both Field and Acceptance Rate turn out to be significant at the 5 % level (Table 12). The result that citations differ by Field (discipline) once we control for other influencing variables would suggest that comparisons of citations across Fields for purposes of evaluation of intellectual contribution of authors is inappropriate. TABLE 11 ANOVA Sum of Squares df Mean Square F Sig. Between Groups 32626.187 3 10875.396 0.359 0.783 Within Groups 16850348.949 556 30306.383 Total 16882975.136 559 TABLE 12 TESTS OF BETWEEN-SUBJECTS EFFECTS Dependent Variable: Citations Source Type III Sum of Squares df Mean Square F Sig. Partial Eta Squared Corrected Model 1141765.004a 4 285441.251 10.064 0.000 0.068 Intercept 2242443.913 1 2242443.913 79.064 0.000 0.125 Acceptance Rate 1109138.817 1 1109138.817 39.106 0.000 0.066 Field 223887.127 3 74629.042 2.631 0.049 0.014 Error 15741210.131 555 28362.541 Total 18616584.000 560 Corrected Total 16882975.136 559 a. R Squared = .068 (Adjusted R Squared = .061) Next, we control for the covariate Acceptance Rate and estimate the marginal means which are reported in Table 13. TABLE 13 ESTIMATES Dependent Variable: Citations Field Mean Std. Error 95% Confidence Interval Lower Bound Upper Bound Accounting 65.008a 16.852 31.907 98.110 Economics & Finance 56.777a 13.323 30.608 82.947 Management 71.797a 12.153 47.925 95.670 Marketing 12.133a 17.575 -22.390 46.655 a. Covariates appearing in the model are evaluated at the following values: Acceptance Rate = .2536. American Journal of Management Vol. 16(3) 2016 27 Further, pairwise comparisons of the mean differences indicate that differences in Fields: marketing field differs from all the fields, but none of the other field differ from each other (Table 14). The F result of the Univariate test also confirms this (Table 15). Thus, differences exists only with reference to the Marketing discipline (Group 4); differences among the other groups, namely accounting, accounting & finance, and management are not significant. TABLE 14 PAIRWISE COMPARISONS Dependent Variable: Citations (I) Field (J) Field Mean Difference (I-J) Std. Error Sig.b 95% Confidence Interval for Differenceb Lower Bound Upper Bound Accounting Econ/finance 8.231 21.496 0.702 -33.992 50.455 Management -6.789 20.848 0.745 -47.739 34.161 Marketing 52.876* 24.224 0.029 5.294 100.458 Economics & Finance Accounting -8.231 21.496 0.702 -50.455 33.992 Management -15.020 17.968 0.404 -50.314 20.274 Marketing 44.645* 22.165 0.044 1.108 88.182 Management Accounting 6.789 20.848 0.745 -34.161 47.739 Econ/finance 15.020 17.968 0.404 -20.274 50.314 Marketing 59.665* 21.932 0.007 16.585 102.745 Marketing Accounting -52.876* 24.224 0.029 -100.458 -5.294 Econ/finance -44.645* 22.165 0.044 -88.182 -1.108 Management -59.665* 21.932 0.007 -102.745 -16.585 Based on estimated marginal means *. The mean difference is significant at the .05 level. b. Adjustment for multiple comparisons: Least Significant Difference (equivalent to no adjustments). TABLE 15 UNIVARIATE TESTS Dependent Variable: Citations Sum of Squares df Mean Square F Sig. Partial Eta Squared Contrast 223887.127 3 74629.042 2.631 0.049 0.014 Error 15741210.131 555 28362.541 The F tests the effect of Field. This test is based on the linearly independent pairwise comparisons among the estimated marginal means. We next examine whether the number of citations differ by the number of External Reviews. As mentioned in the previous section external reviewers add value to the quality of the article by providing critical feedback to the authors. In the selection of the articles, editors depend on the advice of the external reviewers (Coronel & Opthof, 1999). A priori we believe therefore that greater number of reviews would positively impact the quality of the published article and hence the number of citations. We propose therefore the following hypothesis. H5: Number of External Reviews positively affects the number of citations. Tables 16 through 18 present the results of the analysis which indicate that the differences in citations based on the number of reviewers are significant. 28 American Journal of Management Vol. 16(3) 2016 TABLE 16 ESTIMATES Dependent Variable: Citations External Reviewers Mean Std. Error 95% Confidence Interval Lower Bound Upper Bound One 9.775 19.335 -28.203 47.753 Two 56.146 10.725 35.080 77.212 Three or more 71.718 11.659 48.817 94.620 TABLE 17 PAIRWISE COMPARISONS Dependent Variable: Citations (I) External Reviewers (J) External Reviewers Mean Difference (I-J) Std. Error Sig.b 95% Confidence Interval for Differenceb Lower Bound Upper Bound One Two -46.371* 22.110 0.036 -89.800 -2.942 Three or more -61.943* 22.578 0.006 -106.292 -17.595 Two One 46.371* 22.110 0.036 2.942 89.800 Three or more -15.572 15.842 0.326 -46.689 15.545 Three or more One 61.943* 22.578 0.006 17.595 106.292 Two 15.572 15.842 0.326 -15.545 46.689 Based on estimated marginal means *. The mean difference is significant at the .05 level. b. Adjustment for multiple comparisons: Least Significant Difference (equivalent to no adjustments). TABLE 18 UNIVARIATE TESTS Dependent Variable: Citations Sum of Squares df Mean Square F Sig. Partial Eta Squared Contrast 225226.212 2 112613.106 3.766 0.024 0.013 Error 16657748.923 557 29906.192 The F tests the effect of External Reviewers. This test is based on the linearly independent pairwise comparisons among the estimated marginal means. Further, pairwise comparisons (Table 17) suggest that the difference between number of citations from one reviewer process and that from two and three or more reviewer processes is significant; the difference in citations between a two reviewer and three or more reviewer process is however, not significant for pedagogic business journals. It appears that two External Reviews serves as a threshold value. Having more than two External Reviews does not add to the number of citations but having less than two has a bearing on number of citations. Carrying the analysis further we add Acceptance Rate as a covariate i.e., we wished to see if External Review is significant after removing the effect of Acceptance Rate. The results are given in Table 19. Number of External Reviews turns out to be significant when Acceptance Rate is used as covariate. This suggests that both these factors uniquely affect the citation rate. American Journal of Management Vol. 16(3) 2016 29 TABLE 19 TESTS OF BETWEEN-SUBJECTS EFFECTS Source Type III Sum of Squares df Mean Square F Sig. Partial Eta Squared Observed Powerb Corrected Model 1135458.380a 3 378486.127 13.363 0.000 0.067 1.000 Intercept 1661798.535 1 1661798.535 58.673 0.000 0.095 1.000 Acceptance Rate 910232.168 1 910232.168 32.138 0.000 0.055 1.000 External Reviewers 217580.503 2 108790.251 3.841 0.022 0.014 .696 Error 15747516.756 556 28322.872 Observed Powerb Total 18616584.000 560 1.000 Corrected Total 16882975.136 559 a. R Squared = 0.067 (Adjusted R Squared = 0.062) b. Computed using alpha = 0.05; Dependent Variable: Citations Figure 3 presents the plot of the estimated Marginal Means of citations against the External Reviews with Acceptance Rate as a covariate. It is clearly evident that increasing the External Reviews from one to two clearly increases the number of citations while the increase in citations resulting from an increase of External Reviews from two to three is marginal and not significant (refer to aforementioned Table 17). FIGURE 3 MEAN CITATIONS FOR PEDAGOGY ARTICLES BY EXTERNAL REVIEWERS 30 American Journal of Management Vol. 16(3) 2016 DISCUSSION Citations per se do not represent the “truth embodied in an article or author” but is a measure of the ability of the article to influence the scholarship in the field (Johnson, 1997). There is a difference, however, between excellence and influence and the relationship between the two is uncertain (Beed & Beed, 1996). Further, it is plausible, that other authors may have cited an article only to criticize its arguments (Ayres & Vars, 2000). Citations can thus never be projected as an overall objective measure of quality. Nevertheless, to the extent that citation count is a measure of quality, it has become the most common proxy for quality. The aim of the study was to determine if there were meaningful differences among the citations of published articles in pedagogical business journals. While numerous factors may have a bearing on the likelihood of an article being cited, this study was limited to analyzing the impact of a few factors, namely, the period of its publication, the Field or discipline of study, Acceptance Rate, and number of External Reviews. To summarize, we found that number of External Reviews is not independent of the Field or discipline i.e., the discipline has a bearing on the number of External Reviews. In other words, some disciplines tend to employ more number of reviews. The number of citations, however, does not differ by Field when no other factors are taken into consideration. This is as expected and hypothesized since the study is limited to journals of pedagogy. We may make a conjecture here that this finding may not hold true if all journals, and not only pedagogical journals, are considered. Interestingly, when we control for Acceptance Rate as a covariate then both the Field (marketing differs from the other fields) and Acceptance Rate turn out to be significant at the 5 % level. The inference is clear-comparisons of citations across Fields are inappropriate. Acceptance Rate by itself has an effect on number of citations and is significant at 1 % level. This is only to be accepted since a lower acceptance rate implies a more demanding approval process leading to higher quality of articles being published and a resultant increase in citations. More important however, is the finding with respect to the number of external reviews which has practical implications for journal editors. As the pressure to publish increases, editors are faced with ever increasing submissions. Would adding more reviewers lead to better article quality and higher citations of published articles? Editors are however, also under time constraint and more reviews would likely delay the approval process. As hypothesized, number of External Reviews has a significant effect on citations when acceptance rate is held constant. However, pairwise comparisons suggest that more than two External Reviews does not add to the number of citations but having less than two has a bearing on number of citations. Review by two peers thus, seems to be sufficient to ensure ‘quality’ (as judged by citations) of published articles. It is clear therefore, that the pedagogic journals in all the business fields would prosper from a two-reviewer review process rather than using just one reviewer. In others words, “two heads are better than one.” Limitations A possible limitation of the study is that an article’s citations may vary for a variety of reasons. Citation density, i.e., the average number of references cited per article, is likely to vary significantly across disciplines. Some disciplines by the very nature of their field lend themselves to more citations than others. Additionally, if the existing pool of articles is larger in a particular field it can also lead to more citations. Future researchers conducting similar studies may try to take this into account by normalizing the data as suggested by Garfield (1999). This may not be a serious limitation to our study however, since our dataset is restricted to pedagogical journals. Another potential limitation is the implicit assumption of all studies trying to study the impact of External Reviews on citations is that the review is objective. But the review process can never be considered fully objective. Inspite of the blind process, bias may be injected by a reviewer being able to identify the author (Laband & Piette, 1994). There may be other unintended biases (Stanley, 2007) and to that extent the validity of any findings are affected. Finally, it bears noting that no review process can American Journal of Management Vol. 16(3) 2016 31 ensure quality. As pointed out by (Shugan, 2007) many great discoveries were initially rejected and the review process did not safeguard against faulty research being published. REFERENCES Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3, (1), 63-84. Ayres, I., & Vars, F. (2000). Determinants of citations to articles in elite law reviews. Journal of Legal Studies, 29, (1), 427-450. Bedeian, A.G. (2004). Peer review and the social construction of knowledge in the management discipline. Academy of Management Learning and Education, 3, (2), 198-216. Beed, C., & Beed, C. (1996). Measuring the quality of academic journals: The case of economics. Journal of Post Keynesian Economics, 18, (3), 369-396. Bell, R. L., & Chong, H. G. (2010). A caste and class among the relative frequency of faculty's publications: A content analysis of refereed business journals. Journal of Leadership, Accountability and Ethics, 8, (1), 65-89. Bell, R. L. (2013). Management journals and the celebrity researcher effect on tiers. Journal of Leadership, Accountability and Ethics, 10, (5), 44-62. Chong, H. G., & Bell, R. L. (2012). Does hierarchy exists among the refereed accounting journals? International Journal of Business and Public Administration, 9, (3), 60-77. Chow, C.W., Haddad, K., Singh, G., & Wu, A. (2007). On using journal rank to proxy for an article's contribution or value. Issues in Accounting Education, 22, (3), 411-427. Cicchetti D.V. (March, 1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14, (March), 119-186. Coronel, R., & Opthof, T. (1999). The role of reviewer in editorial decision-making. Cardiovascular Research, 43, (2): 261-264 Delgado-López-Cózar, E., & Cabezas-Clavijo, Á. (2013). Ranking journals: could Google Scholar Metrics be an alternative to journal citation reports and Scimago journal rank? Learned Publishing, 26, (2), 101-113. Fei, Q., & Bell, R. L. (2013). Marketing journal ranking, celebrity authors, and the diminishing quality gap. Mustang Journal of Management and Marketing, 3, 16-32. Fei, Q., Chong, H.G., & Bell, R.L. (2015). The diminishing influence of celebrity authors in a diversified world of accounting journals. Journal of Higher Education Theory and Practice, 15, (1), 37-57. Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161, (8), 979-980. Gorman, G.E. (2008). Can we be assured of quality scholarship in a peer review-less environment? Online Information Review, 32, (5), 553-556. Harzing, A.W., & Van Der Wal, R. (2008). Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics, 8, (1), 61–73. Harzing, A.W., & Van Der Wal, R. (2009). A Google Scholar h‐index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60, (1), 41-46. Harzing, A.W.K. (2015). Journal quality list, 55th Edition, http://www.harzing.com/ Johnson, D. (1997). Getting noticed in economics: the determinants of academic citations. American Economist, 41, (1), 43-52 Laband, D.N., & Piette, M.J. (1994). Does the ‘Blindness’ of peer review influence manuscript selection efficiency? Southern Economic Journal, 60, (4), 896–906. Macdonald, S., & Kam, J. (2008). Quality journals and gamesmanship in management studies. Management Research News, 31, (8), 595-606. MacNealy, M. S., Speck, B. W., & Clements, N. (1994). Publishing in technical communication journals from the successful author's point of view. Technical communication, 41, (2), 240-259. 32 American Journal of Management Vol. 16(3) 2016 Miller, C. C. (2006). Peer review in the organizational and management sciences: Prevalence and effects of reviewer hostility, bias, and Dissensus. Academy of Management Journal, 49, (3), 425-431. Mingers, J., & Burrell, Q.L. (2006). Modeling citation behavior in management science journals. Information Processing & Management, 42, (6), 1451-1464. Mingers, J., & Harzing, A. (2007). Ranking journals in business and management: A statistical analysis of the Harzing data set. European Journal of Information Systems, 16, (4), 303-316. Mingers, J., & Xu, F. (2010). The drivers of citations in management science journals. European Journal of Operational Research, 205, (2), 422-430. Oswald, A.J. (2008). Can we test for bias in scientific peer-review? IZA Discussion Paper No. 3665 (August) Peters, D. P., & Ceci, S.J. (1982). Peer-review practices of psychological journals: the fate of published articles, submitted again. Behavioral and Brain Sciences, 5, (2), 187-255. Peters, H.P.F., & Van Raan, A.F. J. (1994). On Determinants of citation scores: A case study in chemical engineering. Journal of American Society for Information Science, 45, (1), 39-49. Phaman, D. (2014) Editorial: A primer for peer review. Journal of Markets & Morality, 17, (1), 1–7. Podsakoff, P. M., Mackenzie, S. B., Bachrach, D. G., & Podsakoff, N. P. (2005). The influence of management journals in the 1980s and 1990s. Strategic Management Journal, 26, (5), 473-488. Seglen, P.O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, (7079), 498-502. Shatz, D. (October, 1996). Is peer review overrated? The Monist, 79, (4), 536-563. Shugan, S.M. (2007). The editor’s secrets. Marketing Science, 26, (5), 589-595. Smith, S. D. (2004). Is an article in a top journal a top article? Financial Management, 33, (4), 133-150. Stanley, C.A. (2007). When counter narratives meet master narratives in the journal editorial-review process. Educational Researcher, 36, (1), 14-24. Starbuck, W.H. (2005). How much better are the most-prestigious journals? The statistics of academic publication. Organization Science, 16, (2), 180-200. Van Noorden, R. (2010). Metrics: A profusion of measures. Nature, 465, 864-866. Retrieved from http://www.nature.com/news/2010/100616/full/465864a.html American Journal of Management Vol. 16(3) 2016 33