ABSEL Research -- A Perspective on the Quality of the Research Presented in the Proceedings Page 15 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 ABSEL RESEARCH -- A PERSPECTIVE ON THE QUALITY OF THE RESEARCH PRESENTED IN THE PROCEEDINGS Steven Gold Rochester Institute of Technology stevengold@saunders.rit.edu Peter Markulis SUNY-Geneseo markulis@Geneseo.edu Daniel Strang SUNY-Geneseo strang@Geneseo.edu ABSTRACT This study builds on and extends the work that has been done in the past by researchers who were attempting to assess the quality of research published in the venues most commonly used by ABSEL members. It reviews and categorizes the work that other researchers have done in their efforts to assess the quality of ABSEL research. In this study, an extensive reference analysis is performed on the articles that appear in the ABSEL Proceedings from 1984 to 2013. Reference analysis is also performed on the articles appearing in Simulation & Gaming from 1996 to 2012. A set of metrics used by SCImago is used to put into perspective the quality of articles in the ABSEL Proceedings. Inferences are made about the likely value for the ABSEL Proceedings based on several key indicators of journal quality used in SCImago but not available for ABSEL Proceedings. The paper then uses reference analysis to compare a “typical” paper from the ABSEL Proceedings to a “typical” paper from Simulation & Gaming. Finally, the authors provide some suggestions to raise the perceived quality of the ABSEL Proceedings. INTRODUCTION How do those who are asked to make important career decisions (i.e., deans, provosts, and presidents) about faculty determine the quality of their scholarly work? While there is no agreed upon standard, most schools are compelled in the tenure and promotion evaluation process to use some kind of rating system as a way to assess the scholarly activity of their faculty. Publishing in scholarly journals is one obvious choice. The question then becomes, “What is a scholarly journal?” To determine that, many schools rate and/or rank journals based on some kind of criteria. Schools use a variety of methods to determine that criteria, including acceptance rate of journal, perceived reputation of journal, relevance of journal to the field or discipline, journal readership, citation analysis, multi- authorship, producer analysis as well as others (McWilliams, Siegel and Van Fleet, 2005; Lazaroiu, 2009; Svensson, 2008; Bell and Chong, 2010; Manton and English, 2007) . In one Association to Advance Collegiate Schools of Business International (AACSB) survey, deans who said that they used scholarly journals as a measure of faculty performance were then asked how they rated the scholarship of those journals (Hynes & Stretcher, 2005). Table 1 shows the relative importance deans attributed to the seven factors mentioned. As Table 1 shows, three factors were clearly the most important criteria to deans when evaluating journal quality: the peer review process (83.33%), the journal's professional reputation (62.22%), and the journal's acceptance rate (54.44%). Over the years, a number of scholarly works have looked at the research conducted in several of the research venues utilized by ABSEL researchers. One principal purpose was to get some reading on the quality of research being fostered by the association. The purpose of this article is to extend and update some of the work that has been done to measure the quality of ABSEL research. This article reports the results of a very extensive review of the research works published in the ABSEL Proceedings and the research works published in the journal most closely associated with ABSEL, Simulation & Gaming. To put the results in a perspective that allows comparison with mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:strang@Geneseo.edu Page 16 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 other scholarly publications, the authors made use of data from SCImago (2007). SCImago Journal & Country Rank is a portal (http://www.scimagojr.com/) that includes the journals and country scientific indicators developed from the information contained in the Scopus® database (Elsevier B.V.). For this article the authors use the following key indicators:  The H Index;  the SJR;  the references per article; and,  the percent of external citations. A group of journals that are in disciplines closely related to the focus of most ABSEL research is selected and comparisons between the key indicators for the selected journals and ABSEL Proceedings and Simulation & Gaming are made. Since results for some of the key measures used by SCImago are not available for the ABSEL Proceedings, the authors provide inferences about these using regression analyses. LITERATURE REVIEW Earlier works addressing the quality of the research done in the ABSEL Proceedings can be categorized by their mode of emphasis. The works can be grouped based upon: 1. research design utilized in the works; 2. analysis of learning outcomes; 3. use of content analysis; 4. profiles of award-winning papers; 5. focus on sub-disciplines; and, 6. reference analysis and citation indexing. 1. Research Design. As early as 1977, Keys (1977), conducted some preliminary analysis that focused on the basic research methodology underlying the works of papers published in the ABSEL Proceedings. Keys’ work was the first of a long list of ABSEL efforts designed to provide insights into the quality of ABSEL research. Incidentally, the name ABSEL Proceedings is being used at this point for convenience. A fuller discussion of a number of issues relating to the use of this name is included later in this paper. Faria and Wolfe (1999) and Faria (2000) provided additional insights into key elements of the quality of research. Faria (2000) delineates three key factors: 1. Post-hoc analysis of factors associated with high game performance or correlates of simulation performance, 2. effectiveness of games in strategic management courses, and, 3. focus on what business games teach for the ABSEL history as dimensions of research efforts. In 2001, Faria broadens the focus to a scholarly group that includes not only ABSEL contributors, but researchers contributing to Simulation & Gaming as well. Gosen and Washbush (2004) add a third measure which focuses on the TABLE 1. RESPONDENTS’ RATINGS OF JOURNAL EVALUATION FACTORS (IN PERCENTAGES)* NOTE: 5 = very important; 1 = not important *SOURCE: Hynes, Geraldine E; Stretcher, Robert H. (2005). “Business Schools' Policies Regarding Publications in Electronic Journals,” Journal of Education for Business, 81.2, 73-80. Level of Importance Acceptance Rate Cabell’s listing Sponsoring organization Peer review Issue frequency No. of cita- tions Professional reputation 5 54.44 30.00 13.33 83.33 2.22 28.89 62.22 4 23.33 22.22 20.00 12.22 3.33 12.22 18.89 3 14.44 17.78 22.22 2.22 17.78 15.56 4.44 2 2.22 12.22 16.67 1.11 26.67 18.89 4.44 1 2.22 14.44 21.11 1.11 43.33 20.00 7.78 No response 3.33 3.33 6.67 0.00 6.67 4.44 2.22 Page 17 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 instruments used to measure the effectiveness of simulations and experiential exercises as teaching/learning systems. Several researchers have been more critical of ABSEL research than Faria, Gosen, and Washburn. Wolfe (1981) and Wolfe and Crookall (1998) are perhaps the most critical and indicate that simulation/gaming literature has progressed relatively slowly in regard to the validity of its various research practices. As early as 1981, Wolfe set the standard of Campbell and Stanley (1963) to push for more rigorous research designs in ABSEL scholarship. While Wolfe stressed the need for carefully constructed research designs, Patz et al. (1999) believed that a more results- oriented approach was more important. Patz dismissively states that “pedagogical research is aimed at producing results—not at advancing the current fashionable and almost fleeting notions of an elite at a local university or editorial staff of a widely distributed journal.” Butler et al. (1985) took a more analytical approach and performed an audit of the papers published in the annual proceedings of ABSEL. Butler and his colleagues recorded the incidence of some of the elements of a “rigorous research design” and defined three key elements of a “rigorous research design” as randomization, control, and treatment. In 2006, Howard, et al. revisited the issue and performed an updated audit of ABSEL articles using the key elements of design as defined in the Butler (1985) paper. 2. Analysis of Learning Outcomes. Butler et al. (1985) performed an audit of the papers published in the annual proceedings of ABSEL, recording and measuring learning outcomes based upon Bloom’s (1956) Taxonomy. In 2006, Howard et al. revisited the issue and performed an updated audit of ABSEL articles based upon educational outcomes again using Bloom’s criteria as operationalized in the Butler (1985) paper. Anderson and Lawton (1997) make a compelling argument for papers addressing learning to specify learning objectives and they resourcefully provide their own classification scheme. 3. Use of Content Analysis. In a seminal, article Krippendorf (1980) advances the concept of content analysis and describes it as “a research technique of making replicable and valid inferences from data to their context.” Krippendorf indicates, “One form of content analysis that is utilized by researchers is classified as semantical content analysis.” Several ABSEL researchers have looked at published ABSEL works and subjected them to a form of semantical content analysis. In 1986, Goosen advances the taxonomy and suggests categories of simulation and experiential research such as design & construction, simulation administration, innovative uses, descriptive uses, and a general category. In 2001, Howard and Strang (2001) conducted a content analysis using incidence of keywords in titles of papers in the ABSEL Proceedings. In a further study, Howard and Strang (2003) performed a content analysis using the key words that had emerged as significant in the earlier study. In the second study, all occurrences of keys words, both in the titles and the text of the papers, were used as the key indicators of content. 4. Profiles of Award-Winning Papers. As a measure of quality with a far narrower scope, Markulis et al. (1991) studied procedures and protocols of the papers that had been selected by ABSEL as award winners. In their work, the authors considered elements of research design, and learning outcomes that have been used by other ABSEL researchers, as well several other dimensions. While not germane to the selection of award- winning papers ipso facto, they note that the blind review process greatly contributed to the quality of the ABSEL Proceedings. 5. Focus on Sub-disciplines. Several evaluative studies have used the specific sub- discipline as the key metric for purposes of analysis. These studies performed research on simulations and experiential exercises categorized by the sub-discipline within the broad umbrella of business such as international business, accounting, marketing, finance, OB, production, policy (Butler, 1999, Graf, 1999). In 1999, Kelley focused on experiential exercises and recorded the articles in each of several sub-disciplines and also recorded the specific activity type such as role-playing, case, etc. 6. Reference Analysis. Several researchers have applied the idea of a reference analysis in its simplest form to assess the ABSEL Proceedings. In 1989, Markulis et al. considered the references per article as one measure by which to assess ABSEL scholarship. Using the Bernie Keys Library, Peach and Platt (2002) categorized the articles from the ABSEL Proceedings in terms of whether they contained references or not. In a later article, Peach and Platt (2007) reviewed selected articles from the ABSEL Proceedings to determine whether the references in the articles were “appropriate” or not. Although Markulis et al. and Peach and Platt can be credited with some preliminary work in reference analysis specifically as it applies to ABSEL research, the current study delves into these issues far more extensively and extends into several areas not covered heretofore. Concerns over Research Quality. Studies on the quality of research at ABSEL have pointed to a number of areas that need improvement. Research design was one of the significant concerns and recommendations were offered by several studies. The most recent study by Howard, et al. (2006) found that papers appearing during the first 15 years of ABSEL did not differ significantly from ABSEL papers during the most recent 15 years in terms of research design or their use of educational learning theory. The study found Page 18 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 that articles published in the ABSEL Proceedings did not employ either a strong research methodology or an effective educational paradigm to track learning and learning outcomes. This finding is consistent with the study by Anderson and Lawton (1997) who were also quite critical of the studies on the analysis of learning outcomes and argued that valid, reliable instruments to assess the “mastery” of learning are rare, and valid measures of “higher level” learning objectives are almost non-existent. But this is not only a critique of the ABSEL Proceedings, but is a broad criticism of the studies on the assessment of learning effectiveness. In terms of content analysis, Howard and Strang (2003) evaluated the patterns of emphasis of research over 29 years of ABSEL’s proceedings. The study found no significant changes in the type of research being evaluated in the ABSEL Proceedings. Focusing on sub-disciplines, Butler (1999) looked at ABSEL’s contributions to experiential exercises in the 1990s. The study looked at frequency counts of ABSEL Proceedings papers, classified by tier and track. Surprisingly, the study found no statistically significant trends in the sub-discipline areas being researched at ABSEL over a nine year horizon; except for the increasing number of papers on exercises in interactive sessions and the emergence of papers on multimedia. With respect to reference analysis, Peach and Platt (2002) presented their work in which they reported the number of articles published in the Bernie Keys Library categorized by whether references were included or not included. The study found two disturbing trends; including a decreasing percentage of papers providing references; and the reference lists were becoming shorter. Additional Issues. The articles mentioned above present the body of research that has been published to gain insights into the quality of the research published in the ABSEL Proceedings or, at least, some aspect of the quality of the research. The authors of several of these articles raised additional issues which although they do not fall into one of the categories listed above, are still noteworthy and warrant consideration by the ABSEL organization. For example, as early as 1989, Markulis et al. reported a high turnover rate of authors. They recorded the incidence of new authors (i.e., authors who had previously not published anything in the ABSEL Proceedings) at 52%. They offer several suggestions that may explain this pattern, but it would seem that they viewed the pattern as negative. Admittedly, this study did not focus on the “quality” or scholarly dimension, but it could be construed as a surrogate measure to ascertain the scholarship quality of ABSEL Proceedings. Perhaps, it is a dimension that others in research investigators might choose to focus on to update ABSEL’s status with respect to this dimension. Strang (2007) reported that one of the renowned authorities in the field of citation analysis, Eugene Garfield (1979), stated “the average scientific article contains approximately 15 citations.” He suggests that citation analysis may be a useful benchmark by which to assess the work of individual authors as well as the scholarship of the journal itself. Peach and Platt (2002) emphatically state in their recommendations, “ABSEL should make the nature and value of the Bernie Key’s Library a focal point of the annual call for papers.” They go on to assert, “The BKL makes it relatively easy for a writer on any topic pertinent to ABSEL’s fields of interest to review and reference relevant prior research.” Additionally, they state “ABSEL has a history of improving the quality of its accepted papers. Effective use of tools such as BKL will further improve the quality of research and submissions, and as a consequence improve the overall impact of ABSEL on its fields of interest.” It is in the context of the work that previous researchers have done, and particularly in response to the suggestions for needed additional research, that the current study was undertaken. METHODOLOGY The purpose of this study is to extend and update the work done on assessing the research quality of the ABSEL Proceedings. The approach taken to assess research quality is to apply reference analysis. To do this, a group of journals that are in disciplines closely related to the focus of most ABSEL research is selected and comparisons between the key indicators for the selected journals and ABSEL Proceedings and Simulation & Gaming are made. Since results for some of the key measures used by SCImago are not available for the ABSEL Proceedings, the authors provide inferences about these using the results of regression analysis. The authors reviewed each printed article in Developments in Business and Simulation Exercises, (hereafter simply referred to as ABSEL Proceedings), listed in Bernie Keys Library for the years 1984 to 2013. Two tallies were recorded. The first tally recorded total references for each article and the second tally recorded the number of references made to previously published works in the ABSEL Proceedings (see Table 2). Since the ABSEL organization has been associated with Simulation & Gaming (ABSEL’s official journal), a tally was also recorded for each article in Simulation & Gaming for the years 1996 to 2012. It should be noted that Simulation & Gaming has undergone some name changes. At its inception the Journal was called Simulation & Games: An International Journal of Theory, Practice and Research; but in March 1989, the name was changed to Simulation & Gaming: An International Journal of Theory, Practice and Research and in 2000, the name was changed to Simulation & Gaming: An Interdisciplinary Journal of Theory, Practice and Research. For the sake of simplicity, name changes will be ignored and the Journal will be referred as Simulation & Gaming (S & G) for this Page 19 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 paper. For S & G., a tally of total references and the references to articles previously published in S & G was recorded (see Table 3). CLARIFICATIONS, & PROVIOS Proceedings Names. The names of the ABSEL Proceedings have also undergone some evolutionary changes from 1974 until 2013. For the first six years of ABSEL, each of the annual proceedings was given a unique name as follows: 1974 Simulation games and experiential learning techniques: On the road to a new frontier 1975 Simulation games and experiential exercises in action 1976 Computer simulation and learning theory 1977 New Horizons in simulation games and experiential learning 1978 Exploring experiential learning: Simulations and experiential exercises 1979 Insights into experiential pedagogy. TABLE 2. RESULTS OF REFERENCE ANALYSIS FOR ABSEL PROCEEDINGS PUBLISHED FROM 1984 TO 2013. TABLE 3. RESULTS OF REFERENCE ANALYSIS FOR SIMULATION & GAMING FROM 1996 TO 2012. Page 20 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Beginning in 1980, ABSEL standardized the name of the annual proceedings to Developments in Business and Simulation Exercises. In this work, the label, ABSEL Proceedings, will be used to denote all works published as proceedings. Condensed Papers in Proceedings. Beginning in 1990, ABSEL began designating papers accepted for publication in the ABSEL Proceedings as either full papers or as condensed papers. Since, in many instances, the authors of condensed papers elected to provide minimal references, condensed papers were excluded from the analysis. Condensed papers are not labeled as condensed, so it was not possible in all cases to draw a clear-cut distinction between condensed papers and full papers. To operationalize the analysis in this study, it was decided that papers of four or less pages would be defined as condensed papers. References on Request or Full List of References on Request. In their work in 2002, Peach and Platt noted the incidence of published papers in the ABSEL Proceedings where the author (or authors) said that “references would be provided on request.” This “problem” first appeared in 1989, and in the years since 1990, there have been various numbers of papers published each year in the Proceedings stating that references would be provided on request, with the fewest being 1 published paper and the most being 11 (refer to Table 2). In addition, the authors of several of the papers during that period of time provided abbreviated sets TABLE 4. RESULTS FROM SCIMAGO BASED UPON SELECTION KEYWORDS Page 21 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 of references and indicated additional references would be provided on request. For this study, those papers that indicated references (full or partial) would be provided on request were earmarked and treated differently than those papers that may have, in fact, utilized no references. Thus, some papers had no references because the author indicated that references would be provided on request and some papers simply had no references. When determinations were made in terms of references per paper--in order not to bias the results--a judgment was made that the denominator for this calculation would exclude those papers that explicitly indicated that references would be provided on request. The reader will note that the practice of authors stating references would be provided on request ceased in 2001, as reported in Table 2. Silver Anniversary Papers. In 1999 ABSEL celebrated its 25thttt anniversary. A collection of “Silver Anniversary Papers” was included in the ABSEL Proceedings that year. Many of these papers were “invited” papers. Some had references and some did not. But because they were not subjected to the blind review process, they were all excluded from the analysis. Exclusions from Analysis of Simulation & Gaming Articles. A number of “specialty items” that appeared in Simulation & Gaming were excluded from analysis. The FIGURE 1. REFERENCES PER PAPER FOR ABSEL FROM 1984 TO 2013. 20152010200520001995199019851980 25 20 15 10 5 Year R e f. / p a p e r ABSEL Ref./paper vs Year Regression Analysis: Ref./paper versus Year The regression equation is Ref./paper = - 974 + 0.495 Year Predictor Coef SE Coef T P Constant -973.8 163.0 -5.97 0.000 Year 0.49475 0.08155 6.07 0.000 R 2 = 56.8% R 2 (adj) = 55.2% Page 22 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 list of excluded items includes—editorials, notes, reviews, ready-to-use-simulation, symposium papers (examples can be found in June 2002 and December 2002), reports, responses, communications, “in conversations”, and short topics (examples are found in September 1998 in the works by Ken Jones). In the tallying process, the ludography that appeared in at least one Simulation & Gaming paper was excluded. These were excluded because they were not subjected to the blind review process. ANALYSIS To put the results reported in Tables 2 and 3 into a broader perspective, the authors applied the SCImago data. SCImago reports extensive data for many scholarly journals and, additionally, assigns a value to each journal for several key indicators. Presumably, these key indicators represent some measure of the journal’s quality. One way to access information from the SCImago website is to select journal search and then to specify a search word. Since the ABSEL organization embodies LEARNING, and SIMULATION, each of these words was used to initiate searches. The keyword, LEARNING, yielded a total of 68 listings. Since at least one of the purposes was to establish an appropriate comparison with ABSEL work, it was decided to include only those listings considered to be relevant: FIGURE 2 REFERENCES PER PAPER FOR S & G FROM 1999 TO2012. 2010200520001995 70 60 50 40 30 20 10 Year R e f/ p a p e r S & G Ref/paper vs Year The regression equation is References per paper = - 1331 + 0.680 Year Predictor Coef SE Coef T P Constant -1331 1053 -1.26 0.225 Year 0.6801 0.5253 1.29 0.215 R 2 = 10.1% R 2 (adj) = 4.1% Page 23 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Social Sciences Business, Management and Accounting Economics, Econometrics and Finance Decision Sciences Using these subject categories as a screen, 41 of the 68 journals were considered relevant. In a similar fashion, SIMULATION, was used as a keyword to conduct a search. In this instance, this resulted in 32 listings with only 5 matching when the criteria stated above were used. Table 4 presents the results. Since the period of time from 2007 through 2011 was readily available and common for the several databases being utilized in the ultimate analysis, only those journals in existence for that entire period of time were included in the analysis. The values for the mean SJR represent the mean value of the annual value reported for the years 2007 through 2011. Similarly, the mean references per document was determined by calculating the mean references per document within each of the 5 years under consideration and then taking a simple average of those values over the 5 year period. The values for percent external that are reported are obtained by dividing the self cites (3 years) by the total cites (3 years) and subtracting that result from 100%. The value of the H Index is simply the value as reported by SCImago. FIGURE 3 PERCENTAGE OF EXTERNAL REFERENCES FOR ABSEL FROM 1984 TO 2013. 20152010200520001995199019851980 90.0% 85.0% 80.0% 75.0% 70.0% Year % E x t. ABSEL % Ext. vs Year Regression Analysis: % External references versus Year The regression equation is % Ext. = 8.31 - 0.00375 Year Predictor Coef SE Coef T P Constant 8.309 1.986 4.18 0.000 Year -0.0037482 0.0009936 -3.77 0.001 R 2 = 33.7% R 2 (adj) = 31.3% Page 24 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 RESULTS References Per Paper—ABSEL Proceedings. References per paper were determined for the ABSEL Proceedings for the years 1984 through 2013 by taking the ratio of the total references in comparison to the numbers of paper for the particular year. Surprisingly, the distribution of references year to year is quite varied. For example, for the years 2007 through 2011, the fewest references in a published paper was zero and the most references in the published papers from 2007 to 2011; respectively, was 58, 85, 69, 68, and 107. It is probably not a coincidence that the lead authors of papers with the most references per paper during that period were some of ABSEL’s most consistent contributors. Starting from 2007, they were Wellington, Wolfe, Wellington, Wellington and Palia. If the number of references in a paper is laudable, Palia, who listed 107 in 2011, is due the highest commendation. Between 1984 and 2013 a number of changes have occurred in terms of the mean references per paper. During that period the fewest mean references per paper was 5.3 and the most 25.6. Figure 1 provides a fuller picture of the changes that have occurred in references per paper during the period 1984 through 2013. During that period the mean references per paper (calculated by accumulating year results) was 14.6 with a standard deviation between years of 5.3. The regression results emphatically support the existence of a significant upward trend in references per paper over the time period 1984 through 2013. The R 2 for FIGURE 4 PERCENTAGE OF EXTERNAL REFERENCES FOR S & G FROM 1996 TO 2012. 2010200520001995 95.00% 92.50% 90.00% 87.50% 85.00% Year % E x t S & G % Ext vs Year The regression equation is % Ext. = - 125961 + 63.3 Year Predictor Coef SE Coef T P Constant -125961 27424 -4.59 0.000 Year 63.28 13.68 4.62 0.000 R 2 = 58.8% R 2 (adj) = 56.0% Page 25 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 this relationship is 56.8%. References Per Paper—S & G. In a similar fashion, references per paper were determined for articles published in Simulation & Gaming for the years 1996 through 2012. Figure 2 presents the results of this analysis. Comparable statistics in terms of mean and standard deviation were determined with a mean of 32.2 and a standard deviation of 10.6. Figure 2 provides a more extensive view of the changes in terms of this criterion for the period. The regression results do not suggest a significant upward trend in the references per paper for the time period 1996 to 2012. The variable year is not statistically significant at the 5% level and the R 2 for the relationship is 10.1% . External References Per Paper—ABSEL Proceedings. Another potential measure of the quality of the printed works is the percentage of references that cite works from other scholarly organizations in comparison to the total references. In this case, a high percentage would mean that the researchers were looking elsewhere for insights versus looking inwardly (i.e. to the work done within a particular scholarly organization). Figure 3 presents the annual mean within each year for the percentage of references that were external to ABSEL. The mean percentage external is 80.8% with the maximum FIGURE 5 SCATTERPLOT OF H INDEX VS. REFERENCES/PAPER 6050403020100 50 40 30 20 10 0 Ref./paper H D i n d e x Scatterplot of H index vs Ref./paper Regression Analysis: HD index versus Ref./paper The regression equation is HD index = 0.03 + 0.516 Ref./paper Predictor Coef SE Coef T P Constant 0.034 5.082 0.01 0.995 Ref./paper 0.5158 0.1416 3.64 0.001 R 2 = 33.0% R 2 (adj) = 30.5% Page 26 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 value during that period at 90.9% and a minimum value of 70.8% The regression results indicate a significant downward (p < 1%) trend in the external references per over the period 1984 through 2013 (with the R 2 for the relationship of 33.7%). External References Per Paper—S & G. Figure 4 presents the annual mean within each year for the percentage of references that were external to Simulation & Gaming for the period 1996 through 2012. The mean percentage external during that period is 90.3%, a maximum value of 95.4% and a minimum value of 85.3%. The regression analysis indicates a significant downward (p < 1%) trend in external references per paper during the period (R 2 = 58.8%). In general, there is a clear pattern for articles in S & G to have a higher percentage of their references as external to the journal than for those articles appearing in the ABSEL Proceedings. The information provided by SCImago can be useful as a way to make quality judgments about both the ABSEL Proceedings and S & G. It turns out that Simulation & Gaming has been evaluated by SCImago. It determined that the H Index for S & G was 22. Since the SJR is reported as an annual figure by SCImago, it seemed appropriate to use a mean for several years. So, the value the mean for the SJR values was determined by taking the mean of the 5 annual values reported for the period from 2007 to 2011. The mean references per document was also determined by taking the mean for the five-year period, FIGURE 6. SCATTERPLOT OF SJR VS. REFERENCES/PAPER 6050403020100 2500 2000 1500 1000 500 0 Ref./paper S JR Scatterplot of SJR vs Ref./paper Regression Analysis: SJR versus Ref./doc. The regression equation is SJR = - 348 + 30.3 Ref./paper Predictor Coef SE Coef T P Constant -347.9 270.2 -1.29 0.209 Ref./paper 30.313 7.525 4.03 0.000 R 2 = 37.5% R 2 (adj) = 35.2% Page 27 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 2007 to 2011. This resulted in the mean SJR value of 360 and the mean references per document of 33 for the period. Since SCImago has not evaluated the ABSEL Proceedings, one might wonder how the Proceedings would fair if SCImago determined values for the H Index and SJR. The authors of this paper thought it might be informative to do some speculation about this question. Although it is totally apparent exactly how SCImago determines these values, it appears that the variable, references per document, may be a direct factor in this determination or, at least a good proxy for some other measure. To explore this possibility, simple regressions were performed on the 25 journals that are reported in by SCImago when the key word, LEARNING, is used for screening. Figure 5 shows the positive relationship between the H Index and the references per paper. The results of the regression using H Index as a dependent variable and references per paper as an independent variable are also presented in Figure 5. A similar analysis was performed for SJR versus references per document (see Figure 6). As mentioned earlier, the value of the SJR was a mean for the five-period for the years 2007 – 2011. Without suggesting that this is a perfect measure, it is, at least, interesting to use these regression results to make some inferences about possible values for the H Index and SJR for the ABSEL Proceedings. In the case of H Index, the simple regression yielded a value for the simple R2 of 37.4%; and in the simple regression predicting SJR, the value of simple R2 was 34.7%. If one uses these regression results to predict the values for the ABSEL Proceedings based upon the historic mean value of 14.6 for references per document, the predicted value of the H Index is 4.6 and the predicted value for the SJR is 88.5. It is interesting to consider where these results would place the ABSEL Proceedings in comparison with the journals reported in Table 3. If the ABSEL Proceedings were including in the listing of the journals that appear using learning as a search word, the Proceedings would be in the lowest quartile of the list based upon either predicted H Index and the predicted SJR. Multiple Regression Results. To gain additional insights into the key variables that SCImago considers in assigning a value for the H Index for a journal, a regression was performed with the H Index as the dependent variable against two independent variables, mean number of internal references and mean number of external references. Results are shown in Figure 7 and are notable in several respects. For the model, the simple R2 is 37.44%, with an adjusted R2 of 31.5%. It is also particularly interesting that the variable, mean number of external references is significant at the 1% level and the variable, mean number of internal references is not significant at the 1% level but is significant at the 10%. So, it would seem that in the determination of a journal’s H Index, external references are more important than internal references. Since the variables recorded in this study were not exactly comparable to the variables employed by SCImago, no attempt was made to quantify the H Index for the ABSEL Proceedings using the results of the multiple regression. However, the relative importance of external, as opposed to internal, references is noteworthy and has implications for ABSEL. CONCLUSIONS The comments made in 2002 by Platt and Peach are clearly prescient. They observed that the Bernie Keys Library (BKL) “makes it relatively easy for a writer on any topic pertinent to ABSEL’s fields of interest to review and reference prior research.” It is interesting to note that the BKL was in its first or second edition when they made this pronouncement. The evidence from this study supports their prediction and helps shed light on several of the patterns. For example, there is a clear pattern of increasing references per paper in the ABSEL Proceedings. It may well be as a result of the “convenience” factor, that is, the convenience for an ABSEL author to thoroughly review all relevant prior work printed by ABSEL by searching the BKL. Another finding of this study is a recent tendency to rely more heavily on prior work published in the ABSEL FIGURE 7. MULTIPLE REGRESSION ANALYSIS: HD INDEX VERSUS MEAN NUMBER INTERNAL REFERENCES AND MEAN NUMBER EXTERNAL REFERENCES The regression equation is HD index = -3.53 + 0.628 internal ref. + 0.577 external ref. Predictor Coef SE Coef T P Constant -3.53431 6.303243 -0.56071 0.580927 Internal ref. 0.628334 0.352418 1.782925 0.089061 External ref. 0.576849 0.183629 3.141378 0.004929 R 2 = 37.44% R 2 (adj) = 31.5% Page 28 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Proceedings than other scholarly work in outside journals. This can be construed as having both positive and negative aspects. For example, it is clear that ABSEL has established a reputation for being a major source for presenting and critiquing new simulations and experiential exercises as well as conducting research on them. Obviously, future submissions will rely heavily on past ABSEL articles as part of the literature review. On the other hand, there are several venues for scholarly research (see Table 4) for simulations and experiential exercises. Serious scholars cannot and should not neglect these sources. However, as deans of schools of business and others try to assess the quality of the work published in the ABSEL Proceedings, they undoubtedly look for reportable measures of quality. The results of this research effort allow interested parties to place the work reported in the ABSEL Proceedings in a continuum of research quality. The regression results of this study suggest that an appropriate measure of the H Index used by SCImago would be roughly 4.6 and the SJR would be 88.5. A glimpse at Table 4 which reports the H Index and SJR for a number of journals in the related fields of study, puts the ABSEL Proceedings into a relative perspective. The findings indicate that SCImago and presumably others, who would like to determine the quality of the scholarship in ABSEL Proceedings, are influenced by two factors that were measured in this study--references per paper and the percentage of references that are external to ABSEL. In that regard, the upward trend that is evident in terms of references per paper is clearly positive and commendable. However, the downward trend in external references, as noted above, has both negative and positive viewpoints. Platt and Peach (2002) showed significant insight when they discussed the potential future impact of the BKL. Both trends revealed in this study (i.e., the increase in references per paper and decrease in external references) are logical outcomes of the availability of the BKL. Researchers are able to easily do a thorough review of all prior works published in ABSEL Proceedings and there is a strong temptation for researchers to not commit as much time to reviewing the literature outside ABSEL because the ABSEL findings are so readily available. Although there is considerable variability between the published papers appearing in the ABSEL Proceedings and in S & G, it is tempting to speculate about what the “typical” paper in each of these outlets looks like, at least with respect to dimension of their references. It is in response to those inclinations that the following is offered. The typical paper published in the ABSEL Proceedings has 22 references, of which roughly 76% are external to ABSEL Proceedings. By comparison, the typical paper published in S & G has 37 references, of which roughly 90% are external to Simulation & Gaming. With full recognition of the caveats provided in 2007 by Platt and Peach, it seems clear that if ABSEL would like its scholarly works to be highly regarded, it needs to be continuously aware of the perceived importance of legitimate references, both in terms of numbers as well as in terms of references external to the organization. REFERENCES Anderson, Philip H. and Leigh Lawton (1997). “Demonstrating the Learning Effectiveness of Simulations: Where We are and Where We Need to Go.” Developments In Business Simulation & Experiential Learning, 24, 68-73. Reprinted in The Bernie Keys Library, 13th edition. Bell, Reginald L; Chong, H Gin (2010). “A Caste and Class among the Relative Frequency of Faculty's Publications: A Content Analysis of Refereed Business Journals,” Journal of Leadership, Accountability and Ethics, 8.1: pp. 1-25 Bloom, Benjamin S. (1956). Taxonomy of Education Objectives: The Classification of Education Goals Handbook I: Cognitive Domain, David McKay Company, Inc. New York, NY. Butler, Jr., John K. (1999). “ABSEL’s Contributions to Experiential in the 90s.” Developments In Business Simulation & Experiential Learning, 26, 1. Reprinted in The Bernie Keys Library, 13th edition. Butler, Richard J., Peter M. Markulis, and Daniel R. Strang (1985). “Learning Theory and Research Design: How Has ABSEL Fared?” Developments In Business Simulation & Experiential Exercises, 12, 86-90. Reprinted in The Bernie Keys Library, 13th edition. Campbell, Donald T. and Julian C. Stanley (1963). Experimental and Quasi-Experimental Designs for Research, Rand McNally & Company, Chicago, Ill. Faria, A.J. (2000). “The Changing Nature of Simulation Research: A Brief ABSEL History.” Developments In Business Simulation & Experiential Learning, 27, 84- 90. Reprinted in The Bernie Keys Library, 13th edition. Faria, A.J. (2001). “The Changing Nature of Simulation Research: A Brief .” Simulation & Gaming: An Interdisciplinary Journal of Theory, Practice, and Research, 32, 1, 97-110. Faria, A.J. and Joseph Wolfe (1999). “ABSEL’s Historical Research Interests.” Developments In Business Simulation & Experiential Learning, 26, 1. Reprinted in The Bernie Keys Library, 13th edition. Garfield, Eugene (1979). Citation Indexing: Its Theory and Application in Science, Technology and Humanities, John Wiley & Sons, NY, NY. Goosen, Kenneth R. (1986). “An Analysis of ABSEL Conference Papers.” Developments In Business Simulation & Experiential Learning, 13, 97-101. Reprinted in The Bernie Keys Library, 13th edition. Gosen, Jerry and John Washbush (2004). “A Review of Scholarship on Assessing Experiential Learning Effectiveness.” Simulation & Gaming: An Interdisciplinary Journal of Theory, Practice, and Research, 35, 2, 270-293. Page 29 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Graf, Lee A. (1999). “ABSEL’s Contributions to Experiential Learning/Experiential Exercises: The Decade of the 1970s.” Developments In Business Simulation & Experiential Learning, 26, 1. Reprinted in The Bernie Keys Library, 13th edition. Gentry, James (ed.) (1991). Guide to Business Gaming and Experiential Learning, Nichols/GP Publishing, East Brunswick. Hirsch, J.E. (2005). “An index to quantify an individual’s scientific research output”, Proceedings of the National Academy of Science, 102(46): 16569-16572. Howard, Barbara, Peter M. Markulis, Daniel R. Strang and Joshua Wixom (2006). “Simulations and Experiential Exercises – Do They Result in Learning? Have We Figured It Out Yet?” Developments In Business Simulation & Experiential Learning, 33, 100-106. Reprinted in The Bernie Keys Library, 13th edition. Howard, Barbara and Daniel R. Strang (2001). “ABSEL: The Way We Talk!” Developments In Business Simulation & Experiential Learning, 28, 111-116. Reprinted in The Bernie Keys Library, 13th edition. Howard, Barbara and Daniel R. Strang (2003). “ABSEL: The Way We Talk! Take II.” Developments In Business Simulation & Experiential Learning, 30, 133- 145. Reprinted in The Bernie Keys Library, 13th edition. Hynes, Geraldine E; Stretcher, Robert H. (2005). ‘Business Schools' Policies Regarding Publications in Electronic Journals,” Journal of Education for Business, 81.2, 73-80. Kelley, Lane & William D. Brice (1999). “The Contributions of ABSEL during the 1980s.” Developments In Business Simulation & Experiential Learning, 26, 1. Reprinted in The Bernie Keys Library, 13th edition. Keys, Bernard (1977). “Review of Learning Research in Business Gaming.” New Horizons in Simulation Games and Experiential Learning, 3, 173-184. Reprinted in The Bernie Keys Library, 13th edition. Krippendorff, K. (1980). Content Analysis: An Introduction to Its Methodology. Beverly Hills: Sage. Lazaroiu, George (2009). “Constructing Journal Rankings and the Reliability Of Prestigious Scholarly Journals,” Economics, Management and Financial Markets, 4.2: pp. 111-115. Manton, Edgar J; English, Donald E. (2007). “The Trend Toward Multiple Authorship in Business Journals,” Journal of Education for Business, 82.3: pp. 164-168. Markulis, Peter M., Phillip Ricci and Daniel R. Strang (1991). “A Critical Review and Assessment of ABSEL’s Award-Winning Procedures and Protocols.” Developments In Business Simulation & Experiential Learning, 18, 61-65. Reprinted in The Bernie Keys Library, 13th edition. Markulis, Peter M., Phillip Ricci and Daniel R. Strang (1989). “A Review of Salient Trends in Proceedings’ Characteristics: A Fifteen Year Profile of ABSEL.” Developments In Business Simulation & Experiential Learning, 16, 200-203. Reprinted in The Bernie Keys Library, 13th edition. McWilliams, Abagail, Siegel, Donald, & Van Fleet, David D. (2005). “Scholarly Journals as Producers of Knowledge: Theory and Empirical Evidence Based on Data Envelopment Analysis,” Organizational Research Methods, 8.2: pp. 185-201. Merton, Robert K. (1988). “The Matthew Effect in Science II: Cumulative advantage and the symbolism of intellectual property,” ISIS, 79: 606-623. Patz, Alan L., J. Bernard Keys and Hugh M. Cannon (1999). “Back from the Future: An ABSEL ‘Merlin’ Exercise for the Year 2005. Developments In Business Simulation & Experiential Learning, 26, 1. Reprinted in The Bernie Keys Library, 13th edition. Peach, E. Brian and Richard G. Platt (2002). “The ABSEL Research Heritage and the BKL: Leveraging Their Value for Future Research.” Developments In Business Simulation & Experiential Learning, 29, 260-264. Reprinted in The Bernie Keys Library, 13th edition. Platt, Richard G. and Brian E. Peach ( 2007). “Inappropriate Use of Citations and Corrupting the Body of Knowledge: Accepting Urban Legends as Truth.” Developments In Business Simulation & Experiential Learning, 34, 314-317. Reprinted in The Bernie Keys Library, 13th edition. SCImago. (2007). SJR-SCImago Journal & Country Rank. Retrieved June 29, 2013, from http://scimagojr.com. Svensson, Göran (2008). “Scholarly journal ranking(s) in marketing: single- or multi-item measures?” Marketing Intelligence & Planning, 26.4: pp. 340-352. Wendl, Michael (2007). "H-index: however ranked, citations need context," Nature, 449 (7161): 403. Wolfe, Joseph (1981). “Research on the Learning Effectiveness of Business Simulation Games—A Review of the State of Science.” Developments In Business Simulation & Experiential Learning, 8, 72. Reprinted in The Bernie Keys Library, 13th edition. Wolfe, Joseph and David Crookall (1998). “Developing a Scientific Knowledge of Simulation/Gaming.” Simulation & Gaming: An Interdisciplinary Journal of Theory, Practice, and Research, 29, 1, 7-19. http://scimagojr.com