THE ABSEL RESEARCH HERITAGE AND THE BKL: LEVERAGING THEIR VALUE FOR FUTURE RESEARCH Developments in Business Simulation and Experiential Learning, Volume 29, 2002 THE ABSEL RESEARCH HERITAGE AND THE BKL: LEVERAGING THEIR VALUE FOR FUTURE RESEARCH E. Brian Peach The University of West Florida Richard G. Platt The University of West Florida ABSTRACT The Bernie Keys Library (BKL) was created to assist researchers in the fields of simulation and experiential learning. A database was created to identify the “citation classics” of the BKL and guide researchers to those BKL articles most frequently cited by ABSEL researchers. It was found that simulation papers tended to use references more than experiential papers, some authors did not include references with published papers due to page limitations and many authors did not use or include reference lists. Some proposals are made to enhance the future utility of the existing BKL, and future research submissions. INTRODUCTION At its annual conference in 2000, ABSEL honored one of its founding fathers by introducing the Bernie Keys Library (BKL). Dr. Keys was one of the guiding leaders of ABSEL through the first 25 years of its history. The updated BKL 2001 edition contains the complete text of all articles as published in the first 28 years of ABSEL. The text along with an index is conveniently provided on a single CD- ROM. Annual updates will continue to keep the BKL current and complete. The BKL is viewed by ABSEL as a boon to researchers in the fields of simulation and experiential learning because it provides the entire published record for ABSEL in an easily searchable format. The BKL was a major step in facilitating access to ABSEL’s research heritage. Accomplishing simple searches for articles, authors, or key words of interest involves only a very minor learning curve which can be accomplished in a matter of minutes. A second contribution was the ABSEL EndNote Database (Platt & Peach, 2001) which provides the capability for more powerful searches of the BKL. In addition, using the ABSEL EndNote Database, the normally tedious task of constructing a set of citations in an appropriate format can be quickly and easily accomplished. Continuing to develop methods and products that facilitate access to ABSEL’s research would be useful in broadening the appeal of preparing and submitting papers to ABSEL’s annual conference. One such effort is the current work in progress proposed at the 2001 conference to identify ABSEL classics. Another such effort is the subject of this paper: developing a list of most cited works. The authors believe that generating a database that identifies the “citation classics” would enhance the utility of the BKL by guiding researchers to those articles perceived by previous researchers to have provided the most benefit and relevance. USING AND REFERENCING PUBLISHED RESEARCH The foundation of any research attempt is the body of work that precedes it as published in the literature relevant to its topic. Only by knowing what has already been done in the field can either the author or reader effectively ascertain the benefit and relevance of the work at hand. The BKL presents the potential for aiding researchers in the fields of interest to ABSEL: simulation and experiential learning. As a searchable database of all past proceedings, it offers ease of access to the thoughts and findings of past contributors. One of the critical aspects of any good research publication is the list of references provided by the authors. References give readers not only insights into the rigor of the current effort, but an immediate and convenient pathway into the relevant literature. The efficacy of the BKL as a rich source of references in addition to the actual content of the articles is predicated on the assumptions that authors (1) do a rigorous literature review and (2) the references are included in the BKL. To the extent that authors do not accomplish a literature review, it is difficult for them to know how their contribution fits with previous efforts. To the extent that references are not available to readers, this lack creates several problems. First, it is difficult for the uninitiated to accurately ascertain the value and level of contribution the paper makes to the overall body of knowledge. Second, it impairs the ability of knowledgeable readers to assess the scope of preparation accomplished by the authors. Third, it makes it more difficult for researchers interested in the topic to identify and access relevant source documents and to build on the paper’s contribution. 260 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 ABSEL, REFERENCE LISTS AND THE BKL It is important for any library such as the BKL that where applicable, contributors accomplish quality literature reviews and that appropriate reference lists are included with published works. As former track chairs for ABSEL, the authors have first hand experience with the uneven quality of literature reviews and reference lists in many of the manuscripts submitted to ABSEL for its annual conference. ABSEL reviewers are forced to downgrade their ratings for potentially quality papers because they demonstrate a lack of familiarity with ABSEL literature. Some papers are submitted which duplicate similar, if not identical, prior efforts published in ABSEL’s Proceedings, Developments in Simulation and Experiential Learning, or Developments in Simulation and Experiential Exercises. This is somewhat vexing when the latest effort is less rigorous than the prior effort. It is very disappointing when a rigorous and thoughtful effort merely duplicates a previous effort. Typically this is attributed to the authors being first time submitters, and that they therefore lack familiarity with ABSEL. Even when the paper appears to make a new contribution, it detracts from its value if the authors fail to clearly place their work in the overall research stream, delineating how their work adds to that of others. Other disciplines have developed listings of published articles that are frequently referenced by other papers published in that discipline (Walstrom & Leonard, 2000). Referred to as ‘citation classics’, these lists provide a guide to popular articles and can be used as the basis for developing reading lists in a specific discipline. DEVELOPING THE CITATION DATABASE The intent of the authors was to identify a core set of ABSEL articles that were frequently cited by other authors. The most frequently cited articles would then be grouped by subject. Potential contributors unfamiliar with ABSEL’s published heritage would still have the capability to search the entire BKL, but they would also have access to a single source where they could quickly identify those articles most frequently associated with their research interest. The easier it is for potential contributors to gain familiarity with ABSEL’s literature, the more likely they are to acquire interest and confidence in adding to this body. Once the initial effort of creating the database of all citations in the BKL for current volumes was accomplished, the maintenance effort would be relatively minor. As the project established its value, future efforts would include citations of ABSEL articles contained in other publishing sources such as journals. Citations were extracted from the 28 volumes ABSEL has produced over the years: Volumes 1-7 as Proceedings, Volumes 8-24 as Developments in Business Simulation and Experiential Exercises, and Volumes 25-28 as Developments in Simulation and Experiential Learning. The citations were placed in an ACCESS database for ease of data manipulation and report preparation. Citations from future editions of Developments in Simulation and Experiential Learning will be added to enhance the database. Placing the citations in a database such as ACCESS allows for the creation of multiple reports that can group the data in any desired manner. For this paper, a report was created listing articles by the frequency of citation by other articles in the BKL. FINDINGS Table 1 lists by volume the number of papers that (a) included references, (b) made no mention of references, (c) indicated that references were available upon request, and (d) the total accepted for that year. The numbers in the column for papers that include references may be misleading as all articles that included any references are counted. Many articles had as few as one to three citations in their reference list, while others had in excess of thirty. Some might argue that in many, if not most instances, citing only one or two references would not constitute an effective literature review. For the purpose of this paper, no distinctions were made based on number of citations. Some of the patterns in Table 1 are readily explained while others are somewhat more difficult to understand. The relatively recent trend for a steady increase in papers that only supply references on request may be explained by the page limitations necessitated by printing costs. Many authors chose to retain article content and forgo including references in the published form if they were approaching their page limits. Such papers ranged from twenty to forty percent annually of accepted papers in the last few years. The pattern for papers that do not include references is not as clear. Volumes 15 and 16 had very few such papers, but the years on either side had relatively high numbers of such papers. It seemed papers without references were trending down in the 1990s, but recent years have again seen higher numbers. The total number of papers accepted has been remarkably stable over ABSEL’s twenty-eight years. Other than 1987's conference at Hilton Head Island, which generated 109 papers, accepted papers have been generally in the range of fifty to seventy. The top nine most frequently cited papers are listed in Table 2. Only the top nine rather than ten papers were listed as an additional ten papers were tied at eight cites. The papers cited nine or more times by other papers in the BKL were all simulation papers. This reflects the tendency of many experiential papers to not provide reference lists. This propensity is revisited later in the discussion section. Other than the Bernie Keys 1976 paper Review of Learning Research (which earned top citation honors at 18) the other top papers seem to fall into two groups: those dealing with construction of simulations and those dealing with learning and simulations. Anyone familiar with ABSEL would not be surprised at this result. Although not listed, all ten of the papers with eight citations were also simulation related. 261 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Nine of these dealt with learning aspects of simulation, and one with the construction of simulation algorithms. For papers with seven cites, experiential papers appear and the variety of topics begins to increase. Table 3 indicates the citation frequency for papers garnering four or more citations. As would be expected, the number of papers for a given level of cites rises almost exponentially as the citation levels decrease. An interesting finding illustrated by the table is that out of 1629 papers examined, only 95 were cited 4 or more times. DISCUSSION Initial analysis of the BKL indicated two disturbing trends: a decreasing percentage of authors were providing references, (See Table 1) and to some extent the reference lists provided were becoming shorter. Volume 28, for the year 2001, may signal a reversal of those trends or may be an anomaly. Approximately 30% of the papers did not include references, although about one fifth of these indicated they were available upon request. After reviewing Table 1, we believe that the usefulness of the database may be degraded by the number of papers categorized as ‘references available’. Many were serious works that likely had substantive reference lists. These missing data became in a sense an intervening variable. Although the majority of papers included references, there was enough missing that it calls for capturing, as far as possible, the missing references. As a follow-on to this paper, we will contact the authors with papers advertising references available on request, and ask for these lists. These lists will then be added to the database, and an analysis conducted to see if there are any material changes in the sequence of papers. A lack of change in the sequence of papers will indicate that the missing lists did not affect the validity of the current data. A material change in the sequence will indicate that references were not provided in a non-random manner and this will reinforce the value of including all data in an analysis. We were somewhat dismayed at the number of authors who do not provide any citations in their works. We accept that some experiential exercises may not require a literature review. But it seems that many of these papers were written without any apparent consideration as to how they fit with similar efforts. The simulation arena was generally noted for providing reference lists, but even here there were papers with remarkably few (e.g., one) references. As noted earlier, the experiential arena was notable for the number of papers without any references at all. It would seemingly be helpful for developers of experiential exercises to reference similar exercises, and discuss the relative features and merits. Readers familiar with similar exercises will more readily understand the purpose of the new exercise, and be able to assess its potential value to them in their classrooms. We hesitate to claim that every paper should have references, but their inclusion will almost always enhance the value of any paper. FUTURE RESEARCH A number of additions to this project are contemplated to improve its value. (1) We intend to collect, to the extent possible, the missing reference lists to enhance the validity of rankings within the list of most cited works. This will require locating and contacting authors and acquiring the missing reference lists. (2) We will add reference lists from future years to add to the richness of data. (3) The most cited papers will be identified and grouped by subject matter to provide prospective authors a ready guide as to which papers are relevant to their research efforts. Analyses can also be conducted to determine the historical pattern of citations. By tracking citation histories, it can be determined if there are differences between total citations, and recent citation patterns. As research streams shift, the most relevant papers may have fewer, but more recent cites. Another possible analysis is to assess the degree of self-citation. Authors with a long-term research stream may tend to cite their previous works in successive papers, thus building their numbers. RECOMMENDATIONS ABSEL should make the nature and value of the Bernie Keys Library a focal point in the annual call for papers. The BKL has clear value for new submitters in providing information and ideas on suitable topics and papers. In addition, a strong recommendation should be made to new and old submitters to review the BKL for prior work similar to the proposed topic. The BKL makes it relatively easy for a writer on any topic pertinent to ABSEL’s fields of interest to review and reference relevant prior research. The ABSEL EndNote Database makes it relatively easy for prospective authors to review articles and build reference lists. Both of these should be prominently displayed in all ABSEL publicity efforts as well as the call for papers. ABSEL has a history of improving the quality of its accepted papers. Effective use of tools such as the BKL will further improve the quality of research and submissions, and as a consequence improve the overall impact ABSEL has on its fields of interest. 262 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Table 1 - Frequency of References in BKL Volume References References References Total Number Included Not Included On Request Accepted 1 19 27 0 46 2 22 21 0 43 3 26 11 0 37 4 35 11 0 46 5 34 14 0 48 6 50 22 0 72 7 36 25 0 61 8 59 18 0 77 9 55 16 0 71 10 31 10 0 41 11 49 13 0 62 12 33 14 0 47 13 54 10 1 65 14 52 56 1 109 15 55 5 0 60 16 39 4 0 43 17 52 22 2 76 18 34 12 4 50 19 51 22 3 76 20 38 26 5 69 21 43 11 2 56 22 43 3 6 52 23 29 6 7 42 24 45 3 15 63 25 32 9 12 53 26 37 12 6 55 27 31 12 18 61 28 29 8 1 48 TOTAL 1123 423 83 1629 263 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Table 2 - Most Cited Papers Cites Article Reference 18 Keys, B. 1976. A review of learning research in business gaming. 14 Pray, T., & S. Gold. 1982. Inside the black box: An analysis of underlying demand functions in contemporary business simulations. 13 Gold, S., & T Pray. 1983. Stimulating market and firm demand - a robust demand system. 11 Wellington, W.E., & A. J. Faria. 1991. An investigation of the relationship between simulation play, performance level and recency of play on exam score. 11 Wolfe, J. 1978. Correlation between academic achievement, aptitude and business game performance. 9 Edwards, W.F. 1987. Learning macroeconomic theory and policy analysis via microcomputer simulation. 9 Faria, A.J., & T.R. Whitely. 1990. An empirical evaluation of the pedagogical value of playing a simulation game in a principles of marketing course. 9 Goosen, K. 1986. An interpolation approach to developing mathematical functions for business simulations. 9 Patz, A.L. 1990. Group personality composition and total enterprise simulation performance. Table 3 - Citation Frequencies Number of Citations 9+ 8 7 6 5 4 Number of Papers 9 10 9 14 20 33 REFERENCES The Bernie Keys Library (version 2) [CD-ROM]. 2001. The Association for Business Simulation and Experiential Learning [Producer and Distributor]. Platt, R. G., & E. B. Peach. 2001. The ABSEL EndNote database: The perfect tool for the Bernie Keys Library. Developments in Simulation and Experiential Learning, 28, 201-204. Walstrom, K. A., & L. N. K. Leonard. 2000. Citation classics from the information systems literature. Information & Management, 38, 59-72. 264 Table of Contents Volume 29, 2001 Threshold Marketer A Family Of Marketing Simulations: Basic Marketer And Advanced Marketer Team Mode And Solo Mode Globalization As An Extended Experiential Exercise The Benefits And Planning Considerations Of Short Term Study Abroad Programs How 2 Setup Your Office Computer To Run Linux For Teaching E-Commerce Without Messing Everything Else Up Incorporating Cosmopolitan-Related Focus-Group Research Into Global Advertising Simulations Demonstration Of Advanced Features In Computer-Assisted Gaming Of International Business A Comparison Of Discrimination-Based Versus Conventional Simulation Game Scoring A Universal Mathematical Law Criterion For Algorithmic Validity The Impact Of Public Policy On Innovation: A Simulation Project For Research And Teaching Participant Identification Of Competitors In A Marketing Simulation Competition Simulation Research In The Hospitality Industry "Computer Simulation, Games And Roleplay: Drawing Lines Of Demarcation" Simulation Distribution Alternatives: Author/User Considerations Managing The Curiosity Gap Does Matter: What Do We Need To Do About It? Use Of External Interventions In A Computer Based Simulation Putting Service Learning Into Orbit It's A Wonderful Life: Simulating The Golden Years Adventures In Creating An Outdoor Leadership Challenge Course For An Emba Program Vbotz: A Pedagogical Cross-Disciplinary, Multi-Academic Level Manufacturing Corporate Simulation Use Of Computer Modeling In Management Accounting Is Simulation Performance Related To Application? An Exploratory Study Learning Cooperatively May Not Be Learning Collaborately! Perception Is Reality: Sharing Frames International Management Virtual Teamwork: A Simulation Financial Plan For Your Life And Career Goals Using Project-Based Experiential Learning Groups In The Principles Of Marketing Course Futures Course: Learning How To Anticipate The Future Of Business Interactive Online Strategic Market Planning With The Web-Based Boston Consulting Group (BCG) Matrix Graphics Package Strategy Learning In A Total Enterprise Simulation Investigation Of The Impact Of Decision Parameters For A Dutch Auction Simulation For Ipo Issues Integrating In-Class Learning With Out-Of-Classroom Experiences Through A Managerial Competency Development Framework War And Peace: Managing Students Learning Experience In A Competitive Simulation Game Virtually Experiential Classrooms Exercise: Conducting Role Plays Using Student Generated Cases Procedural Justice And Acceptance In Group Decision Making The E-Commerce Game: A Strategic Business Board Game Does Student Preparation Matter In A Simulation? A Comparison Of Pedagogical Styles The Game Of Business - A Weekend MBA Course Volume-Dependent Money Exchange Model For Gaming Simulations Implementing Service Learning For Accountants: The Not For Profit Project To Teach Vikings To Behave Among Mandarins: Lessons From Teaching With A Simulation Model Of Applied Business Ethics In International Management Maze Bright Teachers In The Classroom The Validity Investigation Of A Test Assessing Total Enterprise Simulation Learning What Makes Strategy Possible: An Illustration Using Paper And Scissors Learning Micro-OB Skills While Making Top Management Decisions In A Multinational Industrial Firm The Absel Research Heritage And The Bkl: Leveraging Their Value For Future Research New Product Development (Npd) Simulations: Some Challenging Questions And Tough Modeling Issues The Biofeedback Stress Test The Power Circle Exercise Total Enterprise Simulation Learning Compared To Traditional Learning In The Business Policy Course A Business Game Distance Education Application: Learning Outcomes And Experiences Is the Tobin's Q a good Indicator of a Company's Performance? A Critical Examination of the "Experiential" Premise Underlying BUsines Simulation Usage