A CRITICAL REVIEW AND ASSESSMENT OF ABSEL’S AWARD-WINNING PROCEDURES AND PROTOCOLS Development In Business Simulation & Experiential Exercises, Volume 18, 1991 61 “A CRITICAL REVIEW AND ASSESSMENT OF ABSEL’S AWARD-WINNING PROCEDURES AND PROTOCOLS” Peter M. Markulis State University of New York at Geneseo Phillip Ricci State University of New York at Geneseo Daniel R. Strang State University of New York at Geneseo ABSTRACT This paper is the third in a three part series of self- examination reports on ABSEL. The principal purpose of this paper is to report on the procedures and protocols ABSEL has used to designate award-winning papers. The paper also considers the procedures used by other academic organizations and compares these to ABSEL’s. Further, the paper compares the award-winning papers of ABSEL to both ABSEL papers in general and to articles in SIMULATION & GAMES. The paper contains information on the progress of papers after having won an ABSEL award. The paper also reports on the career impact of the award for the award-winning authors. Finally, the paper offers some suggestions for ABSEL in regard to the procedures for designating award-winning papers. INTRODUCTION This paper is the third in a three part series of self- assessment and introspective articles on ABSEL. The first article was a review of the ABSEL Proceedings for the years 1976 to 1984 in terms of degree of rigor of research designs and the degree to which they address various educational objectives (Butler, Markulis, & Strang, 1985). The second article addressed issues of authorship, in ABSEL Proceedings over a 15 year period, including trends in author turnover, multiple authorship, institutional representation, etc., (Markulis, Ricci, Strang, 1989). This article focuses on the award-winning papers, and the process and protocols used by ABSEL to determine award-winners. Most academic organizations provide national or regional forums through which scholars can present and critique the latest research and theory- building in the field. Many of these same organizations also go through a process of selecting the ‘best’ papers or presentations made at these forums. In some fields, competition for the ‘best’ paper or presentation is quite keen, as the award ~s viewed by the indigenous scholars to be a considerable plaudit. In some instances, financial incentives are attached to these awards. It would seem, then, that most academic organizations are proud to publicly recognize their ‘best. ‘ In view of this, one might reasonably argue that an academic organization is, ipso facto, distinguishable by what it considers to be its best work. With this in mind, it is our intention to perform a review of what ABSEL considers its ‘best.’ This paper will cover the following: (1) A profile of authors of award-winning papers. Included in the author profile are comments by award winning authors on the career impact of having received an award, (Appendix 1 contains a complete list of the award-winning papers, authors and/or co- authors, award title and year of award.) (2) A profile of award-winning papers, (3) A comparison of ABSEL’s award-winning papers to other ABSEL papers using the dual criteria of educational objectives and rigor of research, (4) A review of the procedures ABSEL has used to determine and designate award-winning papers since 1978, (the first year ABSEL began giving out awards), (5) A review of procedures and protocols used for designating award-winning papers from several management, as well as non--management organizations. METHODOLOGY During the years 1978 to 1990, 26 papers were designated as award winners. Many of these papers were co-authored, leading to a total of 46 award- winning authors (in this paper, “award-winning author” will refer to all authors of papers with one author, as well as to those with multiple authors). A short questionnaire soliciting information on their papers was sent to each award-winning author. Former ABSEL Program Chairpersons for the period 1978 to 1990 were identified and a letter was sent to them indicating our interest in obtaining information on the procedures and protocols used to determine award-winning papers. The letter presented several questions concerning the procedures that had been used to determine award winners and indicated that one of the authors would be contacting them in a couple of weeks to obtain this information through a telephone interview. Finally, a short questionnaire was sent to the chairpersons of several academic management and non-management conferences. This questionnaire requested information about the procedures and protocols used in determining award- winning papers in their organizations. Development In Business Simulation & Experiential Exercises, Volume 18, 1991 62 RESULTS Twenty-one questionnaires were received from ABSEL’s award-winning authors for a response rate of 46%. Table 1 presents the author profile, which includes information about the author’s degree, teaching area(s) and number of years associated with ABSEL. Table 1. Author Profile Area of Doctorate Number Economics 2 Management 7 Marketing 3 Org. Behavior 5 Miscellaneous Total 21 Major Teaching Areas Frequency Business Policy 5 Economics 1 Information Sys. 1 Management Sc. 3 Marketing 4 Org. Behavior 9 Personnel 5 Psychology 1 Small Business 2 Statistics 1 Total Number of Years Associated with ABSEL Frequency 1 to 3 5 4 to 6 4 7 to 9 1 10 to 12 5 13 or more 4 Total 19* There were two non- responses. As can be seen, all 21 respondents had advanced degrees. Of these, 16 were PhD’s, 3 were DBAs and 2 had EdDs. The field of Organizational Behavior was the most represented in the teaching areas (n = 9), which may seem surprising in that the development of computerized simulations is generally considered to be more of an area for Decision Sciences or Economics. The last data in Table I show the relationship between the number of years affiliated with ABSEL and winning an ABSEL award. Table 2 reports data regarding the subsequent history of ABSEL award-winning papers. Table 2. Paper Profile Resulted in Further Work? Resulted in Publication? Yes 18 Yes 16 No 3 No 4 In Progress 1 * Where Published? Simulation and Games 6 ABSEL 4 Academy of Management Journal 2 Other journals 11 Eighteen respondents indicated that further work was done with the paper with 16 of these stating that a publication resulted. Table 3 provides the authors’ views on the career impact of the ABSEL award. Table 3. Career Impact Impact on Career Number of respondents None 12 Slight 3 Moderate 2 Significant 2 Very Significant 2 Selected Comments: -- “Personal satisfaction” -- “Incentive to do further work” -- “Impact fleeting” -- “Dean unfamiliar with ABSEL” -- “Added to confidence” -- “Helped get promotion to Associate Professor” Table 3 provides the authors’ views on the career impact of the ABSEL award. As Table 3 notes, although most respondents felt that the award ipso facto had little career impact, many of them felt they gained considerable personal satisfaction from having received the award. In an attempt to evaluate the several key characteristics of award winning papers, a comparison was made between the ABSEL award winning papers, articles appearing in SIMULATIONS & CANES (1974-84), and all ABSEL papers (1974-1984). The comparison used the evaluation criteria of research design and educational taxonomy employed by Butler, Markulis and Strang (SIMULATION & GAMES, Vol. 19, No. 1, Mar. 1988, pp. 3-26). Table 4 provides this information. Development In Business Simulation & Experiential Exercises, Volume 18, 1991 63 Table 4. Research Design Characteristics of Articles AWARD S &G ABSEL WINNERS 197 4-84 197 4-84 197 8-90 (%) (%) (%) C 4.1 12.5 0.0 R 0.0 0.0 0.0 T 1.8 1.1 29.2 0 47.6 40.8 0.0 P 19.4 35.6 37.5 C-R 0.6 0.0 0.0 C -T 12.4 3.1 29.2 R-T 0.0 0.0 0.0 C-R-T 14.1 2.6 4.2 100.0 100.0 100.0 (n =170) (n = 458) (n = 26) Note: C (Control) = studies with at least one control group; R (Random) = studies with randomization; T (Treatment) = studies in which treatments were varied; 0 (Other) = articles that do not pertain to research; P (Prescriptive) = articles promoting or describing simulation gaming or experiential exercises with reference to any research methodology. Articles Categorized According to Learning Outcomes AWARD S &G ABSEL WINNERS 1974-84 1974-84 1978-90 (%) (%) (%) A 12.9 9.4 33.3 C 13.5 21.0 8.3 P 0.0 0.0 0.0 A-C 21.8 24.7 37.5 O 51.8 45.0 20.8 100.0 100.0 100.0 (n = 170) (n = 458) (n = 26) Note: A = studies that address learning in the Affective Domain; C = studies that address learning in the Cognitive Domain; P = studies that address learning in the Psychomotor Domain; A-C = studies that address learning in both the Affective and Cognitive Domains; 0 = studies that do not address learning per se. While it should be noted that the time spans for the review are not exactly the same, certain patterns did arise. It is not surprising that ABSEL’s award winning papers seem to be more diligent with respect to research criteria and ~n addressing educational objectives than either ABSEL or Simulation & Games papers. These observations are borne out by the elevated percentage of articles using “treatments” and the larger percentage of articles with explicit educational objectives revealed in Table 4. Another interesting facet of award winners is the selection and designation processes. Eight former Program Chairs were contacted for information regarding the process and protocols used by ABSEL to select award winning papers. It should be noted that the authors were not interested in obtaining information about the so-called ‘politics’ of the process, but rather, what the process was, and how the number of awards to be given was determined. The survey of program chairs indicated that a variety of procedures have been used to select award winners. Program chairs indicated that at this time there are no formal written procedures. Therefore, most chairpersons have considerable discretion in determining both the number of awards as well as the selection process. Although the actual procedures that have been employed have varied over time, chairs did consistently report the use of multiple reviews at different levels. A commonly used procedure was one in which papers were ranked by reviewers on a 10-point scale. If the paper scored high on this scale it was passed on to a second tier of reviewers. In some cases, these reviewers were past ABSEL presidents. The chairs reported variability in how this second review was conducted. It is not clear if the second tier of the review process was always conducted as a blind review. In the last couple of years, the review form has asked reviewers if the paper under evaluation should be considered for a possible award. Most chairs stated that they placed heavy emphasis on precedents. Interestingly, in spite of these assertions there appears to be a great deal of variation in the number of awards as well as the categories of award winners. One piece of information we were unable to obtain was the chairperson’s sense of how many ‘potential’ award- winning papers there were in a given year. To gain a sense of the relationship between ABSEL and other academic groups, a small sample of chairpersons of management and non-management academic conferences was contacted. Information was obtained from these chairpersons about the procedures and protocols used by their organizations to select award-winning papers. This was not meant to be an exhaustive survey, but was simply intended to shed some light as to what other organizations are doing in this regard. Interestingly enough, some organizations such as the American Psychological Association and Eastern Psychological Association do not give awards for papers at their conferences, while other organizations such as the Southwestern Political Science Association have special committees to determine best paper awards and also provide awards for undergraduate papers. In some cases, financial rewards were given for outstanding papers. All 11 organizations that responded to our request for information indicated that they used a standard process to determine award-winning papers. Eight of these said that the program chair had little discretion in determining the award winning process, while 3 did not respond to this question. Most organizations that responded to our query indicated that they use the initial blind review form to identify potential award winning papers. From that point, the ‘typical’ procedure is to use a special committee of reviewers, who then rank order the top papers in order to arrive at the award winning paper or papers. In all cases, the process consisted of a blind review. Development In Business Simulation & Experiential Exercises, Volume 18, 1991 64 Our conclusions will be short as we feel that most of the readers will want to draw their own conclusions--and perhaps offer recommendations-- from the data presented. Our main thrust has been to report on the procedures and protocols ABSEL. uses in determining award-winning papers. As noted, ABSEL has used a variety of procedures in the past, although these procedures have become more standardized in the past few years. We would recommend that ABSEL develop a set of standard procedures for determining award-winning papers, put them in writing and publish them in the ABSEL may also want to consider the possibility of offering various rewards to the authors of award- winning papers. Self-examination and critical self-assessment are important demeanors for any credible academic organization. Reviewing the quality of research and contributions to theory, tracing the organization’s membership, and assessing the protocols used by the organization to designate its ‘best’ represent just few of the kinds of self assessments that organizations should periodically undertake. This paper--like the two previous ones--represents, we believe, part of an invaluable and necessary self-examination process. REFERENCES All ABSEL Award-winning papers (see Appendix 1). Butler, R. J., Peter Markulis and D. R. Strang, “Learning Theory and Research Design: How Has ABSEL Fared?” ABSEL PROCEEDINGS, 1985, pp. 86-90. Butler, R. J., Peter Markulis and D. R. Strang, “Where Are We: An Analysis of the Methods & Focus of the Research on Simulation Gaming” SIMULATION & GAMES, Vol. 19, #1, (March 1988), pp. 3-26. Markulis, P. M., Phillip Ricci and Daniel R Strang, “A Review of Salient Trends in Proceeding’s Characteristics: A Fifteen Year Profile of ABSEL,” ABSEL PROCEEDINGS, 1989, pp. 200-203. APPENDIX 1 AWARD-WINNING PAPERS AND AUTHORS YEAR: 1990 AWARD: Best Computer Simulation Research TITLE: Demand Equations, Which Include Product Attributes AUTHOR: Richard D. Teach UNIV: Georgia Institute of Technology YEAR: 1990 AWARD: Most Innovative TITLE: Executive Evaluation of Student Learning in the Looking Glass AUTHOR: Diana Page & Ralph M. Roberts UNIV: University of West Florida YEAR: 1989 AWARD: Most Innovative TITLE: A Simulating Simulation in International Business Negotiations With a Japanese Company AUTHOR Barry S. Axe UNIV: Temple University YEAR: 1989 AWARD: Best Research TITLE: A Study of the Relationship Between Student Final Exam Performance and Simulation Game Participation AUTHOR: T. Richard Whiteley & Anthony J. Faria UNIV: University of Windsor YEAR: 1988 AWARD: Best Paper - Experiential Learning TITLE: Experimental Analyses of Magnitude and Source of Students’ Inequitable Classroom Perceptions in Three Reward Conditions AUTHOR: John D. Overby & Kay A. Durden UNIV: The University of Tennessee at Martin YEAR: 1988 AWARD: Best Paper – Simulation TITLE: Strategy Design, Process and Implementation in a Stable/Complex Environment: An Exploratory Study AUTHOR: Jerry Gosenpud & Joseph Wolfe UNIV: University of Wisconsin-Whitewater University of Tulsa YEAR: 1987 AWARD: Best Research - Simulation TITLE: Decision Styles and Student Simulation Performance: A Replication AUTHOR: Robert W. Hornaday & Kent E. Curran UNIV: The University of North Carolina at Charlotte YEAR: 1987 AWARD: Most Innovative TITLE: The Use of Expert Systems to Develop Strategic Scenarios: An Experiment Using a Simulated Market Environment AUTHOR: Andrew Varanelli:, Marion Sackson, Denis Cronin, & Carol Lazaro Dulberg UNIV: Pace University YEAR: 1986 AWARD: Best Paper TITLE: The Assessment Center as a Teaching/Learning Device AUTHOR: R. Bruce McAfee & Alex Hawryluk UNIV: Old Dominion University Development In Business Simulation & Experiential Exercises, Volume 18, 1991 65 YEAR: 1985 AWARD: Best Research TITLE: Predicting Performance Over the Course of the Simulation AUTHOR: Jerry Gosenpud, Charles Milton, & Arthur Larson UNIV: University of Wisconsin-Whitewater YEAR: 1985 AWARD: Most Creative TITLE: Developing the Competencies of “Resistance to Stress” and “Accurate Self-Assessment” AUTHOR: Donald S. Kline UNIV: Bloomsburg University YEAR: 1984 AWARD: Best Research TITLE: A Path Analytic Study of the Effects of Alternative Pedagogies AUTHOR: Alvin C. Burns & Dan L. Sherrell UNIV: Louisiana State University YEAR: 1983 AWARD: Outstanding Research TITLE: Simulating Market and Firm Level Demand -A Robust Demand System AUTHOR: Steven C. Gold & Thomas F. Pray UNIV: Rochester Institute of Technology YEAR: 1983 AWARD: Outstanding Paper TITLE: Role-Playing Based on Video-Tape Scenarios: An Application of Modelling to Building Supervisory Skills AUTHOR: James C. Faltot & John R. Ogilvie UNIV: University of Delaware YEAR: 1982 AWARD: Best Experiential TITLE: The Value of Conjoint Analysis in Enhancing Experiential Learning AUTHOR: Leonard Greenhalgh & Scott A. Neslin UNIV: Amos Tuck School, Dartmouth YEAR: 1982 AWARD: Best Research TITLE: The Effects of Different Team Sizes Business Game Performance AUTHOR: Joseph Wolfe & Thomas I. Chacko UNIV: University of Tulsa Iowa State University YEAR: 1981 AWARD: Best Simulation Research TITLE: Providing a Real World View of the Personnel Function: A Simulation AUTHOR: George E. Stevens & Eileen K. Burton UNIV: Arizona State University YEAR: 1981 AWARD: Innovative Experiential TITLE: Finding an Effective Means of Teaching Managerial Behavioral Skills: Two Different Experiential Teaching Methods Compared AUTHOR: Daryl G. Mitton & Betty Lilligren- Mitton UNIV: San Diego State University YEAR: 1981 AWARD: Best Experiential Research TITLE: Problems in Evaluation of Experiential Learning in Management Education AUTHOR: Lane Kelley & Jeffrey Easton UNIV: University of Hawaii University of Lancaster YEAR: 1981 AWARD: Innovative Simulation TITLE: Teaching Performance Appraisal Skills: An Experiential Approach AUTHOR: Eileen K. Burton UNIV: Arizona State University YEAR: 1980 AWARD: Best Research TITLE: Associations Between Individual Cognitive Processing Variables and Business Game Performance and Play AUTHOR: Joseph A. Wolfe UNIV: University of Tulsa YEAR: 1979 AWARD: Best Experimental TITLE: Personality Development and Conflict Dynamics: An Experimental Design to Study the Effects of Teaching Methodologies on Conflict Resolution AUTHOR: Tom Basuray, Jerry J. Gosenpud, & Steven A. Scherling UNIV: University of North Dakota YEAR: 1978 AWARD: Most Innovative TITLE: Emergent Simulation in Administration Courses AUTHOR: Claude t. Wynn, George E. Crawford UNIV: Weber State University Table of Contents Volume 18, 1991 Personality Types and Total Enterprise Simulation Performance Using DIS 'n DAT as a Decision Support System for a Marketing Simulation Game Theoretical Derivation of a Market Demand Function for Business Simulators The Ethnographic Case Study: An Experiential Approach to Teaching Retail Management Electronic Bulletin Board Systems (BBS): Support Software for Computer Simulations The New Budget Game Negame: A Cross-Cultural Role-Play to Introduce Students to the Familiarization Stage of Negotiations Modeling Short-Run Cost and Production Functions Using Sheppard's Lemma in Computerized Business Simulations Increasing Simulation Realism through the Modeling of Step Costs Predicting Simulation Performance: Differences Between Groups and Individuals A Facility Location Case to Stimulate Classroom Interaction Educational Effectiveness of Business Simulation Gaming: A Comparative Study of Students and Practitioner Perspective Ethical Dilemmas in Experiential Learning: Issues and Strategies A Critical Review and Assessment of ABSEL's Award-Winning Procedures and Protocols Political Risk: A Simulation for Business Practitioners Upside Down: A Cross-Cultural Game in Experiential Learning Gorby's Dilemma: From Communism to Free Enterprise in Two Hours Strategic Market: Planning with the COMPLETE Product Portfolio Analysis Package: A Marketing Decision Support System Career Concepts and Total Enterprise Simulation Performance Experiential Learning in Human Resources: A Performance Appraisal Application Managerial Motivation and Realism Among MBA Student as Viewed through The Looking Glass, Inc. Simulation An Experiential Approach to Teaching Data Analysis Using MYSTAT: Rationale, Procedures and Results Practicing What Was Preached: A Sequential Learning Model put to the Test Student Attitudes about Policy Course Simulations An Investigation of the Relationship Between Simulation Play, Performance Level and Recency of Play on Exam Scores The Effect of Leadership and Cognitive Processing Styles upon Peer Performance Evaluation: Implication for the Utilization of Simulations in Business Pedagogy On the Transfer of Market Oriented Business Games to Socialist Cultures An Application of Financial Analysis of the Business Firm in a Simulated Competitive Environment Collective Bargaining Simulation: An Exercise based on a Familiar Theme Making Business Policy a Strategic Management Experience A Student Exercise for Intergrating the Concepts of Power and Motivation The Boundaries Extended: An Experiment Comparing Dialectical Inquiry, Devil's Advocacy and Consensus Using the Executive Game Using a Simulation Package to Develop a Simulation Exercise in Cost Accounting Success Factors in Experiential Training for Creative Problem-Solving Teams An Experiential Approach for teaching Quality management Critical Success Ratios: A Comparison of Two Business Simulations in a Multi-Year Environment Stocklogs: A Classroom Exercise for Teaching the Logistical Relationship of Location and Inventory The Organizational Leadership Program Simulating Business Decision-Making: Using Statistical Cases for Classroom Exercises Scripting for the Classroom Upgrading the Business Strategy and Policy Game Developing Student Team-Building and Leadership Skills Using Computer-Aided Experiential learning Strategies Organizing and Outward Bound Field Trip An Architecture for Extensible Simulation Games Performance in the Capstone Business Course: What is the effect of Pedagogy, Learning Styles, and Student Motivation? Operational Strategy with Participant-Modifiable Parameters An Example of a Personal Selling Case Transformed into a Role Play Scenario Instructional Software: It's Evolution and Current State of the Art in the Business Curriculum Ascertaining Performance Variables for use in Determining Student's Grades in Courses Employing a Business Simulation Designing Management Seminars Using Business Simulations The Accounting Information Systems Course: Bridging the Gap between the Classroom and the Real World The Good Cooks Guide to Training Excellence: Working with Passion A Demonstration on Multiple Data Collection Methods: Seeing Strategic Issues Through the Looking Glass Simulation Systems Analysis and Design: Why Undergraduate Education gets a Failing Grade Accommodating Organizational Culture: An Evaluation of Management Development Delivery Modes in Varying Organizational Cultures Modeling Total Quality into Business Simulations The Political Futures Game Meeting Meeting Objectives