A TESTBANK FOR MEASURING TOTAL ENTERPRISE SIMULATION LEARNING Developments in Business Simulation & Experiential Learning, Volume 26, 1999 A TESTBANK FOR MEASURING TOTAL ENTERPRISE SIMULATION LEARNING Jerry Gosen and John Washbush, University of Wisconsin-Whitewater, Alan Patz, University of Southern California, Tim Scott, Mankato State University, Joseph Wolfe, Experiential Adventures LLC, Dick Cotter ABSTRACT This paper describes a procedure to construct a test bank of 120 items categorized according to pre-determined learning objectives designed to assess whether or not learning takes place from playing a total enterprise simulation. The paper also contains the list of forty learning objectives and examples of items in the test bank. Finally, a pilot study is presented showing reliability and validity data for a selected sample of test bank items. BACKGROUND The fact that simulations are used in universities suggests that those who teach with them expect students to learn from them. In addition, scholarly attention has been paid to the concept, learning from the simulation. But there is little consensus as to what kind of learning occurs from playing a simulation, and there is some disagreement as to whether measurable learning takes place at all. Apparently, part of the problem is that there have been few if any available ways to objectively measure whether learning takes place. According to Anderson and Lawton (1 997a): There is relatively little hard evidence (emphasis theirs) that simulations produce learning or that they are superior to other methodologies. Much of the reason for the inability to make supportable claims about the efficacy of simulations can be traced to the selection of dependent variables and the lack of rigor with which investigations have been conducted1. These authors are criticizing the definition and measurement of learning in most of the previous research generated on the construct. According to Gentry et a!. (1998), one of the problems in most attempts to measure learning is that what is intended to be learned has not been clearly specified. Anderson and Lawton (1997a) have agreed: Virtually all research designed to measure the outcomes produced by engaging in an activity requires by necessity assumptions concerning the expected outcomes produced by performing that activity. We cannot construct an assessment activity without knowing what it is we expect to measure. Previous efforts to design objective2 measuring devices to ascertain whether learning takes place from a simulation are relatively few. Corner and Nichols (1996) used course grades to measure learning from a simulation, and Raia (1966) and Wellington and Faria (1991) used course exams. There have been studies in which pre-defined objectives have been used to guide simulation learning measurement, but the objectives have often been very general. Faria and Whiteley (1990) and Whiteley and Faria (1989) divided learning in quantitative skills and applied knowledge. Pearce (1978-9) tested recognizing the firm’s external environment and the role of the company’s self-concept and its objective setting in the business policy formulation process. And Wheatley, Horneday and Hunt (1988) examined goal-setting abilities. One study in which specific learning goals were articulated was performed by Wolfe 82 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 (1976). His focus was on the effects of game participation on learning strategic management and organizational goal setting. His more specific objectives included ‘administer a preconceived strategy’ and ‘create the components of a business policy system.’ Wolfe (1976) developed an instrument from these objectives, and this same instrument was used to measure Strategic Management in three other studies (Keys, 1977, Wolfe and Chacko, 1983, and Wolfe and Guth, 1975). After reviewing the above studies it appears that in only one study (Wolfe, 1976) were specific objectives used to guide the development of an instrument measuring simulation related learning, and in none were measurement devices developed from specific objectives emerging from the simulation itself3. The purpose of this project was to develop a bank of test items and other measurement mechanisms that game administrators can use to measure learning from simulations. The bank was developed with specific objectives inherent in the simulation as a guide. These objectives were created by simulation scholars, authors and administrators. These contributors were asked what they expected their students to learn from playing a simulation. Items for many of the objectives were furnished by an overlapping set of scholars, authors, and administrators, who were asked to furnish items appropriate for measuring whether the learning objective in question were accomplished. The end result is intended to be a test bank of usable items, the objectives from which they emerge, and reliability and discrimination statistics. The intention is also to create simulation-learning related scales and validity statistics for each scale. METHOD AND PROCEDURE Procedure of objective and item creation. This effort was initiated in a seminar for instrument design assessing the effectiveness of simulations at the ABSEL conference in New Orleans in 1997 (Anderson and Lawton, 1997b). In the following year, five scholars, namely Dick Cotter, Jerry Gosen, Alan Patz, Tim Scott, and John Washbush created a list of learning objectives. Originally, the objectives were classified according to Bloom Taxonomy of educational objectives (Bloom 1956, Krathwohl et. al 1964). An overlapping set of scholars, namely Jerry Gosen, Al Patz, John Wasbush, and Joe Wolfe, furnished items that could be used to measure whether an objective could be accomplished. Phil Anderson, Bill Biggs, Dave Fritzsche, Denise Markovitch, Dick Teach, and Precha Thavikulwat also contributed. Objectives and Items. The result was 40 objectives and 120 items that could test whether a student was learning certain skills and concepts from the simulation. The resulting list of objectives is presented in Exhibit 1. Examples of items appear along with the list. Two of the objectives are sub-divided, one, #8, because the two objective authors provided different wording and the other, #37, because it involved team dynamics, a conceptual field of comparatively little interest for this scholarly effort. An appendix of items is available from the first author of this paper. Of the items, 102 were multiple choice, eight were short essay, six involved analysis of simulation-generated financial statements, four involved analysis of an hypothetical income statement, and one asked students to study hypothetical marketing information. Some of the items seem to fit with two or more objectives. These are noted in the appendix.4 In Exhibit 1, the number of items in the test bank for each objective is specified at the end. An asterisk (*) following an objective signifies the existence of short essay questions to measure the objective. 83 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 PILOT STUDY A pilot study was undertaken during the first six weeks of the Summer Session of 1998. Forty-one questions were randomly assigned to two forms of an examination designed to evaluate student cognitive abilities used in playing total enterprise simulations. Each test consisted of items totaling 50 points. The forms of the test were administered to undergraduate business students enrolled in and completing the BBA program at the University of Wisconsin-Whitewater. Thirty-five students were enrolled in the class, and 32 took Form 1 as a pretest at the beginning of the term. All 35 took the post-test (used as the course final examination) at the end of the term. The post-test was worth 15% of the course grade. Learning scores were determined for the 32 students who took both forms. Learning was defined as the difference between the percent correct score from the post- test minus the per-cent correct score from the pre- test. Percent correct score was determined for each student for each test by dividing the points awarded by points possible (i.e., raw scored divided by 50). The students played Micromatic (Scott, et al., 1992). One round of play was used for practice and then students played nine rounds (9 quarters) at the rate of about two decisions per week. Students were grouped into 10 company teams containing 3-4 members in one Micromatic industry (5 teams of 3, 5 teams of 4). Simulation performance was also graded by using the game’s scoring routine. Like learning, it was also worth 15% of the course grade. The mean pre-test percent score was 52.94%, and the mean post-test percent score was 60.47%. Since substantial variance difference occurred (pre-test 164.06; post-test 64.00), a conservative approach was used in comparing the mean scores by using a t-test assuming unequal variances. For this calculation, t = 2.821, p = .007 (two tail) and was significant beyond .01. Thus there is evidence that significant learning occurred over the course of the term. It is reasonable to suppose that the learning scores would have been higher over a normal semester than they were in the intense 6- week summer session. Normally, semester play extends for at least 12 quarters and there is more time for learning curve effects to have impact under those conditions. Reliability of the two forms was evaluated in two ways. First for both tests, reliability scores were determined using alternating item scores. In the case of the post-test, the subjective items were divided so that items assigned to each half totaled 10 points. Correlations between half-scores were calculated for each form of the exam. Reliability coefficients for each form were determined using the Spear-man-Brown (S-B) formula employing the correlation between halves as described in Guilford (1965, pp. 459-460). The results of these calculations were: Form Reliability Pre-test .8512 Post-test .5062 Reliability was also determined from item-test correlations using average item-test correlations as an estimator or average item intercorrelation and calculating the reliability using SB as described in Guilford (1965, p. 463). The results of these calculations were: Form Reliability Pre-test .7833 Post-test .71 11 At first glance these results do not prove these 84 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 tests to be reliable. However, the results do not prove the tests to be unreliable either, given the small sample size, brief and intense instructional period, the sensitivity of split-half methods to item assignment technique, and the Spearman-Brown results. DISCUSSION This paper summarizes an effort to build a test bank from pre-agreed upon objectives assessing whether learning takes place from playing a simulation. Scholars (Anderson & Lawton, 1997; Gentry et a!., 1998; Thavikulwat et a!., 1998) have called for efforts such as this. To our knowledge, the developmental work documented here is the first effort of its kind, given that the bank comes from pre-agreed upon specific objectives that emerge from a simulation. While a draft the bank is finished, the process of developing a reliable, valid set of items is in its early stages. The process is unfinished as there are no items provided for seventeen of the objectives and fewer than five items for thirteen more. There has only been one validity-reliability study of only one run of a simulation to assess the bank. Much needs to be done to further the process. More items are needed. More validity-reliability studies are needed. Item analyses have yet to be undertaken. While a pilot study used a sampling of the items to test the validity of the simulation, no studies have yet been performed to assess the items of the test bank. That means developing criteria to assess the items against. In addition, face validity feedback on the items is needed from experts in the field. The intention is to distribute items in the bank to anyone who wants them, and to provide data to users. The hope is that those who use the items will provide reliability and validity data to us so we can refine the bank. The objectives presented in this paper are different than the original set in two ways. First, there have been the normal conceptual and language-related improvements. The language has been modified, overlaps either made explicit or eliminated, complex objectives simplified. Second, the structural characteristics of Bloom’s taxonomy (Bloom, 1956; Krathwohl, et al., 1964) has been dropped. We believe that there would be too much contention regarding whether given items were to reach higher or lower level objectives and that the arguments would divert the field from the practical, important task of trying to create appropriate items. While we believe that the list of objectives is better developed than the item list, it too, is incomplete. Additions, subtractions and modifications are welcome. It should be pointed out that the data shows significant differences between post and pretests, supporting the notion that simulation is a valid learning exercise. The reliability statistics are mixed for the tests used in the pilot. In particular, the Spearman-Brown scores for the post-test were only .5062, suggesting concern and a trip back to the drawing board. Well, we are there anyway. 85 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 REFERENCES Anderson, P.H. & Lawton, L. (1997a). Demonstrating the learning effectiveness of simulations: where we are and where we need to go. Developments in Business Simulation & Experiential Exercises, 24, 68-73. Anderson, P.H. & Lawton, L. (1997b). Designing instruments for assessing the effectiveness of simulations. Developments in Business Simulation & Experiential Exercises, 24, 300. Bloom, B. (ed.) (1956). The Taxonomy of Educational Objectives. Handbook 1: The Cognitive Domain. New York: David McKay. Corner, L.B. & Nichols, J.A.F. (1996). Simulation as an aid to learning: How does participation influence the process? Developments in Business Simulation & Experiential Exercises, 23, 8. Faria, A.J. & Whiteley, T.R. (1990). An empirical evaluation of the pedagogical value of playing a simulation game in a principles of marketing course. Developments in Business Simulation & Experiential Exercises, 17, 53-57. Gentry, J.W., Commuri, S.F., Burns, A.C. & Dickenson, J.R. (1998). The second component to experiential learning: A look back at how ABSEL has handled the conceptual and operational definitions of learning. Developments in Business Simulation & Experiential Exercises, 25, 62-68. Gosen, J. & Washbush, J. (under editorial review). An exploration of the concept, perceived learning. Developments in Business Simulation & Experiential Exercises. Keys, B. (1977). A comparative evaluation of the management of learning grids applied to Business Policy learning environments. Journal of Management, 3, 33-39. Lawson, L. & Anderson, P. H. (in press). Measuring what is learned from simulations: Future directions for research. Simulation & Gaming: An International Journal. Guilford, J.P. (1965). Fundamental Statistics in Psychology and Education, 4th ed. New York: McGraw-Hill. Pearce, J.A.II (1978-1979). Developing business policy skills: A report on alternatives. Journal of Educational Technology Systems, 7, 361-371. Raia, A. P. (1966). A study of the educational value of management games. Journal of Business, 39. 339-352. Scott, T.W., Strickland, A.J., Hofmeister, D.L. & Thompson, M.D. (1992). Micromatic: A Management Simulation. Boston: Houghton Mifflin. Washbush, J. & Gosen, J. (under editorial review). Learning in total enterprise simulation: Do players learn, does learning vary with performance, why do some players learn more than others. Simulation & Gaming: An International Journal. Wheately, W.J., Hornaday, R.W. & Hunt, T.G. (1988). Developing strategic management goal setting skills. Simulation & Games, 19, 173-185. 86 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 Whiteley, T.R. & Faria, A.J. (1989). A study of the relationship between student final exam performance and simulation game participation. Simulation & Games, 20, 44-64. Wolfe, J. (1976). Correlates and measures of the external validity of computer-based business policy decision making environments. Developments in Business Simulation & Experiential Exercises, 7, 411-438. Wolfe, J. (1997). The effectiveness of business games in strategic management course work. Simulation & Gaming: An International Journal, 28, 360-376. Wolfe, J. & Chacko (1983). Team size effects on business game performance and decision-making behavior. Decision Sciences, 14, 121-133 Wolfe, J. & Guth, G.R. (1975). The case approach versus the simulation in the teaching of business policy: An experimental evaluation. Journal of Business, 48, 349-364. 1 There is disagreement on this point. For example Wolfe (1997) argues that there is evidence that simulations bring about learning in Strategic Management Courses. 2 Many studies measure perceptions of learning. See Anderson and Lawton (1 997a) or Lawton and Anderson (in press) for a review and Gosen and Washbush (under editorial review) for a research study. This study focuses on attempting to measure learning objectively and chooses to avoid player perceptions of learning 3 It should be noted that studies in this paragraph by Keys (1977), Pearce (1978-9), Wolfe (1976), and Wolfe and Chacko (1983) all attempted to assess the simulation by comparing it with other learning methodologies in terms of learning outcomes. In the present study, simulation learning was assessed, but there was no intention to compare the simulation with other methods. 4 At the end of the opening statement of the item, cross reference objectives are noted in parentheses. 87 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 EXHIBIT I: OBJECTIVES WITH EXAMPLES OF ITEMS 1. Correctly use game rules to make decisions (note: items emerging from this objective must be game specific). (0) 2. Attend to detail, such as ordering raw material, accounting for employee turnover, or distributing the sales force, so that poor performance does not result. (5*) EXAMPLE Manufacturing costs per unit sold will be lowest when a. Worker wages are reduced b. Labor capacity are used optimally c. Materials are purchased in small quantities d. Interest expenses are reduced 3. Understand the consequences of specific decisions, such as ordering materials at a discount or issuing stock. (34) EXAMPLE Which of the following actions will normally increase the market price of a firm’s stock? a. maintaining earnings b. continuing dividends c. buying back stock d. issuing stock 4. Understand the meaning of business related concepts such as inventory, raw materials, equity, or strategy type. (31 *) 5. Apply models involving cash flow, growth, profits, assets and dividend payments. (4*) EXAMPLES Deferred taxes are a. Long term debt obligation b. Payable in the next accounting period c. Current liability d. None of the above An important contributor to profits in a manufacturing firm is keeping cost of goods sold as low as possible. How can a firm do that? _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ 88 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 6. Understand and effectively interpret the game’s financial statements. (22*) 7. Enhance an understanding of group dynamics, structures and processes. (0) 8. (1) Understand and distinguish between market structure, rivalry and other economic forces that affect the firm. (2) Understand and distinguish between market characteristics, competitor behavior, and other economic forces that influence decisions. (7 items for both*) EXAMPLE Which of the following is the best course of action in a stagnant market? a. Raise marketing expenditures and reduce work force size. b. Improve manufacturing efficiency. c. Increase market research expenditures. d. Market penetration. 9. Derive and implement effective and efficient decisions which address situations, problems, and opportunities that arise during the simulation. (20*) 10. Appropriately use financial statements in decision making. (6) 11. Appropriately apply strategic concepts to decision making. (17) EXAMPLE If a large company, which is currently producing a full range of cars in the automobile industry, wished to expand its operations to include the manufacture of major home appliances such as washing machines, stoves and refrigerators, its organizational form should change in which of the following fashions: a. From a simple structure to a functional structure b. From a divisional structure to a conglomerate structure c. From a functional structure to a divisional structure d. From a conglomerate structure to a divisional structure 12. Know strategic concepts well enough to identify strategic issues. (2) 13. Apply knowledge about strategic plans to identify and differentiate strategic plans. (0) 14. Apply economics theory. (1) 15. Create and Implement internally consistent strategies. (3) 89 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 EXAMPLE If you use the best materials in your product, you should probably also a. advertise more b. train workers in craftsmanship c. borrow money at higher interest rates d. implement a high volume strategy 16. Use pro forma or what if analyses to assess the probable impacts of decisions. (3) 17. Assess periodic performance in terms of costs, profits, sales, assets, equity, and stock price. (3*) 18. Analyze a company’s marketing mix and by accurately estimating the relative influence of each of the marketing factors, competitors’ behavior, demand and the degree to which industry effort matches demand. (2*) 19. Allocate costs on a per unit basis. (0) 20. Forecast product demand with reasonable accuracy. (0) 21. Use financial statements to enhance decision making. (3*) EXAMPLE Identify the problems with the company producing the following financial statements. Suggest what to do about these problems. If you do not think there is a problem, say so. (21) Company A Income Statement 1994 1995 1996 1997 1998 Sales 200 210 220 230 250 Cost of goods sold 150 160 170 180 200 Sales & Admin Expense 20 21 22 23 25 Warehouse Expense 4 4 4 4 5 Interest 2 2 2 2 2.5 NIBT 24 23 22 21 17.5 ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 22. Understand the relationships between cash flow, growth, profits, asset value, and dividend payouts. (3*) 90 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 23. Utilize incomplete data to enhance effective decision making, under time constraints. (0) 24. Effectively use analytic techniques in decision making. (0) 25. Develop a strategy that is coherent, internally consistent, and appropriate to past events and market conditioned. (4) EXAMPLE The successful implementation of a Generic Strategy of Cost Focus requires the firm to a. Spread its unit costs over as many products as possible. b. Create a unique product for each of the industry’s product/market segments. c. Simplify and automate production. d. Purchase inexpensive parts and sub-assemblies from foreign manufacturers. 26. Formulate a strategic plan. (0) 27. Adapt strategy and decision making to changing circumstances. (0) 28. Reason carefully about strategic options under conditions of limited information. (0) 29. Properly balance marketing, production and financial factors in decision making. (0) 30. Apply and integrate marketing, finance, accounting, management, and production knowledge to decisions. (0) 31. Apply strategic planning models. (9) 32. Apply models involving cash flow, growth, profits, assets, and dividend payouts. (2) 33. Select and effectively use appropriate analytical methods. (1) 34. Assess business performance in terms of evaluative indices. (3) 35. Relate the game’s principles and skills to one’s career. (0) 36. Evaluate and improve the effectiveness of strategies. (0) 91 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 37. Team-related (implies peer assessment). (0) a. Behave effectively in teams, which solve problems. b. Communicate clearly with peers. c. Resolve and manage conflicts effectively in a group setting. d. Manage organizational processes by which strategies get formulated and implemented. 38. Implement a strategic plan successfully. (0) 39. Utilize experience, practice, and coaching to enhance effectiveness. (0) 40. Understand and apply the lessons offered by experience. (0) a. Apply concepts to the business world. 92 Table of Contents Volume 26, 1999 ABSEL's Historical Research Interests Back From the Future: An ABSEL Merlin Exercise for the Year 2005 The Contributions of ABSEL During the 1980's ABSEL's Contributions to Experiential Exercises in the 90's ABSEL's Contributions to Experiential learning/Experiential Exercises: The Decade of the 1970's Business Simulations - Algorithms and Model Enhancements: A 25 year Review A Study of the ETS General Field Test as an AACSB Assessment Tool and the Impact of Experiential Exercises and Simulation on Learning A Framework for Assessing the Competencies Reflected in Simulation Performance Developing A Learning Culture: Assessing Changes in Student Performance and Perception Financial Simulation Using Distributed Computing Technology Analyzing Managers' Judgements and Decisions with an Educational Business Simulation Understanding Your Business through Home-Made Simulator Development An Examination of a Reanalysis of the Impact of a Market Leader on Simulation Competitors' Strategies Applying Shocks to TE Simulations: A Demonstration Increasing Efficiency of Management Skill Assessment A Testbank for Measuring Total Enterprise Simulation Learning Developing Leadership Skills - Video Live! LEADSIMM: Collaborative Leadership Development for the Knowledge Society A Team Approach to Producing Multi-Media Laptop and Video Formatted Presentation Tinkertoys Revisited: Exploring Trust Based Relationships Overall Dominance in Total Enterprise Simulation Performance Success or Bankruptcy: The Relationship between Personal and Goal Orientation and Simulation Performance Assessing the Effects of Feedback: Muti-method and Muti-directions in Multi-pedagogical Courses The Missing Ingredients in Experiential Learning Purpose and Learning Benefits of Business Simulations: A Design and Development Perspective Building Capabilities for Change through Laboratory Simulations Modeling Innovation as a Process of Design in Educational Business Simulation The Need to Measurer variance in Experiential Learning and a New Statistic to do so Assessing Effectiveness of an Experiential Oriented Course Over Time Developing Participant Satisfaction Models of Experiential Exercises in Business Education Perceptions of Learning in TE Simulations Students' View on the Use of Business Gaming in Hong Kong Management Gaming's Lost Opportunity: Meditations about the Russian Experience and Prospects for the Future Unanticipated Enhancements in the Business Strategy and Policy Game when Running in Windows 95 Using a Business Game to Demonstrate Broad Business Concepts A New Model for Business Courses (Getting the Student Connected) Seeing the Forest and the Trees: Integrating Knowledge Using Large Scale Simulations in Capstone Business Strategy Classes Is It Here To Stay? A Roundtable Discussion of the Inter-Group Interaction Interactive Tools Used in Applying Financial Concepts Strengthening Essential Skills through a Finance Exercise: Calculation of Beta A Model of Currency Exchange Rates Understanding Currency Exchange Rates: A three-part Exercise Student Experiences in the World Intercollegiate Business Game Competition Student Experiences in the International-Collegiate Business Policy Game Competition Using Boards of Directors in Simulation Environments: Comments From Board Members Sharing Best Practices: Teaching Smarter, Not Harder Star Power: A Simulation for Understanding Power and Empowerment The Marketing Game: A Marketing Principles Simulation Alexander Islands: GSSM Tiny Business Simulator on the WWW Putting Strategy into Strategic Business Games Industry Analysis, Porter's Five Forces Model and Strategic Group Maps in the Business Strategy Game Simulation The Use of Concept Mapping to Improve Student Performance and Understanding of Strategic Management Concepts: A Comparison of Techniques Transformational thinking in the Organizational Behavior Course: The Use of Metaphor as an Assessment Tool Creation of a Virtual Learning Community for the Global Virtual Enterprise Project Inter Institutional use of Educational Resources: Joint Use of Management Simulation Games Business Insights: Theory and Practice with the Aid of a Business Simulation Progress: An Experiential Exercise in Development Marketing The ABC's of teaching the Theory of Constraints to Undergraduate Business Students Demonstrating Principles of Organizational Purchasing Behavior through an Experiential Exercise The Virtual Manager: A Different Simulation for Managing Complexity When the Rules are changing and Chaos Breeds Innovation: Recapturing the Value of Constructive Thinking and Play in Simulation Training The Pitfalls, and Potential, of Actual Events as Problem Drivers Using Business Games to teach Environmental Awareness and Green Management: The International Experience with the ENSIM Game A Review of my ABSEL-Related Work Simulation of Government: A Workshop Using GEO Creating Internet-Based Games Using Perl and JavaScript Who's on First? Exploring the Concepts of Problem-Based Learning, Experiential Learning, and Lifelong Learning Students Learn Customer Service and Selling while Conducting Research So You Want to Run an NFL Football Team–An Honors Interdisciplinary Project Supervised Internship: The Employer's Perspective Using Computer Assisted Simulation to Teach International Business Strategy: A Case of the Multinational Management Game (MMG) Multi-Cultural Experiential Learning: A Computer Simulation in China An Appreciative Stance on Diversity as We Move into the 21st Century: A Timeline Exercise to Identify Key Experiences in Good Work Relationships between Black and White Peers A Day in the Life of an Interactive, Real Time, and Internet Delivered Course: A Demonstration Comparing Internet Search Engines: An Experiential Learning Exercise Total Enterprise Simulations and the Internet: Assessing Student Perceptions and Preferences Teaching Accounting Information Systems in a Practicum Format Providing Experiential Learning in Accounting through a Field Study Payroll Project A Spreadsheet Based Business Simulation Game Computer-Behavioral Simulations Training for Project Managers Simulation Scenarios - Rationale and Illustration