THE TOURNAMENT CONCEPT: EXTENDING THE IDEA OF USING SIMULATIONS AS INSTRUMENTS OF ASSESSMENT Developments in Business Simulation and Experiential Learning, Volume 28, 2001 THE TOURNAMENT CONCEPT: EXTENDING THE IDEA OF USING SIMULATIONS AS INSTRUMENTS OF ASSESSMENT Precha Thavikulwat, Towson University (pthavikulwat@towson.edu) ABSTRACT Defining a competitive event as a setting wherein all competitors begin at the same or equivalent positions and a tournament as a collection of two or more competitive events, the tournament concept was tested in a very complex total-enterprise simulation that was organized into a tournament of 15 competitive events. The data supports the effectiveness of the tournament, and suggests that participants remained challenged for the entire 30 decision periods of the competition. Problems encountered that have since been resolved were the illiquidity of excess inventory, immobility of equipment, and unenforceability of organizational structure. The issue of breaking down overall scores into components corresponding to specific skills remains to be studied. INTRODUCTION The argument that a business simulation is an ideal setting for the assessment of business skills has been observed to be compellingly reasonable. “After all, what more logical candidate for measuring student potential in a real work situation than performance in an exercise designed to simulate a real situation?” (Anderson, Cannon, Malik, & Thavikulwat, 1998, p. 36). The idea itself is far from new, reaching back over three decades to Vance and Gray (1967), whose “hope for a major breakthrough” (p. 37), however, has apparently remained unrealized. Anderson, Cannon, Malik, and Thavikulwat (1998) have suggested that the problem lies in the absence of a theory of simulation game performance. They asserted that “the validity of both the educational approach and the assessment measures are dependent on the educational outcomes—the key skills needed for success. Without knowing what these are, we have no way of knowing whether the educational approach and assessment measures are valid” (p. 36). In a follow-up article, Cannon and Burns (1999) suggested that the appropriate framework for such a theory is Bloom’s classical taxonomy, which breaks down performance into a hierarchy of cognitive, affective, and psychomotor domains that must be learned in succession (Bloom, Englehart, Furst, Hill, & Krathwohl, 1956; Krathwohl, Bloom, & Masia, 1964). Within that framework, Cannon and Burns suggested a three-dimensional approach consisting of educational objectives, management skills, and multiple items. Yet, as Jones (1998) has pointed out, “Teachers love aims, but authors love ideas much more. To stipulate that an author should start with an aim then follow it by devising parameters is, to my mind, unrealistic and inefficient” (p. 342). The work reported here came from an idea presented by Pogossian (1998, 1999), who suggested a tournament whereby participants would be assessed by their collective scores across a number of competitive events. The method does not rely on any theory of learning. It merely assumes that each participant possesses a stable set of strategies with respect to the competition in question, and that the set of strategies available to each participant is individualized. Within each competitive event, each participant will put into play that participant’s best strategy for the event, given the strategies played by all other competitors. Thus, in a tournament of many competitions, each participant’s set of strategies will be competitively tested against many other sets, resulting thereby in scores that measure the competitive value of the set of strategies possessed by each participant. In other words, skill is ownership of strategy. Those with superior skills possess superior strategies. Pogossian’s theory is silent as to how individuals come into possession of their strategies. They may have learned them in a formal setting, or they may have figured them out by themselves, or they may even have been born with them, as instincts that are manifestations of genes. The point is that for the purpose of assessment the historical source of a strategy does not matter, so learning theory is superfluous. One participant’s learned strategy may be tested against another’s self-developed strategy and against a third’s instinctual strategy. What matters is that strategies are stable, that is, that what one is able to do today one is able to do tomorrow. From this perspective, assessment depends on a tournament of competitive events each of which must be functionally real, so that the skills superior in the simulated setting are the same as those that would be superior also in the everyday setting. The simulation must present “reality of function in a simulated environment” (Jones 1982/1998, p. 333). Participants must accept the relevance of the duties and responsibilities assigned to them. They must not “merely go through the motions and have fun and games, or play it for a laugh, or ham it up” (Jones 1982/1998, p. 333). DEFINITION OF CONCEPT To avoid confusion, a tournament must be distinguished from an event with multiple measures. An event is a setting wherein all competitors begin at the same or equivalent positions. Although multiple measures may be taken in an 225 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 event, the positions of the competitors are not reset to the start after each measurement. A tournament is collection of two or more events. The events may be repetitive, related, or different. In all cases, however, competitors start at the same or equivalent positions at the beginning of each event. APPLICATION TO TOTAL ENTERPRISE SIMULATIONS Total-enterprise simulations are commonly administered as batch-processed single events of between 4 and 16 periods (Anderson & Lawton, 1992; Rollier, 1992), each of which may be called a quarter or a year. A trial run may precede the real competition (Fritzsche & Cotter, 1990). Each simulated company is managed by a team of participants (Wolfe & Chacko, 1981a, 1981b) whose membership is generally stable for the duration of the competition. The reward for performance is points towards grade, often moderated by peer ratings, supplemental assignments, and the instructor’s judgment (Anderson and Lawton, 1992). Performance measures, objective or subjective, unidimensional or multidimensional, (Biggs, 1978, 1990), profit based or forecast based (Hand & Sims, 1975; Teach, 1990, 1993a, 1993b; Wolfe, 1993a, 1993b) may be taken after each period or collection of periods. The conditions of the companies, however, are not equalized at the end of any period to launch a new event. Accordingly, the minimum action needed to convert a total-enterprise simulation event into a tournament is to reset the conditions of companies such that all companies are equalized one or more times before the end of the competition. For a 4-period competition, for example, resetting the companies at the end of the second period gives rise to a tournament of two events; for a 12-period competition, resetting it at the end of every even-numbered period gives rise to a tournament of six events. Thus, the conversion of a single- event competition to a tournament can be accomplished by partitioning the box that contains the simulation, without having to expand the container. The catch in this solution has both a hard side and a soft side. On the hard side, the companies that are equalized must truly be competitively equal, in the mathematical sense. Otherwise, the advantage that some companies will have over others will compromise the fairness of the subsequent competitive event, and consequently the validity of the scores. On the soft side, the resetting must appear to participants as legitimate, so as to maintain reality of function. A participant who does not accept the reality of the change may “abandon the function [of the assigned role] and become actor, author, comedian or saboteur” (Jones, 1982/1998, p. 333), thereby undermining the validity of the assessment. The dilemma is that if the companies are reset to identical conditions, assuring hard-side, competitive equality, the action may appear contrived. How is it ever conceivable that companies starting identically would become identical again after an interval of substantial business activity? The dilemma can be resolved by resetting the companies to different but equivalent positions. Achieving accounting equivalence is easy. Mandating dividend payments (or angel investments in the case of losses) such that the net worth of all companies becomes identical accomplishes it. Achieving operating equivalence is more difficult. The simulation program must allow companies to dispose of excess inventory, to make up for resource shortages, to restructure fixed assets, and to refinance as needed, without crippling penalties. Although companies that are different can never become completely equal, the remaining inequality may be controlled by a sufficiently extensive computer program supporting the simulation. Thus, any of the commonly available total-enterprise simulation can be converted from single-event competitions to tournaments by administratively mandating dividend payments (or angel investments) from time to time. Moreover, the transition from one event to the next can be enriched by reorganizing the teams. The resetting and reorganization may resolve also the matter of early dominance, whereby the relative standing of teams change little after the first four periods (Rollier, 1992; Patz, 1992, 1999, 2000), although Peach and Platt (2000) reported not see the effect before eight periods. HYPOTHESES One way to gauge the effectiveness of applying the tournament concept is to examine the correlation of scores between events. A high correlation suggests that the events are redundant. In this case, all events except one can be dropped with little loss in the assessment power of the system, so the complexity of the tournament is unnecessary. On the other hand, if the scores do not correlate more frequently than would be expected from chance, chaos is suggested, implying that the scores measure nothing beyond the noise of the system. This second of the two boundary considerations leads to the first null hypothesis, as follows: • H1. Scores between different events of the tournament do not correlate more frequently than would be expected from chance. When a tournament is created by segmenting an integrated simulation experience, scores that are correlated should generally arise from events that are proximal to each other, as compared with events that are distant from each other. This leads to the second null hypothesis: • H2. The temporal distances between events whose scores do correlate will be the same as those between events whose scores do not correlate. Moreover, scores that are correlated should generally arise from events occurring later in the integrated simulation experience, because chance factors would attenuate as the experience progresses. This leads to the third null hypothesis: 226 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 • H3. The earlier event time of events whose scores do correlate will be the same as the earlier event time of events whose scores do not correlate. (In a tournament with an integrated simulation experience, every correlation between scores is between scores of an earlier event and those of a later event.) Finally, if the simulation achieved reality of function, participants’ actions would have been consistent with their assigned roles in the simulation. In particular, if participants are allowed to invest in the companies of the simulation, then their investment decisions should be explainable by the performance of the companies in the immediately preceding events. This leads to the last null hypothesis: • H4. Multiple regressions of stock purchases by participants with respect to the earnings of companies in preceding events will show no fit. THE GAMING SIMULATION The tournament concept was executed by segmenting the CEO total-enterprise gaming simulation, without extending its length. CEO is a gaming simulation with special assessment characteristics. Notable among these characteristics are the tracking of individuals and the gaming of product and stock markets. Similar to other total-enterprise simulations, CEO simulates companies managed by teams of participants. Different from them, CEO assigns each participant a periodic income that each can use to buy stock in the companies of the competition and to buy products made by those companies. Thus, in addition to team performance scores, CEO supplies individual performance scores based on the extent of each participant’s consumption, which is affected both by the participant’s skill as a consumer shopping for the highest value products and by the participant’s skill as an investor buying shares in the companies of the competition. In essence, CEO is a superset of the typical total-enterprise simulation. It may properly be called a very complex total- enterprise simulation, or a total-economy simulation. A schematic showing how a typical total-enterprise simulation is embedded in CEO is given in Figure 1. The validity of CEO’s individual performance score as a measure of business skills has been supported by a four-year study showing its correlation with business-students’ grade point average (Pillutla & Thavikulwat, 1998). The same study also showed a correlation between individual performance scores and grades in a course entitled Quantitative Methods for Business I, but not with other lower-level core business courses typical of an AACSB-accepted curriculum, therefore suggesting that the individual performance score was an incomplete measure of business skills. The study supplied no data on team performance scores, although one might infer that because individual scores arise from team scores (i.e., company profits), if the individual scores are reasonably valid, the team scores must be reasonably valid also. In this particular execution of the tournament concept, participation in the CEO simulation was the sole activity of a semester-long, senior-level, required course called Management Experience Simulation. The participants in the exercise were from a day section (38 students) and an evening section (36 students) of the course. Day students attended classes twice a week; evening students, once a week. The simulation was administered to each section independently of the other, in other words, each section constituted its own industry. Grading included pluses and minuses. Thus, although all students were reasonably assured of passing the course, a small difference in game performance could result in a difference of grade. 227 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 Consumption Points of Individual Participants Product-Purchasing Decisions of Individual Participants Inheritance & Periodic Entitlements of Individual Participants Dividends Received by Individual Participants Stock-Purchasing Decisions of Individual Participants Dividend Payments of Firm #2 Earnings of Firm #2 Dividend Payments of Firm #1 Earnings of Firm #1 Typical Total- Enterprise Simulation Measurement Point Decision Point Parametric Setting Figure 1 CEO as a Superset of the Typical Total-Enterprise Simulation 228 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 PROCEDURE The CEO gaming simulation was administered for its standard length of 30 periods, evenly divided over 5 phases. Generally, participants progressed through two periods within each 75-minute class session. Processing of decisions was automated and handled asynchronously (Thavikulwat, 1996) through programs and files installed on a local area network. This meant that at times some participants finished their work early and left early whereas some others could not finish in the available time and had to complete their work either outside of the class session or in the following session. In any class session, time was of the essence because the market for products was best when the greatest number of participants (i.e., customers) were present and the fewest number of competitive offerings were available. Accordingly, team members felt endogenous market pressures to work quickly so that their products would be available in the marketplace early. In sum, the game ran for many more decision cycles each of which required a higher intensity of effort than is generally practiced with total enterprise simulations. In administering the tournament, an event was defined as two periods, with mandated dividend payment at the end of each even-numbered period equaling total earnings over the two periods. The scenario presented to participants was that each period represented half a year, and that company policy mandated complete payment of all earnings as dividends at the end of each year. To enable companies to maintain sufficient financing despite the total loss of retained earnings in each alternate period, the simulation was configured to allow virtually unlimited borrowing. The effect of this policy was to create a tournament of 15 integrated competitive events, each of which spanned two periods. The time-based breakdown of the simulation experience is shown in Figure 2. The five phases of the simulation experience were of progressive difficulty. Each participant managed a company in the first phase; four- or five-person teams managed companies in the remaining four phases. Teams were instructed to be organized by business function in Phases 2 and 3, and by product line in Phases 4 and 5. The president of each company job-hopped to a lower-status position at another company between phases, and jobs were rotated then to accommodate the turnover. As a consequence, no team was completely intact for more than a single phase, but the turnover in each team was limited to one person between phases. The solo phase was first. This was actually a qualification phase in which companies were limited to one plant and run completely by one participant, who could execute repeated runs of the phase without limit. Sales were confined to a market whose demand was modeled, with no interdependence between companies either in the acquisition of resources or in the sales of products. Participants received full credit towards grades for reaching fixed earnings targets, set at levels only slight higher that the company’s cost of capital. Although almost everyone was able to reach the targets, those who completed their assignments earliest and achieved the highest earning levels were rewarded by being designated presidents of companies that would continue with the competition. Remaining companies essentially failed to qualify, so their founders had to secure jobs with the companies that qualified. By this merit-based selection system, the 74 companies of Phase 1 were reduced to 17, 9 in the day section and 8 in the evening section. The expansion phase was second. In this phase, companies could have up to three plants, each of which produced a different product requiring a different mix of resources. The products could be sold to participants, who now began receiving a periodic income that could be used to buy those products, in order to get points towards grades. The phase ended with a complete withdrawal of capital on top of the regular withdrawal of earnings. As a consequence, all companies ended Phase 2, and would thus begin Phase 3, with no net worth. The policy phase was third. Companies began this phase by each making presentations to investors, which meant the class. Following the presentation, all participants were able to purchase shares in companies they would not be managing in that phase. Within the phase, each manager received periodic stock options, valid only if exercised in the period issued. Accordingly, managers who were absent for any period had to forgo their stock options for that period, the only form of endogenous compensation for their efforts. Tournament 1 Phase 1 2 3 4 5 Event 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Period 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Milestone Form Teams First Public Sales of Stock Second Public Sales of Stock Third Public Sales of Stock Organization Solo By Function By Function By Product Line By Product Line Figure 2 Breakdown of Simulation Experience The strategy phase was fourth. As with the previous phase, companies began this phase by making presentations and selling additional stock. Prior to their presentations, 229 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 however, the companies could adjust up to 73 parameters that constrain their individual companies, with the limitation that no single parameter could be changed by more than 20% in either direction and that the absolute percentage sum of the changes could be no more than 100 percentage points. Moreover, the point-value of products, previously fixed, now became variable so that at any given time, the point value of each product as compared with other products was highest for some consumers and lowest for others. On average, however, all products had equal value. The total quality phase was fifth. In addition to all the features of the previous phase, two hidden assignable causes of defective products became active. This meant that if a company could not find and address the causes, it would have to either absorb the high cost of screening out defective products or sell products with high rates of defects. Inasmuch as a consumers who bought defective products received no point despite money spent, companies selling products with high rates of defects came under severe market pressure to reduce their prices. RESULTS The performance of companies as measured by their earnings in each event is summarized in Table 1. The table excludes the solo phase (Events 1 through 3), because teams had not been formed at that time. Neither the mean nor the variance of cumulative earnings over all 12 listed events of the day section is statistically different from that of the evening section, but the variances of earnings from Events 6, 10, and 15 are statistically different between sections at the conventional five-percent level of significance. Thus, sectional differences are not altogether negligible. Controlling for the effect of the section on company earnings, the partial correlations among earnings across events are given in Table 2. Of the 66 correlations, 7 a statistically significant at the five-percent level or higher frequency greater than would be expected by chance, Z 2.087, p < .05. Accordingly, H1 is rejected. Means and Standard D Company Earn Day (N = 9) Event M SD 4 3,769 7,340 5 98,601 98,070 6 165,216 103,579 7 153,414 106,049 8 195,880 107,305 9 237,537 144,811 10 248,551 100,662 11 191,595 201,518 12 184,297 223,140 13 162,960 129,686 14 132,556 138,437 15 44,928 62,765 4-15 1,801,197 774,485 *p < .05, **p < .01 Table 1 eviations of Company Earnings ings ($) Evening (N = 8) M SD F (Equal Variance) t (Equal Mean) 1,499 3,104 2.625 0.847 112,048 161,994 0.481 -0.204 254,013 196,008 5.442* -1.147 253,775 133,545 1.084 -1.702 214,695 309,908 3.759 -0.163 93,938 211,764 0.466 1.612 458,790 304,221 11.081** -1.866 103,569 76,712 2.087 1.215 115,028 116,140 0.527 0.815 347,145 535,614 2.235 -0.945 160,868 220,107 1.517 -0.313 268,866 349,029 0.811** -1.789 2,384,235 1,206,135 2.876 -1.170 re , a = 230 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 Of the significant correlations, five are between events that are either temporally adjacent or within one event of each other, and the remaining two are separated by no more than three events. The average temporal distance between events whose scores are significantly correlated is 1.43 events, whereas that between events whose scores are not significantly correlated is 3.56 events, t(64) = 2.014, p < .05. Accordingly, H2 is rejected. Moreover, of the significant correlations, the earliest of the earlier event times is Event 8. The average earlier event time of events whose scores are significantly correlated is 10.0 events from the start of the exercise, whereas that of events whose scores are not significantly correlated is 7.0 events from the same start, t(64) = 2.910, p < .01. Accordingly, H3 is rejected. Results of multiple regressions of stock purchases by participants with respect to company earnings in the three events of the immediately preceding phase are given in Table 3. Every regression explains about half of the variability in stock purchases, and all are statistically significant at the five-percent level or higher. Accordingly, H4 is rejected. Table 3 Regressions of Stock Purchase on Company Earnings in Immediately Preceding Events β Immediately Preceding Event Stock Offering First Second Third R2 F First .427 .438 .145 .497 4.273* Second .344 .592** .317 .609 6.739** Third .675** .055 .236 .580 5.985** *p < .05, **p < .01, df = 13. Every null hypothesis has been rejected, yet, it is still reasonable to ask if the simulation experience remained a challenge in the later periods, especially considering Rollier’s (1992) observation that “using a game of reasonably high complexity…, by the sixth [period], … there was no longer much uncertainly about how to stay profitable” (p. 447); Patz’s (1992, 1999, 2000) repeated finding, from administering different total-enterprise simulations to different populations over 10 or fewer periods, that dominant teams at the end of the competition had established and maintained an early lead; and Peach and Platt’s (2000) finding that “ten to twelve decisions periods are optimal for achieving learning and then reinforcing belief and acceptance” (p. 247). Table 2 Partial Correlation Coefficients of Earnings Event 5 6 7 8 9 10 11 12 13 14 15 4 -.128 .012 .099 -.048 .239 -.008 -.014 .287 .038 .458 .285 5 .239 .072 -.186 .010 .488 .026 -.006 .044 -.004 .039 6 .220 .006 -.207 -.075 -.136 -.207 -.140 -.235 -.400 7 .111 .269 -.105 .043 .020 -.226 .337 .103 8 -.036 .628* .341 .339 .339 .088 -.292 9 .087 .579* .467 -.111 .583* .494 10 .364 .580* .579* .259 -.181 11 .197 .243 .359 -.064 12 .631* .654** .131 13 .442 -.232 14 .316 *p < .05, **p < .01, df = 13. If the simulation experience had ceased to be a challenge past 12 periods, participants’ would be able to predict the relative standing of companies in the events that followed. Based on that prediction, their investment decisions would have been explainable by the earnings of companies in the subsequent events. To see if this might be so, multiple regressions of stock purchases by participants with respect to the company earnings in the three events immediately following investment decisions were computed. The results are presented in Table 4. None of the regressions is statistically significant at the five-percent level, although a trend toward greater significance in the later stock offerings is apparent. The trend suggests that by the third stock offering, Period 25, the tournament was approaching the point where it would no longer be a challenge, but it apparently remained a challenge even then. Table 4 Regressions of Stock Purchase on Company Earnings in Immediately Following Events β Immediately Following Event Stock Offering First Second Third R2 F First -.206 -.197 -.024 .079 0.373 Second .047 .282 .521 .406 2.956 Third -.344 .479 .256 .439 3.131 *p < .05, **p < .01, df = 13. 231 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 CONCLUSION Considering that all the null hypotheses have been rejected by the data, this study concludes that the tournament concept can be effectively applied to a complex total-enterprise simulation by segmenting an integrated experience, without extending the length of the experience. The study also shows that if the experience is sufficiently enriched and if the supporting computer programs are sufficiently powerful, participants can be kept challenged for at least 30 decision periods. This number of decision periods should alleviate concerns expressed by Teach (1990) that total enterprise simulations mislead by rewarding shortsighted decision-making. In the course of conducting the study, three problems became apparent that have since been resolved. The first was the illiquidity of excess inventory; the second, the immobility of equipment; and the third, the unenforceability of organizational structure. The liquidity and mobility problems limited the extent to which companies could be made competitively equal at the beginning of each event. The enforceability problem made it difficult to trace any particular decision back to the participant who actually made it. With respect to the liquidity problem, the original simulation did not allow companies to liquidate excess material inventory. Consequently, companies that had excess material inventory at the end of an event had to absorb the excess carrying cost over the next event. In a few instances, the material inventory was so excessive as to make it impossible for the companies to realize a profit without administrative intervention in the form of no- interest loans. This problem has since been resolved by allowing companies to liquidate material inventory, subject only to a fixed liquidation cost. With respect to the mobility problem, the original simulation did not allow companies to move equipment from a plant where it may be idle to a plant where it could be used. As in the previous case, the handicap of equipment in the wrong plant could be inherited from an earlier event. This problem also has been resolved by allowing companies to move equipment at will. With respect to the enforceability problem, this is a long-standing problem common not only to total-enterprise simulations, but to other simulations as well. In reviewing a number of computerized general management simulations, Wolfe and Rogé (1997) observed that “many games, although recommending various job responsibilities and alternative organization structures, were not themselves structured in ways that reinforced whatever formal structure was chosen by the students when they played the game” (p. 436). Moreover, in a critique of computerized simulations generally, Jones (1991) writes, “The fact that a computer is a box with a keyboard designed for a single person physically alters behavior and authority…. Once they face the machine, the whole group can come under the influence of the arcade dichotomy of operator and spectators” (p. 236). The problem is especially acute for simulations designed for assessment, because an unenforceable structure means decisions may not be correctly attributed to the person who was assigned to make them. In this study, the administrator attempted to reinforce the assigned product-line structure of Phases 4 and 5 by giving each product-line manager partial credit for the accounted earnings of that manager’s product line. Despite the reinforcement, the administrator observed instances in which the decisions of a product line were not made by the manager assigned to it. Although Jones faults the computer for the problem, the problem in inherent to any centralized data entry scheme. The problem has been resolved, not by dispensing with the computer, but by augmenting the computer program to decentralize data entry. The new scheme requires each team member to identify himself or herself to the computer program, which then restricts that member to entering only those decisions for which that member has authority. Although the scheme can be defeated if members choose to share their individual passwords, defeating it generally will not be in the interest of members because it would mean the loss of individual control and privacy. Problems lead to solution, which brings forth more problems to solve. So it is with respect to this study. The problem of using simulations for assessment led to the tournament solution. The tournament solution surfaced liquidity, mobility, and enforceability problems, all of which have nevertheless been resolved. The next problem to be resolved may be that of breaking down overall scores into components corresponding to specific skills. Presentation skills, for example, can be measured objectively by the sales of stock, but the validity of this measurement has yet to be systematically studied. The measurement of product- and stock-purchasing skills also might be investigated. Thus, although it may be compellingly reasonable to use simulations for assessment, the path is not well paved. In time, however, perhaps this way will become a superhighway. REFERENCES Anderson, P., Cannon, H. M., Malik, D. & Thavikulwat, P. (1998). Games as instruments of assessment: A framework for evaluation. Developments in Business Simulation & Experiential Learning, 25, 31-37. Anderson, P.H. & Lawton, L. (1992). A survey and methods used for evaluating student performance on business simulations. Simulation & Gaming, 23, 490-498. Biggs, W. D. (1978). A comparison of ranking and relational grading procedures in a general management simulation. Simulation & Games, 9, 185-200. Biggs, W. D. (1990). Introduction to computerized business management simulations. In J. W. Gentry (Ed.), Guide to 232 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 business gaming and experiential learning (pp. 23-35). East Brunswick, NJ: Nichols/GP. Bloom, B. S., Englehart, N. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives—The classification of educational Goals, Handbook I: Cognitive Dominance (New York: David McKay). Cannon, H. M. & Burns, A. C. (1999). A framework for assessing the competencies reflected in simulation performance. Developments in Business Simulation & Experiential Learning, 26, 40-44 CEO. Thavikulwat, P. (1997-2000). Towson, MD: Towson University (Dept. of Management, Towson University, Towson, MD 21252, USA, http://www.towson.edu/~precha/ceo.html). Fritzsche, D. J. & Cotter, R. V. (1990). Guidelines for administering business games. In J. W. Gentry (Ed.), Guide to business gaming and experiential learning (pp. 74-89). East Brunswick, NJ: Nichols/GP Publishing. Hand, H. H. & Sims, H. P. Jr. (1975). Statistical evaluation of complex gaming performance. Management Science, 21, 708-717. Krathwohl, D. R., Bloom, B. S., & Masia B. B. (1964). Taxonomy of educational objectives—The classification of educational Goals, Handbook I: Affective Dominance (New York: David McKay). Jones, K. (1982/1998). Simulation as examinations. Simulation & Gaming, 29, 331-341. (From Simulation/Games for Learning, 12). Jones, K. (1991). Using computer-assisted simulations and avoiding computer-hindered simulations. Simulation & Gaming, 22, 234-238. Jones, K. (1998). What happens when students design and run their own simulations? Simulation & Gaming, 29, 342- 347. Patz, A. L. (1992). Personality bias in total enterprise simulations. Simulation & Games, 23, 45-76. Patz, A. L. (1999). Overall dominance in total enterprise simulation performance. Developments in Business Simulation & Experiential Learning, 26, 115-116. Patz, A. L. (2000). One more time: Overall dominance in total enterprise simulation performance. Developments in Business Simulation & Experiential Learning, 27, 254- 258. Peach, E. B. & Platt, R. G. (2000). Total enterprise simulations and optimizing the decision set: Assessing student learning across decision periods. Developments in Business Simulation & Experiential Learning, 27, 242- 247. Pillutla, S. & Thavikulwat, P. (1998). Observing general ability in a total enterprise gaming simulation. Developments in Business Simulation & Experiential Learning, 25, 224-229. Pogossian, E. M. (1998). Development of management skill assessment. Developments in Business Simulation & Experiential Learning, 25, 29-30. Pogossian, E. M. (1999). Increasing efficiency in management skill assessment. Developments in Business Simulation & Experiential Learning, 26, 80-81. Rollier, B. (1992). Observations of a corporate facilitator. Simulation & Gaming, 23, 442-456. Teach, R. D. (1990). Profits: The false prophet in business games. Simulation & Gaming, 21, 12-26. Teach, R. D. (1993a). Forecasting and management ability: A response to Wolfe. Simulation & Gaming, 24, 63-72. Teach, R. D. (1993b). Forecasting ability as a performance measure in business simulations. Simulation & Gaming, 24, 476-490. Thavikulwat, P. (1996). Activity-driven time in computerized gaming simulations. Simulation & Gaming, 27, 110-122. Vance, S. C. & Gray, C. F. (1967). Use of a performance evaluation model for research in business gaming. Academy of Management Journal, 10, 27-37. Wolfe, J. (1993a). Forecasting and management ability: A rebuttal to Teach’s response. Simulation & Gaming, 24, 73-75. Wolfe, J. (1993b). On the propriety of forecasting accuracy as a measurement of team management ability: A preliminary investigation. Simulation & Gaming, 24, 47- 62. Wolfe, J. & Chacko, T. I. (1981a). The effects of different team sizes on business game performance, Developments in Business Simulation & Experiential Exercises, 9, 232- 236. Wolfe, J. & Chacko, T. I. (1981b). Mean size effects on business game performance and decision-making behavior, Decision Sciences, 14, 212-233. Wolfe, J. & Rogé, N. R. (1997). Computerized general management games as strategic management learning environments. Simulation & Gaming, 28, 423-441. 233 Table of Contents Volume 28, 2001 Using A Web Based Tutorial Program To Enhance Student Learning Reflecting On Reflection: An Examination Of Reflective Learning And Assessing Outcomes Overcoming Obstacles To A Cross-Global Collaborative Classroom Panel Discussion On Building And Maintaining Trust In The Abselesque Classroom Cosmopolitan-Based Cross National Segmentation In Global Marketing Simulations Strategic Management And The Case Method: Survey And Evaluation In Hong Kong Designing Interactive Self-Learning Modules Using Macromedia Director Teaching Financial Markets Using The Wall Street Journal Core Course Outcomes Assessment And Program Continuous Improvement Using An Integrative Business Plan Œ An Empirical Evaluation Using A Mock Trial Activity For Interdisciplinary Learning Fidelity, Verifiability, And Validity Of Simulation: Constructs For Evaluation 2001 International Collegiate Business Strategy Competition: 37 Years Of Collegiate Competition Motivating Students: An Initial Attempt To Operationalize The Curiosity Gap Model E-Learning: The Next Wave Of Experiential Learning Helping New Game Adopters: Four Perspectives An Initial Validity Invesitigaion of A Test Assessing Total Enterprise Simulation Learning Administering A Marketing Simulation - Common And Varied Practices Among Instructors Issues In Case Method Instruction: Class Discussion Leadership Sex Composition, Cohesion, Consensus, Potency And Performance Of Simulation Teams Integrating Management Curriculum: Linking Courses With Frontpage Absel: The Way We Talk! Teaching Strategic Management: A Case Study In The Diffusion Of Innovation In Education A Demonstration Of Bussim Strategy A Total Enterprise Simulation Teaching Methods: Etic Or Emic Antecedents Of Work Team Performance In A Business Simulation: Personality And Group Interaction When Experiential Educators Reflect Upon Their Craft: A Qualitative Research Report Strategic Sensemaking In Organizations: Model Formulation And Operationalization Panel Discussion: A Model For Initiative Development: Discussion And Ideas Of Individual Spirit Harmonics Assessing The Efficacy Of Experiential Learning In A Multicultural Environment Are Business Schools Producing 21st Century Managers? Blue Horizon Cruises: An International Simulation Exercise To Illustrate Conflicting Economic And Cultural Models Different Perceptions Of The Same Thing: An Experiential Exercise Current Use Of Agribusiness Simulation Games: Survey Results Of University Agribusiness And Agricultural Economics Programs Team Building 101 For Accountants An Online Evaluation Of The Compete Online Decision Entry System (Codes) Total Enterprise Simulation Winners And Losers: A Preliminary Study Gilding The Lily, Enhancing The Pedagogical Experience, Or Fixing What Isn™t Broke: The Dilemma Of New Simulation Versions The Absel Endnote Database: The Perfect Tool For The Bernie Keys Library Decision-Making Exercises (A), (B), And (C): Examining The Role Of Conflict In Group Decision-Making Group Decision Simulation (A)-(F): Examining Procedural Justice In Group Decision-Making Processes Evaluation Of Performance In Business Games: Financial And Non Financial Approaches A Framework For Evaluating Simulations As Educational Tools Congruence II™: A Strategic Business Board Game The Tournament Concept: Extending The Idea Of Using Simulations As Instruments Of Assessment The Economic And Political Transition Of The Mexican State In The Threshold Of Twenty First Century: From The Entrepreneurial State To The State Of Entrepreneurs. Experiential Learning For Accountants: The Not For Profit Project An Investigation Of The Environmental Awareness Attained In A Simple Business Simulation Game A Daily Exercise In Theory Application: Is This Why I Had To Take Marketing, Management, Finance –? An Alternative View to Scoring Selected Simulation Games and Experiential Exercises