The Effectiveness of SDM Method in Business Simulation Game Page 302 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 ABSTRACT Many researchers acknowledge that debriefing is important in simulation game. One of the key aspects of debriefing is to acquire knowledge. In this paper, we give attention to knowledge acquisition in simulation game. Thanks to the advancement of IT, most simulation games have been computerized and widely utilized online. Such development of IT makes new methods for knowledge acquisition available. Surrogate decision maker (SDM) method is one of such methods. The SDM is a computer based numerical simulation with intelligent agents and human-generated gaming data. It can be used for different types of participants to acquire different types of knowledge from/for various gaming simulation activities. This paper focuses on the effectiveness of SDM method for knowledge acquisition in computer supported simulation game. First, we briefly introduce the different types of activity and participant that involved in simulation games and their relationship, and identify possible types of knowledge acquisition in section 2. Then, defining the SDM method, we see how the method can be used in different activities in section 3. In section 4, we give some usage of the method in business simulation game, and suggest possible usage of the method for each type of activities. In section 5, we present two experiments to evaluate the effectiveness of SDM method in both in-game and post- game debriefing. Finally, we discuss how IT supported methods can facilitate knowledge acquisition in simulation game activities. Keywords: surrogate decision maker method, business simulation game, knowledge acquisition, experiential learning, debriefing, game model characteristic exploration INTRODUCTION Many researchers acknowledge that debriefing is important in simulation game. David Crookall is a representative among them. He mentioned that debriefing is perhaps the most important part of a simulation/game (Crookall, 1992). Crookall (2010) also pointed out that the learning comes from the debriefing, not from the simulation game itself, and debriefing is the processing of game experience to turn it into learning. What is the purpose of debriefing? Knowledge acquisition is a possible answer to the question. Then the question will be what kind of knowledge we can learn from simulation game. In previous research, the authors identified three types of participant and four kinds of activity in simulation game, and relationship between each type of participant and activity, and pointed out that each type of participant can acquire different kinds/levels of knowledge from each activity. Moreover, the authors proposed a method, called SDM, which aims at comprehensive debriefing, or comprehensive knowledge acquisition. The SDM method is a computer based simulation in which decision-making data set yielded in simulation game is used as the decision input for the analytic model that has the same structure as the original simulation game, and a machine agent is substituted for a specific player. Thanks to the advancement of technology, most simulation games have been computerized and widely utilized online. It makes new methods for knowledge acquisition available. The SDM is one of such methods. The SDM method can be used in both two generally categorized debriefing types: in-game debriefing and post- game debriefing. In-game debriefing is a phase in which the data can be processed to provide material for feedback during play (Crookall, 2010). In in-game debriefing, the SDM method can be used to analyze future and past market situations, and make player’s decision (what-if analysis, goal seek analysis, and sensitivity analysis, etc.). Post-game debriefing is an instructional process that is used after a game, simulation, role-play, or some other experiential activity for helping participants reflect on their earlier experiences to derive meaningful insights (Thiagarajan, 1992). In in-game debriefing, the SDM method can be used to detect the game model characteristic, to analyze the market situation in the past and the future, to make player’s decision, and to examine possible strategy in the gaming world. In post-game debriefing, the SDM can be used to acquire knowledge about the game structure, to test the effectiveness of player’s model building during the game play, and to provide further opportunities where player can analyze another ‘as-if’ worlds; these analysis might provide further understanding of the given game structure and real business structure. Moreover, the SDM can be used for not only debriefing but also for other purposes, such as game design. However, these have not been fully examined. To explore what is effective usage of SDM, the authors The Effectiveness of SDM Method in Business Simulation Game Beiyu Yang Yokohama National University yang-beiyu-ky@ynu.ac.jp Motonari Tanabu Yokohama National University tanabu@ynu.ac.jp Page 303 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 designed and conducted two sets of experiments. In this paper, we evaluate the effectiveness of the SDM method for the acquisition of the specific knowledge in both in- game and post-game debriefing in simulation game. This paper focuses on the effectiveness of the SDM method in knowledge acquisition in computer supported simulation game. First, we briefly introduce the different types of activity and participant that involved in simulation games and their relationship, and identify possible types of knowledge acquisition in section 2. Then, defining the SDM method, we see how the method can be used in different activities in section 3. In section 4, we give some usage of the method in business simulation game, and suggest possible usage of the method for each type of activities. In section 5, we present two experiments to evaluate the effectiveness of SDM method in both in-game and post-game debriefing. Finally, we discuss how IT supported methods can facilitate knowledge acquisition in simulation game activities. RELATIONSHIP BETWEEN PARTICIPANT AND ACTIVITY IN BUSINESS SIMULATION GAME Participants involved in the business game can be categorized into three types from both the educational and game design point of view. From an instructional perspective, participants can be categorized into instruction designer, teacher and learner. From a design perspective, all participants are regarded as learner who acquires different types of knowledge through participation in simulation game. Such participants are categorized into game designer, game facilitator and game player. Each type of instruction designer, teacher and learner corresponds to game designer, facilitator and player respectively. Game designers design/build games not only in order that players learn something, but also in order that they do their researches, for example, validate the game model or investigate player behavior in a specific environment. Game facilitators can also acquire knowledge from players’ behavior through facilitation. For example, facilitator may learn about better strategies during game session. Game players can obtain domain knowledge and, if any, do their research. In this paper, we mainly focus on learner/player. Activities in simulation game can be divided into four types: game model building, model characteristic exploration, decision strategy exploration, and domain knowledge acquisition. Game model building here particularly includes game model modification. Model characteristic exploration refers to finding or detecting characteristics of the game model. Decision strategy exploration refers to examining the participants’ own strategy. Domain knowledge acquisition is an activity in which the players learn about domain-specific knowledge such as marketing, accounting, finance, logistics, etc. Instruction/game designer will be involved in first three types of the activities. Teacher/facilitator will be involved in exploration of model characteristics and decision strategy. Since the players can not only acquire domain knowledge for the learning purpose but also explore their own strategy for the research purpose, learner/ player will be involved in the last two types of the activities. Table 1 shows how participant can be involved in different activities. For game model building, the instruction/game designer can design the simulated and structured world in which player will experience something similar to the designer’s experience or thought, and modify the game model based on the feedback. The teacher/ Table 1 Participants involvement in different activities Game model building Model characteristic exploration Decision strategy exploration Domain knowledge acquisition Instruction designer/ Game designer Design the simulated world in which player will experience something similar to the designer’s experience or thought Game model modification Detect the game model to find if there is any place which should be revised (Model validity) Explore effective strategy under the model Design the simulated world to help learners learn Teacher/Game facilitator Instruct the learners to play the game and give their feedback to help designer to modify the model Detect the game model to get knowledge about the game model characteristic and find if there is any place which should be revised and report to the designer Explore effective strategy under the model Understanding of player’s behavior Facilitate learners learn by simulation game Learner/Game player Play the game and give their feedback which helps designer to modify the model Detect the game model to get knowledge about the game model characteristic Examine the possible strategy in gaming and real world Get knowledge aimed to be acquired in the game (learning objectives) Get “something” acquired in debriefing Page 304 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 facilitator instructs the learners to play the game and can give their feed back to help designer to modify the model. The learner/player plays the game and can give their feedback so that designer modifies their model. By the activity of model characteristics exploration, the instruction/game designer can judge whether the game model should be revised, that is, whether model is valid. The teacher/facilitator can know about the characteristic of game model, and what facilitator learned can be fed back to the designer in order that the original model is more sophisticated. The learners/players also learn about the game model characteristic. For decision strategy exploration, the instruction/game designer and teacher/ facilitator can explore effective strategy under the model. The learner/player can examine the possible strategy in the gaming world and the real world. For domain knowledge acquisition from an educational perspective, the instruction/ game designer can design the simulated and structured world to help learners to learn. The teacher/facilitator can understand player behavior through facilitation of simulation game. The learner/player can gain knowledge that is aimed to be acquired (learning objectives) and get “something” acquired in debriefing. SDM METHOD EPISTEMOLOGY OF SIMULATION/GAMING Simulation and gaming is used in various fields not only as an educational tool but also as a research approach. In the context of which simulation/gaming is used as a research methodology for practical problem solving, for example, in policy science rather than training tools for developing decision making ability, the methodology of simulation/gaming could coincide with that of participatory paradigm, whereas participatory paradigm does not emphasize the ways of knowing by making a meta-critical comparison between critical subjectivity of experiences in a “simulated and structured” another world and that of the real world. Based on the characterization of gaming simulation by Arai (2004) and the epistemology of participatory paradigm, Tanabu (2011) characterized epistemology of simulation & gaming as subjectivity derived from a comparison between critical subjectivity of experiences in “simulated and structured” world and past experiences. SDM METHOD Computer based simulation that deals with social problems needs appropriate dataset as well as simulation model. When a game designer or researcher explores the decision strategy that meets a specific requirement under a given simulation game, it is usually hard for the explorer and computer agent to simulate complex, and sometimes unclear and irrational decision of human players. The surrogate decision maker (SDM) method was originally designed to carry out business simulations for the purpose of player’s strategy exploration. The original SDM method refers a computer simulation that uses a dataset made by human players. The dataset consists of decision values of all decision items of all players of all rounds in a simulation game. The simulation model that yielded the dataset is used again in the SDM simulation. A series of decisions made by a specific player in most cases will be replaced with the data that will be made by computer agent. By changing the decision data of a specific player, the explorer can examine various decision strategies under pseudo “game” situation that might take place. The SDM was originally designed for the exploration of decision-making strategy. However, it also can be used Figure 1 Debriefing forms Page 305 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 in game design such as model building and parameter fitting. Required the model modification, if the designer or researcher does not have enough knowledge of player’s decision strategies in advance, the SDM method will be used to design the game balance and examine it. In the context of examining a specific decision strategy under given simulation model, changing data yielded by the human player causes interpretation problem. For example, in the simulation game in which the market share is calculated by the selling price every round, if some players were setting the selling price that is different from the original, all players might make different decision in the next round because the result of each decision is referred by the players in the succeeding rounds. However, in the case of simulation game in which demand is distributed by the selling price, if we assume that the demand for each player is influenced by the variation of players’ decision in selling price than that of total demand, we can interpret the result of the SDM simulation by simple modification of total demand. Since, the demand for each player is usually calculated by dividing total demand proportionally in terms of decreasing function of selling price, we can modify the total demand so that the demand for all players other than the computer agent (that is also calculated by the modified total demand) is just same as that of gaming, by multiplying the original total demand by the coefficient that is calculated from the human generated dataset. This means that by using the method of SDM with demand modification, strategy explorers can interpret their SDM result into the original gaming situation. Although to change the original dataset causes the impossibility of logical interpretation of the SDM result and the method of SDM with total demand modification allows us to interpret the SDM result into the original situation, the authors observed that the SDM simulation without total demand modification could be used instead of the demand-modification SDM in most cases. The SDM simulation we will introduce later is simple SDM that does not modify the total demand. SDM METHOD IN DIFFERENT TYPES OF ACTIVITY In section 2, we illustrated different types of participant and activity in simulation game, and explained how participants can be involved in different types of activities. SDM method can be used in these activities to achieve those purposes, and debriefing is the most appropriate phase where SDM method can be applied. Debriefing According to Steinwachs (1992), a debriefing is a time to reflect on and discover together what happened during game play and what it all means. Debriefing, even without a facilitator, usually move of their own power through the three phases of description, analogy/analysis, and application. In the description phase, participants air their experiences and impressions, and also need to listen to the other participants and so be filled in on the whole picture. In the analogy/analysis phase, participants systematically examine the simulation game model as just played and as designed, identifying and exploring parallels with real- world situations. In the application phase, participants focus on the reality presented by the simulation game. As Petranek, Corey and Black (1992) presented, Gillespie (1973) observes that games are not self-teaching and need a good debriefing session to assist students in reflecting on their behavior and the purpose of the simulation. Most instructors who use simulations and games move to the second level of learning by following the simulation with a session designed to help students reflect on their learning (Thatcher, 1990). Debriefing is the occasion and activity for the reflection on and the sharing of the game experience to turning it into learning (Crookall, 2010). Debriefing consists of an oral discussion session in which students and teachers engage in a question and answer session designed to guide students through a reflective process about their learning. Coppard and Goodman (1979, p. 41) write, “According to many designers and facilitators, such discussion (debriefing) and analysis are the most important Figure 2 SDM method in debriefing Page 306 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 elements in gaming/simulation in terms of the learning process involved.” Thatcher (1990) argues convincingly that participants who can reflect on the game are in a better position to recognize what they learned in the game. All these definitions of debriefing pointed out the importance and purpose of debriefing. However, some of them over emphasis the oral discussion, and will make people think that debriefing equals to oral discussion. It is incorrect or misunderstood. We should pay more attention on the essence of debriefing, reflection, rather than the method, oral discussion. Oral discussion is useful, but it is only based on one game situation, and the knowledge we can get from this one situation is limited. We think such reflection is not thorough. In this sense, new method such as numerical analysis could also be considered as part of debriefing. We particularly appreciate analogy/analysis which means to systematically examine the simulation game model as just played and as designed, identifying and exploring parallels with real-world situations. According to the epistemology discussion, we can also find the importance of reflection and bridging the gap between the simulation world and the real world. SDM method is such kind method. It can provide a variety of situations for participants to do in-depth debriefing, or what we called in the introduction “comprehensive debriefing”. Debriefing forms Debriefing is generally categorized into two types: in- game debriefing in which the data can be processed to provide material for feedback during play (Crookall, 2010) and post-game debriefing which is an instructional process that is used after a game, simulation, role-play, or some other experiential activity for helping participants reflect on their earlier experiences to derive meaningful insights (Thiagarajan, 1992). Different combinations of in-game and post-game debriefing produce different debriefing forms. The authors concluded four forms cover all kinds of combinations of in- game and post-game debriefing for one game session. SDM in debriefing SDM method can be used in all debriefing phase mentioned in the four forms. Figure 2 gives an example of using SDM method in debriefing phase. In Figure 2, D’ stands for debriefing with SDM method. Again, any combinations of debriefing form can be used when applying SDM method. Different types of participant can use SDM method in different debriefing phase to achieve their different purpose. First, player can use SDM method both in in-game and post-game debriefing. If the player is a novice to the game, he/she has no knowledge about the game model, so in- Figure 3 Global optimal solutions in profit Page 307 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 game debriefing is unsuitable, or the facilitator should give them SDM simulation model in this case. If the player has already had some knowledge of the game, then he or she can build his/her own simulation models. When using SDM method in in-game debriefing, the player can detect the game model characteristic, or analysis the past market situation, predict the future market, and make his own decision and examine the possible strategy in gaming world. When using SDM method in in-game debriefing, the player can get knowledge that aimed to be acquired in the game (that is, learning objectives), and further, can get “something” acquired in debriefing for transfer the knowledge to the real world. Combining all of these information both get from the in-game and post-game debriefing together with the game experience, the player can give their feedback which helps designer to modify the model. Second, facilitator can also use SDM method in both and post-game debriefing. When using SDM method in in- game debriefing, the facilitator can analysis the market situation and understanding of player’s behavior, and give advice to the player who has problem with make decision by himself or herself. When using SDM method in post- game debriefing, the facilitator can explore effective strategy under the model, detect the game model to get knowledge about the game model characteristic and find if there is any place which should be revised and report to the designer, and give the players’ feedback to the designer to help designer to modify the model. Last, designer can only use SDM method in post-game debriefing for game model modification, model characteristic exploration, and decision strategy exploration. USAGE OF SDM METHOD PRELIMINARY THM ANALYSIS The Bakery Game In this section, we introduce a usage of SDM method, which is to do actual numerical analysis for specific purpose after game. The game conducted is the Bakery Game, developed by Shirai (2008) and implemented on Yokohama Business Game (YBG) platform. In the bakery game, each player manages one virtual bakery shop and tries his/her best to maximize the profit in the competitive circumstance. The shop produces and sells the bread and competes against the other shops in maximizing profit. If a player wants to produce a loaf of bread, he/she has to place an order for frozen dough to the supplier and the ordered item will be delivered on the next day. The production process of the bread consists of thawing out the frozen dough and baking it in the oven. This process also takes one day. Therefore, it takes two days from ordering the frozen dough to selling finished product. The decision variables that should be decided by the players in the game are (1) selling price of a loaf of bread, (2) production Figure 4 Profit to throughput by different sell-out coefficients Page 308 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 quantity of loaves of bread, and (3) procurement quantity of frozen dough. The selling price of all players influences the customer demand for each player, which also influences the decision making of each player in production quantity and procurement quantity. That is, the higher selling price than others brings fewer customers, and conversely the lower selling price brings more customers. Thus under low selling price, player would expect more customers so that the player needs to set both production and procurement quantity at high level to meet the customer demand, otherwise the player would lose customer in the next round. On the other hand, under high selling price, the player needs to set them at low level to avoid loss of disposal because unsold bread must be disposed of. In terms of procurement, the player has to order enough quantity of frozen dough to ensure required production volume one day after the ordering. Throughput maintaining strategy In general, THM is defined as a strategy that focuses on the flow of material in internal and/or external processes such as procurement, production and distribution. However, in our experiment, throughput-maintaining (THM) strategy refers to keep procurement and production quantity as constant to reduce the complexity of decision- making which caused by the complex structure of the Bakery Game mentioned above and the uncertainty of other player’s decision making. For more simplicity, we set the production quantity and the procurement quantity as equal. When we make use of the THM, we need to solve the following questions. Can we achieve the best or better performance in profit? And if so, how much throughput should we set? What is the relationship between selling price and appropriate throughput? Moreover, is there the global optimal throughput that brings the highest profit among possible THM settings? We test the possible combination of constant selling price and constant throughput by using SDM method. Experiment In this SDM experiment, we just conducted the post- game debriefing, and no in-game debriefing. The data is collected from the real gaming data played by the MBA students at East China Normal University. Two sets of real human data altogether, both of them consist of 11 same player-teams. Then, we replaced specified one player’s decision with the agent that uses the THM strategy, and other player’s decisions were reused for SDM simulation. To see the credibility of SDM, SDM simulations were conducted for all of the 11 players in both two data sets. As to the experiment design, we just introduce one performance variable, profit, in this paper. More precisely, we focus our discussion on profit and its relationship with price and throughput. In order to see whether the THM player can achieve the higher performance in profit, possible all combinations of price and throughput were used for the simulation. The selling price for a loaf of bread ranges between 300 JPY and 1000 JPY, and the standard is 700 JPY. The authors chose 101 values for selling price, each of which lies between 500 JPY and 1000 JPY at step of 5. For the throughput, the default procurement quantity and production quantity is 100. The authors chose 51 values for the throughput, each of which is between 50 and 300 at step of 5. We did SDM simulation for each combination of selling price and throughput. Number of all combinations of two values is 5151 (=101*51). There were 11 possible decision makers, each of which was substituted by the computer agent respectively. Therefore, 56661 (=5151*11) simulations were done for each of two data sets of gaming to see how SDM result varies depend on the gaming data. Figure 4 shows profit for each throughput under some specific selling prices. The horizontal axis stands for the throughput, and each curve represents the relationship Figure 5 The process of experiment 2 Page 309 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 between profit and throughput under fixed selling price. Each marker corresponds to one SDM simulation and the figure was generated from 5151 SDM simulations with a fixed surrogate decision maker. We can find from figure 5 that there exists unique optimal throughput that maximizes profit for any selling price. Moreover, we find the unique maximum value in the set of maximum profit for each selling price, that is, there exists the global optimal solution in profit. In this example, the global optimal solution is 105 of throughput and 850 of selling price. Figure 4 shows just a set of SDM simulation results for a fixed surrogate decision maker, and because of the space limitation, we do not present all of the figures generated. But actually, we generate such figures for different surrogate decision makers, and can observe the existence of global optimal solution for other surrogate decision makers, since a large number of figures show the same tendency. This implies some sort of credibility of SDM method. Revised THM analysis In the previous section, the computer agent has a predefined pair of values that is used for its decision, namely, selling price and throughput. The original notion of throughput-maintaining decision strategy is to keep the production and procurement volume constant, whereas selling price is not necessarily to be constant. In this section, we modify the previous THM simulation so that the SDM agent calculates the appropriate selling price to sell out the planned amount of the products. First, we assume that the elasticity of demand (number of customer visits) with respect to the selling price is known. This is to indicate that SDM agent can estimate the demand function based on the previous decision data, and also calculate the sell-out price. For SDM simulation in this section, the SDM agent estimates total demand and average selling price in order to estimate the demand function of next round, and calculates the sell-out price at which all the target amount of products can be sold with less shortage. The target sell- out amount of products is automatically determined by multiplying throughput by the sell-out coefficient of which value lies between 100% and 200%. Thus, the SDM simulation parameters are throughput and sell-out coefficient. The 200% of sell-out coefficient means 200% of throughput is expected to be sold out. In this case, the selling price must be lower than the price at which 100% of throughput is expected to be sold out. Figure 4 is a result of the revised THM simulation with SDM methods. THE EXPERIMENTS In order to evaluate the effectiveness of SDM method in both in-game and post-game debriefing in simulation game, two experiments have been designed. In this section, the authors present these two experiments in order to explore what impact does SDM method have on players’ knowledge acquisition. Both quantitative and qualitative evaluation is used so that we can get the players’ understanding of both the gaming world and the real world. GAME DESIGN Since there are two debriefing types: in-game and post- game debriefing, we designed two experiments respectively. (1) Comparison between ‘post-game debriefing without SDM’ and ‘post-game debriefing with SDM’. The purpose of this experiment is to see the impact of SDM in post-game debriefing, and in what sense, it is effective. 1a. post-game debriefing without SDM 1b. post-game debriefing with SDM Table 2 The original experiments plan Purpose The effectiveness of SDM method for knowledge acquisition and decision making in business simulation game Participants 20 players (2 players/team * 10 teams) (with the knowledge of SCM) Game The Bakery Game Environment PC room (with 22 PCs or more; all of the PCs have installed MS-Excel ver. 2007 or higher, and can access Yokohama Business Game Website simultaneously; with projec- tor) Procedure The experiment is divided into two parts (The two parts of the experiment will be con- ducted respectively.) The procedure of these two parts is almost same and as follows: Explain the Bakery Game, let the students get familiar with the game environment and make a trail (about 50 minutes) (Rest) Game play (about 70-80 minutes) (the two parts need different time) (Rest) Exam (about 20 minutes) Interview (about 40 minutes) (Each part needs 3-3.5 hours; Two parts need about 7 hours altogether) Time Middle of October Page 310 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 In this experiment, we asked several student player teams to play the Bakery Game without in-game debriefing. After the game, all player teams were divided into two groups. For one group, we asked them to do usual oral debriefing without using SDM method. For another group, we taught them SDM analysis method, and asked them to do debriefing with the result of SDM analysis. (2) Comparison between ‘in-game debriefing without SDM’ and ‘in-game debriefing with SDM’. The purpose of this experiment is to see the impact of SDM in in-game debriefing. 2a. in-game debriefing without SDM 2b. in-game debriefing with SDM In this experiment, we asked the same student player teams as above to play the Bakery Game. We divided them into two groups just the same as in previous experiment. That is, the teams used SDM method in experiment 1 would still use SDM method in experiment 2. Then, we asked the two groups of students to play the game again. During the game, we asked the students to do in-game debriefing. For the group, which acquired the SDM method, we asked them to do debriefing with SDM analysis. For the other group, only oral debriefing is used for in-game debriefing. After the game play, we asked all of the student player teams to do post-game debriefing. Figure 5 illustrates the process of experiment 2. At first, we designed 3.5 hours respectively for both of these two experiments. The original experiments plan can be observed in table 2. However, because of the time limitation of the participants, after negotiation, the experiment time was condensed into 4.5 hours altogether. 21 players participated in the real experiments at last. All of them were Chinese undergraduate students from business administration and accounting departments (grade 2, 3 and 4). Those who were grade 2 and 3 had already learned supply chain management, and those who were grade 2 hadn’t. They were divided into 11 player teams, 10 of which have 2 student players, and another team has only one player. The experiments were conducted at a Chinese university on 1:00-5:30pm Nov. 4th, 2012. The schedule of the experiments can be observed in table 3. EVALUATION DESIGN As mentioned above, we conducted both qualitative and quantitative evaluation for the students to collect not only the objective data but also the subjective data. A) Performance measure Performance measure means directly evaluate the players’ game performance. From the game result, which was automatically collected from the game platform, we got knowledge of and could compare the players’ performance. Performance measure belongs to objective evaluation. It is useful for evaluating the effectiveness of in -game debriefing with SDM method, since the performance is influenced by in-game debriefing, and we can compare the performance of both using SDM method and without using SDM method. However, performance measure is not suitable for post-game debriefing case, since post-game debriefing does not influence the performance. The variables we will measure in the experiment are as follows: profit, sales, number of disposal, sell out rate, service level, and etc. In this paper, we focus on profit only. B) Achievement test Achievement test means let students to do test after post-game debriefing. It can be used for both evaluating the effectiveness of in-game debriefing with SDM method and post-game debriefing with SDM method, since we can compare the test result of both two groups of students. Achievement test also belongs to objective evaluation. Since the aim of this research is to see the effectiveness of SDM method in players’ domain knowledge acquisition, there is no need and impossible for us to test all the domain related knowledge. Thus, we just focus on one important concept, the break-even point, and to test the level of students’ grasp of this concept. At first, we planed to ask students to do a paper test to explain what is break-even point to see if they know the concept itself, to ask them to calculate the breakeven point to see if they understand the concept, and to ask them make decisions Table 3 The schedule of the experiments 1:00 - 1:30pm Introduction; Explanation of the process of the experiments; Explanation of the Bakery Game 1:30 – 3:00pm Game session 1 (11 rounds)—experiment 1 3:00 – 4:00pm Introduction of SDM method; Debriefing phase 1 (half of the teams use the SDM analysis tool); Rest (15 minutes break) 4:00 – 4:50pm Game session 2 (half of the teams use the SDM analysis tool) (10 rounds) —experiment 2 4:50 – 5:30pm Debriefing phase 2 (half of the teams use the SDM analysis tool); Conclusion (Conclude the experiment) Page 311 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 under a given scenario to see if they would consider or apply the concept when they make decision. However, because of the time limitation in real experiment, we changed to ask them answer the question about this concept. Though their oral explanation, we judged their understanding of this concept. C) Interview Interview is subjective. The questions in interview were open-ended. It was not only about the players’ understanding of the gaming world, but also the players’ understanding of the relationship between the gaming world and the real world. The questions were as follows: What’s happened in the game? What did you consider when making decision? What strategy did you use? Will you apply the strategy in the real world? What kind of experience you get from the gaming world can be used in the real world? Interview was used for both evaluating the effectiveness of in-game debriefing with SDM method and post-game debriefing with SDM method, since we could compare the feelings of different groups. GAMING RESULTS The gaming results of experiment 1 and experiment 2 can be observed respectively in table 4 and table 5. Figure 6 and figure 7 reflect the progress of two experiments respectively. The evaluation of the effectiveness of SDM method will be based on these data, implemented with interview data. EVALUATION To evaluate the effectiveness of SDM method in both in-game and post–game debriefing, we mainly focus on the gaming result. Table 4 and 5 shows the result of experiment 1 and 2 respectively. In this paper, we just compare the change of the ranking in profit of different teams, since all teams increased a lot in profit in experiment 2. We can see in the table, the column “Using SDM” stands for if the team uses SDM method or not, Y means “yes” and N means “No”. Compare these two tables, we can find 6 teams used SDM method, and two of them ranked higher in second game session, one of them same with the first time, and three of them went down in ranking. It seems the effect is not obvious at first glance. However, when we analyze the result in depth, we will find we cannot get such conclusion. Team 5 ranked first place in first experiment. Although his ranking went down for one position, it still showed the very good performance. If we observe the progress of experiment 2, we can find that team 5 kept the first rank for the previous seven rounds. It shows SDM had no bad impact on Team 5, while it had very obvious effects on team 11. Team 6 ranked 9th and 11th position in two experiments respectively, although they use SDM. However, through the interview and discussion, we found that team 6 didn’t use the SDM method in their decision; instead they tried to use another strategy—game theory strategy, which was different from the game’s objective. That’s the reason why they didn’t show good performance in both of two game sessions. Team 7’s ranking went down from position 3rd to 8th. According to discussion and interview data, they think the SDM method was better to be used from very beginning to help them explore the game model characteristics. However, since they have already got some knowledge of game model characteristics, they didn’t used it very much, but just used to see the trend of ranking in profit of different teams. On the contrary, team 11 and 3 shows great improvement in ranking. Both of these two terms enjoyed the merit of SDM in post-game debriefing in first experiment, which made them notice the importance of relationship between production quantity and selling price. Since they didn’t notice this point, they lost Table 4 The gaming result of experiment 1 Ranking in the profit Profit Team No. Using SDM (post-game) Remarks 1 34830 5 Y 2 28569 9 N 3 -4265 7 Y 4 -24074 4 N 5 -28895 11 Y 6 -52480 2 N 7 -52876 1 Y 8 -56780 8 N 9 -72710 6 Y 10 -85579 10 N 11 -163404 3 Y Page 312 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 the game in the first experiment. However, when they gave attention to balance this two data, their performance improved. Team 3 mentioned in discussion feedback, they didn’t use SDM very much in experiment 2, so their performance was still not good. However, team 11 used this method in the game, so they improved very much and went up to the first position. From the analysis, we can find that SDM may have positive impacts on player of business simulation game both in in-game and post-game debriefing. However, the degree of the impact will depend on the degree the player use it. If the player just uses it superficially, it may be not effective or the effectiveness may be not obvious. If the player understand it deeply and uses it in-depth, its effectiveness may be obvious. It may be also useful in supporting player’s strategy exploration. CONCLUSION In this paper, we focus on the evaluation of the effectiveness of SDM method in knowledge acquisition in computer supported simulation game. Rather than from the game itself, knowledge can be acquired from debriefing. Simulation games are now computerized and widely utilized online, which make new method for debriefing become available. SDM is one of such methods. First, we briefly introduce the different types of activity and participant involved in simulation games and their relationship, and identify possible types of knowledge acquisition. In this paper, we just focus on one types of the participant—game player/learner. SDM method is a computer based numerical simulation with intelligent agent and human generated gaming data. The epistemology of gaming simulation makes the usage of SDM method sensible. Debriefing is the suitable phase where SDM method can be used. After explaining the concept and definition of the method, we illustrate how the method can be used in different activities for certain purpose. Then, we give some usage of the method in business simulation game, and suggest possible usage of the method for each type of activities. After that, we present two experiments to evaluate the effectiveness of SDM method in both in-game and post- game debriefing. We find that SDM may have positive impacts on player of business simulation game both in in- game and post-game debriefing. However, the degree of the impact will depend on the degree the player use it. Finally, we discuss how IT supported methods can facilitate knowledge acquisition in simulation game activities. From the discussion of this paper, we can point that IT supported method such as numerical analysis could also be considered as part of debriefing. Analogy/analysis is particularly appreciate which means to systematically examine the simulation game model as just played and as designed, identifying and exploring parallels with real- world situations. According to the epistemology discussion, we can also find the importance of reflection and bridging the gap between the simulation world and the real world. One game session provides only one simulated situation. Oral debriefing based on this one situation is limited. IT supported method can provide a variety of situations for participants to do in-depth debriefing, or what we called in the introduction “comprehensive debriefing”, so that they can acquire more knowledge from simulation game and can transfer them to apply to the real world. Although we get some evidences to show the effectiveness of SDM method on players’ knowledge acquisition from two experiments, they are not enough. We will do more experiments in the future to explore in what circumstances SDM is most effective. Moreover, we will also evaluate the effectiveness of SDM method in not only business simulation game, but also in the other field’s simulation game. ACKNOWLEDGEMENT This work was supported by KAKENHI (23530430). REFERENCES Crookall, D. (1992). Editorial: Debriefing. Simulation & Gaming, 23: 141-142. Crookall, D. (2010). Serious Games, Debriefing, and Simulation/Gaming as a Discipline. Simulation & Figure 6 The progress of experiment 1 Page 313 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 Gaming, 41(6): 898-920. Thiagarajan, S. (1992). Using Games for Debriefing. Simulation & Gaming, 23: 161-173. Arai, K. (2004). Gaming Simulation, Journal of the Operations Research Society of Japan, 49(3): 143-147. (in Japanese) Tanabu, M. (2011). Education for Business Intelligence. Communications of JIMA, 20(5): 243-250. (in Japanese) Steinwachs, B. (1992). How to Facilitate a Debriefing. Simulation & Gaming, 23(2): 186-195. Petranek, C.F., Corey, S., Black, R. (1992). Three levels of Learning in Simulations: Participating, Debriefing, and Journal Writing. Simulation & Gaming, 23(2): 174- 185. Gillespie, J. A. (1973). The game doesn’t end with winning. Viewpoints, 49: 21-27. Thacher, D. C. (1990). Promoting learning through games and simulations. Simulation & Gaming, 21: 262-273. Coppard, L. C., Goodman, F. L. (1979). Urban gaming/ simulation. Ann Arbor, MI: School of Education, University of Michigan. GAME AND SOFTWARE The Bakery Game. Shirai, H. (2008). A consideration to University Programs Using Business Games. Yokohama Business Review, 29(3), 171-188. Table 5 The gaming result of experiment 2 Ranking in the profit Profit Team No. Using SDM (in-game) Remarks 1 273960 11 Y up 2 246525 5 Y down 3 246010 4 N 4 239484 9 N 5 227000 8 N 6 224216 3 Y up 7 206942 1 Y same 8 203700 7 Y down 9 196870 10 N 10 195770 2 N 11 156120 6 Y down Figure 7 The progress of experiment 2 Table of Contents Volume 40, 2013 Leo Investor SERIOUS PLAY: ON-LINE PEPULATOR SERIOUS PLAY:ON-LINE GAME "MANAGE-ART" The Game Within the Simulation Game the Research Method Concept and Project With Game Design Implementation Managing Human Resources Simulation Rising Stars: A Study of Creativity as Experiental Learning in County Government Why do Gen Y Students Study Abroad? The Relationship Between Individual Growth and the Intent to Study Abroad Our Favorite Assignment: The In-Basket Activity Mirror-Mirror: How Can We Better Understand Our Consumption Behavior? Reflective Thinking, Writing and Imagery as Tools to Better Understand Abstract Marketing Concepts Increasing Intuitive Decision Making Speed and Accuracy by Further Understanding Intuitive Decision Making Using Emotional Means Group Exams: Are they Relevant and Reliable as a learning Tool? Integrative Learning: Exploring Opportunities in Business Simulations Simulated Tabletop Exercise for Risk Management - Anti Bio-Terrorism Multi Scenario Simulated Tabletop Exercise Simulated Tabletop Exercise for Electric Power Saving Management in a Small Organization Merging the Case Method and Simulation in Management Education: Is it Possible? Experiential Learning in Accelerated Human Resource Management Courses Digital Game Building as Assessment: A Study of Secondary Students' Experience Using an Investment Fund Simulation Integrated with a Business Game Creating Simulation Conditions Based On Real World Data Simulation Games in Training New Management Methods A Summary Overview of Cultural Differences in Higher Education Innovations And Future Directions In Education: Case Review For Best Online Teaching Practices An Implicit Measure of Forecasting Accuracy Immediate Feedback and Assessment Technique (IF-AT) testing forms: An overview of the tool and uses Teaching Secondary Mathematics: Pre-service Teachers' Digital Game Design, Pedagogy and 21st Century Skills Follow The Leader II Business Simulations: From Punch Cards to Web-Based University Engagement: Good Neighbors Budget Allocation Exercise Customizing Business Simulations: An Exploration and a Hierarchy A Review of the Simulation Research in the Academy of Management Journal: Suggestions for Strengthening the Research Conducted by ABSEL Members ABSEL Reflections: 40 Years Of Excellence, Now Going Forward How Many Options do Multiple-Choice Questions Really Have? The Management/Accounting Simulation Now 100% Web-based Design, Server Side Programming, and Implementation A Large-Scale Game Test of Early-Determined Game Finishes Large-Scale Business Games for Assurance of Learning Purposes The Gamification Of Education OB Simulation: A Hands-On Demonstration Converting Simulations for the Online Environment: The New Ginseng Game Repositioning Brands with the Web-Based Product Positioning Map Graphics Package Implications of Regulatory Focus Theory for Simulation and Experiential Learning Riskware: A Game for Teaching Software Project Risk Management Teamwork for Decision-Making Through Games: The Case of On-Line "Manage-Art" Virtual Pepulator, a Model for Teaching Negotiation Small-Scale Business Games for Assurance of Learning Purposes Incorporating Intellectual Property Issues into a Business Simulation© Nationalization and Privatization in a Computer-Assisted Business Game An Investigation of the Relationship of Plan Quality, Forecast Accuracy and Earnings Performance Under Equal and Unequal Starting Positions in Marketing Positions in Marketing Simulations Measuring the Performance Ranking Curve in Marketing Simulation Games The Effectiveness of SDM Method in Business Simulation Game 40 years of Brazilian S&G - Analysis and Perspectives Accounting For Externalities: Harnessing The "Face-In-The-Mirror" Phenomenon The Role Of Simulations In Organizational Learning: Building Individual Absorptive Capacity Assessing Project Management as an Academic Learning Outcome (ALO) At the Inflection Point: Designing Army Assessments "Economics in Practice" A Simulation Game for High School Students Teaching the Basics of Economics and Entrepreneurship Fueling the Force: Exploring Leader Priorities Implementing Mental Models: Extending Insight and Whole Person Learning Using Business Simulations To Introduce Business Concepts