Learning Transfer from a Business Simulation: How are you Situated? Page 213th - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 ABSTRACT Studies have shown that learning is situated, where learning in one context does not easily transfer to a new context. So, does learning from a simulation game situation transfer to other contexts or do simulation game participants simply become better game players? To explore these questions, we asked 275 university students on a final examination to apply knowledge gained during the play of an operations management simulation, called The Fresh Connection, to both familiar and unfamiliar contexts. In the learning transfer literature, this would be classified as near low-road transfer. We hypothesized that if the context of the exam questions were both familiar (improving student comprehension of the question) and offered simulation related cues (stimulating recall of simulation learning), performance on those questions would be enhanced. We also anticipated that some learning would transfer to unfamiliar contexts. Unfortunately, our data did not unambiguously support these hypotheses. The current literature suggests that two confounding factors may have affected our experimental study: student engagement level and metacognition responses. Future research that controls for these two new variables may offer stronger support. INTRODUCTION Businesses have been vocal in their dissatisfaction with the skill level of recent university graduates. Data collected by the National Association of Colleges and Employers, on eight competencies that employers deem necessary for graduates entering the workplace, indicates that students believed themselves more proficient than their new employers do (Bauer-Wolf, 2018). Similar results were obtained in a study done by PayScale. This study indicated that nearly 90% of recent college graduates considered themselves prepared for the their new jobs, while employers indicate that 60% of new graduates lacked the critical thinking skills needed for their new jobs (Berr, 2018). Clearly, there is perceived gap between student and employer perceptions of skills. Is this perceived gap real? Is there actually a skills gap as described in the business press (Dishma, 2016; Hanc, 2017; Hora, 2017; Mirza, 2017; Tugend, 2013; White, 2018)? In a test given at 200 colleges across the U.S., The Wall Street Journal concluded that there has been a failure of America’s higher-education system to prepare graduates with the analytical reasoning and problem- solving skills needed in the fast changing and increasingly global, job market (Belkin, 2017). The magnitude of the skills gap was recognized in Business News Daily where they observed that the skills-gap is leaving more than 6 million jobs unfilled, thus, stalling business growth and innovation (Caramela, 2018). A Deloitte survey noted the disconnect between the knowledge students gained in college and the skills their employers valued most in the workplace. This survey concluded that because of an actual skills gap employers must invest heavily in training and development to prepare college recruits to make meaningful contributions to their employer’s organization (Deloitte, 2015). Thus, it appears a skills gap does exist on two dimensions; a gap in perceptions between new graduates and employers, and the gap in new graduate skills relative to the job sought (Munk, 2016). If we assume that institutions of higher education exist to prepare students for future job opportunities, an assumption questioned by some (Munk, 2016), higher education should be seeking ways to close these gaps. There may be various reasons for these gaps; however, the one receiving significant academic attention is the concept of learning transfer, with a search of Google Scholar indicating over 2,000 academic articles published in 2017. Gick and Holyoak (1987) define learning transfer as a change in performance of a task because of performance in a prior task. Thus, the study of learning transfer seeks to understand how learning in one context can be transferred to a different context. Studies have shown that learning is situated, where the skills learned in one context are locked in that context and do not easy transfer to new contexts. This is called the theory of situated learning (Bennett, Dunne, & Carré, 2000; Patel, 2018). Subscribing to this theory means that knowledge that is obtained in one environment, for example an economics class, does not easily transfer to another environment, for example a marketing class. Situated learning theory often claims that little learning transfer can and does occur in higher education (Gray & Orasanu, 1987). The theory of situated learning is sometimes used to explain the actual and perceived skills gaps that have been observed. If one believes this theory, there is little hope for higher education closing the skills Learning Transfer from a Business Simulation: How are you situated? Christopher M. Scherpereel Northern Arizona University Chris.Scherpereel@nau.edu Susan K. Williams Northern Arizona University Susan.Williams@nau.edu Scott E. Hoefle Northern Arizona University Scott.Hoefle@nau.edu mailto:Chris.Scherpereel@nau.edu mailto:Susan.Williams@nau.edu mailto:Scott.Hoefle@nau.edu Page 214th - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 gap. Fortunately, other research on transfer has provided evidence supporting learning in higher education. In a comprehensive review of teaching for transfer, Billing (2007) identified nine teaching principles that improve and maximize learning transfer: 1. Motivation (especially interest) affects whether deep (e.g., self-regulated goals – interesting and useful) or surface learning (e.g., extrinsic goals – getting good grades) occurs, with deep learning resulting in greater transfer. 2. Meta-cognitive strategies and skills (including reflection) development increases students ability to explain concepts and the conditions for application of learning, thus, increasing the potential for transfer. 3. Learning in natural contexts increases transfer, but specificity of context can reduce transfer. 4. Principles, rules, and schemata acquisition is critical and needs to be attached to pre-existing knowledge structure to increase transfer effectively. 5. Similarity and analogy are effective in bridging new with existing knowledge structures and increasing transfer. 6. Varied examples and contexts facilitate learning and increases transfer. 7. Reducing cognitive load allows for forward reasoning (i.e., reasoning from learned examples/experiences to novel problems) and pattern recognition. Forward reasoning improves learning transfer. 8. Active learning can be used to induce cognitive conflict, teach intermediate steps, generate relations, test predictions, and integrate knowledge, enabling better transfer. 9. Learning by discovery allows learners to create and augment their own schemata and search for new strategies. By discovering how the unfamiliar resembles the familiar, learning transfer increases. While these nine teaching principles seem to be good teaching practice and their efficacy is supported by research, it is still challenging to quantify the magnitude of learning transfer that has taken place in individual students. To measure learning, students are frequently evaluated using exam questions that relate to the specific contexts presented in the course. Even if case studies are used to improve student application of specific knowledge in a course, assessment is seldom done using new unfamiliar situations. Thus, much of our testing is situated and is failing to assess learning transfer. In this paper, we attempt to measure learning transfer from a specific operations management simulation context to unfamiliar but related contexts. We will show that some learning transfer has occurred and will hypothesize that other cognitive factors (not included in Billing (2007)) may play a role in learning transfer. Near versus Far Transfer To this point, we have defined learning transfer broadly: how learning in one context can be transferred to a new context. However, to actually measure learning transfer we need to be more precise in our definition. The relevant literature has made a distinction between two types of learning transfer; far transfer and near transfer (Perkins & Salomon, 1992). It is generally understood that there is a continuum of learning transfer from very near to very far; where very near transfer can be illustrated as students taking an exam that includes problems of the same type and with the same context as they have practiced in homework assignments. In contrast, very far transfer refers to the transfer of learning to situations that appear unrelated. A common example used to illustrate far transfer is using a principle from chess regarding controlling the center of the board to investment schemes, medical diagnoses, political policies, or military strategies (Alfieri, Brooks, Aldrich, & Tenenbaum, 2011). Unfortunately, the literature does not seem to provide a clear metric of closeness in distinguishing near and far transfer of learning. Thus, we are left to use these terms as a broad characterization and imprecise codification (Perkins & Salomon, 1992). Because we are looking at the transfer of course-related material on a student examination with the application of this course-related material from one context to a related but unfamiliar context, we would classify our research as an exploration of near learning transfer. Low-road versus High-road transfer In addition to identifying the closeness of learning transfer (e.g., near verse far transfer), a taxonomy has been developed to identify the level of abstraction in the transfer, with a continuum from low-road to high-road transfer. Low-road transfer (i.e., limited abstraction) occurs when a skill/learning is practiced to the point of automaticity in one context and then becomes stimulated by conditions in another context (Salomon & Perkins, 1987). Thus, the focus here is not on the context but on application itself. If the application of learning is similar in the new context, we are talking about low-road transfer, and if the application of learning is dissimilar (i.e., abstract) we consider it to be high-road transfer. For example, a student driver who has learned to drive a passenger vehicle can transfer this learning to driving a truck. Low-road transfer reflects the automatic application of well-practiced routines to circumstances similar to the original learning (Perkins & Salomon, 1988). In contrast, high-road transfer involves the deliberate mindful abstraction from the learned application to another application (Salomon & Perkins, 1987). With high-road transfer (i.e., high level of abstraction) a person learns something in one application and abstracts it to a new application (Perkins & Salomon, 1988). For example, a concept learned in college calculus might successfully be applied by a supply chain manager to improve throughput, when no prior application to this context was ever made. The manager was able to abstract learning that occurred in a calculus course to a new problem in an unrelated context. Again, a precise codification of the low-road/high-road continuum has not been developed in the literature. Therefore, one is left to classify imprecisely the abstractness of the learning transfer. In our study, we would classify the transfer as testing two levels of low-road transfer. On the lowest road is the transfer from a familiar homework problem to a contextually equivalent examination question. This is standard practice in most recall-based examinations. We also test on a slightly higher road when we ask students to apply their learning to a new context. Because we consider the level of abstraction limited in this latter case, we Page 215th - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 classify our research as exploring low-road transfer. ABSEL’S INTEREST IN LEARNING TRANSFER Researchers in the use of simulations and experiential learning in education have a history of attempting to quantify learning transfer. The quantification comes from both training (e.g., Human Resource Development) and university education research (e.g., Instructional Psychology). The training literature uses the term training-transfer to refer to the amount of training that result in on-the -job behavioral/performance change. Empirical estimates by Saks (2002) suggest that only about 51% of training investments result in a positive change for employees and a 47% positive change for their organization. Unfortunately, the data on training-transfer does not attempt to distinguish between near and far transfer or high and low road transfer. In a meta-analysis by Burk and Hutchins (2007), the authors identify three areas for future empirical and theoretical research on training transfer: 1. Future empirical research should directly assess transfer as the variable being predicted (i.e., the criterion variable). 2. Future research should validate the utility of various transfer practices in organizations to provide a closer connection between practice and research. Most of the current research is limited to case studies and/or conceptual articles. 3. Research should theorize and assess training as a multidimensional phenomenon with multilevel influences. Conducting training-transfer research studies and validating measures is challenging. Access to organizational data, gathering multiple measures from multiple sources, and randomization of study participants are all problematic (Burk & Hutchins, 2007). Packer (2001) concludes that learning transfer is hard to define, difficult to investigate, and controversial. Research methods that are able to overcome these obstacles will contribute greatly to our understanding of training-transfer and by association learning- transfer. Unfortunately, studying training transfer in real business settings remains challenging. The use of simulations and experiential learning activities in academic institutions has encouraged research regarding learning transfer. Although still presenting some methodological and implementation challenges, academic settings can offer greater control. Research in academic settings is lower risk and allows greater control over the time required to complete the study (Gagnon, 1982). Access to data is easier and less costly. Studies can be quickly implemented, and the variables adapted as changes are indicated by preliminary results. Probably the most important advantage is the familiarity of the researcher with the setting; offering more precise control over the variables in the study. Thus, empirical research in academic settings continues to be a valuable contributor to our understanding of learning transfer. The half century of research regarding business simulations and experiential learning has produced abundant articles trying to justify the advantages of these pedagogies from a learning, attitudinal, or behavioral perspective (Anderson & Lawton, 2009). Articles that actually focus on cognitive learning transfer from business simulations are rare (Burns, Gentry, & Wolfe, 1990). Despite the lack of empirical studies demonstrating the effectiveness of simulations and experiential activities (Girard, Ecalle, & Magnan, 2013), several studies seem to confirm their potential to enhance learning work-related knowledge and skills. Sitzmann (2011) shows that participation in a simulation increased declarative knowledge by 11%, procedural knowledge by 14%, retention by 9%, and self-efficacy by 20%. In a study focused on near low-road transfer, Lunce (2006) suggests that simulations can facilitate learning transfer in a classroom where the simulation approximates a situated learning context. Finally, Micklich (2014) showed that near transfer learning occurs between subsequent rounds of a simulation game. Learning transfer from simulated/experiential environments has been studied more extensively in hands-on fields such as health professions and aviation. In a meta-analysis, Shin, Park, and Kim (2015) reported that health profession simulation-based learning increased scores on knowledge-based examinations, enriched clinical application of learning, and improved overall learning outcomes. Specifically, Rutherford-Hemming (2012) showed a significant growth in clinical competency from participation in simulations prior to working on actual patients. A meta-analysis on the use of flight simulators by Hays, Jacobs, Prince, and Salas (1992), showed that 90% of research studies confirmed that a combination of flight simulator use and aircraft training was superior to aircraft training alone. It is clear that simulations and experiential learning pedagogy has some impact on learning transfer, unfortunately, most research relies on participant self-reporting of impact (Anderson & Lawton, 2009). Current research also fails to recognize the underlying taxonomy of the learning transfer; near versus far transfer (closeness), and high- versus low-road transfer (abstractedness). We attempt to empirically measure near-low-road transfer of learning and try to understand better this elusive phenomenon. METHODOLOGY Proponents of active, problem-based, simulation and experiential learning have long contended that context is important to learning. If a person experiences learning in a particular context can they transfer that learning to a new unfamiliar context? To answer this question, we asked students in an operations management class to complete several final-exam, multiple-choice questions that required they apply course material in both familiar and unfamiliar contexts. We anticipated that: H1: Students would perform better on the questions that were presented in a familiar context (FC) rather than an unfamiliar context (UC) H1: mFC > mUC Page 216th - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 H2: Students would perform better on context free (CF) questions than on unfamiliar context (UC) questions H2: mCF > mUC H3: Students would perform better on familiar context (FC) questions than on context free (CF) questions. H3: mFC > mCF For H1, the rationale is that contextual cues provided by the familiar context will improve student recall of information and improve performance on these questions (Saufley, Otaka, & Bavaresco, 1985). Additionally, if there is content overlap between what the student already knows and the context in text passage (or test question), understanding should be improved (Kintsch, 1994). Therefore, if the context of the exam questions is both familiar (improving student comprehension of the question) and offer cues (stimulating recall of simulation learning), performance on those questions should be enhanced. If the context of a question provides unfamiliar cues, it should be more difficult to answer these questions and students’ performance will likely decline. This is consistent with research that suggests that unfamiliar context cues are prone to create confusion (Schatz & Baldwin, 1986). The second hypothesis, H2, suggests that putting questions in unfamiliar context will lower performance due to this increased confusion. Even though context-free questions typically lack familiar cues (stimulating recall of simulation learning) they should not create confusion. Thus, students should perform better on context-free questions than those presented in an unfamiliar context. Finally, H3 suggests that questions in a familiar context will be the least confusing and contain the cues necessary to stimulate recall. The familiar context for all questions was based on a supply chain management simulation called The Fresh Connection (TFC). The simulation was conducted during the final four weeks of an introductory operations management class. Students participating were immersed in an intensive team-learning experience where they had to apply supply-chain principles to run a competitive supply chain. In teams of four, students were tasked with making typical supply-chain decisions in an attempt to improve their company’s supply chain performance. The team with the best performance, as measured by ROI, was declared the winner and received the highest grade for the simulation exercise. In addition to the simulation grade, the students were tested on TABLE 1 EXAM QUESTIONS FOR THIS STUDY Type Question N Pool 1 (P1) CF 1. The shelf life of your product is ___ weeks from the date of manufacture. You have promised a shelf life of ___ weeks to your customer and have achieved an average of ___ weeks of remaining shelf life on past de- liveries. You have been discarding ___ of your product that can’t be shipped due to the agreement. At the request of your customer, you have increased the shelf life agreement to ___ weeks. Which of the following do you expect to happen? A. You will discard more product. B. You can purchase less raw materials. C. You will offer a discount to your customer. D. Your quality will decrease. 100 FC 2. The shelf life of TFC juices is ___ weeks from the date of manufacture. TFC has a shelf life agreement of ___ weeks with its grocery customer and has achieved an average of ___ weeks of remaining shelf life on past deliveries. TFC has been discarding ___ of the juice that can’t be shipped due to the agreement. At the request of the grocery customer, TFC has increased the shelf life agreement to ___ weeks. Which of the fol- lowing do you expect to happen? A. TFC will discard more product. B. TFC can purchase less raw materials. C. TFC will offer a discount to its grocery customer. D. TFC’s quality will decrease. 93 UC 3. The shelf life of Joy Cone ice cream cones is ___ weeks from the date of manufacture. Joy Cone has prom- ised a shelf life of ___ weeks to Baskin Robbins and has achieved an average of ___ weeks of remaining shelf life on past deliveries. Joy Cone has been discarding ___ of the cones that can’t be shipped due to the agreement. At the request of Baskin Robbins, Joy Cone has increased the shelf life agreement to ___ weeks. Which of the following do you expect to happen?  Joy Cone will discard more product.  Joy Cone can purchase less raw materials.  Joy Cone will offer a discount to Baskin Robbins.  Joy Cone’s quality will decrease. 82 Total 275 Page 217th - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 their understanding of the supply-chain concepts as part of a comprehensive final examination. A total of 275 students participated in the simulation, took the final examination, and agreed to have their data included in the study. Three pools of questions (P1-P3) were created to test the hypotheses. Each pool contained a question that was identified as context-free (CF), familiar context (FC - i.e., simulation context), and unfamiliar context (UC). The three questions in each pool were constructed to maintain as much similarity in both construction and level of difficulty. The three pools and their questions can Pool 2 (P2) CF 1. Your company has decided to produce each product type more frequently, from every ___ days to every ___ days. If all your products have a limited shelf life and no other changes are made to the production/ distribution process, what do you expect to happen?  Fewer products will expire.  More warehouse space will be used.  Your suppliers will more easily meet delivery requirements.  Total time spent changing over equipment will decrease. 86 FC 2. TFC lowers their production interval from every ___ days to every ___ days for all products. No other changes are made. What do you expect to happen?  Product obsoletes will decrease.  Outbound warehouse space usage will increase.  Supplier reliability will increase.  Bottling changeover time will decrease. 103 UC 3. The Purina company has decided to produce each type of dog food more frequently, from every ___ days to every ___ days. If all products have a limited shelf life and no other changes are made to the dog food pro- duction/distribution process, what do you expect to happen?  Fewer Purina products will expire.  More warehouse space will be used.  Purina suppliers will more easily meet delivery requirements.  Total time spent changing over equipment will decrease. 86 Total 275 Pool 3 (P3) CF 1. You have a policy of ordering ___ weeks stock of a component used to make your product. If demand for this component is ___ units per week, and the supplier ships in ___ unit boxes, what quantity will be ordered in each lot?  1,000 units  2,000 units  900 units  300 units 92 FC 2. TFC has a policy of ordering ___ weeks stock of their Vitamin C component. If demand is ___ liters per week, and the supplier ships Vitamin C in filled IBCs (an IBC contains ___ liters), what volume of Vitamin C will be ordered in each lot?  1,000 liters  2,000 liters  900 liters  300 liters 93 UC 3. W.L. Gore Medical Products has a policy of ordering ___ weeks stock of Micro-Renathane® Implantation Tubing, a component used to make heart catheters. If demand is ___ feet per week, and the supplier ships the tubing in rolls containing ___ feet, what quantity of tubing will be ordered in each lot?  1,000 feet  2,000 feet  900 feet  300 feet 90 Total 275 Page 218th - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 be seen in Table 1. Random students received a random question from each pool, thus, we implemented a completely randomized experimental design. Approximately one third of the study participants received each question from each pool as indicated by N in Table 1. RESULTS Data collected on the final exam for 275 students are in Table 2. It is worth noting the discrimination of each item in each pool. Discrimination is measured on a per question basis using the Pearson product-moment correlation coefficient for the correlation between a student’s overall exam scores and their individual question score. The discrimination values range from − 1 (a negative correlation) to 1 (a positive correlation). According to El-Uri and Malas (2013), questions with discriminators ≥ 0.40 would be considered excellent. Good discriminators score in the range of 0.30–0.40, fair discriminators scores range from 0.10 to 0.30, and poor discriminators from 0.001 to 0.099. Questions with a score of 0.00 are considered non-discriminators. In our three pools, all questions would be considered excellent to fair discriminators. We can also infer from the average percent correct in Table 2 that Pool 3 contained the easiest questions and Pool 2 contained the hardest questions. It should be noted that if the question is too easy the hypothesized relations might not hold, with random variation accounting for the differences. Similarly, if the question is too difficult, students may just be guessing and the data may not support the hypotheses. From Table 2, it does not appear that any of the questions would be considered too hard or too easy. A statistical analysis was performed to test the three hypotheses. The results left us somewhat puzzled. Hypotheses 1 and 2 were somewhat supported by the data in both questions pools 1 and 2 but not at all by pool 3. In fact, pool 3 seemed to generate the opposite effect from those hypothesized. Hypothesis 3 was not supported by any of the pools, and similarly, the data suggested the opposite effect was present. With these mixed results, we decided to look at alternative theories that might offer an explanation and formulate some new hypotheses that could be tested in this continuing research. CONCLUSION AND DISCUSSION Because of the ambiguous results, we must consider this experimental study to be exploratory. It appears that something is happening to reverse the expected effect stated in hypothesis 3. This hypothesis was generated based on the assumption that the familiar context would provide cues to help the student answer the related questions. We were expecting that the familiarity with the simulation would avoid any confusion and ambiguity that might be generated by an unfamiliar context or one that was context free. What we realized is that our context free questions were not really context free but rather applying the question to what might be considered a personal context. That personal context, although generic, may have prompted students to spend more mental effort trying to answer the question. TABLE 2 QUESTION ANALYSIS Type Discrimination Percent Correct Attempts Average* Standard Deviation Standard Error Pool 1 (P1) CF 0.39 70.00 100 2.10 1.39 0.14 FC 0.48 64.52 93 1.95 1.45 0.15 UC 0.30 57.32 82 1.72 1.5 0.17 Pool 2 (P2) CF 0.21 46.52 86 1.40 1.51 0.17 FC 0.25 41.75 103 1.26 1.49 0.15 UC 0.44 31.40 86 0.95 1.41 0.16 Pool 3 (P3) CF 0.33 80.44 92 2.42 1.20 0.13 FC 0.33 72.05 93 2.17 1.36 0.15 UC 0.34 88.89 90 2.67 0.95 0.10 * Average score - giving 3 points for correct answers and 0 points for incorrect answers Page 219th - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 A conceptual paper by Tofade, Elsner, and Haines (2013) provides a taxonomy that might help us better understand the results. Using what are called question circles, questions are posed in the subject matter, personal response, and external reality domains. The subject matter domain includes factual, conceptual, and procedural knowledge that is derived directly from the course material. The personal response domain solicits responses on the student’s reactions, perceptions, experiences with the course material. The external reality domain focuses on the broad context in which the course material exists and its relationship with the real world. These three question circles overlap to create enriched questions (i.e., overlap of two domains) or dense questions (i.e., overlap of all three domains), shown in Figure 1. Using the question circle model, we can better classify our questions. The first question, CF (i.e., context free), in each pool TABLE 3 STATISTICAL ANALYSIS Hypothesis df t score p value (left tail) Pool 1 (P1) H1 (mFC > mUC) 173 1.030 0.152 H2 (mCF > mUC) 180 1.858 0.032** H3 (mFC > mCF) 191 0.734 0.768 Pool 2 (P2) H1 (mFC > mUC) 187 1.459 0.073* H2 (mCF > mUC) 170 2.178 0.015** H3 (mFC > mCF) 187 0.639 0.738 Pool 3 (P3) H1 (mFC > mUC) 181 3.014 0.999 H2 (mCF > mUC) 180 1.633 0.948 H3 (mFC > mCF) 183 1.378 0.915 * result is significant at p < .10 ** result is significant at p < .05 FIGURE 1 QUESTION CIRCLE MODEL Source: Modified from (Tofade et al., 2013) Page 220th - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 affect their learning. Future research may include an assessment of students’ attitudes toward the simulation to see if this affects their performance on the exam questions. making, they experience varying levels of success. In addition, students’ attitudes toward the simulation seem to differ. We have noted that some students describe anxiety and others frustration with the simulation. We expect attitude toward the simulation to would land in the enriched area between personal response and the external environment or the PE domain. The PE domain would suggest a question that elicits a personal reaction to a real-world situation. The FC (i.e., familiar context) questions would be classified as a subject-matter question, or the S domain. In the S domain, students could use factual knowledge obtained directly from the course. Finally, the UC (i.e., unfamiliar context) questions would likely be in the enriched area between the external environment and the subject-matter circles or the ES domain. ES domain-type questions would apply factual knowledge from the course to a real-world situation. Given these classifications, hypothesis H3: mFC > mCF could be rewritten as H3: mS > mPE, implying that a question within the subject domain would be easier than a question in the personal/external domain. Our data unambiguously shows the opposite relationship, mPE > ms. Performance on questions in the personal/external domain was better in all three pools. Using the three circles classification model, we might assume that these questions took the students outside of the course subject domain, increasing uncertainty, and encouraging them to engage their metacognition. Couchman, Miller, Zmuda, Feather, and Schwartzmeyer (2016) noted that when college students encounter an exam question that causes uncertainty (i.e., the PE questions in our study) they recognize their uncertainty and use metacognition to control their response. According to Couchman et al. (2016) they are likely to think about the question in new ways rather than relying on their intuition. Their study concluded that on test items causing the greatest uncertainty, initial instincts were correct less than half the time and students should revisit those questions and consider revising their answers. In the context of our study, students may have higher confidence in the familiar, simulation-based, questions and therefore rely on their first answer (often intuitive), achieving poor results. The questions with the least familiarity, most uncertainty, caused them to use their metacognition to slow their response and think harder about the question. Thus, they achieved better performance by not relying on their intuitive first response. We will call this the metacognition effect. It is possible that the metacognition effect caused the ambiguity we saw in hypothesis 1, mFC > mUC, and hypothesis 2, mCF > mUC. From the data collected in this study, there is no way to assess how big the metacognition effect might have been. However, question pool 3 contains challenging questions on which the students performed exceptionally well. We might hypothesize that students were less confident in these questions, and thus, thought harder about those questions. Again, the metacognition effect might be playing a role. If we only consider question pool 1 and pool 2, the data seems consistent with hypothesis 1, mFC > mUC, and hypothesis 2, mCF > mUC. We anticipated a greater magnitude difference, but are hopeful that this is an indication that near learning transfer is occurring. What makes us confident that this is the case is the relative performance on questions from the three different pools. It is clear that pool 2 was the most challenging across all question types and questions from pool 3 were the easiest. Still, all three question types exhibited the same relative performance within each pool. FUTURE RESEARCH Due to the metacognition effect, we would propose a reverse relationship in hypothesis 3. Because level of confidence has been shown by Couchman et al. (2016) to trigger the metacognition response, we plan to include a measure of confidence in the student responses to these questions. If students are confident in their responses to questions that they answered incorrectly, it is likely they answered these questions without thinking carefully. We anticipate that students will indicate greater confidence in the subject-related (i.e., simulation) questions and therefore are more likely to rely on their initial response to the question as being correct. In addition to adding a question regarding student confidence, to try and measure metacognition effect, we believe that measuring student effort and attitude may also be fruitful. In the current study, we assumed that the students had a similar experience and level of interaction with the simulation. The Fresh Connection is a challenging, dynamic, and complex simulation, so participating students will certainly have different learning experiences. With some students putting greater effort into their decision- Page 221st - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 REFERENCES Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of educational psychology, 103(1), 1. Anderson, P. H., & Lawton, L. (2009). Business simulations and cognitive learning: Developments, desires, and future directions. Simulation & Gaming, 40(2), 193- 216. Bauer-Wolf, J. (2018). Study: students believe they are prepared for the workplace; employers disagree. Retrieved from https://www.insidehighered.com/ news/2018/02/23/study-students-believe-they-are- prepared-workplace-employers-disagree Belkin, D. (2017). Exclusive Test Data: Many Colleges Fail to Improve Critical-Thinking Skills. Retrieved from https://www.wsj.com/articles/exclusive-test-data-many -colleges-fail-to-improve-critical-thinking-skills- 1496686662 Bennett, N., Dunne, E., & Carré, C. (2000). Skills Development in Higher Education and Employment: ERIC. Berr, J. (2018). Employers: New college grads aren't ready for workplace. Retrieved from https://www.cbsnews.com/ news/employers-new-college-grads-arent-ready-for- workplace/ Billing, D. (2007). Teaching for transfer of core/key skills in higher education: Cognitive skills. Higher education, 53(4), 483-516. Burk, L., & Hutchins, H. (2007). Training transfer: an integrativ literatur review. Human Resource Development Review, 6, 263-296. Burns, A. C., Gentry, J. W., & Wolfe, J. (1990). A cornucopia of considerations in evaluating the effectiveness of experiential pedagogies. Guide to business gaming and experiential learning, 253-278. Caramela, S. (2018). Bridge the Gap: What Companies Can Do About the College Skills Gap. Retrieved from https:// www.businessnewsdaily.com/10729-college-not- closing-skills-gap.html Couchman, J. J., Miller, N. E., Zmuda, S. J., Feather, K., & Schwartzmeyer, T. (2016). The instinct fallacy: the metacognition of answering and revising during college exams. Metacognition and Learning, 11(2), 171-185. doi:10.1007/s11409-015-9140-8 Deloitte. (2015). College Grads Face Skills Gap in Workplace. Retrieved from https://deloitte.wsj.com/ cio/2015/05/21/college-grads-face-skills-gap-in- workplace/ Dishma, L. (2016, 2016-05-17). These Are The Biggest Skills That New Graduates Lack. Retrieved from https:// www.fastcompany.com/3059940/these-are-the-biggest -skills-that-new-graduates-lack El-Uri, F. I., & Malas, N. (2013). Analysis of use of a single best answer format in an undergraduate medical examination. Qatar medical journal, 1. Gagnon, R. J. (1982). Empirical Research: The Burdens and the Benefits. Interfaces, 12(4), 98-102. Gick, M. L., & Holyoak, K. J. (1987). The cognitive basis of knowledge transfer Transfer of learning: Contemporary research and applications (pp. 9-46): Elsevier. Girard, C., Ecalle, J., & Magnan, A. (2013). Serious games as new educational tools: how effective are they? A meta‐ analysis of recent studies. Journal of Computer Assisted Learning, 29(3), 207-219. Gray, W. D., & Orasanu, J. M. (1987). Transfer of cognitive skills Transfer of learning: Contemporary research and applications (pp. 183-215): Elsevier. Hanc, J. (2017, 20170607). With Innovation, Colleges Fill the Skills Gap. Retrieved from https:// www.nytimes.com/2017/06/07/education/with- innovation-colleges-fill-the-skills-gap.html Hays, R. T., Jacobs, J. W., Prince, C., & Salas, E. (1992). Flight simulator training effectiveness: A meta-analysis. Military psychology, 4(2), 63-74. Hora, M. T. (2017). Beyond the Skills Gap. Retrieved from http://www.naceweb.org/career-readiness/trends-and- predictions/beyond-the-skills-gap/ Kintsch, W. (1994). Text comprehension, memory, and learning. American Psychologist, 49(4), 294. Lunce, L. M. (2006). Simulations: Bringing the benefits of situated learning to the traditional classroom. Journal of Applied Educational Technology, 3(1), 37-45. Micklich, D. L. (2014). Does Learning Occur In One-Shot, Non -Cooperative Games? Paper presented at the Developments in Business Simulation and Experiential Learning: Proceedings of the Annual ABSEL conference. Mirza, A. (2017). Students Graduate With Mediocre Skills for the Workplace. Retrieved from https:// www.usnews.com/news/stem-solutions/articles/2017- 04-03/study-most-students-graduate-with-mediocre- skills-for-the-workplace Munk, J. (2016, 2016-05-08 08:00:00). Universities can’t solve our skills gap problem, because they caused it. Retrieved from http:// social.techcrunch.com/2016/05/08/universities-cant- solve-our-skills-gap-problem-because-they-caused-it/ Packer, M. (2001). The problem of transfer, and the sociocultural critique of schooling. The Journal of the learning sciences, 10(4), 493-514. Patel, C. (2018). Situated Learning: Macat Library. Perkins, D. N., & Salomon, G. (1988). Teaching for transfer. Educational leadership, 46(1), 22-32. Perkins, D. N., & Salomon, G. (1992). Transfer of learning. International encyclopedia of education, 2, 6452-6457. https://www.insidehighered.com/news/2018/02/23/study-students-believe-they-are-prepared-workplace-employers-disagree https://www.insidehighered.com/news/2018/02/23/study-students-believe-they-are-prepared-workplace-employers-disagree https://www.insidehighered.com/news/2018/02/23/study-students-believe-they-are-prepared-workplace-employers-disagree https://www.wsj.com/articles/exclusive-test-data-many-colleges-fail-to-improve-critical-thinking-skills-1496686662 https://www.wsj.com/articles/exclusive-test-data-many-colleges-fail-to-improve-critical-thinking-skills-1496686662 https://www.wsj.com/articles/exclusive-test-data-many-colleges-fail-to-improve-critical-thinking-skills-1496686662 https://www.cbsnews.com/news/employers-new-college-grads-arent-ready-for-workplace/ https://www.cbsnews.com/news/employers-new-college-grads-arent-ready-for-workplace/ https://www.cbsnews.com/news/employers-new-college-grads-arent-ready-for-workplace/ https://www.businessnewsdaily.com/10729-college-not-closing-skills-gap.html https://www.businessnewsdaily.com/10729-college-not-closing-skills-gap.html https://www.businessnewsdaily.com/10729-college-not-closing-skills-gap.html https://deloitte.wsj.com/cio/2015/05/21/college-grads-face-skills-gap-in-workplace/ https://deloitte.wsj.com/cio/2015/05/21/college-grads-face-skills-gap-in-workplace/ https://deloitte.wsj.com/cio/2015/05/21/college-grads-face-skills-gap-in-workplace/ https://www.fastcompany.com/3059940/these-are-the-biggest-skills-that-new-graduates-lack https://www.fastcompany.com/3059940/these-are-the-biggest-skills-that-new-graduates-lack https://www.fastcompany.com/3059940/these-are-the-biggest-skills-that-new-graduates-lack https://www.nytimes.com/2017/06/07/education/with-innovation-colleges-fill-the-skills-gap.html https://www.nytimes.com/2017/06/07/education/with-innovation-colleges-fill-the-skills-gap.html https://www.nytimes.com/2017/06/07/education/with-innovation-colleges-fill-the-skills-gap.html http://www.naceweb.org/career-readiness/trends-and-predictions/beyond-the-skills-gap/ http://www.naceweb.org/career-readiness/trends-and-predictions/beyond-the-skills-gap/ https://www.usnews.com/news/stem-solutions/articles/2017-04-03/study-most-students-graduate-with-mediocre-skills-for-the-workplace https://www.usnews.com/news/stem-solutions/articles/2017-04-03/study-most-students-graduate-with-mediocre-skills-for-the-workplace https://www.usnews.com/news/stem-solutions/articles/2017-04-03/study-most-students-graduate-with-mediocre-skills-for-the-workplace https://www.usnews.com/news/stem-solutions/articles/2017-04-03/study-most-students-graduate-with-mediocre-skills-for-the-workplace http://social.techcrunch.com/2016/05/08/universities-cant-solve-our-skills-gap-problem-because-they-caused-it/ http://social.techcrunch.com/2016/05/08/universities-cant-solve-our-skills-gap-problem-because-they-caused-it/ http://social.techcrunch.com/2016/05/08/universities-cant-solve-our-skills-gap-problem-because-they-caused-it/