Assessing Systemic Thinking in Undergraduates: An Exploratory Study Using a Total Enterprise Business Simulation Page 53 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 ASSESSING SYSTEMIC THINKING IN UNDERGRADUATES: AN EXPLORATORY STUDY USING A TOTAL ENTERPRISE BUSINESS SIMULATION Michelle L. Washington Arcadia University washingm@arcadia.edu Raghu Kurthakoti Arcadia University kurthakotir@arcadia.edu Annette L. Halpin Arcadia University halpina@arcadia.edu Steven Byrd Arcadia University sbyrd@arcadia.edu ABSTRACT Assessing the extent to which systemic thinking is developed through total enterprise simulations is critical in measuring the effectiveness of this tool in the business curriculum. This exploratory research aims to quantitatively evaluate and assess the level of systemic thinking that undergraduate students in a first-year business course develop during the use of a business simulation. Students provided feedback on decision summary reports which was content analyzed to develop a rubric that was used to evaluate the extent of systemic thinking development among students. Further analyses were conducted to relate the level of systemic thinking to organizational performance. Results indicate that students who start with a low level of systemic thinking show an increased level of systemic thinking as the simulation progresses. We also found a positive relationship between systemic thinking and organizational performance both in the short and long run. INTRODUCTION The real world is complex and ambiguous and poses a myriad of challenges to today’s business leaders (Gregory & Miller, 2011). The interdependence of people, ideas, and beliefs challenge managers as they ‘make sense’ of the reality around them. Decision makers need to understand the complexity of the current environment and appreciate that there is no ‘simple’ solution to the problems they face (Caldwell, 2012). Gregory and Miller (2011) suggest that “the strategic models on which management decisions are based need to be more holistic than ever due to a tighter coupling among different components of the environment” (p. 5). Furthermore, incomplete information is a characteristic of our business environment so lengthy analysis is often not possible, especially in entrepreneurial enterprises. Without much time for deliberation, developing skills to become a manager who is comfortable making decisions in an uncertain environment and with limited time should lead to greater success (Noel & Erskine, 2013). “Systems thinking is a way of understanding reality that emphasizes the relationships among various components in a process, rather than the independent constituents of the process” (Gregory & Miller, 2011, p. 259). It involves an understanding of many smaller systems, subsystems, which exhibit patterns that can be discernible and understood in a larger context (Gregory & Miller, 2011). Systemic thinking views a problem in terms of its properties – the interaction of the parts with the whole. It is an attempt to see the ‘big picture’ by identifying relationships and patterns (Allio, 2003; Senge & Fulmer, 1993). Managers with a systems view of the world may be better able to conceptualize the structures Page 54 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 that exist in their organizations since the foundation of this approach is based on logic and an understanding of causality (Henning & Chen, 2012). If we are to strengthen the skills of the next generation of managers to think holistically, business education programs need to incorporate training with an eye on systems thinking and an understanding of how learning occurs in organizations (Allio, 2003; Noel & Erskine, 2013). One tool that may prove effective at developing and assessing systemic thinking skills is the total enterprise computer simulation (hereafter ‘simulation’). Often used in business programs at colleges and universities, these software packages model the functional areas of an enterprise along with external factors and offer students an opportunity to experiment with strategies to achieve financial success (Romme, 2004). This ‘practice’ method involves making decisions that lead to stronger performance in some cases and unintended consequences in others (Caldwell, 2012). Understanding that decision making is influenced by the logic of the decision maker, self-interest, and personal values simulations require a holistic approach to problem solving which may help future business leaders hone their skills in creative thinking (synthesis) and judgment (evaluation), two learning goals suggested by Bloom (Bloom, Englehart, Furst, Hill, & Krathwohl, 1959; Caldwell, 2012). In post-secondary business education, there are challenges to incorporating a systems thinking view of the world. These include: 1) the compartmentalization of the business curriculum, 2) the a-contextual nature of business education, and 3) the relationship between theory and practice (Gregory & Miller, 2011). When individual disciplines are considered more important than the whole students are getting short changed since they are limited in the development of skills to analyze complex problems. Secondly, most business texts are based on USA-focused research so future leaders have limited exposure to culturally diverse contexts within which many business situations occur. Finally, a background in theory alone is not the answer to business education. Future business leaders have much to gain from practical experience so a combination of theory and practice should be the norm. Simulations provide a means for assessing student learning (Anderson, Cannon, Malik & Thavikulwat, 1998). It is common in studies using simulations to have samples with upper-level undergraduate students, MBA students, and professionals. Evaluating decision making skills and assessing student learning are generally thought to be best measured by those nearing completion of an undergraduate degree or those in the post-graduate stage (Bloom et al., 1959; Halpin, 2013; Hornyak, Peach & Snyder, 2007; Wolfe, 1979). But if we are intent on developing and studying systems thinking in future business leaders we should begin to assess this phenomenon in earlier stages of business programs. The current research proposes that business programs should seek to develop and assess its students’ systemic thinking skills earlier in the education process. Incorporating more simulations into programs – and at earlier stages – will prove beneficial in preparing students to navigate the uncertain and highly competitive business environment into which they will enter after graduation. We attempt to defend this argument by evaluating the degree of and changes in the systemic thinking of students in the early stage of an undergraduate degree. Our research builds on prior research using computer simulations by evaluating statements by decision makers and linking their understanding of the parts (marketing, management, and operations) to the whole (firm performance). We organize the paper in the following manner: Our literature review begins with a discussion of the value of systemic thinking for managerial proficiency and, ultimately, organizational success. We then argue the importance of using simulations as a tool to assess systemic thinking in underclassmen. This is followed by the presentation of our propositions and a description of our research methods and analytical techniques employed. Then, we share our results, conclusions, and suggestions for future research. LITERATURE REVIEW The examination of any system recognizes that its characteristics help define it and that the ‘whole’ has properties independent of its parts (Allio, 2003). A business entity can be thought of as a system with set goals and objectives and whose achievements go beyond what its individual parts are able to achieve. This view implies that success will only be achieved when members recognize a system’s rules and understand its patterns and logic (Henning & Chen, 2012). Other considerations in the study of systems include the interrelationships of a system’s members with one another as well as the interactions that occur between a system and its environment. Feedback received from this complex series of iterations guide decision makers as they adapt their organization while struggling to maintain its core properties (Caldwell, 2012). For a leader, being a systems thinker goes beyond learning individual facts about a system to understanding those facts within a broader context. Understanding ‘how’ a system works is less challenging than being able to explain ‘why’ a system works (Henning & Chen, 2012). Grasping issues in their totality, interpreting feedback, identifying patterns in behavior, and problem solving are recognized as skills needed to cultivate sustainability in an organization (Gregory & Miller, 2011). These traits, in particular identifying patterns, can help decision makers move from a position of helplessness and feeling ‘out of control’ to one which provides simplification and a clear view of what is developing in a situation (Senge & Fulmer, 1993). Systems thinking then, as a method of identifying and analyzing relationships, has practical value for managers charged with managing the complexity of their business Page 55 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 organizations while improving performance (Allio, 2003; Gregory & Miller, 2011). Business programs at the university level are charged with teaching business skills and concepts to students. To address the need for complex thinking by future leaders, business schools and programs can incorporate into the curriculum a series of challenges and situations that business leaders face. Russell Ackoff suggests that we move students beyond learning a set of vocabulary terms and principles to incorporating hands-on learning including apprenticeships and mentoring experiences (Allio, 2003). Considering the competitive environment for job seekers in today’s job market, we can no longer view a business degree as an automatic “ticket of admission” to a job (Allio, 2003, p. 20). There must be more at the core of this learning experience. If we look at Bloom’s Taxonomy of Learning there is a systemic ordering of educational outcomes, two of which involve higher-level learning - Objective Synthesis and Objective Evaluation. Relating these to management education, we might say that decision makers arrive at solutions to problems using creative thinking (Synthesis) and judgment based on detailed analysis and logic (Evaluation) (Bloom, Englehart, Furst, Hill & Krathwhol, 1959). With a focus on these upper-level capabilities the learning process and activities within an undergraduate curriculum should develop the systems thinking abilities of students which can then be carried into the work environment and, hopefully, produce successful leaders. To assess what learning is taking place within a curriculum, a variety of tools are used including case analysis, essays, class discussions, problems, and computer simulations. Independent of the teaching method, assessment verifies whether planned outcomes are achieved at institutions (Anderson, et al., 1998). Criticisms of case studies are that one learns only one or two business principles or that the solutions offered are not creative (Allio, 2003). They generally focus on improving the performance within one functional area of an organization, which ignores the richness of most business situations involving the parts interacting as a whole with properties distinct from the individual units. Improving the performance of the parts alone may not necessarily result in improved performance of the whole (Allio, 2003). Simulations help students develop ‘mental models’ of business organization through the testing of strategy alternatives, review and analysis of feedback, and operating in an uncertain environment (Gregory & Miller, 2011; Senge & Fulmer, 1993; Thavikulwat, 1994). Gaming parameters, set by administrators, determine the complexity of the environment and allow for adjustments in costs, product quality, financial constraints, etc. The automation of this tool has increased their ease of use and the speed with which results are generated and feedback provided (Thavikulwat, 1994). These improvements have led to the increased use of this tool for both teaching and research purposes. Recognizing the need to improve the systemic thinking of future decision makers, business schools need to identify teaching methods and tools that will help students view connections more holistically when analyzing business situations (Gregory & Miller, 2011). Thinking systemically is a learned skill involving more than definitions of what a system is. It is knowledge that leads to wisdom about a complex entity. In a business environment it moves decision makers away from focusing primarily on the parts (marketing, operations, etc.) to viewing the whole as having its own set of unique characteristics and qualities. To understand systemic thinking it important to analyze the cognitive skills used by individuals to understand a system (organization) and how these influence decision making and, ultimately, the success or failure of the entity (Henning & Chen, 2012). The current study argues that the use of simulations in business programs is an effective tool to assess the level of systemic thinking of learners in the initial stage of a business program. Accepting that understanding the ‘why’ of how a system operates leads to creative thinking and better judgment by decision makers, we argue that assessing this skill should occur earlier in a program than most studies would suggest. Drawing from extant literature we offer the following. Proposition 1: The use of business simulations is related to the systemic thinking of students enrolled in an introductory business course. Business programs establish objectives for student learning in several domains such as theory, critical thinking, and literacy. Simulations are tools that are used at both the undergraduate and graduate levels to assess performance by students in these areas. Quantitative measures such as Earnings, Return on Investment, and Return on Equity have been used to assess student performance within and between programs (Wolfe, 1979). Qualitative measures such as perceptions of academic preparation and value shed light on student reactions to this tool. Content areas such as project management have been evaluated using a total enterprise simulation. In a 2007 study, individuals assessed team member skills associated with planning, teamwork, and project delivery (Hornyak et al., 2007). Studies using simulations have examined the decision making process (Early, Northcraft, Lee, &Lituchy, 1990; Gladstein& Reilly, 1985) as well as constructs in organizational behavior (Boone, Van Olffen, & Van Witteloostuijn, 2005; Ellis, 2006; Waller, 1999), strategy (Chesney & Locke, 1991; Knight, Durham, & Locke, 2001; Mathieu & Schulze, 2006), finance (Seo & Barrett, 2007; Seo, Goldfarb, & Barrett, 2010), marketing (Smith, Mitchell, & Summer, 1985), and operations (Aggarwal & Dhavale, 1975). Performance, a common variable of interest, is operationalized using measures such as market share (Seijts, Latham, Tasa, & Latham, 2004), stock portfolio value (Earley, Northcraft, Lee, &Lituchy, 1990; Page 56 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Seo& Barrett, 2007; Seo, Goldfarb, & Barrett, 2010), and financial outcomes such as Earnings Per Share, Return on Sales, and Net Income (Chesney & Locke, 1991; Mathieu & Schulze, 2006). Work by Noel & Erskine (2013) studied decision-making by evaluating written feedback about perceptions of progress over several periods of operating a business. Using content analysis, they found that participants using action-oriented language had stronger performance than those who did not. Simulations are not only useful because of their ability to measure progress towards program goals and overall performance. Students themselves offer feedback to administrators regarding their value as an aid in learning. Early research by Wolfe (1979) found that students from different institutions found participating in a simulation exercise both challenging and useful. Work by Romme (2004) looked at three student populations (lower-level undergraduates, upper-level undergraduates, and MBA students) to link perceptions of the value of using simulations to understanding relationships in the business world. While lower-level undergraduates perceived the ‘added value’ for learning using a simulation less than upper-level undergraduates and MBA students, upper-level undergraduates and MBA students rated the ‘added value’ of simulations as high relative to their previous courses. The proposed research moves away from investigating perceptions to analyzing decision making in specific areas (marketing, management, and operations) and relating them to measures of performance. Drawing from extant literature, we propose a second proposition. Proposition 2: The level of systemic thinking is related to organizational performance by students enrolled in an introductory business course. METHODOLOGY AND DATA ANALYSIS In this study we used an online business simulation to assess the systemic thinking skills of undergraduate students in an introductory business course, International Business and Culture. Students were enrolled in a small liberal arts university located in northeastern US. Given the strong emphasis on the development of global perspectives among the students at this university, the leadership of the business school decided several years ago to eliminate the traditional “Introduction to Business” course as one of the first required courses in the business curriculum and replace it with an international business course. The majority of students in this course are in their first year while some take this introductory course as an elective. A requirement for this course is for students to manage a coffee shop using an internet-based simulation called BizCafe that is licensed by Interpretive Simulations (James & Deighan, 2012). Students work alone to make marketing (price, promotion, product), management (hire, fire, compensation), and operations (hours, days, coffee purchase, cups purchase) decisions. In addition, instructors can assign ‘special decisions’ (incidents) such as offering Wi-Fi, purchasing equipment insurance, etc. BizCafe provides feedback at the end of each period to aid students in understanding the implications, financial and otherwise, of their decisions. These include balance sheet, income statement, checkbook, and inventory report. In this course, students are introduced to the simulation via an in-class lecture followed by several practice sessions during which each student navigates the simulation in “benchmark” mode. This allows each participant to “play” a few periods at his/her own pace. After the practice rounds, the instructor changes to “direct competition” mode wherein students compete directly against one another and enter decisions for each period according to a schedule set by the instructor. Students manage their cafes for a total of seven periods. Starting after Period 4 the instructor requires students to submit, for grading, Decision Summary Reports which provide feedback about decisions made in the three functional areas addressed in this simulation. These reports are not required earlier than period 4 to allow students to gain experience with the simulation since most participants are using a simulation for the first time. The report format was developed by the instructor and is shown in Appendix A. The content of each of the reports is the same in that it asks students to reflect upon their café’s performance in the prior period, to indicate what changes in decisions will be made in the next period, and explain how these changes will affect the café’s performance on multiple dimensions. To test our first proposition (P1), we had to assess the level of systemic thinking that students exhibited over the course of the simulation. We developed a rubric to quantitatively assess systemic thinking by students. The post-Period 4 and post-Period 6 reports (hereafter ‘first’ and ‘second’) were used to ascertain changes in the level of systemic thinking among students. The development of the rubric was guided by our intent to conduct content analysis of the written feedback of students. Content analysis is a procedure involving the gathering of text, developing categories related to the text, coding information, and analyzing data (Frey et al., 1999). We now discuss the process of developing and refining the rubric. Students were expected to turn in Decision Summary Reports three times during the simulation, post-Periods 4, 5 and 6. In order to develop a rubric to assess systemic thinking, the three authors who did not instruct the course used a sample of the post-Period 5 reports to evaluate the information students provided. We used this report (post- Period 5) to develop the rubric instead of the post-Period 4 or 6 reports for a few reasons: (a) the use of either post- Period 4 or post-Period 6 reports would have affected our sample frame for the study since we would have had to exclude these students from the main analysis and (b) this report was likely to be better written than the first (post- Period 4) and provide more meaningful statements to gauge the thinking of students. Roughly 12 reports, randomly Page 57 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 selected by the instructor, were reviewed. The rubric was refined after multiple rounds of discussion among three of the authors. In the final stage, the instructor was brought in to offer insights after which the rubric was approved to rate the level of systemic thinking of students (See Appendix B). Two of the authors acted as raters for all student feedback. The instructor of the course had access to student reports for nine semesters dating back to spring 2009. This amounted to about 300 students in total. However, for the sake of assessment, a subset of 81 students was randomly selected from this pool. This sample size is comparable to prior studies whose samples have ranged from 22 to 105 students (Hornyak et al., 2007, Noel & Erskine, 2013, Thavikulwat, 2001, Wolfe, 1979). Stratified random sampling was used to generate the sample such that students from each semester were proportionately represented in the sample. For the selected 81 students, contents from the first and second reports were analyzed and quantified using the prepared rubric. The instructor removed any identifiers such as student and café names as well as any verbiage related to the reporting periods (post- Period 4 or post-Period 6). Two authors, who were not the instructors of the course, performed Content Analysis of student reports. This was done to minimize bias and prior knowledge, which might interfere with the rating process. A total of 162 reports were rated by each of the two raters (81 students) but due to issues with the reports of four students, only 154 responses (77 students) could be used in the final analysis. Each report was rated across 6 items, each item on a 7- point scale indicating a progression of systemic thought. In total the raters rated a total of 924 items each. The inter- rater reliability was assessed by both consensus and consistency estimates (Stemler, 2004). The percentage of agreement for both raters was 79.2% indicating a high degree of consensus between the raters. Additionally, Cohen’s Kappa was estimated for each item rated by the two raters. The Kappa values ranged from 0.628 – 0.845 (p < 0.001) again indicating a high degree of consensus beyond chance. For each of the six items on the rubric, inter-rater consistency as measured by correlation, ranged from 0.769 to 0.908. The reliability estimates, as measured by Cronbach’s alpha, also showed that reliability estimates ranged between 0.856 to 0.952 indicating that the raters were not only in consensus on rating each item of the report, but were also consistent in their rating. Any differences in the rating between raters were resolved through discussions. Once agreement was reached the final scores were used in further analysis. A paired sample t-test was conducted to test P1. The expectation was that, on an average, the rating on the second report would be higher than the first report. Since the study was designed as a within-subjects study, a paired sample t-test was appropriate (Stevens, 2002). For the purpose of testing, scores on all six items for each report were averaged to create a summated scale. This decision was both conceptually sound and statistically valid. The main intention of the rubric was to assess systemic thought. However, the reports were structured in such a way that each item measured only one dimension of systemic thought as envisioned in the course. So, to gauge the extent Table 1 Descriptive Statistics of Full Samples (n = 77 and n = 41) FIRST REPORT SECOND REPORT Details of the Measure Variable Mean Standard Deviation Mean Standard Deviation Systemic Thinking - Average Rating for the Reports (n=77) 4.21 1.09 3.98 1.18 Ratings on all six items of the rubric for each report was averaged to get the summat- ed score. Simulation Performance Metrics in the Period Immediately Following Each Report (n=41) Net Income -12.66 1795.66 2160.63 1744.66 In $ per decision period Revenue 8611.85 2534.72 9598.95 2436.54 In $ per decision period Cups Sold (Sales) 2490.20 589.09 2817.63 650.80 Actual number of cups (irrespective of size) per decision period Return on Sales -5.43 31.12 21.51 15.32 As a percentage (Revenue/Sales) Brand Awareness 58.38 26.89 79.03 25.50 Score between 0-100 determined by the simulation Page 58 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 of development of systemic thought, all items of the rubric had to be considered simultaneously. Additionally, a Principal Component Analysis (Hair et al., 2009) on the rubric ratings for both reports indicated that all variables loaded on to a single component. Therefore, a summated scale was created and used to test P1. Proposition 2 addresses the use of systemic thinking in decision-making and tests for a relationship between student scores and performance. BizCafé provides a variety of measures that could be used to evaluate student performance. Based on previous work (Thavikulwat, 2001) and available metrics in the simulation, the instructor chose the following – profits, sales, revenue, and brand awareness. To test this relationship, a series of regressions on various simulation performance metrics (profits, brand awareness, etc.) were performed using the summated scores from the Decision Summary Reports. Due to data limitations, we could only test P2 on a subset of the sample tested for proposition 1. RESULTS AND DISCUSSION As explained in the Methodology and Data Analysis section, to test P1 we used a rubric to conduct a content analysis to translate student feedback into quantitative variables. Students’ levels of systemic thinking were then used for further analysis. In this section, we will discuss the results of our analysis. Table 1 presents some descriptive statistics of the sample. When we look at these statistics, we find that the average rating on the rubric for the FIRST report (post- Period 4) is greater than the SECOND report (post-Period 6). This indicates that the level of systemic thinking declines as the simulation progresses (4.2 to 3.98). To confirm whether this difference is statistically significant we performed a paired sample t-test. The results indicate that this difference in rating between the two reports is statistically significant (t= 2.634; p < 0.05). This result seems counter intuitive as one would not expect that as the simulation progresses, systemic thinking diminishes. In order to further probe the possible reasons for these results, we performed a series of additional tests. We first organized the data based on the average rating of the FIRST report. These data were divided into three equal subgroups – those scoring low, those scoring average and those scoring high on systemic thinking. The descriptive statistics for the high and low scoring subgroups are summarized inTable2. When we look at Table 2, we find that the low scoring group actually shows some improvement in their scores on TABLE 2: Descriptive Statistics for Two Subgroups – High and Low Scorers on Systemic Thinking Rubric LOW Group HIGH Group FIRST REPORT SECOND REPORT FIRST REPORT SECOND REPORT Variable Mean SD Mean SD Mean SD Mean SD Average Sys- temic Think- ing Rating on Two Reports (nlow =24; nhigh=25) 3.05 0.76 3.24 1.01 5.09 0.65 4.68 0.79 Word Count for Two Re- ports (nlow =24; nhigh=25) 146 72.26 132.04 86.80 241.44 84.85 222.44 77.43 Simulation Performance Metrics in the Period Immediately Following the Report (nlow =7; nhigh=18) Net Income -$ 340.29 $2,351.09 $2,082.57 $2,280.86 -$ 0.56 $1,822.15 $1,995.56 $1,478.51 Revenue $8,488.57 $2,372.21 $9,751.71 $2,420.45 $8,684.28 $2,738.62 $9,453.28 $2,249.30 Cups Sold (Sales) 2557.43 506.96 3067.29 735.88 2517.50 721.10 2820.33 697.06 Return on Sales -8.63 35.33 19.29 19.97 -5.83 29.63 20.75 11.34 Customer Sat- isfaction -181.43 635.28 18.26 103.94 -32.47 361.52 46.32 40.81 Brand Aware- ness 56.71 35.27 79.43 36.63 59.30 24.51 78.02 24.32 Page 59 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 the report whereas the students who were judged to have a higher level of systemic thinking initially show a decline in their scores towards the end of the simulation. A paired t- test for the scores on the reports for these two subgroups indicates that the improvement in the low scoring group (3.05 to 3.24) was not statistically significant. On the other hand, the difference between the scores on the two reports for the high scoring group (5.09 to 4.68) was statistically significant. Table 3 summarizes the results of the t-tests. As shown in Table 2, we also analyzed one characteristic of the written reports, word count. The expectation was that low performers might be less articulate in their summary report, which could account for lower scores on systemic thinking. However, although not statistically significant, both groups show a decline in the length of their reports. This suggests a factor, common to both groups, might be at work here. We offer that ‘fatigue’ might be a condition affecting all members of the sample. Students were expected to provide a Decision Summary Report three times during the course of the simulation. The structure and format of this report was identical for all three reports. Additionally, these reports were due for submission during consecutive weeks in most semesters (and in some cases two reports were due within the same week with only a couple of days gap between reports). The frequent and repetitive nature of the report might have led to fatigue in students resulting in shorter and less detailed reports. While students may have developed systemic thinking, it was not explicitly expressed in their reports due to the repetitiveness of the assignment. Another possible reason for the decline for high scorers could be the fact that each report was meticulously reviewed and returned to students with detailed feedback on its content. It is possible that low scoring students received critical feedback that prompted them to ‘think differently’ in future periods resulting in improved scores on future reports. High scoring students, on the other hand, received little negative feedback on their earlier reports and may have gotten complacent in their later reports. This, combined with the fatigue effect, might be related to the drop in average scores on the SECOND report. A final reason for the decline in scores for the high scorers might be that decisions made following the SECOND report was the final decision of the simulation. It is possible that some students were preparing an end-of- game strategy that may not reflect systemic thinking. The ultimate goal for many students may have been to ‘win’ the simulation so this might have meant taking drastic measures to do well in the competition even if it meant abandoning an approach to decision making that would reflect systemic thinking. To test P2, we tested the differences in organizational performance at different points of time. As shown on Tables 1 and 2, irrespective of the subgroup studied, there appears to be a significant improvement in various performance metrics in the latter stages of the simulation. Unless one can demonstrate a relationship between the students’ systemic thinking and performance, this improvement could be attributed to learning curve effects. To demonstrate the link between systemic thinking and performance we conducted correlation analysis on the total sample (n=77) to identify relationships between report ratings and performance. The full sample was used because the number of observations in the low/high subgroups was too small to be meaningful for interpretation. Results of the correlation analysis are summarized in Table 4. When we look at the correlations in Table 4 we see that, as expected, ratings on the first and second report are highly correlated. Similarly, most of the performance measures are highly correlated with each other during the same decision period. These performance measures are highly correlated across decision periods indicating that the past performance is related to future performance. However, the relationships of interest for P2 are in columns 1and 2 of Table 4. Contrary to what one might expect, there seems to be no relationship between the ratings on the SECOND report and the performance metrics in the period immediately following. This provides additional support for our belief that end-of-game strategies might have influenced performance and might not be consistent with a systemic thought process. There was some weak correlation (p < .10) between the FIRST report and performance in the period immediately following this report. This suggests that systemic thinking may be positively related to firm performance. To explore in greater depth the weak correlation between systemic thinking and performance (as measured by systemic thinking scores and Net Income), we performed additional regression analysis. The results of the regressions are summarized in Table 5. Table 5 presents some interesting findings. Model 1 indicates that Net Income in the final decision period is unrelated to the systemic thinking score on the second report. Consistent with the correlations results (Table 2) this could be due to the use of end-of-game strategies that may be unrelated to systemic thinking about the business by students wanting to excel in the simulation. TABLE 3 T-test Results for the High and Low Subgroups Subgroup Pair compared Mean difference t df sig Low Avg2 – Avg1 0.18 0.871 23 > 0.10 High Avg2 – Avg1 -0.40 -2.646 24 < 0.05 Page 60 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 T A B L E 4 C o rr el a ti o n A n a ly si s Page 61 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Again, confirming the correlation analysis results (Table 2), Model 2 indicates that simulation performance early on is related to systemic thinking leading up to that decision period. This positive relationship, although weak, seems to indicate that systemic thinking is positively related to firm performance. One might be tempted to think that Model 3 has nothing additional to offer this study given a non- significant F for the regression model. However, given the exploratory nature of this study, we believe this model offers insights that extend the findings of Models 1and 2. Although the overall result for Model 3 is non-significant, it is important to note that compared to Model 1, the inclusion of the ratings of the FIRST report as an explanatory variable significantly increases the adjusted R2 for the model. This indicates that this variable (systemic thinking on the first report) alone is responsible for explaining about 5% of the variance in student performance on the simulation. In addition, similar to Model 2, this variable shows a positive impact on the firm’s Net Income at a later stage and irrespective of any end-of-game strategy that may have been adopted by students. Thus, it appears that the level of systemic thinking early on in a simulation not only has some immediate effect but also has a persistent effect since it is related to performance at later stages. This finding, combined with the information in Table 1, shows higher systemic thinking in the early stages is related to higher systemic thinking and stronger firm performance immediately and over the long run. IMPLICATIONS, LIMITATIONS, AND FUTURE RESEARCH Results from these analyses have pedagogical and theoretical implications. From a pedagogical standpoint there are two things that may be taking place – respondent fatigue and end-of-game strategies. Students with stronger systemic thinking scores early on offered less feedback in the subsequent report (as measured by word count) and had a lower systemic thinking score on the second report. This could be due to writing three reports over a short period of time (10 days). Students may have felt their responses were redundant and, perhaps, were becoming bored with this writing assignment. End-of-game strategies may also account for lower systemic thinking scores at the end and weaker performance since students were preparing their last period of decisions as they wrote the final report. Grades on the simulation were based on Profits, which might have caused unusual decisions regarding spending, marketing, and/or operations. To eliminate these conditions in future studies, researchers should consider reducing the number of reports required or increasing the period of time between reports and gathering information on the final report at least two periods prior to the last period of the simulation. The theoretical implications of this research suggest that systemic thinking is related to organizational performance not only in the short-term but also in the long-term. This study was based on data gathered from students who were primarily first-year TABLE 5 Results of the Regression Analysis Variables Model1 Model2 Model 3 B SE t B SE t B SE t Intercept 2720.68 1018.302 2.672 -2134.88 1212.62 -1.761 1289.89 1227.96 1.050 Average Rating- Second Report -135.737 237.644 -.571 -456.63 282.62 -1.616 Average Rating- First Report 480.72* 267.64 1.796 624.01* 320.54 1.947 Adjusted R2 -0.017 0.053 0.051 F 0.326 3.226* 2.070 Df (n,d) 1,39 1,39 2, 38 Sig F < 0.57 <0.08 < 0.140 #Change in R2 0.090 #F for change in R2 3.790* #Dependent Variable Net Income_Period7 Net Income_Period5 Net Income_Period7 Note: #These values in Model 3 show an improvement of R2 and related test statistics over Model 1 *p < 0.10 ** P < 0.05 Page 62 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 undergraduates. Regardless, we were still able to demonstrate that systemic thinking may occur at the early stage of a business program. These results extend the simulation and systems thinking literatures through the development of a formal instrument to measure systemic thinking within the context of managing a simulated business. This exploratory investigation suffers from several limitations but ones that can be incorporated in future investigations of the topic. The rubric developed for this research measures systemic thinking involving three functional areas – marketing, management and operations. It will require adaptation for use with business simulations that involve decision making related to additional functional areas such as finance and production. Analyzing data from one course in an undergraduate program limits the generalizability of the results shown here. Incorporating the rubric in two or three courses across the business curriculum will provide data to support its use as a general tool to measure systemic thinking and will allow for time series analysis. Additional insight might also be gained in the future through the deployment of an experimental design. Finally, the authors were limited in their ability to control for other factors that might relate to systemic thinking and performance such as student age, year in school, and major. CONCLUSION Within the management education literature a debate continues around ways business programs can effectively prepare students to lead in the increasingly fast-paced, global business environment. As part of this discourse, Atwater, Kannan & Stephens argue in their 2008 article that to enhance student preparedness business curricula should incorporate pedagogical tools that encourage students’ development of systemic thinking. We contribute to this dialogue by offering two research questions grounded in both the systems thinking and simulation literatures. First, we ask if the experience of managing a business enterprise using a business simulation relates to students’ systemic thinking. Second, we ask whether systemic thinking relates to organizational performance. We explore these questions further with analyses using data from multiple semesters of a first-year undergraduate international business course. Our analyses generated meaningful results. Students with low levels of systemic thinking early on in the simulation experience display higher levels of systemic thinking later. Contrary to expectations, the systemic thinking of those with high levels early on declines later. We also found a positive and persistent relationship between systemic thinking and performance as measured by Net Income. REFERENCES Aggarwal, S.C. & Dhavale, D.G. (1975). A simulation analysis of a multiproduct multiechelon inventory distribution system. Academy of Management Journal, 18, 41-54. Allio, R.J. (2003). Russell l. Ackoff, iconoclastic management authority, advocates a “systemic” approach to innovation. Strategy and Leadership, 31 (3), 19-26. Anderson, P, Cannon, H.M., Malik, D., &Thavikulwat, P. (1998). Games as instruments of assessment: A framework for evaluation. Developments in Business Simulation and Experiential Learning, 25, 31-37. Atwater, J. B., Kannan, V.R. & Stephens, A.A. (2008). Cultivating Systemic Thinking in the Next Generation of Business Leaders. Academy of Management Learning & Education, 7(1), 9-25. Bloom, B.S., Englehart, M.D., Furst, E.D., Hill, W.H., & Krathwohl, D.R. (1959). Taxonomy of educational outcomes: The classification of educational goals. Handbook 1: Cognitive Domain, New York, NY: David McCay Company, Inc. Boone, C.W., Van Olffen, A., & Van Whitteloostuijn, A. (2005). Team locus-of-control composition, leadership structure, information acquisition, and financial performance: A business simulation study. Academy of Management Journal, 48, 889-909. Caldwell, R. (2012). Systems thinking, organizational change and agency: A practice theory critique of Senge’s Learning Organization. Journal of Change Management, 12 (12), 145-164. Chesney, A.A. & Locke, E.A. (1991).Relationships among goal difficulty, business strategies, and performance on a complex management simulation task. Academy of Management Journal, 34, 400-424. Earley, P.C., Northcraft, G.B., Lee, C., &Lituchy, T.R. (1990).Impact of process and outcome feedback on the relation of goal setting to task performance. Academy of Management Journal, 33, 87-105. Ellis, A.P.J. (2006). System breakdown: The role of mental models and transactive memory in the relationship between acute stress and team performance. Academy of Management Journal, 49, 576-589. Frey, L.R., Botan, C.H., & Kreps, G.L. (1999). Investigating communication: An introduction to research methods (2nded.). Boston, MA: Allyn& Bacon. Gladstein, D.L. & Reilly, N.P. (1985). Group decision making under threat: The tycoon game. Academy of Management Journal, 28, 613-640. Gregory, A. & Miller, S. (2011). Waving or drowning? Re- evaluating the place of systems thinking in business and management learning and curricula. Presented at the Organisational Learning, Knowledge and Capabilities Conference, Hull, UK. Page 63 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Hair, Jr., J., Black, W.C., Babin, B.J. & Anderson, R. E. (2009), Multivariate Data Analysis 7th Ed. Upper Saddle River, NJ:Prentice Hall. Halpin, A.L. (2013). A review of the simulation research in the Academy of Management Journal: Suggestions for strengthening the research conducted by ABSEL members. Developments in Business Simulation and Experiential Learning, 40, 164-179. Henning, P.B. & Chen, W-C. (2012). Systems thinking: Common ground or untapped territory? Systems Research and Behavioral Science, (29), 470-483. Hornyak, M.J., Peach, E.B., & Snyder, S. (2007). Assessment and simulations: Measuring the academic learning compacts within. Developments in Business Simulation and Experiential Learning, 34, 215-222. James, S.W. & Deighan, M. (2012). BizCafe [Computer Software] Charlottesville, Va: Interpretive Simulations Knight, D., Durham, C.C. & Locke, E.A. (2001).The relationship of team goals, incentives, and efficacy to strategic risk, tactical implementation, and performance. Academy of Management Journal, 44, 326-338. Mathieu, J.E. & Schulze, W. (2006).The influence of team knowledge and formal plans on episodic team process- performance relationships. Academy of Management Journal, 49, 605-619. Noel, T & Erskine, L. (2013). The silent story: using computer-aided text analysis to predict entrepreneurial performance. Journal of Entrepreneurship, 22(1), 1- 14. Romme, A.G.L. (2004). Perceptions of the Value of Microworld Simulation: Research Note. Simulation & Gaming, (35), 427-436. Seijts, G. H., Latham, G. P., Tasa, K., & Latham, B. W. (2004). Goal setting and goal orientation: An integration of two different yet related literatures. Academy of Management Journal, 47, 227- 239. Senge, P.M. & Fulmer, R.M. (1993).Simulations, systems thinking and anticipatory learning. The Journal of Management Development, 12 (6), 21-33. Seo, M. & Barrett, L.F. (2007). Being emotional during decision making—good or bad? An empirical investigation. Academy of Management Journal, 50, 923-940. Seo, M., Goldfarb, S.B, & Barrett, L.F. (2010). Affect and the framing effect within individuals over time: Risk taking in a dynamic investment simulation. Academy of Management Journal, 53, 411-431. Smith, K.G., Mitchell, T.R., & Summer, C.E. (1985). Top level management priorities in different stages of the organizational life cycle. Academy of Management Journal, 28, 799-820. Stemler, S.E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability, Practical Assessment, Research & Evaluation, 9(4). Retrieved October 10, 2013 from http://PAREonline.net/getvn.asp?v=9&n=4 Stevens, J. P. (2002). Applied Multivariate Statistics for the social sciences, 4th Edition. Mahwah, NJ:Lawrence Erlbaum and Associates. Thavikulwat, P. (1994). CEO II: A gaming simulation for assessment. Developments in Business Simulation and Experiential Learning, 21, 173. Thavikulwat, P. (2001). The tournament concept: Extending the idea of using simulations as instruments of assessment. Developments in Business Simulation and Experiential Learning, 28, 225-233. Waller, M.J. (1999). The timing of adaptive group responses to non-routine events.Academy of Management Journal, 42, 127-137. Wolfe, J.A. (1979). A demonstration of the business simulation-game as a curriculum assessment device.Developments in Business Simulation and Experiential Learning, 5, 191-195.