STIB: The Systemic Thinking Inventory For Business Page 250 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 ABSTRACT This paper presents a scale to measure systemic thinking among business students called the Systemic Thinking Inventory for Business (STIB). Based on literature related to cognitive styles, three dimensions of systemic thinking were identified -Locus of Attention, Inter-Relatedness and Flexibility. A 16-item instru- ment was developed to capture the three dimensions. The scale's validity and reliability were assessed through an explor- atory factor analysis and the three dimensional structure was supported. Pre- and Post-simulation analysis of scores on STIB showed that students significantly improve their locus of atten- tion as they progress through the simulation. Individual-based simulations were also found to exhibit a diminished level of flexible thinking. An attempt to link STIB scores to simulation performance indicates that the Inter-Relatedness dimension of systemic thinking was positively related to average time spent making a decision as well as average stock price. Implications and future research are also discussed. INTRODUCTION Management education is tasked with developing decision makers that can manage in a global and, often, turbulent envi- ronment. Given that managers coordinate people, ideas, and beliefs in implementing strategies how these managers think will play a role in their decision making process and, ultimately, the organization’s results. Systemic thinking, an approach to understanding reality, recognizes that systems have characteris- tics and patterns independent of their parts. As suggested by Allio (2003), “(Managers) have to understand how the interac- tions of the parts, and the parts with the whole and its environ- ment, create the properties of the whole” (p. 3). An exploratory study by Washington, Kurthakoti, Halpin, & Byrd (2014) measured the change in the level of systemic thinking of undergraduate students using a total enterprise busi- ness simulation. Using a rubric developed for the study, the researchers completed a content analysis of statements made by decision makers in an early and then later stage of the simula- tion. Results showed an increase in the systemic thinking skills of students as they progressed in the exercise. In addition, high- er levels of systemic thinking in early periods of the simulation were positively related to subsequent performance (p < 0.10). Although insightful as to the importance and impact of systemic thinking on performance, this study had some limitations. First, the rubric developed for the study was specific to the simulation used which limited the generalizability of the findings. It was impractical to suggest that a different rubric be developed for each simulation used in business schools to assess a student’s systemic thinking skills. Second, while a rubric may be useful for assessing systemic thinking in a simulation environment it is not necessarily an appropriate methodology for assessing this skill across different pedagogical approaches such as lectures, experiential exercises, and cases. The current study aims to ad- dress these issues by offering a comprehensive scale to assess systemic thinking in students running two different simulations. Once tested for validity and reliability, we believe the scale can be used across disciplines and pedagogical approaches. LITERATURE REVIEW “Systems thinking is a way of understanding reality that emphasizes the relationship among various components in a process, rather than the independent constituents of the pro- cess” (Gregory & Miller, 2011, p. 259). Recognizing that a sys- tem has characteristics and patterns independent of its parts provides a rationale for business leaders to acquire skills that draw on their ability to view an entity in a holistic way (Allio, 2003; Henning & Chen, 2012). Systemic thinking integrates analysis and synthesis and is said to lead to greater understand- ing and better decision making. According to Laszlo (2012), “Analysis answers the questions ‘what’ and ‘how’…Synthesis answers the ‘why’ and ‘what for’ questions” (p. 97). Research on individual cognitive styles provides a starting point for us to identify key dimensions of systemic thinking. How one organizes and processes information is known as one’s cognitive style. When applied to how one completes a task or responds in a decision making situation, some individu- als may focus on the individual parts of the task while others take the set of information and process it in a global context. These different methods of thinking are thought to be relevant in problem solving situations and may help predict the success rates of decision makers (Sadler-Smith & Badger, 1998). Stern- berg & Wagner (1998) offered thirteen thinking styles and cre- ated a scale, the Mental Self-Government Thinking Styles In- ventory (MSG), which attempted to categorize individuals based on how they approach problem solving situations. They identified a number of tendencies in decision makers such as rule making, goal setting, and flexibility. Choi, Koo, & Choi (2007) contributed further to our understanding of thinking styles with their Analysis-Holism Scale, which distinguished between individuals who view the world in a holistic way and those with a focus on the world as a set of independent compo- nents. One of the four domains examined in this work, locus of attention, is a dimension of the construct proposed in the current study. Business schools and programs seek to graduate students with the knowledge and skills to manage in a global economy. Facing increasingly complex environmental factors, solutions to STIB: The Systemic Thinking Inventory For Business Raghu Kurthakoti Arcadia University kurthakotir@arcadia.edu Annette L. Halpin Arcadia University halpina@arcadia.edu mailto:kurthakotir@arcadia.edu mailto:halpina@arcadia.edu Page 251 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 problems today are neither apparent nor satisfactory (Caldwell, 2012). Given the current business climate, decision makers must arrive at solutions in shorter periods of time and with less than complete information (Noel & Erskine, 2013). Leaders who are systemic thinkers may be able to adjust to time con- straints and make sense of situations where the relationships or patterns may not have occurred previously. This requires high- er-order thinking such as analysis, synthesis, and evaluation as defined by Bloom, Englehart, Furst, Hill, & Krathwohl (1959) or, more recently, the ability to analyze, evaluate, and create (Krathwohl, 2002). From the perspective of management educa- tion, identifying teaching methods that strengthen the systemic thinking skills of students is a reasonable goal. A range of tools including computer-based simulations, experiential exercises, and case analysis are recognized as effective methods of teach- ing business concepts. The focus of the proposed study uses a simulation to test several hypotheses. Business simulations allow students to work alone or in groups to test their decision making skills. Moschella & Moti- walla (1997) argue that strategic thinking skills are improved using a simulation by requiring participants to engage in goal setting, strategy formulation, and planning. Lovelace, Eggers, and Dyck (2015) found critical thinking skills improved in those completing different web-based simulations across differ- ent classes - business strategy, human resources, and organiza- tional behavior. Understanding the range of business course requirements that exist, we would argue that the more complex the concepts being studied the more complex the simulation should be resulting in a more challenging exercise. A total en- terprise simulation fits this category since students are expected to consider how marketing, operations, finance, and manage- ment factors influence one another and affects the overall per- formance of an organization. Strategies used by participants in a simulation that are consistent with the environment they face are thought to indicate that learning is taking place (Wellington, Faria, & Whiteley, 1998). Given the above discussion, we pro- pose: Hypothesis 1. Participants improve their systemic thinking skills as they progress through a total enter- prise business simulation. The current research proposes the use of a newly developed survey instrument, the Systemic Thinking Inventory for Busi- ness (STIB), to assess systemic thinking. Items on the scale capture three dimensions the authors propose are major compo- nents of this construct. Included are: Locus of Attention: Problem solving skills require an un- derstanding of a situation and the ability to sort through vast amounts of data to select the most relevant and meaningful in- formation (Moschella & Motiwalla, 1997). Locus of Attention pertains to what a decision maker focuses on while making de- cisions. Some individuals pay more attention to the parts of a task when developing a response while others take the infor- mation and process it all within a holistic context (Sadler-Smith & Badger, 1998). Systemic thinking requires a more holistic approach - an ability to view the whole task with qualities and characteristics distinct from its parts. In two studies using a business simulation as a tool there were differences in holistic cognitive perceptions across industries (Wellington, Faria, & Whiteley, 1998) and a persistent positive relationship between systemic thinking and performance over several periods (Washington, et.al., 2014). This dimension is captured in the STIB inventory through various statements that assess what a student pays attention to while performing tasks. (E.g. When working on a task, I like when I need to pay attention to details; I like tasks where I can focus on general ideas, rather than spe- cifics.) We predict: Hypothesis 2a. Participants improve their level of Locus of Attention running as they progress through a total enterprise business simulation. Inter-Relatedness: Successful decision making by managers requires an appreciation of the interconnectedness of the parts of a task or issue. This is a challenge for business leaders who are charged with identifying which elements of a situation are the most pressing and in need of attention (Moscella & Moti- walla, 1997). Organizations facing complex decision making situations can employ techniques such as chunking and speciali- zation (Cannon, 1995) or mapping to visually represent the vari- ables in a situation (Wallis & Wright, 2015). Modeling helps individuals conceptualize the system they are trying to under- stand. Identified as conceptual knowledge this is what enables a learner to understand how a system functions, how the parts interact with one another, and how the properties of the parts differ from the properties of the whole (Krathwohl, 2002; Rich- ardson, 2008). This dimension of systemic thinking is captured in the STIB instrument through various statements that assess how students perceive the relationship between the whole and the parts. (E.g. When working on a task, I like to see how what I do fits into the overall picture; Everything associated with a task is somehow related to each other.) Based on the above, we predict: Hypothesis 2b: Participants improve their level of Inter- Relatedness as they progress though a total enterprise business simulation. Flexibility: In addition to thinking in a holistic way, seeing the relationship between the components of a task, a systemic thinker should also be creative and responsive to changing con- ditions while solving problems. This means having the ability and the will to be flexible in one’s approach to problem solving. As organizations strive to remain relevant, successful leaders often frame situations they face in a way that allows for flexibil- ity (Zaccaro, Gilbert, Thor, & Mumford, 1991). It is not uncom- mon for experienced decision makers to develop a range of re- sponses to a particular problem. These experiences or ‘scripts’ are acquired over time and stored to memory – to be recalled at a later date. This increases the flexibility of a manager’s deci- sion making style and can lead to a greater ability to be creative in solving problems (Gioia & Poole, 1984). It is not the routine decision making situation that requires flexibility so much as the atypical one which calls on a leader to search through his/ her array of past experiences to arrive at a range of possible solutions. Thus, we believe that a scale measuring systemic thinking should include a dimension we call ‘flexibility’. This component is captured in the STIB inventory through state- ments that assess the extent of students’ rigidity in completing tasks. (E.g. When considering ways to complete a task, I tend to approach it in a traditional way; When working on a task, I like to do things in new ways not used by others in the past). We predict: Hypothesis 2c: Participants improve their level of Flexibility as they progress through a total enterprise business simulation. Understanding the thinking process of decision makers may Page 252 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 help explain the quality of the solutions they offer and, ulti- mately, whether there is a relationship between thinking style and performance (Gioia & Poole, 1984). As schools of business are charged with developing managers with a broad perspective and an ability to make decisions in a complex world the use of experiential learning methods, multidisciplinary courses, and group exercises may aid in the development of successful lead- ers (Moschella & Motiwalla, 1997). Knowing that quantitative results in a business setting are used to assess the decision mak- ing skills of business leaders a final hypothesis is proposed which relates the level of systemic thinking to performance on a total enterprise simulation. Hypothesis 3: Systemic Thinking is positively related to sim- ulation performance. METHODOLOGY In this study we assess the systemic thinking skills of un- dergraduates enrolled in two business courses that traditionally run a total enterprise simulation. One course is an introductory business class taken in the first or second year by business and non-business majors. The second is a senior capstone class (business policy) taken only by business majors in the final year. All students are enrolled in a small comprehensive univer- sity located in northeastern US. This research had three main objectives – (a) assessing the structural validity of a systemic thinking inventory for business; (b) assessing whether students in a business class improve their scores on the inventory after participating in a simulation, and (c) testing the relationship between the score on the systemic thinking inventory and performance on the simulation. In the first phase of the study, we identified 25 statements we believe capture three essential dimensions of systemic think- ing, based on work done in prior studies as well as face validity. Each statement was rated by students using a 5–point Likert Scale (Strongly Disagree to Strongly Agree). Administration of the inventory was done during the Fall 2014 and Spring 2015 semesters. The instrument was administered twice for each par- ticipant – before they started the simulation and after its com- pletion. The simulation used in the introductory business course was BizCafe: The Business Essentials Simulation (James & Deighan, 2015). In the business policy class students worked with Micromatic: A Strategic Management Simulation (Scott, Kaliski, & Anderson, 2013). BizCafe was administered as an individual simulation where students independently managed a business while Micromatic was a team-based exercise with groups of 3-4 students managing a firm. The rationale for col- lecting data from both types of users (individuals and groups) was to look closely at the three components of the construct in decision makers facing situations with changing conditions. Individual vs. group decision making were thought to have the potential to influence results. Given the small size of the business program at the institu- tion, the survey was administered in multiple sections over two semesters, Fall 2014 and Spring 2015. In the Fall term, the sur- vey was administered to 108 students in the introductory busi- ness course. In the Spring, it was administered to 40 students in the introductory business course and 45 students in the business policy class. Participation was voluntary and students received no compensation (financial or extra credit) for their participa- tion. The survey for the introductory business classes was ad- ministered face-to-face, while the survey for the business policy sections was administered via SurveyMonkey through individu- alized invites to each student. Each student participating in the study was provided with a unique code that had to be entered while taking the survey. This approach helped us to match the pre-and post-surveys for participants and, for the policy stu- dents, match the survey results with simulation performance EXHIBIT 2 PAIRED T-TEST RESULTS: PRE- AND POST-SIMULATION SCORES ON STIB (N = 126) Test Pair (Post - Pre) Pre-Simulation Average Post-Simulation Average Difference in Mean t statistic p value Locus of Attention 2.952 3.124 0.172 3.290 < 0.01 Inter-Relatedness 4.034 4.068 0.034 0.846 > 0.10 Flexibility 2.298 2.182 -0.116 -2.258 < 0.05 Overall (Pooled) – H1 3.095 3.124 0.029 1.076 > 0.10 Overall (Intro Class; n = 104) 3.109 3.132 0.023 .754 > 0.10 Overall (Policy Class; n = 22) 3.028 3.085 0.057 1.073 > 0.10 EXHIBIT 1 TEST OF SCALE RELIABILITY Factor # of items Cronbach Alpha Locus of Attention (pre simulation) 7 0.791 Locus of Attention (post simulation) 7 0.819 Inter-Relatedness (pre simulation) 6 0.661 Inter-Relatedness (post simulation) 6 0.720 Flexibility (pre simulation) 3 0.666 Flexibility (post simulation) 3 0.577 Page 253 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 using email information. Of the 193 total participants, matching data (pre- and post- surveys) could be obtained for 126 partici- pants (104 for the introductory business class and 22 for busi- ness policy) so this became our sample frame for analysis. DATA ANALYSIS AND RESULTS Determination of Factor Structure Reliability analysis using Cronbach’s alpha was initially performed on the different sets of data independently (Fall 2014; Spring 2015; introductory business class; and business policy class). Reliability tests indicated that all sets of data dis- played similar reliability levels for the three theorized factors of the STIB (Locus of Attention, Inter-Relatedness, and Flexibil- ity). As a result, the data from both semesters and across both classes was pooled to boost sample size for testing the scale structure and validity. Given the size of our sample and the number of items on the scale, exploratory factor analysis (EFA) was deemed appro- priate for testing the factor structure. Although a confirmatory factor analysis is more robust for testing factor structures, given our limited sample size, we could not perform a confirmatory analysis and expect reliable results. Hair et al. (2009) suggest a sample size of at least 5 observations for each variable to be analyzed. Driven by this suggestion, an EFA was performed on the pooled data. Factors were allowed for free estimation based on eigen values as well as were restricted to three factors as theorized. Oblimin rotation was used initially to rule out the possibility of inter-factor correlation. Results indicated that inter -factor correlation was not significant and so a revised analysis using Varimax rotation was performed. All references to factor structure and its use in analysis are based on the results of EFA with Varimax rotation. Allowing for free factor estimation re- sulted in the extraction of a total of 6 factors that explained 58- 60% of variability (for the pre- and post- sets). Forcing the fac- tor structure to only 3 factors resulted in a 42-44% variance extraction. However, when we looked at the composition of the factor structures, we observed that the three additional factor structures created primarily consisted of those scale items that were heavily cross loaded in the 3 factor solution. Going back to the survey instrument, it was found that these cross loaded items contained words that might be subject to a different inter- pretation. So the cross loaded items were removed and the EFA was performed again. This resulted in generating a 3 factor so- lution with most of the items for each factor loading on to its hypothesized factor. The total variance extracted in this solution was between 48-50%. The final solution resulted in three fac- tors – Locus of Attention (7 items), Inter-Relatedness (6 items) and Flexibility (3 items). The factor loadings for individual items on the final scale for both pre- and post-simulation condi- tions are provided in the Appendix. Reliability analysis of the three factors indicates they all had very high reliability, especially for an exploratory study. Factor analysis and reliability tests were conducted separately on both pre-simulation and post-simulation surveys and it was consistent. Results of the Reliability analysis are provided in Exhibit 1. Testing Hypotheses 1 & 2 A paired sample t-test was conducted to test our first two hypotheses. Since the study was designed as a within-subjects study, a paired sample t-test was deemed appropriate (Stevens, 2002). Scores for items corresponding to each factor of the STIB were averaged to create a summated scaled for each fac- tor. Paired t-tests were conducted with the results shown in Ex- hibit 2. Hypothesis 1 stated that students would improve in their systemic thinking when working on a simulation. That is, the overall post-simulation STIB score would be higher than the overall pre-simulation score. Examining differences for the total sample as well as two sub-groups (students in an introductory business class and students in a business policy class) the results shown in Exhibit 2 do not support this hypothesis (rows 4 through 6). Levels are moving in the direction hypothesized, positive, but the differences are not significant. H1 is, therefore, not supported. The factors Locus of Attention and Flexibility show signifi- cant differences between the pre- and post-simulation levels. Locus of Attention is in the direction hypothesized so we con- clude that H2a was supported. The dimension Flexibility pro- EXHIBIT 3 PAIRED T-TEST RESULTS FOR FLEXIBILITY: PRE- AND POST-SIMULATION SCORES ON STIB Course Pre Simulation Average Post Simulation Average Difference in Mean (post-pre) t statistic p value Introductory Course (n=104) 2.298 2.141 -0.157 -2.677 < 0.01 Business Policy (n=22) 2.303 2.378 0.075 0.816 > 0.10 EXHIBIT 4 PAIRED T-TEST RESULTS FOR OVERALL STIB SCORE EXCLUDING FLEXIBILITY Test Pair (Post - Pre) Pre Simulation Average Post Simulation Average Difference in Mean t statistic p value Overall (Pooled; n = 126) 3.493 3.596 0.103 2.915 <0.01 Overall (Intro Class; n = 104) 3.515 3.628 0.113 2.802 <0.01 Overall (Policy Class; n = 22) 3.390 3.438 0.047 0.803 >0.10 Page 254 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 vides a most surprising result. There is a significant difference between the pre- and post-periods but in a direction contradicto- ry to our hypothesis. In other words, flexibility scores were sig- nificantly lower at the end of the simulation than at the begin- ning. A difference occurred but not in the direction hyposta- sized so H2c is not supported. The Inter-Relatedness score re- mained stable over the period so the results are not significant and H2b is not supported. This may have been due to a high level at the pre-simulation stage (4.034 on a 5-point scale). Per- haps there is not much room for improvement when respond- ents start out at such a high level on a factor. The unexpected results on the Flexibility dimension led us to examine this construct in greater depth by considering other conditions facing students running a simulation. As stated earli- er, Flexibility in one’s thinking is expected to foster creativity in problem solving plus improve overall decision making when facing stressful and uncertain conditions. Reflecting on the en- vironment facing the participants in this study, those in the in- troductory business class operated their firm alone, not in groups. On the other hand, business policy students worked in groups. In addition, the simulation was only a small part of the overall grade for the introductory class but a major component in the business policy class. Given the group dynamics effect and importance of simulation performance to those in the policy class it is possible that the data on Flexibility might need to be parsed further to get a clearer picture of what is happening. Stu- dents in the introductory business class worked alone so there was no opportunity for them to discuss their strategy/plans with others, compromise on any issue, or face a challenge about their point of view. This may have resulted in these individuals be- coming less flexible and less creative in their decision making. To test for this possibility, we split the flexibility data by class (introductory course and policy class) and performed some t- tests. Details are provided in Exhibit 3 below. As we can see from Exhibit 3, the score on Flexibility de- creased as the simulation progressed for those in the introducto- ry course and increased in the business policy course. Working alone with sole responsibility for decision making (introductory course) might reduce a person’s ability to see alternatives in a decision making situation. For those working in groups (Policy class) there was improvement on this dimension but given the small sample size of this population, the effect was not pro- found. We conclude that the aggregated data shown earlier in Exhibit 2 are likely skewed by the larger number of individuals in the introductory class. This negative directionality of the Flexibility score is likely to have interacted with the positive impact of Locus of Attention, thereby rendering the overall score on the STIB non-significant. As we can see from Exhibit 3, Flexibility had a significant adverse impact on students work- ing alone on a total enterprise simulation. Recognizing the pos- sible impact of this negative relationship, the overall STIB score was re-estimated excluding the Flexibility score. T-tests were conducted on the re-estimated Pre- and Post-Simulation scores with the results shown below in Exhibit 4. As shown in Exhibit 4 the revised Overall score on the STIB (Locus of Attention and Inter-Relatedness) does show a significant increase in the post-simulation scores for students in the introductory class. Thus, we conclude that H1 is partially supported. Testing Hypothesis 3 Any study of pedagogical effectiveness is useful only if it can demonstrate the benefits of the technique in improving stu- dent performance. Although Exhibits 2 and 3 indicate that changes occur in the systemic thinking of those involved with a simulation, it would further substantiate the value of the STIB if we demonstrate a relationship between a student’s STIB score and performance on the simulation. Data collected from the introductory business class was anonymized and so it was not possible to match student responses on the STIB to their simula- tion performance. On the other hand students in the business policy course responded to the STIB as individuals, but partici- pated in a simulation as a group. Therefore, relating individual responses to group performance would not be an appropriate analysis. However, students in the policy course were required to run a version of the simulation on their own as part of their final exam. Therefore, simulation results on this individual as- signment could be used to test whether STIB scores are related to performance. Various metrics were available from the simulation to be proxies for student performance. These included Return on Sales, Profits, Inventory Turnover, Stock Price, etc. Of those reviewed, stock price was thought to best represent a firm’s overall performance as it was determined by the market and was, therefore, selected. We also had access to the time students spent processing decisions during the exercise so that data were included. We expected that stronger systemic thinking skills were related to time spent making decisions. That is, the higher the systemic thinking level the longer the time spent making decisions. This added the variable ‘Average Time per Decision’ to the correlation analysis. To examine the relationship between the three dimensions of systemic thinking and the two dependent variables (stock price and the time spent), a correlation analysis was performed. These results are provided in Exhibit 5. Results from Exhibit 5 indicate that Inter-Relatedness cor- relates positively with both Average Time per Decision and Performance (Average Stock Price). For business policy stu- EXHIBIT 5 CORRELATION BETWEEN THE THREE DIMENSIONS OF SYSTEMIC THINKING AND SIMULATION PERFORMANCE (TIME PER DECISION, AND STOCK PRICE) FOR THE BUSINESS POLICY CLASS (N = 22) * p < 0.10 Variable Average Time per Decision (Minutes) Average Stock Price ($) Locus of Attention -0.315 -0.333 Inter-Relatedness 0.376* 0.379* Flexibility 0.109 0.268 Page 255 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 dents understanding connections between the components of a total enterprise simulation results in more time spent making decisions. Understanding these connections is also significant and positively related to Performance (Stock Price). Greater skills on the dimension of Inter-Relatedness result in longer periods between decisions but with better results, something rewarded in business. Interestingly, Locus of Attention and In- ter-Relatedness exhibit a strong negative correlation for these students running a total enterprise simulation. Those paying more attention to details (Locus of Attention) have a greater interest in the parts of a task than the exercise as a whole. This may work in opposition to thinking that is focused on tying all of the parts of a task together. Additional Analysis In an attempt to further strengthen the correlation results, two sets of regressions were conducted on the student perfor- mance data. Average Stock Price at the end of the test was used as the dependent variable in the first model and the Average Time Spent Per Decision (in minutes) was used as the depend- ent variable in the second. The Post-Simulation Scores on each the three dimensions of the STIB (Locus of Attention, Inter- Relatedness, and Flexibility) were the independent variables in both models. However, given the small sample size and strong correlation between two of the independent variables (Locus of Attention and Inter-Relatedness) we did not expect the regres- sion models to yield any significant insights. This was the case as neither of the models turned out to be significant. Additional- ly, the model exhibited high collinearity due to the high correla- tion between the Locus of Attention and Inter-Relatedness di- mensions. DISCUSSION Based on the analysis several things about systemic think- ing come to light. First, consistent with Gregory & Miller’s (2011) definition of Systemic Thinking, our analysis provides support for the multidimensional nature of the systemic thinking construct. EFA and scale reliability tests indicate systemic thinking to be a three factor construct – Locus of Attention, Inter-Relatedness and Flexibility. Second, students demonstrated improved systemic thinking as they progressed through the simulation. This supports the argument of Moschella & Motiwalla (1997) that a simulation encourages systemic thinking by requiring strategy thinking and planning. Changes in levels of the individual components of the construct include some interesting findings. As hypothesized, Locus of Attention (focusing on the whole task as opposed to the parts) shows significant improvement after running a simu- lation. These results are consistent with Sadler-Smith & Badger (1998) proposition that systemic thinkers are more likely to process info in a holistic context. All simulation participants improve on this dimension whether working alone (introductory business class) or in groups (business policy class). Third, with regard to Flexibility, there is a significant dif- ference between the pre- and post-periods but not in the direc- tion hypothesized. The significant negative relationship reveals that participants appear to be more rigid or less likely to consid- er alternate methods of completing tasks after participating in a simulation. Additional analyses to explore this difference show that there are differences between the two groups of students. Those in the introductory class work alone on the simulation which means there is no collaboration or discussion with others to arrive at a strategy for their firms. Acting alone is likely to limit one’s perspective and interest in developing alternative solutions in a given situation. Additionally, these students have had limited experience making business decisions. As argued by Gioia & Poole (1984) this results in a shallower repertoire of options to consider during task completion. Students in the busi- ness policy class work in groups. Their Flexibility scores in- creased over time, in the direction hypothesized, but the change is not significant which we think is due to the small sample size. Working in groups appears to foster flexibility in one’s think- ing. The simulation for the policy class is more complex and presents more dynamic situations under which decision making occurs. As Gioia & Poole (1984) state, the more complex and atypical a situation the greater the need for flexibility. Those in the policy class have more familiarity with business practices and, therefore, a richer repertoire of experiences to tap into when faced with complex decision making situations. This is, perhaps, why we see a reduction in flexibility among those in the introductory class and an increase in the policy class. Fourth, a revised analysis of the Overall Score without Flexibility shows a significant increase in the Post-Simulation scores for both the entire sample and those in the introductory business class. This suggests that flexibility may have a stronger role in the systemic thinking construct by moderating the devel- opment of systemic thinking for those running a business simu- lation. Since such a phenomenon was not expected and given the limited sample size of the study, this proposition was not explored further. Future studies may want to focus on the role of flexibility in systemic thinking. Fifth, this analysis suggests that systemic thinking may have an impact on performance. Greater inter-relatedness be- tween parts of a task was positively associated with both time required to make decisions and stock price in the simulation for those in the policy class. This indicates that as students start seeing the relationship between the various parts of a task, they take more time to analyze available information and this delays decision making. This is consistent with Hill & Cox (1994) who found a positive relationship between complexity and duration of the simulation. Finally, the development of one instrument that can meas- ure systemic thinking in business students regardless of teach- ing method used (simulations, cases, experiential exercises) will lead to more research in this area and a better understanding of the benefits of each. This methodology also addresses the chal- lenge faced in earlier studies where the use of a simulation- based rubric reduced the generalizability of the findings (Washington, Kurthakoti, Halpin, & Byrd, 2014; Lovelace, Eg- gers, & Dyck, 2015). LIMITATIONS AND FUTURE DIRECTIONS Given limited samples we were only able to perform an Exploratory Factor Analysis. Although this is a good approach to identify important items that make up a construct, a more rigorous confirmatory factor analysis is preferred to identify and determine the factor structure as well as its validity and reliabil- ity (Hair, 2009). Our limited data precluded us from performing these rigorous analyses. Gathering responses from larger sam- ples would help create a more robust and valid scale to measure systemic thinking. As stated in our discussion, the exact nature of Flexibility and its impact on systemic thinking could not be explored due to the limited sample size. Large sample analysis is likely to make the data more amenable for additional analysis that could be used to generate deeper insights into the nature of this di- mension and its impact on the other dimensions of systemic Page 256 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 thinking. Another limitation is linked to the data collection pro- cedures as we were unable to relate individual performance of those in the introductory class with their score on the STIB. Anonymized data collection fosters candid and frank responses yet makes it hard to do further analysis due to lack of identifia- ble information. The limited data we had for the policy class (n=22) suggests a positive relation between systemic thinking and simulation performance, but the sample size was clearly not sufficient to demonstrate a strong relationship. Our overarching goal through this research is to develop a scale that not only measures systemic thinking but also can aid in developing appropriate pedagogical techniques that foster the systemic thinking of business students. To that effect, testing the scale across various pedagogical approaches used in busi- ness schools (lecture only, lecture + discussion, case analysis, simulations, etc.) would be a good extension of this study and provide validity to the STIB. This will also help us compare and assess various pedagogical approaches regarding their impact on systemic thinking. CONCLUSION In this study we proposed that when students participate in a total enterprise simulation they develop systemic thinking skills. We developed an instrument called the Systemic Think- ing Inventory for Business (STIB) and, using factor analysis, identified three components of the construct: Locus of Atten- tion; Inter-Relatedness; and Flexibility. We argued that the overall score as well as the levels of each component would increase with the use of a total enterprise simulation and that systemic thinking and simulation performance are related. What we found was that the thinking of participants using a total en- terprise simulation changes over time in ways that have impli- cations for schools of business. These findings provide evidence that systemic thinking can be assessed and developed with the use of a business simulation and that one of the underlying components of the construct, Flexibility, has some interesting properties. Allio, R.J. (2003). Russell I. Ackoff, iconoclastic management authority, advocates a “systemic” approach to innovation. Strategy and Leadership, 31 (3), 19-26. Bloom, B.S., Englehart, M.D., Furst, E.D., Hill, W.H., & Krath- wohl, D.R. (1959). Taxonomy of educational outcomes: The classification of educational goals. Handbook 1: Cog- nitive Domain, New York, NY: David McCay Company, Inc. Caldwell, R. (2012). Systems thinking, organizational change and agency: A practice theory critique of Senge’s Learning Organization. Journal of Change Management, 12 (12), 145 -164. Cannon, H. (1995). Dealing with the complexity paradox in business simulation games. Developments in Business Sim- ulation and Experiential Learning, 22, 96-102. Choi, I., Koo, M., & Choi, J.A. (2007). Individual differences in analytic versus holistic thinking. Personality and Social Psychology Bulletin, 33, 691-705. Hall, J.J.S.B & Cox, B.M. (1994). Complexity: Is it really that simple? Developments in Business Simulation and Experi- ential Learning, 21, 30-4. Gioia, D.A. & Poole, P.P. (1984). Scripts in organizational be- havior. The Academy of Management Review, 9, 449-459. Gregory, A. & Miller, S. (2011). Waving or drowning? Re- evaluating the place of systems thinking in business and management learning and curricula. Presented at the Or- ganisational Learning, Knowledge and Capabilities Confer- ence, Hull, UK. Hair, J.F. Jr., Black, W.C., Babin, B.J., & Anderson, R.E. (2009). Multivariate Data Analysis (7th Edition), Prentice Hall, Upper Saddle River, NJ. Henning, P.B. & Chen, W-C. (2012). Systems thinking: Com- mon ground or untapped territory? Systems Research and Behavioral Science, (29), 470-483. James, S.W. & Deighan, M. (2015). BizCafe: The business es- sentials simulation. Interpretive Software, Inc., Char- lottesville, VA USA. Krathwohl, D.R. (2002). A revision of Bloom’s Taxonomy: An overview. Theory Into Practice, 41(4), 212-218. Laszlo, K.C. (2012). From systems thinking to systems being: The embodiment of evolutionary leadership. Journal of Organizational Transformation & Social Change, 9 (2), 95 -108. Lovelace, K.J., Eggers, F., & Dyck, L.R. (2015). I do and I un- derstand: Assessing the utility of web-based management simulations. Academy of Management Learning & Educa- tion.2013.0203; published ahead of print October 23, 2015, doi:10.5465/amle.2013.0203. Moscella, P. & Motiwalla, L. (1997). For today’s dynamic and changing world, business simulations need to be expanded to encompass much greater complexity (A demonstration). Developments in Business Simulation and Experiential Learning, 24, 311-319. Noel, T & Erskine, L. (2013). The silent story: using computer- aided text analysis to predict entrepreneurial performance. Journal of Entrepreneurship, 22(1), 1-14. Richardson, K.A. (2008). Managing complex organizations: Complexity thinking and the science and art of manage- ment. E:CO, 10(2), 13-26. Sadler-Smith, E. & Badger, B. (1998). Cognitive style, learning and innovation. Technology Analysis & Strategic Manage- ment, 10 (2), 247-265. Scott, T.W., Kaliski, J.A. & Anderson, P.H. (2013). Micro- matic: A strategic management simulation. Oak Tree Simu- lations, LLC. Sternberg, R.J. & Wagner, R.K. (1991). Mental self government thinking styles inventory manual. Unpublished manual. Stevens, J.P. (2002). Applied Multivariate Statistics for the So- cial Sciences (4th Edition). Lawrence Earlbaum and Associ- ates, Mahway, NJ. Tait, A., & Richardson, K.A. (2011). Guest editorial: From the- ory to practice. E:CO, 13(1-2), v-vi. Wallis, S. (2009). The complexity of complexity theory: An innovative analysis. E:CO, 11(4), 26-38. Wallis, S.E. & Wright, B. (2015). Strategic knowledge map- ping: The co-creation of useful knowledge. Developments in Business Simulation and Experiential Learning, 42, 1- 18. Washington, M.L., Kurthakoti, R., Halpin, A.L., & Byrd. S. (2014). Assessing systemic thinking in undergraduates: An exploratory study using a total enterprise business simula- tion. Developments in Business Simulation and Experien- tial Learning, 41, 53-63. REFERENCES Page 257 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 Wellington, W., Faria, A.J., & Whiteley, T.R. (1998). Holistic cognitive strategy in a computer-based marketing simula- tion game: An investigation of attitudes towards the deci- sion-making process. Developments in Business Simula- tion and Experiential Learning, 25, 246-252. Wong, E.M., Ormiston, M.E., & Tetlock, P.E. (2011). The ef- fects of top management team integrative complexity and decentralized decision making on corporate social perfor- mance. Academy of Management Journal, 54(6), 1207- 1228. Zaccaro, S.J., Gilbert, J.A., Thor, K.K., & Mumford, M.D. (1991). Leadership and social intelligence: Linking social perspectiveness and behavioral flexibility to leader effec- tiveness. Leadership Quarterly, 2(4), 317-342. APPENDIX A FACTOR LOADINGS FOR STIB IN PRE- AND POST-SIMULATION CONDITIONS Factor Loading Structure – Pre Simulation Item # Component 1 (LA) 2 (IR) 3 (Flex) Item 3 .644 Item 5 .568 Item 8 .692 Item 10 .687 Item 12 .663 Item 15 .751 Item 21 .627 Item 4 .629 Item 9 .554 Item 13 .514 Item 23 .652 Item 24 .624 Item 25 .641 Item 6 .841 Item 16 .649 Item 18 .813 Factor Loading Structure – Post Simulation Item # Component 1 (LA) 2 (IR) 3 (Flex) Item 3 .796 Item 5 .618 Item 8 .789 Item 10 .709 Item 12 .658 Item 15 .695 Item 21 .537 Item 4 .637 Item 9 .636 Item 13 .501 Item 23 .820 Item 24 .594 Item 25 .685 Item 6 .795 Item 16 .663 Item 18 .683 Contact the authors for additional information about the scale description.