REFLECTING ON REFLECTION: AN EXAMINATION OF REFLECTIVE LEARNING AND ASSESSING OUTCOMES Developments in Business Simulation and Experiential Exercises, Volume 28, 2001 REFLECTING ON REFLECTION: AN EXAMINATION OF REFLECTIVE LEARNING AND ASSESSING OUTCOMES MICHAEL L. BRUCE, GENEVA COLLEGE mlbruce@geneva.edu ABSTRACT Reflection has been identified as a critical factor in the learning process. This experiment examines some of the linkages between reflection and outcomes of the simulation experience. The experiment attempted to measure reflection-for-action, reflection-on-action, and reflection- in-action during a undergraduate business course using a marketing simulation. Teams in this experiment that outperformed their peers appear to demonstrate improved decision making as evidenced by decreased time spent in the various dimensions of reflection. The teams, which did not perform as well, were able to improve their outcome measures of net income but at a slower rate. For these teams, a pattern of decreased reflection time was not evident. Does the use of computer simulations in business education actually contribute to learning? The answer to this question is elusive, evading our attempts of capturing it. We believe that the experience and learning environment of a simulation component in our courses affords an opportunity for learning that cannot be achieved any other way. Arguably one of the challenges facing researchers is the measurement mechanism for assessing that learning. Calls for measuring the quality of learning tend to focus on the ability to demonstrate specific measurable skills. Simultaneously, those seeking to measure process look at the improvement dimension of learning, i.e., can a student demonstrate a more rapid attainment of specific knowledge or skills? Business managers, of course, are interested in both of these behaviors. Students should be able to demonstrate competency in business knowledge/skills and do so very quickly. The pace of change in the current business world demands quick and accurate learning on the part of managers and students alike. This paper will attempt to assess both facets of learning. LITERATURE SEARCH Learning is a difficult concept to succinctly define. Kimble (1961) provides a widely accepted definition of learning as “a relatively permanent change in behavioral potential as a result of [reinforced] practice.” Others describe is as both a process and a product (Walker, 1996). The process aspect views learning as the acquisition of knowledge or information. The product aspect, on the other hand, views learning as the knowledge or information acquired and stored. And still others refer to learning as the acquiring of knowledge and information as a result of an organism’ interacting with its environment (Kolb, 1984). In that interaction process is the opportunity for reflecting upon some concrete experience. Reflecting is often described as contemplating the results of a given experience within the context of the impact on the entity, enterprise, or organism. When this reflective process leads to a change of behavior, it is called reflective learning (Boud, Keogh, and Walker, 1985). Work within this sphere of experiential learning has suggested that reflection can focus on a number of different dimensions (Daudelin, 1996; Rosenorn and Kofoed, 1998; Schon, 1987; Swenson, 1997). Three of these dimensions have been labeled “reflection-in-action,” “reflection-on-action,” and “reflection-for-action.” Reflection-in-action (RIA) is described as the consideration of the action being undertaken. The learner considers the potential interaction between the action and the environment. Reflection-on-action (ROA) is the contemplation of the experience itself. In this reflective process, the learner reflects on the outcomes an action has produced in the environment. And reflection-for-action (RFA) directs attention with a future orientation. In other words, it considers a more directive global perspective than simply the immediate action and associated experience. The learner in this realm seeks a mental correlation between the desired goal and the intermediate action steps deemed necessary to achieve that goal. HYPOTHESES Hypothesis 1: Reflection-in-action time will decrease over the length of the simulation experience. Reflection-in-action (RIA) involves incorporating the data and information gathered from experience into an immediate short-range action. This action may be considered “making the next quarter decision.” Reacting to the environment should, after an undetermined amount of time, involve “fine-tuning” rather than “coarse- adjustment.” In addition, the longitudinal nature of the monitoring process of the other forms of reflective learning should enable team members the ability to “see” a more global perspective and reduce decision times. If learning is indeed occurring, it is believed that the amount of time necessary to make decisions will decrease. 8 Developments in Business Simulation and Experiential Exercises, Volume 28, 2001 Table 1 Sample Retained Earnings Statistics Characteristic Value Mean $12,413,376 Median $11,723,947 Standard Deviation $2,727,507 Minimum $9,132,068 Maximum $17,346,216 Hypothesis 2: Reflection-on-action will decrease over the simulation experience. Reflection-on-action (ROA) involves the data gathering activity and necessary analysis of that data. This reflection is very action oriented in that the student must use various analysis tools to convert the data gathered into information that will be useful to the success of the enterprise. It is believed that the process of gathering the data and manipulating it will become more routine and mechanical. In this type of learning, it is believed that the amount of time necessary to accomplish this stage of the reflective process will decrease. Hypothesis 3: Reflection-for-action will remain constant during the simulation experience. Reflection-for-action (RFA) is associated with considering whether the past actions have been successful in achieving the desired outcome or goal. This reflection is more global and long-term in perspective rather than directed at a specific task or activity. Accordingly, it is believed that the amount of time spent in reflection-for- action will remain essentially unchanged. Hypothesis 4: The reflective learning of high performing teams in its various dimensions will differ from the learning of other teams. One of the critical questions relating to reflective behavior is whether changes are simply spurious and random occurrences. For this experiment, teams with retained earnings above the median will be described as “high performing (HP) teams while teams with retained earnings below the median will be labeled “low performing (LP) teams. In an attempt to determine if occurrences are a result of intentional action, the performance of high performing teams will be compared with the results of the remaining teams. This comparison should minimize the impact of guessing or “lucky” decisions throughout the simulation experience. That is, the probability of long-term successful guessing is decidedly low. METHOD The design of this experiment focused on capturing the reflective learning behaviors via a number of operational constructs. Measurement involved assessing the quality and quantity of time spent reflecting on the different aspects of the simulation experience. These reflective behaviors were correlated to the overall performance of the team. The reflection-on-action (ROA) was equated as the time spent analyzing the results of the team’s previous decisions such as mathematical computation, interpretation, and evaluation of the results. The reflection-in-action (RIA) period was identified as the time spent considering the various decisions to be made by the team. And reflection-for-action (RFA) directs attention with a future orientation. This reflection was operationalized as being the time spent examining strategy, i.e., are we going the right direction or are we doing the right things? One measurement instrument for this study was a modified journal-like format. Students were assigned the task of reporting the amount of time spent in the various reflective periods as well as the quality of that time. In addition, teams were also asked to identify the topics discussed and elaborated upon during the reflective periods. Arguably, simply measuring the time spent does not totally capture the essence of the construct of reflective behavior. That time is, however, a necessary Reflection-In-Action 0 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 10 11 12 Decision Quarter A ve ra ge T im e (M in ut es ) HP Teams LP Teams Figure 1 0 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 10 11 12 Decision Quarter A v e r a g e T i m e ( M i n HP Teams LP Teams Figure 2 Reflection-On-Action 9 Developments in Business Simulation and Experiential Exercises, Volume 28, 2001 but not sufficient factor in determining the learning that is taking place during the reflective phase of contemplation. Simply put, there must be both “quality and quantity” dimensions in the measurement process to effectively gain the needed insights into this phenomenon. This experiment seeks to measure the quality of the different reflective periods for each team. Each team was asked to rate and record their perception of the productive quality of each period during their team meetings. The rating scale was a seven point odd-numbered scale anchored from one for “Definitely Unproductive” to seven for “Definitely Productive.” This experiment was conducted using undergraduate students enrolled in a business course in marketing, eight teams of two members. Using the BUSSIM® simulation, students made decisions similar to those associated with a manufacturing company. Students made decisions regarding the quantity, type, quality, and price of the product as well as promotion, distribution, research, selling activity management, and media choices. The simulation was conducted for twelve quarters (three years) to allow for learning curve effects to stabilize. Decisions were separated by at least forty-eight hours (two days) to allow for students to meet as a team and make the next decision. Students were asked to record the amount of time spent on the different types of decision-making activities as well as the quality of that time. RESULTS This experiment made several measurements of the identified constructs. Data values were collected via modified journal forms that teams completed and turned in with the designated quarterly decision. The other source of data came from the financial and operational reports of the simulation programs. An alpha level of 0.05 was used for all statistical tests. One of the outcomes measured was retained earnings of the firms in the study. These values contain the net income values for the twelve quarters in addition to the starting value. The summary descriptive statistics of these data are shown in Table 1. Hypothesis 1 stated that the time spent in considering and decision making for all teams would decrease over the length of the simulation. The relationship between the amount of time spent by all teams and specific decision quarter was analyzed using simple linear regression. This hypothesis is not supported by the data (r = -0.107, p = 0.332, two tailed, n = 84). Thus there is insufficient evidence to support the first hypothesis. Reflection-in- action differs significantly, however, between HP teams and LP teams as shown in Figure 1. The HP teams demonstrated on average a consistent decrease in the amount of time spent on decision making. The relationship between the amount of time spent in decision-making and the decision quarter was statistically significant (r = -0.497 p = 0.001, two tailed, n = 44). This constancy supports the idea that teams who perform well are much better at arriving at decisions than their lower performing counterparts. The relationship, on the other hand, for lower performing teams between the time spent in decision- making and the decision quarter is not statistically significant (r = -0.026, p = 0.869, two tailed, n = 43). The time spent increases midway through the experience as these teams strive to improve their performance. The second hypothesis stipulated that analyzing and reflecting on the previous decision and actions (ROA) for all teams would also decrease over the simulation period, i.e., the analysis phase of reflective learning would become more routine and mechanical. This hypothesis was not supported by the data (r = -0.229, p = 0.033, two tailed, n = 85). Reflection-on-action findings differ similarly between the HP teams and LP teams. This is show in Figure 2. The HP teams demonstrated a progressive decrease in the amount of time spent analyzing their data was statistically significant (r = -0.498, p = 0.001, two tailed, n = 42). Admittedly, this could be attributed to making the analysis phase more routine. LP teams had a more volatile, erratic pattern to the amount of time spent in analysis, which was not statistically significant (r = 0.048, p = 0.759, two tailed, n = 43). 0 20 40 60 80 100 120 1 2 3 4 5 6 7 8 9 10 11 12 Decision Quarter A ve ra ge T im e (M in ut es ) HP Teams LP Teams Figure 3 Reflection For Action The third hypothesis relates to the reflection associated with more strategic directional issues such as “is the team going in the right direction”? Data analysis of the sample did support the hypothesis (r = -0.184, p = 0.178, two tailed, n = 86) in that the null hypothesis could not be rejected. This information is shown in Figure 3. But the data for all teams pertaining to this construct were not statistically significant. In comparing HP teams and LP teams, there is a noticeable difference in the data. The difference, however, is not statistically significant for HP teams (r = 0.314, p = 0.041, two tailed, n = 43). It should be noted that there is still a practical significance between the reflective time and the decision quarter. LP teams tended to have spent less reflective time in the early phase of the simulation experience. These teams did, however, increase the amount of time spent in reflective behavior later in the experience. The relationship between the time spent and the decision quarter for LP teams is likewise not statistically significant. 10 Developments in Business Simulation and Experiential Exercises, Volume 28, 2001 The last hypothesis indicated that HP teams would have different reflection behaviors than LP teams. Since the length of the simulation experience was long enough to insure that “lucky guesses” or “bad decisions” should be offset by good decision making, it was believed that HP teams should learn faster and adapt to the environment more rapidly. Thus it is useful to compare the reflection behaviors of both type of teams. The quality ratings for the reflection-for-action are reported for both HP and the lower performing teams. Interestingly, the lower performing teams reported a more productive perception than that of the HP teams. The differences between the two groups ratings are statistically significant (t = -6.568, p = 0.000+, two tailed, df = 11). Both categories of teams reported an increasing value for the productive quality of reflective period. The lower initial ratings by HP teams suggest that the teams felt that the time was not as productive as desired. Later in the simulation, however, the HP teams reported a high rating, similar to that of the lower performing teams. Reflection-on-action ratings were similar to the reflection- for-action. The teams with retained earnings below the median reported a higher rating for the productive nature of their analysis time. While the ratings of the HP teams are lower than that of the lower performing teams, the differences are statistically significant (t = -6.721, p = 0.000+, two tailed, df = 11). These findings seem to indicate that lower performing teams feel that the time used to reflect on previous decisions was productive despite the fact that their outcome performance was below median. Similarly, the time spent for reflection-in-action is that time when the team considers the potential impacts and consequences of the upcoming decision. Like the previous reflective quality rating values, these ratings of the lower performing teams are higher than those of the HP teams. These ratings, in addition, remain stable and basically unchanged during the simulation period. These ratings have statistically significant differences (t = -4.802, p = 0.000+, two tailed, df = 11). The ratings of the HP teams, however, are decidedly lower at the beginning of the simulation experience. The last hypothesis indicated that HP teams would have different reflection behaviors than LP teams. Since the length of the simulation experience was long enough to insure that “lucky guesses” or “bad decisions” should be offset by good decision making, it was believed that HP teams should learn faster and adapt to the environment more rapidly. Thus it is useful to compare the reflection behaviors of both type of teams. The quality ratings for the reflection-for-action are reported for both HP and the lower performing teams. Interestingly, the lower performing teams reported a more productive perception than that of the HP teams. The differences between the two groups ratings are statistically significant (t = -6.568, p = 0.000+, two tailed, df = 11). Both categories of teams reported an increasing value for the productive quality of reflective period. The lower initial ratings by HP teams suggest that the teams felt that the time was not as productive as desired. Later in the simulation, however, the HP teams reported a high rating, similar to that of the lower performing teams. Reflection-on-action ratings were similar to the reflection-for-action. The teams with retained earnings below the median reported a higher rating for the productive nature of their analysis time. While the ratings of the HP teams are lower than that of the lower performing teams, the differences are statistically significant (t = - 6.721, p = 0.000+, two tailed, df = 11). These findings seem to indicate that lower performing teams feel that the time used to reflect on previous decisions was productive despite the fact that their outcome performance was below median. Similarly, the time spent for reflection-in-action is that time when the team considers the potential impacts and consequences of the upcoming decision. Like the previous reflective quality rating values, these ratings of the lower performing teams are higher than those of the HP teams. These ratings, in addition, remain stable and basically unchanged during the simulation period. These ratings have statistically significant differences (t = -4.802, p = 0.000+, two tailed, df = 11). The ratings of the HP teams, however, are decidedly lower at the beginning of the simulation experience. 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 1 2 3 4 5 6 7 8 9 10 11 12 Decision Quarter A ve ra ge N et In co m e HP Teams LP Teams Figure 4 Average Net Income of Teams In light of the reflective time spent and productive quality ratings, it is helpful to examine the average net income for the HP and LP teams through the simulation experience. This is shown in Figure 4. It is apparent from the graph that HP teams earned higher profits during the first two years (eight quarters of the simulation period). Following that period, the LP teams were able to earn slightly higher net income than their HP counterparts. This outcome is likely due to a “mimicry” behavior by the LP teams. Since teams were well aware of the time frame of the simulation experience, LP teams may have imitated the decision variables of the HP teams in an effort to end up with a better final result. CONCLUSIONS 11 Developments in Business Simulation and Experiential Exercises, Volume 28, 2001 All teams did not seem to profit equally from the reflective learning dimensions throughout the simulation experience. HP teams in this experiment demonstrated the ability to decrease the amount of time spent in reflecting on previous decisions through analyses and more successfully incorporate that information in the decision making process. HP teams spent more time during the early quarters of the simulation experience than that their LP counterparts. These HP teams were generally more critical of the quality of their team meeting time when compared to the LP teams. Building on the successes of the early periods, HP teams were able to decrease the decision making time spent later in the simulation and rated these latter times as more productive. LP teams tended to spend more time in analysis, decision making, and strategy later in the simulation experience. Interestingly, these same teams rated the productive quality of the time spent as higher during the later quarters than the rating from the earlier. But it must be pointed out that LP teams were able to produce comparable net income during the last periods of the simulation experience. Thus it appears that the LP teams were able to successfully use the information from previous decisions to reinforce their performance, albeit at a slow rate. The reason for this behavior is not totally clear. Possible explanations include analytic abilities of the team, decision-making skills of the team, and strategic thinking processes by team members. REFERENCES Boud, D., Keogh, R., and Walker, D. (eds). (1985) Reflection: Turning experience into learning. London: Koan Page. Daudelin, M. W. (1996) “Learning from experience through reflection. Organizational Dynamics, 24, 36- 48. Jordan, David A. and Bruce, Michael L. (1999) BUSSIM Marketing: A Marketing Management Simulation. Beaver Falls, PA: David A. Jordan, Enterprises. Kimble, G. A. (1961) Hilgard and Marguis’ Conditioning and Learning. (2nd ed.). New York: Appleton-Century. Kolb, David A. (1984) Experiential Learning. Englewood Cliffs, NJ: Prentice-Hall, Inc. Rosenorn, Torben and Kofoed, Lise Busk (1998) “Reflection in Learning Processes through Simulation/Gaming.” Simulation & Gaming, (29,4), 432-440. Schon, Donald A. (1987) Educating the Reflective Practitioner. San Francisco: Jossey-Bass. Swenson, David X. (1997) “Requisite conditions for team empowerment.” Empowerment in Organizations, 5(1), 15-25. Walker, James T. (1996) The Psychology of Learning: Principles and Processes. Upper Saddle Falls, NJ: Prentice-Hall, Inc. 12 Table of Contents Volume 28, 2001 Using A Web Based Tutorial Program To Enhance Student Learning Reflecting On Reflection: An Examination Of Reflective Learning And Assessing Outcomes Overcoming Obstacles To A Cross-Global Collaborative Classroom Panel Discussion On Building And Maintaining Trust In The Abselesque Classroom Cosmopolitan-Based Cross National Segmentation In Global Marketing Simulations Strategic Management And The Case Method: Survey And Evaluation In Hong Kong Designing Interactive Self-Learning Modules Using Macromedia Director Teaching Financial Markets Using The Wall Street Journal Core Course Outcomes Assessment And Program Continuous Improvement Using An Integrative Business Plan Œ An Empirical Evaluation Using A Mock Trial Activity For Interdisciplinary Learning Fidelity, Verifiability, And Validity Of Simulation: Constructs For Evaluation 2001 International Collegiate Business Strategy Competition: 37 Years Of Collegiate Competition Motivating Students: An Initial Attempt To Operationalize The Curiosity Gap Model E-Learning: The Next Wave Of Experiential Learning Helping New Game Adopters: Four Perspectives An Initial Validity Invesitigaion of A Test Assessing Total Enterprise Simulation Learning Administering A Marketing Simulation - Common And Varied Practices Among Instructors Issues In Case Method Instruction: Class Discussion Leadership Sex Composition, Cohesion, Consensus, Potency And Performance Of Simulation Teams Integrating Management Curriculum: Linking Courses With Frontpage Absel: The Way We Talk! Teaching Strategic Management: A Case Study In The Diffusion Of Innovation In Education A Demonstration Of Bussim Strategy A Total Enterprise Simulation Teaching Methods: Etic Or Emic Antecedents Of Work Team Performance In A Business Simulation: Personality And Group Interaction When Experiential Educators Reflect Upon Their Craft: A Qualitative Research Report Strategic Sensemaking In Organizations: Model Formulation And Operationalization Panel Discussion: A Model For Initiative Development: Discussion And Ideas Of Individual Spirit Harmonics Assessing The Efficacy Of Experiential Learning In A Multicultural Environment Are Business Schools Producing 21st Century Managers? Blue Horizon Cruises: An International Simulation Exercise To Illustrate Conflicting Economic And Cultural Models Different Perceptions Of The Same Thing: An Experiential Exercise Current Use Of Agribusiness Simulation Games: Survey Results Of University Agribusiness And Agricultural Economics Programs Team Building 101 For Accountants An Online Evaluation Of The Compete Online Decision Entry System (Codes) Total Enterprise Simulation Winners And Losers: A Preliminary Study Gilding The Lily, Enhancing The Pedagogical Experience, Or Fixing What Isn™t Broke: The Dilemma Of New Simulation Versions The Absel Endnote Database: The Perfect Tool For The Bernie Keys Library Decision-Making Exercises (A), (B), And (C): Examining The Role Of Conflict In Group Decision-Making Group Decision Simulation (A)-(F): Examining Procedural Justice In Group Decision-Making Processes Evaluation Of Performance In Business Games: Financial And Non Financial Approaches A Framework For Evaluating Simulations As Educational Tools Congruence II™: A Strategic Business Board Game The Tournament Concept: Extending The Idea Of Using Simulations As Instruments Of Assessment The Economic And Political Transition Of The Mexican State In The Threshold Of Twenty First Century: From The Entrepreneurial State To The State Of Entrepreneurs. Experiential Learning For Accountants: The Not For Profit Project An Investigation Of The Environmental Awareness Attained In A Simple Business Simulation Game A Daily Exercise In Theory Application: Is This Why I Had To Take Marketing, Management, Finance –? An Alternative View to Scoring Selected Simulation Games and Experiential Exercises