ABSEL Contribution to the Assurance of the Quality of Learning – workshop results with discussion Page 35 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 ABSTRACT The paper presents results of two workshops conducted during ABSEL conferences in 2013 and 2014 in Oklahoma City and Orlando. The objectives of the initial workshop were to conduct an ice-breaker event and to build awareness among ABSEL community with respect to assurance of the quality of learning. After the workshop, the authors collected the papers where people noted their ideas, and analyzed the content using qualitative key words in context analysis. One year later, the authors presented the results of the initial workshop and performed a small experiment on giving additional wage and insight into the results from the initial workshop. The paper also proposes a discussion of the results and a model fundations of learning environment, which took shape in the process of crowdsourcing and elimination. Keywords: gamestorming, qualitative analysis, assurance of quality of learning, accreditation standards INTRODUCTION Quality of learning systems are storming the world of academic teaching nowadays. Accreditation standards (e.g. AACSB, EQUIS, AMBA) and the growing need to deliver value to students create pressure upon many academic institutions. ABSEL members have also noticed the increasing importance of this phenomenon. In order to bring this topic closer to ABSEL audience, a special conference track has been created to address the issue of the quality of learning delivery from the perspective of accreditation standards. Moreover, a special workshop devoted to the topic was delivered during the conference held in 2013 in Oklahoma City on the occasion of 40th anniversary of ABSEL. The results of this workshop are presented in this paper. A year after, during the 41st ABSEL conference in Orlando, the initial workshop results were presented to the group of conference participants. The presentation featured a short follow-up experiment was conducted to collect further data and arrive at new findings. INITIAL WORKSHOP SCENARIO The workshop was conducted in one of the general sessions during 40th ABSEL conference. The duration of the workshop was 45 minutes. The aim of the workshop was both to integrate people around ABSEL and build some ideas around the topic of how we can contribute to Assurance of Learning. The workshop scenario has been designed based on “gamestorming” methodology (Gray, Brown & Macanufo, 2010). First of all, such techniques have a very large ice-breaking potential, and one of the main goals of this workshop was to integrate people into ABSEL community; this activity is especially important for those who attend the conference for the first time. Secondly, the aforementioned area of assurance of quality of learning can be emphasized this way, showing the importance of this topic to ABSEL audience. The general sessions involved people sitting at round tables and since they did not know the content of the workshop, they took their places more or less at random. At the beginning of the workshop and after initial invitation, the general issue was presented to the audience and explained in brief. “How can we improve the assurance of learning with experiential learning and simulation games” Pieces of paper where placed on every table and the participants were then invited to play along and follow the prepared scenario: 1. Introduction - 5 min. 2. Teams/tables organization - 5 min. 3. Use the piece of paper to write down 3 to 5 key elements of games and experiential learning exercises contributing to the quality of learning – 5 min. 4. Share them with the people in your team/at your table – 10 min. 5. Based on the aforementioned elements, create 1 or 2 project ideas on how we can support the process of assurance – 10 min. 6. Share your project ideas with the people in your team/at ABSEL CONTRIBUTION TO THE ASSURANCE OF THE QUALITY OF LEARNING – WORKSHOP RESULTS WITH DISCUSSION Marcin Wardaszko Kozminski University Centre for Simulation Games and Gamification wardaszko@kozmiski.edu.pl Anna Pikos Kozminski University apikos@kozminski.edu.pl mailto:wardaszko@kozmiski.edu.pl mailto:apikos@kozminski.edu.pl Page 36 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 your table – 10 min. 7. Choose 1 or 2 best ideas in your team/at your table – 5 min. 8. Each team/table presents their chosen project ideas – 5 min. Each step of the scenario was shown to everyone using a video projector. The workshop was very well received and the author was even surprised with the number of people engaged in the process. Moreover, a lot of people approached the author after the workshop, sharing their thoughts and ideas. At the end of the workshop, the author asked the workshop participants to hand over the pieces of paper they were using. In total, 36 sheets of paper were collected during the workshop itself; some people also sent scanned versions by e-mail, because they wanted to keep the original copy for themselves. RESULTS OF THE INITIAL WORKSHOP The analysis of the collected sheets of paper reveled a very diversified content, ranging from some sketched notes and loose thoughts to fully structured proposals and ideas. Thus, for the sake of a proper analysis of the content, the authors opted for a linguistic analysis in the form of key-word-in-context analysis (Weber, 1990) and word frequency lists (Zipf 1932, 1965; FIGURE 1. An example of the collected sheets of paper. Own photo. Page 37 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 Weber, 1990) as a methodological approach. All sheets of paper have been analyzed and grouped according to key words and phrases and afterwards marked additionally in the case of reappearance of a given key word or phrase. The authors took into consideration both the appearance of particular phrases and words and the context of the words and phrases in longer sentences. Phrases/key words have been presented in order of appearance. Looking at the most frequently appearing phrases, we can observe the following key expressions: 1. Collaboration - as teamwork and knowledge sharing – 10 2. Contextual and clear feedback – 8 3. Economic fidelity/ real – 8 4. Application of theory – 8 5. Problem solving/challenge – 7 6. Unfreeze key ideas/critical thinking – 6 7. Reflection/debriefing – 6 8. Immersion – 6 Looking at the whole table, we can clearly say that all of the named elements are important, but the top eight of them TABLE 1 Summarized findings from text analysis of the sheets of paper collected at the initial ABSEL workshop in Oklahoma in order of appearance. Own work. Phrase/key word Number of appearances Contextual and clear feedback x x x x x x x x 8 Connecting results with measures x x x x 4 Economic fidelity/ real x x x x x x x x 8 Interesting story or scenario x x x 3 Implement games in the curriculum x x 2 Affective response x x 2 Making mistakes x 1 Safe environment x x 2 Responsibility for own actions x 1 Application of theory x x x x x x x x 8 Reflection/debriefing x x x x x x 6 Immersion x x x x x x 6 Emotions/empathizing x x x 3 Collaboration/knowledge sharing x x x x x x x x x x 1 0 Fun x x x 3 Following students interests x x x 3 Personalization of learning goals x x 2 Game as the best teacher x x x 3 Going beyond the class x x x 3 Unfreeze key ideas/critical thinking x x x x x x 6 Problem solving/challenge x x x x x x x 7 Interdisciplinary x x x 3 Using tools x x x x x 5 Decision making skills x x x x 4 Social construction x x x 3 Simple x x x 3 Page 38 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 give us some direction on what the quality system should focus on. Collaboration and knowledge sharing is a crucial element of gaming simulation and experiential learning, and the authors do agree with the view that it should be emphasized in the face of creating a high-quality learning systems. If we look at Dale’s (1969) methodology, it is the most effective way to learn, i.e. teaching/explaining to others. It is also the only way to build shared mental models and team learning according to Senge (1990) through a learning organization. Moreover, looking at the revised Bloom’s taxonomy (Anderson & Karthwoll et al., 2001) reaching higher levels of knowledge requires learners to evaluate and create knowledge transforming different domains into meta-cognitive knowledge. Other two points of our shortlisted key phrases also fit perfectly into this picture. With respect to problem-solving and challenge, and unfreezing key ideas as critical thinking, the authors decided to include both of these into one phrase because they have appeared in this particular context. The didactic experience of the authors allows them to say that in many cases, students feel under–challenged, or just directed to a particular solution. Simulation games and experiential learning allows them to ask difficult questions in a safe and structured environment, putting a question mark to the existing order. Another issue is to motivate students to solve problems on their own in a creative way. Stressing the problem of motivation among contemporary students is also a very important element. The Millennials are different type of people, and motivating them and keeping them focused is a growing problem for many academics (Hove & Strauss, 2007). Simulation games and experiential learning have been there for many years to learn and test learners’ skills, so nowadays it is also expected from us to pass knowledge forward, because students feel more motivated to learn more theory through e.g. a simulation game. Thus, application of theory and economic fidelity of realistic learning environment becomes more viable in today’s curriculum of courses based on simulation games and experiential learning. The Millennials have a strong need for the ability to instantly understand problems/ideas and be able to implement the solutions in practice. Therefore, learning, which is close to reality, has a chance to keep them focused long enough to make them absorb the knowledge. The authors decided to treat the contextual and clear feedback and reflection and debriefing as separate elements, because the cognitive process connected to them is different (Thiagarajan 1993, Kriz 2003, 2007). Clear and instant feedback is a mandatory learning tool for the Millennials, and building knowledge through hypotheses and assumptions testing process and putting it in the right context is a part of brining learning closer to reality. Debriefing and reflection is connected to the cognitive process of self-evaluation and collaborative knowledge sharing, which is the essence of gaming simulations (Crookall, 2010). An interesting element of the shortlisted key phrases is immersion. Over the last few years, an increase in the number of papers on that topic has been observed – including among ABSEL community (Giambatista & Hoover 2009, Micklich 2011, 2012, Hoover 2011, 2014). Immersion becomes an TABLE 3 List of key words/phrases from the post-workshop stage. Own work. Phrase/key word Number of appearances Application of theory 6 Decision making skills 6 Problem solving/challenge 5 Collaboration/knowledge sharing 5 Reflection/debriefing 4 Using tools 4 Fun 4 Interdisciplinary 4 Going beyond the class 4 Economic fidelity/ real 3 Game as the best teacher 3 Connecting results with measures 3 Unfreeze key ideas/critical thinking 2 Contextual and clear feedback 2 Following students interests 2 Implement games in the curriculum 1 Making mistakes 1 Immersion 1 Affective response 1 Simple 1 Responsibility for own actions 1 Page 39 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 important factor when talking about student motivation again. However, it is the only soft-skill-oriented element on the list. In the authors’ opinion, it is not a coincidence that this element places high among the ideas that ABSEL scholars have in mind when talking about the ability to create high-quality learning environments through simulation gaming and experiential learning. A first glimpse at this problem reveals a small conflict of immersion and economic fidelity and realistic approach. Immersion is more related to free-from games (Huizinga 1985) and ilinx type games according to Caillois’ (2001) typology. However, if we gain a deeper insight into that problem, the answer to this dilemma would be rather a balance between reality and fiction, then a conflict. A pursuit of more realistic simulation and experiential environments/settings may lead to favoring more foreseeable and thus more biased decisions, strategies and solutions. Thus, even a very serious problem and realistic setting can be given “unrealistic” context or story in order to create a more immersive environment or situation. A very good example of this kind of setting is gamification (Zichermann & Cunnigham 2010, Herger 2014), which can create very unrealistic contexts to quite important and serious problems like speeding, health problems, fighting overweight or looking for cancer cure. Through storytelling and fun, they create immersive environments. At further stages of analysis, the authors decided to split all phrases into two categories and analyze the balance between more course- and process-oriented elements and more human- and skill-oriented elements. Dividing the aforementioned elements into these two groups is not an easy task because many of them belong partially to both of them. The authors made their decisions based on experience and a seemingly “stronger” belonging to one of either of the subgroups, and it can be a subject of the discussion. There are more human/skill-oriented elements than system/ course-oriented elements. However, judging from the number of appearances in the presented ideas, the system/course- oriented elements appear much more often. Conclusion drawn from this distribution can be of double type. First of all, when it comes to simulation gaming and experiential learning, the course and system setup is extremely important. Our job is to secure safe and suitable environment for our learners, and without this, no learning quality can be achieved. Secondly, the right course/system setup is not enough for simulation gaming and experiential learning to be effective and, thus, of high quality. Skill-oriented and human behavior-oriented elements are the other part of essence to creating high-quality learning, and there are many aspects of human behavior we should take into consideration. Special care should be given to the ability of creating space for collaboration and knowledge sharing as a unique feature of simulation gaming and experiential gaming. RESULTS OF THE POST-WORKSHOP EXPERIMENT One year later, the authors presented the results of the initial workshop during the 41st ABSEL conference in Orlando. The session and the presentation was attended by about 15 people. During the presentation stage, a small gamestorming (Gray, Brown & Macanufo 2010) experiment was conducted. The authors presented a part of the initial workshop results, showing the discovered key words/phrases and describing their meaning in the context of the experiment, but the number of appearances was not disclosed. Then, the authors asked the audience to join the discussion, use the previously distributed pieces of paper to write down 4 to 6 most frequently appearing key words/phrases from the most frequent to the least frequent. After some time, full results were disclosed and session participants were asked to award themselves with points if they hand named key word/phrases from the list of top eight. The winner of this small experiment, to the utter surprise of the people gathered, had chosen all 6 from the top 8 list. When the winner was announced, the authors finished the presentation and a discussion followed. At the end, the authors asked the audience once again to hand over the pieces of paper with written solutions; in total, 14 pieces of paper were collected. The aim of this second activity was further investigation of the significance and meaning of the discovered elements. Out of the 14 pieces of paper collected, 4 people picked elements none of which were on the list of top 8 list, and the other 10 had on average 2.4 key words/phrases from the top 8 list. Out of 26 displayed key words/phrases, 21 reappeared on the collected pieces of paper. Furthermore, all of the 8 shortlisted words/phrases appeared on the list of choices. However, only four shortlisted phrases reappeared most frequently (more than 4 times):  Application of theory - 6  Problem solving/challenge - 5  Collaboration/knowledge sharing - 5  Reflection/debriefing - 4 Other phrases reappeared less frequently. Additionally, because the authors asked the participants to put the phrases in a specific order, there was a possibility to add weight to their answers. The authors decided to add the following weight to chosen phrases:  5 points for the first place on the list  3 points for the second place on the list  2 points for the third place on the list  1 point for every answer from position 4 to 6 on the list After that, all points were added and a new table with weight-based score was created. The weight-based score changes the landscape of the results. “Game as the best teacher”, “clear and contextual feedback”, “fun” and “economic fidelity/real” moved up significantly in the top ranking because they appeared most frequently (3 times) on the list of reappearing words/phrases. However, “collaboration/knowledge sharing”, “interdisciplinary”, “using tools” and “going beyond the class” fell down in the ranking, because they reappeared quite frequently – but mostly in lower positions of the lists. Collaboration/knowledge sharing is the best example of this, as the most popular phrase in the initial workshop and the second most popular at the post-workshop stage went down by 4 places Page 40 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 after applying weight, because it reappeared 5 times in the shortlist in the post-workshop session, but rarely in the top 3 choices of the participants’ lists. The results of the weight-based score table shows that 6 out of 8 phrases from the initial shortlist of the most popular elements are at the top of the table of the post-workshop ranking, i.e. have more than 8 points. Concluding remarks and discussion ABSEL community gathers people from different areas and backgrounds. Both shared and individual knowledge is very extensive, and the magnitude of experience can be counted in thousands of years. Using crowdsourcing techniques in the form of gamestorming method was both a pleasure and a challenge at the same time. One of the aims of the workshops was to take a sneak-peek behind the scenes of the shared knowledge and experience of this extraordinary community. The material gathered through the initial workshop and the post-workshop session may appear to be not applicable at first sight, but after a careful analysis, it seems that it leads to interesting and insightful observations. To answering the initial question formed at the workshop in Oklahoma City in 2013, we can build a preliminary model foundations of how the “perfect” model of high-quality learning environment should look like for simulation gaming and experiential learning. We should build our learning environments on the basis of theory, but it should involve both contemporary and applied approach so that our students can pick up the challenge of testing it and putting it in practice. Instead of creating courses and classrooms, we should build learning environments allowing for and promoting collaboration and knowledge sharing, or maybe even forcing teamwork and collaboration in a positive way. Maybe we should try not to teach anything but just be a source of theory and advice on one hand, and a source of clear feedback and encouragement on the other. The central point of the learning structure should be feedback and reflection in the form of debriefing and discussion. The main driving force behind the debriefing and discussion should be simulation games and experiential learning exercises together with current economic challenges and problems of the modern world. Ultimately, we should try to make our learning environments immersive through storytelling, role-playing and free-from games stimulating creativity and meta-cognitive knowledge creation. TABLE 4 List of key words/phrases from the post-workshop stage with weight-based score. Own work. Phrase/key word Number of appearances Waged score Application of theory 6 19 Decision making skills 6 18 Problem solving/challenge 5 15 Fun 4 14 Game as the best teacher 3 11 Reflection/debriefing 4 10 Contextual and clear feedback 2 10 Collaboration/knowledge sharing 5 8 Economic fidelity/ real 3 8 Interdisciplinary 4 7 Using tools 4 6 Going beyond the class 4 5 Making mistakes 1 5 Unfreeze key ideas/critical thinking 2 4 Connecting results with measures 3 3 Following students interests 2 3 Simple 1 3 Implement games in the curriculum 1 2 Responsibility for own actions 1 2 Immersion 1 1 Affective response 1 1 Page 41 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 REFERENCES Anderson L.W., Krathwohl D.R., Airasian P.W., Cruikshank K.A., Mayer R.E., Pintrich P.R., Raths J., Wittrock M.C., (2001) A Taxonomy for Learning, Teaching and Assessing. A revision of Bloom’s Taxonomy of Educational Objectives. New York: Addison Wesley Longman. Caillois R. (2001) Man, Play, and Games. Champaign: University of Illinois Press. Crookall, D. (2010). Serious Games, Debriefing, and Simulation/Gaming as a Discipline. Journal of Simulation & Gaming, vol. 41, 898—920. Cunningham C., Zichermann G. (2011) Gamification by Design. Sebastopol, California: O’Reilly Media Inc. Dale E. (1969) Audiovisual methods in teaching. New York: Dryden Press Edition. Giambatista R.C., Hoover J.D. (2009), An Empirical Test of “Behavioral Immersion” In Experiential Learning, Developments in Business Simulation and Experiential Learning, ABSEL, vol. 36, pp. 124-130. Gray D., Brown S., Macanufo J. (2010), Gamestorming. A Playbook for Innovators, Rulebreakers, and Changemakers. O’Reilly Media, Inc. Sebastopol, CA. Herger M. (2014), Enterprise gamification. Engaging people by letting them have fun. Enterprise Gamification Corp., Lexington, KY. Hoover D.J. (2011), Complexity Avoidance, Narcissism and Experiential Learning, Developments in Business Simulation and Experiential Learning, ABSEL, vol. 38, pp. 255-260. Hoover D.J. (2014) Experiential Learning in the 21st Century: Personal Reflections and Insights from Burning Man. Developments in Business Simulation and Experiential Learning, ABSEL, vol. 41, pp. 323-333. Howe, N., Strauss, W. (2007). Millennials Go to College (2nd ed.). Great Falls, VA: LifeCourse Associates. Huizinga J. (1985), Homo lundes. Zabawa jako źródło kultury. Warsaw: Czytelnik. Kriz W.C. (2003) Creating Effective Interactive Learning Environments through Gaming Simulation Design. Journal of Simulation & Gaming, 34 (4), pp. 495–511. Kriz W.C. (2007) Planspiele für die Organisationsentwicklung. Schriftenreihe: Wandel und Kontinuität in Organisationen (Bd. 8). Berlin: Wissenschaftlicher Verlag. Micklich D.L. (2011), Examining the Cognitive, A ffective, and Psychomotor Dimensions in Management Skill Development through Experiential Learning: Developing a Framework. Developments in Business Simulation and Experiential Learning, ABSEL, vol. 38, pp. 261-272. Micklich D.L. (2012), The Effect of Affective Domain Characteristics on Behavioral or Psychomotor Outcomes, Developments in Business Simulation and Experiential Learning, ABSEL, vol. 39, pp. 164-172. Senge, P. M. (1990). The Fifth Discipline. The Art & Practice of The Learning Organization. New York: Currency Doubleday. Thiagarajan S. (1993) How to maximize transfer from simulation games through systematic debriefing. In: F. Percival, Sh. Lodge, D. Saunders (eds.), The Simulation and Gaming Yearbook 1993. London: Random House, pp. 45–52. Weber R.P. (1990), Basic content analysis. Series: Quantitative Applications in the Social Sciences. Sage Publications Inc., London. Zipf, G.K. (1932), Selected studies of the principle of relative frequency in language. Cambridge MA, Harvard University Press. Zipf, G.K. (1965), Psycho-biology of Language. Cambridge, MIT Press.