Process-Oriented Research Method for Teamwork Effectiveness Assessment in Business Simulation Games Page 233 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 ABSTRACT The current state of the art in teamwork effectiveness research in business simulation games pictures this field to be quite ex- tensively examined already. Nevertheless, a substantial – and still growing – number of studies does not offer any explicit conclusions on effectiveness in computer-assisted learning, es- pecially in teams. Many authors point, amongst other reasons, to the lack of overarching research methodology to be the cause of this difficulty. In this paper some of the troublesome method- ological aspects of teamwork effectiveness research in business simulation games are addressed by proposing a research meth- odology that combines process- and outcome-oriented ap- proaches and measuring tools together. This paper describes the process of implementation and some sample results of a quasi-experimental comparative study based on two, compared in pairs, groups of students – participants of business simula- tion games courses. As a result, there emerges a methodology that brings particular elements of qualitative and quantitative paradigms and methods – as complementary – together in order to bring a broader picture of the research situation to the scene. The authors argue that this approach might make it possible to obtain more conclusive and easier to interpret research results with respect to teamwork effectiveness in business simulation games. Key words: team communication, research methodology, seri- ous games, team roles, experimental INTRODUCTION Although there are many studies on effectiveness in simula- tion games, there is no standardized methodology for conduct- ing research in this scope. This results in lack of sound empiri- cal evidence in this field of study, which is caused by a diversi- ty of measures for effectiveness assessment, a multitude of methods of data collection, and overall suboptimal study de- signs (All, Nuñez Castellar & Van Looy, 2014). Having said that, we have to admit that there is no clear empirical evidence of effectiveness of serious games. Yet, there is also a series of arguments raised in literature regarding serious games to be powerful tools for learning. Therefore, the necessity of further exploration of this matter by experimental studies is also sug- gested by several researchers like (Annetta, Minogue, Holmes & Cheng, 2009; Wrzesien & Raya, 2010, Hainey, Connolly, Stansfield & Boyle, 2011; Girard, Ecalle, & Magnan, 2013). There is a broad field of interest for theoreticians and prac- titioners in investigation of the patterns of group processes and their changes over time. For over four decades of research on group development models, numerous models have been devel- oped (Chidambaram & Bostrom, 1996). Chidambaram and Bostrom in their meta-analysis of these models distinguished two approaches used for group development models construc- tion: (1) process- and (2) outcome-oriented. Thus, this well- established distinction was chosen to be the basic principle of the general logic and methodological approach of this study, also referring to the interdisciplinary approach towards business simulation games research methods recommended by Duke and Geurts (2004) – especially in the face of the fact that researchers in the field of computer-assisted learning effectiveness tend to focus either on quantitative or solely on qualitative methods and tools, and also taking into account the state-of-the-art tendency of quantitative studies significantly outnumbering the qualita- tive ones (All et al., 2014). In this article we will use both ap- proaches, but would like to still elaborate more on the qualita- tive measures we have adopted for its purpose, as we have put into work and transformed a set of selected psychological tools to this end (a video analysis of trained behavioral experts and a questionnaire measuring: communication patterns of team members interaction, level of engagement of team members, and socio-emotional/task oriented preferences in teamwork). Although authors in the field of serious games generally disagree on the way learning effectives is reached and meas- ured, the majority of them agree on the importance of the role of communication in experiential learning and serious game- based learning courses played in teams (Kriz, 2000;2003; Kayes, Kayes & Kolb, 2005; Kriz & Hense 2006, Hergeth 2007; Kriz & Nöbauer 2008). METHOD The aim of this research was to create and test a compound methodology applicable for teamwork effectiveness examina- tion in business simulation games, one that takes into account the dynamics of the game processes and encompasses both – qualitative and quantitative – measures. In order to achieve that, Process-Oriented Research Method for Teamwork Effectiveness Assessment in Business Simulation Games Anna Ruszkowska Center for Simulation Games and Gamification Kozminski University aruszkowska@kozminski.edu.pl Marcin Wardaszko Center for Simulation Games and Gamification Kozminski University wardaszko@kozminski.edu.pl Page 234 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 a quasi-experimental research concept was employed to exam- ine a group of master level business students taking part in a business simulation games course. An independent variable: teamwork effectiveness (quantitative) – was measured by the financial results of each examined team, expressed by stock price of their company (in euro). This variable served to divide the examined teams into two categories: “winners” or “losers”. The sample table below depicts how this division was made, showing the financial results and the final positions of the teams in comparison to others in the same simulation game course. Dependent variable: teamwork effectiveness (qualitative) – measured by indicators formulated in two steps. First, some main characteristics of well-developed groups were chosen (Chidambaram & Bostrom, 1996) to be operationalized in the next step as follows: 1) Socio-emotional and task needs balance. Group mem- bers focus on balancing both types, but it is suggested in the reference literature that an excessive focus on only one of them might restrain the other and can lead to a regression of group development (Bales, 1950). Operationalization: First, the preferred team roles of the team members were diagnosed by Belbin’s Self-Perception In- ventory (Belbin, 1981). The results of each member were classi- fied according to two categories: (a) socio-emotional or (b) task -oriented – using Fisher, Hunter and Macrosson method (Fisher, Hunter & Macrosson, 1998); next, the ratio of both categories was assessed as balanced or not by the rule of thumb as suggest- ed by the authors (Fisher et al., 1998). See table below for Belbin’s role categorization. The main preferred team role of each team member was taken into account in this analysis. As a result, we obtained “profiles” of teams, indicating balance or imbalance with an inclination towards either task or socio-emotional needs. 2) Effective communicative behaviors. Well-developed group members share a common group identity, understand common goals of the group, and communicate openly, even if there occur some conflicting opinions (Chidambaram & Bostrom, 1996). Operationalization: the ratio of communicative and non- communicative behaviors was observed in team interactions by qualified behavioral experts. The experts assessed and catego- rized the observed behaviors based on a video recording of a game play situation and a set of given behavioral categories. There were three categories of behaviors defined: a) Communicative behavior (code: k) – based on the defini- tion by Goldin-Meadow and Mylander (Goldin-Meadow and Mylander, 1984, after: Tomaszewski, 2008), such be- havior: (1) contains an intention directed to a partner of interaction and (2) the behavior is not an operation on an object serving other purpose but communication. This dif- ferentiation encompasses speech acts accompanied by ges- tures of hands/body/head and also independent gestures with no verbal communication at the same time. b) Individual work (code: pi) – as this study focuses on com- puter-assisted learning, we created an additional category to include into the behavior analysis: it is associated with op- erations on objects, when a person does not communicate directly with other team members as the person is busy with individual activity. For example: reading course mate- rials, making necessary calculations with computer, tablet or phone, looking through the simulation outcomes. Such behaviors, when displayed, were categorized as indirect communication of cooperation and work to other team members. c) Non-communicative behavior (code: n) – the category was set as one opposing the definition of communicative behav- ior mentioned above. This included all behaviors contain- ing no intention towards the partner of interaction and serv- ing other purposes than communicating with another team member. A more specific description of those behaviors for the purpose of the experts’ categorization instructions was created based on negative social-emotional behavior cate- gory descriptions by Bales (Bales, 1950). 3) Active engagement in team activities. Continuous en- gagement and cooperation during team activities – especially when in conflictual situations or under pressure of competitive conditions – are the necessary conditions for a good group de- velopment (Chidambaram & Bostrom, 1996). Operationalization: The level of active engagement in team activities was assessed by trained behavioral experts based on a video recording in a game play situation. The experts were as- sessing the observed engagement on a 5-leve Likert scale, where: 1 = “no active engagement in team activities, the team is not working” and 5 = “all team members actively engaged in team activities, full engagement”. PROCEDURE Subjects. The research was conducted at the authors’ uni- versity, during “Strategic Business Games” courses, where last year, part-time business students of master’s-level studies played “TOPSiM” managerial game. Four student classes were researched in total, two of which were profiled in business man- agement, and two others specializing in finance. Adequate game scenarios were applied for these two specializations: A – a less- developed, 4-round scenario for Management, and B – a more developed, 5-round scenario for Finance. Each decision round was exactly 90 minutes long, students were making decision in TABLE 1 EXAMPLE OF TEAM FINANCIAL RESULTS. Stock price in EUR Final rank A1 Round 0 Round 1 Round 2 Round 3 Round 4 Team 1 100 106,6 136,2 189,2 255,5 winner 2 Team 2 100 120,1 78,8 145,6 158,7 loser 1 Team 3 100 173,5 128,5 182,8 235,8 loser 2 Team 4 100 104,7 171,4 214,3 275,3 winner 1 Page 235 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 the classroom and under time pressure. The students were aged from 21 to 42 (mean age = 27), there were 39 women and 41 men examined in their class groups. All of the students had pre- viously finished their courses in finance, accounting, strategic management, marketing, etc., and therefore held an equal theo- retical preparation for practical application of this knowledge in the simulation game. Direct preparatory information given be- fore the strategic business games course started was provided in the supplied user’s manual reading, plus by means of a lecture and basic instructions given by the class instructor. Researching procedure. On the starting day of the business simulation game play, when the participants gathered in their classrooms, they were informed by the teacher about the oppor- tunity to take part in a research study regarding group process- es. The teacher would introduce the researcher, inform about the lack of consequences of the participation in the research with respect to the grade for the course, and leave the classroom (to boost the sense of the scientific nature of the study, and to separate the research from the standard course of education). The researcher would introduce herself and describe the re- search procedure, underlining two components of the research: (1) Team Role Self-Perception Inventory survey and (2) video registration of the game play situations by two video cameras set in the classroom. The information on the collected data be- ing anonymous, confidential and voluntary given, additional questions were answered, and the procedure followed with an invitation to the research. After the willing participants filled the Team Role Self-Perception Inventory surveys, the research- er collected them and provided her contact details in case of any additional questions. The participants were informed about the questionnaire feedback receiving term and the teacher was in- vited back to the classroom and started the class by instructing participants to form 4-6 person teams, as recommended by Wolfe, Joseph & Chacko (1983) and Wolfe & Thomas (1988). All teams worked in the open space of the classroom, where two video cameras were registering their behavior. We also allowed students to create their teams by themselves in order to have a natural team composition, allowing them a better com- munication from the start (Faria & Wellington, 1994; Wolfe & McCoy, 2008; 2011; Thavikulwat, & Chang, 2010; 2012; 2015), bearing in mind all pros and cons of such composition. Qualitative data analysis procedure. The Team Role Self- Perception Inventory was analyzed according to its formal re- quirements. A behavioral analysis by qualified experts was per- formed in several steps. First, the video material was registered by two video cameras that were set in the room (diagonally, both with wide-angle lenses to cover the whole space of the classroom from two different perspectives at one take). Next, the video material was processed for the use of the behavioral experts: from each decisive round (one and half an hour each), the first 15 and the last 15 minutes section were selected for analysis. The experts were given assessment sheets, instructions regarding behavioral categories, and received a training. The training was conducted on a pilot video material presenting a sample of previously examined teams that were not the subjects of the research in question. The training took 4 hours in total and included: definitions of the categories, instructions on as- sessment sheets plus a trial assessment of the sample video ma- terials. The trial assessment of two 15-minute samples gave over 80% of accordance among individual experts, so, to pro- vide highest internal consistency of assessment, the experts were asked to work in pairs – discussing to agree on a certain behavior categorization if in doubt. The research material was then categorized according to the instructions given: communi- cative and non-communicative behaviors were noted and cate- gorized every 30 seconds of the video, for every person from each team. Active engagement in group activities assessment took place every 60 seconds for the whole team on a 5-level Likert scale. Dominant behaviors were noted down. SAMPLE RESULTS The sample results presented below depict the structure of the measures employed and designed for this research. All teams were divided according to winning and losing categories, based on their financial results (stock price in euro), and com- bined in pairs for a process-oriented, dynamic analysis. The result sheets presented above make it possible for us to focus on the dynamics and compare the components of the wid- er picture of simulation game, including: financial results (stock price changes), communicative behavior changes, and active engagement dynamics of team members. Second, we can detect significant events across all of those researched areas. The sam- ple results obtained using this method already suggests some interesting coexisting patterns and characteristics for both win- ning and losing team. LIMITATIONS AND DISCUSSION As this paper focuses on teamwork effectiveness, it was necessary to reflect on the definition of a ‘group’ and a ‘team’. The psychological standpoint focused on group dynamics de- fines groups by two processes: first (1) – social identification process – based on a belief in a “we-they” division, which en- compasses cognitive (world categorization) and emotional (evaluative judgment of the world) components, and second (2) – social representation process, which derives from values, ide- TABLE 2 BELBIN’S ROLE CATEGORIZED INTO TWO SUB-GROUPS. Preferred group role categories Socio-emotional oriented Task oriented Coordinator Shaper Implementer Plant Resource Investigator Monitor Teamworker Completer-Finisher Page 236 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 Preferred roles of team members: FIGURE 1 CHART A2 – TEAM 3 (WINNER 1) Page 237 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 FIGURE 2 CHART A2 – TEAM 2 (LOSER 1) Preferred roles of team members: Page 238 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 as and believes about the world of the group members (Levi, 2007). Levi, when defining groups in detail, lists the qualities that describe them, particularly stressing the importance of in- terpersonal interactions among the group members – and espe- cially communication as the main and most important group process of all of those listed (Levi, 2007). Therefore, communi- cative behaviors of team members considered in our research took one of the central and most elaborated points. Also, the definition of a team by Katzenbach and Smith, stating that “A team is a small group of people that complement each other, are engaged in accomplishing shared goal, present similar ap- proaches towards work and hold themselves responsible for the outcomes of their own work” (Bitkowska, 2009), underlines team performance, complementary skills of the members, and direct interactions among them. We believe these qualities to be substantial components for a reliable and highly relevant exami- nation of group processes, having in mind the research design that would let us deliver reliable and adequate answers for our research questions. Also, the analysis of communicative interactions has a well -established and verified methodology in the field of psycholo- gy, and can be transferred to the field of organizational behavior studies. Psycholinguistics offers some precise methods and tools, especially in the context of verbal and non-verbal com- munication analysis. The transfer of these to the field of busi- ness simulation games that we have made, allowed us to engage this qualitative approach and highlight the process dynamics and to reason from its measures, as opposed to outcome- oriented approach that measures and allows one to reason from final results and some measurements made at certain points of time only. The method described in this paper offers continuous measures of analysis of communicative behaviors and engage- ment of group members. In conjunction with the feedback based on financial results for each decisive round, it offers continuous, interlinked and therefore more insightful observations. Another issue addressed by our method is the measurement of effectiveness factors for a team. Although there is a wide range of studies in computer-assisted learning effectiveness, not many of them put a team as an entity to be the subject of the research, as we attempted to do here. Of course, there is a list of limitations and difficulties related to such approach, which ob- viously accompanies its innovative feature: the communicative and engagement behavior measurement. Firstly, we could not study the semantics of each team member’s utterances, but only observed the behaviors they displayed, but it would have been impossible to focus on this aspect in more detail with this amount of research data due to the time limitations. Secondly, the video recordings for the behavioral analysis were conducted for the team as whole. The setting of both of the used video cameras was designed to capture the widest spectrum possible, but there are some minimal data losses occurring in the process. Using a separate camera for each team member would probably eliminate this issue, but it would most probably distract and stress the participants and thus introduce a serious interfering variable to the research situation. Also, in the case of the per- formed communicative behavior analysis, the behavioral ex- perts were categorizing dominant behaviors of every team member. As each observation took 30 seconds, there was a pos- sibility of exclusion of some minor behavioral observations, but according to the debriefing discussion of the experts, this exact time period encompassed cohesive behaviors of each member – even in 4-6 person team interactions. Therefore, for the purpose of high accuracy of our analysis, a numerical indicator of the summarized observations was expressed in percentage and served to express ratios. This limitation did not touch on active engagement evaluation, as it was conducted in the form of an experts’ assessment, based on Likert scale. We also kept in mind that the Self Perception Inventory by Belbin is not considered a valid psychometric tool. However, when researching the socio-emotional and task oriented prefer- ences of the participants, it is worth noting though that it was used for a basic differentiation and testing purposes, and in- structions of Fisher et al. were followed as well (Fisher et al., 1998). Finally, the quasi-experimental concept of the research could be considered a cause of limitations of this study, as, for example, the members of the examined teams were not assigned to their teams randomly, or the games scenarios were adjusted for each student group specialization, etc., like in a classic ex- periment design. We are aware that this is why we are not al- lowed to conduct a cause-and-effect reasoning here. Neverthe- less, the quasi-experimental framework allowed the most accu- rate approximation of a real educational simulation game situa- tion, and it lets us indicate the differences in the researched fac- tors when reasoning. We believe the fidelity of the educational simulation game situation to be worthy of this trade-off at this early stage of creating and testing our method. Further research is recommended in order to eliminate all limitations possible and to optimize the researching and reasoning processes of this method, as it might become a basis for a valid research frame- work of group processes, one which focuses on dynamics of interactions during game play. CONCLUSIONS During the test, we found the data for these particular two groups pointing to a high imbalance with respect to the pre- ferred team roles, and those teams who had more task-oriented roles had on average better in-game results than those with more socio-emotional roles. The teams that scored better pre- sented more consistent patterns of communication. However, some of the losing teams had very consistent communication patterns and a high level of engagement, which did not prevent them from failing at all eventually. If we look at the communi- cation pattern from the beginning and the end of the decision round, we can notice quite big differences. The majority of the teams had more balanced and consistent communication pat- terns in the first half of the decision round (20 cases out of 36 had 55% of communicative behaviors), and as they progressed to the second half and towards the end of the decision round, the patterns became more and more rugged with a tendency to communicate (only in 9% of the cases the communicative be- havior ratio was lower than that at the beginning). Successful teams, however, had more consistent changes between the deci- sion round halves. If we look at the first two decision periods compared to the last two decision periods, we can say that the winning teams tended to increase their communicative behavior (k+pi) as the games progressed, while the losing teams dis- played, in majority, an opposite trend. Looking at the communi- cative behavior composition with regards to the preferred roles, we can say that teams that adapted to the situation in the most consistent way had more chances to succeed than those who just maintained the communication pattern. It can be clearly ob- served when looking at the communicative behavior pattern change in critical moments, i.e. rapid decrease or increase in the score. However, although the results of the study and findings are quite interesting, we cannot make generalizations based thereon for two reasons. First, the scale of the initial experiment is not large enough and there is no basis for statistically significant Page 239 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 dependencies. Second, the research method forged for the pur- pose of this study is still untested and needs a lot of fine tuning and further calibration. Nevertheless, both the method and its results are very promising and have already created many new question and research ideas. One of them is the issue of feed- back frequency for the teams. In the presented research, the teams received just a single case of feedback communication from the game in one round. In the future, we would like to increase the number of feedback messages to teams during one decision round, which would let us extend the number of valid observations in a given time frame. Such observations will make it possible for us to build dynamic models of communica- tion with the use of different variables and situations. Thus, a dynamic analysis of communicative behavior changes in critical events for unbalanced teams can lead us to new and interesting findings in the future. All, A., Nuñez Castellar, E. P., & Van Looy, J. (2014). Measu- ring Effectiveness in Digital Game-Based Learning : A Methodological Review. International Journal of Serious Games, 1(2), 3–21. Annetta L.A., Minogue J., Holmes S.Y.&Cheng M.-T. (2009) Investigating the impact of video games on high school students’ engagement and learning about genetics. Comput- ers & Education 53, 74–85. Bales, R. (1950). A Set of Categories for the Analysis of Small Group Interaction. American Sociological Review, 15(2), 257–263. Belbin, M. (1981). Management Teams; W hy They Succeed or Fail. Guilford, UK: Biddles Ltd. Bitkowska, A. (2009). Zarządzanie procesami biznesowymi w przedsiębiorstwie. Warszawa: Vizja Press & IT. Chidambaram, L., & Bostrom, R. P. (1996). Group Develop- ment ( I ): A Review and Synthesis of Development Mod- els, 187, 159–187. Duke R.D., Geurts J.L.A. (2004) Policy games for strategic management. Amsterdam: Dutch University Press. Faria A.J., Wellington W. (1994) The effect of time pressure, team formation and planning on simulation/game perfor- mance. In: D. Crookall, K. Arai (eds.), Simulation and gaming across disciplines and cultures, SAGE Publicatons. Fisher, S. G., Hunter, T. a., & Macrosson, W. D. K. (1998). The structure of Belbin’s team roles. Journal of Occupational and Organizational Psychology, 71(3), 283–288. Girard, C., Ecalle, J., & Magnan, A. (2013). Serious games as new educational tools : how effective are they ? A meta- analysis of recent studies. Journal of Computer Assisted Learning, 29(March 2012), 207–219. Hainey T., Connolly T., Stansfield M. & Boyle E.A. (2011) Evaluation of a game to teach requirements collection and analysis in software engineering at tertiary education level. Computers & Education 56, 21–35. Hergeth H. (2007) Team behavior and team success: results from a board game simulation. Developments in Business Simulation and Experiential Learning, Volume 34, [Available from http://www.absel.org] Kayes D.C., Kayes A., Kolb D. A. (2005) Experiential learning in teams. Simulation and Gaming, 36(3), pp. 330–354. Kriz W.C., Hense J. (2006) Theory-oriented evaluation for the design of and research in gaming and simulation. Journal of Simulation & Gaming, 37, 2, pp. 268–283. Kriz W.C., Nöbauer B. (2008) Teamkompetenz. Konzepte – Trainingsmethoden – Praxis, Göttingen: Vandenhoeck & Ruprecht. 4. Auflage. Kriz W.C. (2000) Lernziel Systemkompetenz. Planspiele als Trainingsmethode. Göttingen: Vandenhoeck & Ruprecht. Kriz W.C. (2003) Creating Effective Interactive Learning Envi- ronments through Gaming Simulation Design. Journal of Simulation & Gaming, 34 (4), pp. 495–511. Levi, D. (2007). Group dynamics for teams (2nd editio.). Sage Pulications, Inc. Thavikulwat, P., & Chang, J. (2010). Pick your group size: A better procedure to resolve the free-rider problem in a busi- ness simulation. Developments in Business Simulation and Experiential Learning, 37, 14-22. [Available from http:// www.absel.org] Tomaszewski, P. (2008). Interactions of deaf preschoolers: a comparison of the communicative behaviors of deaf chil- dren of deaf parents and of deaf children of hearing parents. Psychology of Language and Communication, 12(2), 69– 87. Wolfe, J., Thomas B. (1988), Team Cohesion Effects on Busi- ness Game Performance, Simulation and Games, Vol 19, Number 1 March, 1988, pp. 82 - 98. Wolfe, Joseph and T. L Chacko, Team-Size Effects on Business Game Performance and Decision Making Behaviors, Deci- sion Science, Vol 14, 1983. pp. 121 133. Wolfe, J., & McCoy, R. (2008). Should business game players choose their teammates: A study with pedagogical implica- tions. Developments in Business Simulation and Experien- tial Learning, 35, 315-328. [Available from http:// www.absel.org] Wolfe, J., & McCoy, R. (2011). Those who do and those that don’t: A study of engaged and disengaged business game players. Developments in Business Simulation and Experi- ential Learning 38, 383-389. [Available from http:// www.absel.org] Wrzesien M. & Raya M.A. (2010) Learning in serious virtual worlds: evaluation of learning effectiveness and appeal to students in the E-Junior project. Computers & Education 55, 178–187. REFERENCES