The New Space Race: A Classroom Game about the Maturation Process of New Markets Page 322 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 INTRODUCTION Military leaders must marshal their staffs to prepare for future operations and anticipate challenges or opportunities that they may entail. Army leaders plan for future operations utilizing the Military Decision Making Process (MDMP). This detailed planning methodology entails a process of developing courses of action, testing those courses of action for viability, and publishing an order directing subordinate units to execute the selected course of action (Department of the Army, 2014; Department of the Army, 2012a). For a detailed description of the MDMP, refer to appendix G. A vital part of this process is testing courses of action for viability also known as the Course of Action Analysis commonly known in Army doctrine as the “wargaming” step of MDMP. The focus of this study was to examine the effect of simple paper and pencil role-playing games on improving performance during the wargaming step of MDMP. In this study, the paper and pencil role-playing game utilized was Kriegspiel, a 19th century Prussian wargame considered by many to be the first simulation of its type. Kriegpiel is an analog simulation. Specifically, the intent of this study was to document the perceived quality of wargaming for groups who played Kriegspiel prior to wargaming and compare those results with groups who did not. PROBLEM The problem that researchers in this study examined was related to the notion that quality wargaming is an art form that many Army staff members struggle to master. Combat Training Center trends spanning the last 20 years consistently document wargaming challenges faced by training units (Center for Army Lessons Learned, 1997; 1999; 2004; 2007; 2008; Combined Arms Center, 1999). Wargaming is an essential step for unit staffs to test and refine developed courses of action and visualize situations, both threats and opportunities, which may emerge in the future. Therefore, anticipation and foresight enabled by creativity and The Effect of Simple Role-Playing Games on the Wargaming Step of the Military Decision Making Process (MDMP): A Mixed Methods Approach Richard A. McConnell U.S. Army Command and General Staff College Mark Gerges U.S. Army Command and General Staff College John Dalbey U.S. Army Command and General Staff College Typhanie Dial U.S. Army Command and General Staff College George Hodge U.S. Army Command and General Staff College Marty Leners U.S. Army Command and General Staff College Joel Miller U.S. Army Command and General Staff College Jacob Mong U.S. Army Command and General Staff College Patrick Schoof U.S. Army Command and General Staff College Page 323 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 innovation are important aspects of wargaming. Experiences from 15 years of military operations indicate that operational environment complexity will only continue to increase. Therefore, improving wargaming skills for staffs will help them better identify challenges, opportunities, planning shortfalls, and unforeseen opponent options. Discovering the unforeseen during execution is a common occurrence during military operations. Improving the quality of the Course of Action Analysis (Wargaming) step of MDMP can potentially reduce the amount of surprises experienced during rehearsals and execution. Military leaders who seize opportunities to improve their subordinate’s wargaming abilities through gaming experience may have a higher success rate in their operations, as well as developing critical and creative thinking subordinate leaders than those who do not. PURPOSE The purpose of this mixed methods study was to determine the degree to which simple paper and pencil role-playing games (such as Kriegspiel) can improve the quality of wargaming and thus visualization skills. Paper and pencil role-playing games include a similar iterative process as wargaming; action, reaction, and counter reaction refined through adjudication. It is possible that such games can reinforce the understanding of seminal wargaming concepts by removing the modern complexities of warfare. The focus of this study was to determine if groups of staff members who play Kriegspiel prior to the wargaming step of MDMP, would understand the rudimentary elements of this process, improve their visualization skills, and thus experience more effective wargames. This purpose was accomplished by comparing student groups at the U.S. Army Command and General Staff College (CGSC) who played Kriegspiel prior to wargaming to peer groups who did not. HYPOTHESIS Student participants who play Kriegspiel (the test group) prior to the wargaming step of the MDMP will be more effective at visualizing their environment and thus will anticipate threats and opportunities more readily than student participants who do not (the control group). RESEARCH QUESTIONS R1. What was the level of student effectiveness at visualizing and describing key aspects of an operation if they have played Kriegspiel prior to the wargaming step of MDMP? R2. What was the level of student effectiveness at visualizing and describing key aspects of an operation if they have not played Kriegspiel prior to the wargaming step of MDMP? R3. How did faculty members describe the effectiveness of wargaming sessions performed by students who played Kriegspiel prior to the wargaming step of MDMP? R4. How did faculty members describe the effectiveness of wargaming sessions performed by students who did not play Kriegspiel prior to the wargaming step of MDMP? DEFINITIONS Visualization: the capability to combine observations of the environment with experience and education to identify clues to impending threats and opportunities. Visualization loosely correlates with intuition used to make decisions. Intuition skills in visualization can improve with practice and may include some subconscious aspects of thinking used to enable prudent decisions(Gladwell, 2007). Certainty: Certainty refers to the level to which players of games and decision makers can gain a belief in the certainty or lack thereof that they understand their environments (Tadelis, 2013). In regards to decision making, such certainty can assist a decision maker to address an emerging threat or opportunity thereby gaining an advantage and seizing the initiative over an opponent before the opponent can act (Department of the Army, 2012d). Common Knowledge: Common knowledge is the level to which a player of a game might believe that possible solutions to a problem encountered by opponents might experience higher of lower levels of difficultly based on the comparative knowledge and experience of the opponents. In relation to decision making, common knowledge is the level to which decisions might be easy or difficult based on the commonness to the players of the knowledge required to reach a reasonable decision (Tadelis, 2013). Environmental Complexity: Environmental Complexity describes the level to which game players and decision makers experience a range of complexity from static to dynamic. Static environments include simple variables that do not readily change whereas dynamic environments are complex, changing, and require constant monitoring for the need for new decisions (Tadelis, 2013). In regards to decision making, decisions in static environments may require one time only solutions whereas decisions in dynamic environments may call for multiple decisions requiring understanding of constant environmental change caused by multiple drivers and resisters (Lewin, 1943; Tadelis, 2013). Page 324 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Information: The information continuum in regards to game players and decision makers describes the level of completeness that might be perceived by gamer players and decision makers to act with a level of confidence that they have enough information to pursue reasonable actions (Tadelis, 2013). Rationality: Rationality is the level to which a player of a game might believe they are able to determine if their opponent is rational; what might they do or not do. In relation to decision making, rationality is the level to which decisions might be able to be logically reasoned or not (Tadelis, 2013). Time: Time presents a challenge for game players and decision makers as they attempt to reach a belief that they have been allowed adequate time in which to understand the environment. In regards to decision making, each decision maker engages in a process balancing all of the above variables given what they think is adequate time to make an informed decision (See figure 2) (Tadelis, 2013). Kreigspiel vs. Wargaming step of the Military Decision Making Process (MDMP). Kreigspiel is a simple board game that features chance and friction approximated using rolling dice, two-sided free play, etc. The wargame step of MDMP might appear to the civilian wargame community as a rehearsal using an umpire or a thought experiment designed to stress test a plan or determine the stronger option among competing hypotheses (see figure 2). For the purposes of this study, the term wargame will refer to the course of action analysis step of MDMP and the term Kriegspiel will refer to the simple turn based board game. BRIEF LITERATURE REVIEW Introduction The practice of testing plans using a wargame is not a new concept to the military institution. However, due to the frenetic pace of military operations over the last 15 years, perhaps the practice of wargaming has atrophied as a skill among military planners and leaders. What follows is a discussion of the literature and doctrine governing wargaming and supporting an examination of wargaming through simple role playing games. Content There are two competing doctrinal concepts that contribute to challenges faced by unit planners to conduct wargaming. These concepts are the art of command and the science of control (Department of the Army, 2014). Both concepts require critical and creative thinking. US Army doctrine balances these competing concepts through the principles, tasks, and roles defined as “Mission Command.” However, some leaders may emphasize the science of control more than the art of command. This may be because the science of control is one of the first things that military leaders learn and it is all about synchronization, coordination, and practical aspects of the operation. The science of control is also easier to grasp because it is concrete, objective, and relies on well established processes and procedures. Hence, the science of control may be the domain of staff officers who offer up their scientific/analytical products to the commander for decision-making. However, if all organizational leaders are to improve their visualization skills through wargaming the art of command is an essential skill for staff members and commanders alike. The art of command is the creative application of intuition, innovation, and foresight, which is much harder to teach and grasp. “Art" in its nature, does not conform to structure. It relies on experience, judgment, intuition, and of course is an area where commanders ultimately assume risk. Mastering the art of command requires experience and transmission from one generation to the next through mentorship and coaching. Conceptually, it sometimes follows that you must be a good "factory worker and scientist" before you can truly practice the art of command. For this reason, military leaders have historically used training exercises and combat experience to effectively mentor and coach subordinates how to apply the art of command. However, it is possible that since military operations in the last decade have been quick, repetitive, and focused on long term results, that time to share the skills necessary to successfully apply the art of command (requisite skills for wargaming) have atrophied. Numerous documents establish this challenge in wargaming. The United States Army keeps meticulous records of training trends for units attending training exercises especially during field training and computer assisted exercises. For the last twenty years, wargaming has been listed is a significant challenge for unit staffs observed (Center for Army Lessons Learned, 1997; 1999; 2004; 2007; 2008; Combined Arms Center, 1999). These combat training center reports list three significant challenges experienced by staffs attempting to execute the wargaming step of MDMP. First, some units skipped the wargaming step entirely. The last decade of persistent conflict included quickly changing situations with limited time to plan. Many commanders issued directed courses of action and subordinates simply skipped the wargaming step. This may have occurred because course of action comparison (where staff members develop recommendations regarding strengths and weaknesses between two competing courses of action) follows wargaming. If there is no course of action to compare to the directed course of action, some staffs may have assumed that a wargame was superfluous since the commander already decided which course of action to use. However, this assumption if false; the doctrinal term of the wargaming step of MDMP is actually titled, Course of Action Analysis. Regardless of the number of courses of actions considered, each are analyzed in order to improve the plan. Page 325 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Second, the rehearsal step of Preparation Activities in the Operations Process devolved into a wargame. The wargame is a vital opportunity to determine if courses of action (whether a single directed or multiple to be considered) are viable. The wargame may also be the first time during MDMP when all staff members come together with their running estimates to collaborate on a thought experiment criticizing their own plans. Wargaming exists for a purpose. Since some units skipped the wargame, the only time to address planning shortfalls or emerging threats and opportunities became the rehearsal just before execution. In such cases, instead of using the rehearsal for synchronization, of subordinate units and ensure they understand the commander's visualization, planners then waste time to address problems that they should/could have been fixed during wargaming. Finally, some units execute a wargame that appear to be the first iteration of the rehearsal. As planners at the unit level executed wargames less frequently, wargaming skills atrophied and the tenants of wargaming became less understood. Wargaming is the opportunity for planners to put on their lab coats and experiment with their courses of action against an uncooperative enemy traditionally role-played by the Intelligence Officer (S2). Wargaming is time-consuming, requiring planners to question assumptions, identify planning shortfalls, and use intuition to anticipate the unforeseen. Wargaming is a collaborative application combining the art of command with the science of control, emphasizing the former. Since the science of control is easier to teach and grasp, the art of command has suffered through the lack of use during the wargaming step of MDMP. Therefore, since rehearsals tend to emphasize the science of control, it should be not surprising that wargames look more like rehearsal’s than creative applications of the art of command. Perhaps one way to determine how to improve military planners wargaming skills is to examine their perceptions of wargaming. There are plenty of examples of studies using subject perceptions as a key metric. Holt, Bjorklund, and Green (2009) conducted a quantitative study of perceived expectations of leader qualities based on cultural and family norms. Augustijnen, Schnitzer, and Esbroeck (2011) conducted a qualitative study of executive coaching using grounded theory to determine perceptions of executives coached and what they found most effective in coaching received. Elston and Boniwell (2011) conducted a qualitative grounded theory study to determine perceptions of women coached to identify strengths and find ways to apply coached skills within the workplace. Cilliers (2011) conducted a qualitative case study of leaders within a large financial organization utilizing positive psychology leadership coaching. These studies are additional sources to provide FIGURE 1 The three cognitive planning domains. The figure depicts the three domains within which planners may operate while de- veloping courses of action. The factory produces Concepts of Operations (CONOPS) constrained by time, the laboratory analyzes plans for validity, and the art institute imagines and anticipates possibilities. Source: Richard McConnell. Page 326 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 deeper understanding of coaching and leadership development studies from perspectives of those receiving it, which may influence subject perceptions of wargaming. Additionally, and what may be more relevant to this research at CGSC, Fallesen (1995) conducted a qualitative study at CGSC of “practical thinking” instruction that was intended to address how the attitudes of students may influence their thinking and encourage students to take different perspectives while planning. Such instruction would be especially useful for planners and leaders executing the wargaming step of MDMP. Wargaming is an established planning process that FIGURE 2 The three cognitive planning domains applied to the scientific method and MDMP. The figure depicts the three cognitive domains as they relate to the scientific method and the Military Decision Making Process (MDMP). Source: Typhanie Dial & Rich- ard McConnell Page 327 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 encourages planners to take different perspectives and question their assumptions. What follows is a discussion of the cognitive theoretical model for this study combining planning domains that coalesce to facilitate the science of control and the art of command (see figure 1). The Cognitive Planning Domains Figure 1 depicts a visualization of three cognitive lenses through which planners and decision makers might view their environment. One-way to view planning and decision making is through the lenses of three planning domains: the Factory, the Laboratory and the Art Institute. In the Factory, planners swiftly address emerging situations by synchronizing and coordinating while focused on operational realities. These realities are usually constrained by time and are repetitious. Operations planners in the Factory planning domain meet needs quickly given time constraints. In the Laboratory, planners take more time to test ideas for feasibility, relevance and validity. Operations planners in the Laboratory put on their white lab coats to experiment with their plans to determine if they might work – this process takes longer than the factory. In the Art Institute, planners combine innovation, foresight, creativity, and imagination in the effort to meet needs through imagining and anticipating the unexpected or unforeseen. Operations planners in the Art Institute take more time that in the other planning domains engaging in a creative process. Where these three planning domains intersect becomes the confluence of the Art of Command and the Science of Control. These two doctrinal concepts provide a framework for understanding the origins of leader decisions. Scientists using the scientific method apply similar concepts to decision making. Figure 2 below provides a visual depiction of how the cognitive planning domains relate to the scientific method and MDMP. As depicted in Figure 2, MDMP supports decision-making in ways similar to the scientific method. The process of mission analysis in the formulation of a problem is similar to the theorizing that takes place in the scientific method that generally occurs in the cognitive planning domain of the Art Institute. The process of developing, analyzing, and comparing courses of action is similar to developing, testing, and determining the hypothesis with the most probable chance of success which occurs in the cognitive planning domain of the laboratory. Operations order production is similar to research protocol implementation leading to a clinical trial and takes place in the cognitive planning domain of the factory. Figure 2 above depicts three ways to look at decisions. Although the communities that might apply these cognitive paradigms are different, their methods for arriving at decisions have several similarities. Decision makers in these contexts may experience variables existing in the environment categorizing aspects of environmental complexity. FIGURE 3 The anatomy of a decision. The figure depicts variables influencing decisions made by rational actors as they balance levels of information available, certainty, rationality, given levels of environmental complexity, level of common knowledge required, and time. Source: Richard McConnell Page 328 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Theory Game theory provides a theoretical structure useful for describing actors that may compete or cooperate, decisions they may make, and the environments within which these actions take place. As the planning domains describe perspectives of planners as they attempt to address threats and opportunities within their environments, game theory provides descriptions of why planners might choose to operate within a given planning domain (See figure 3). As decision-makers engage in a process deciding how to address threats and opportunities within their environment, they examine numerous variables that might influence the level of depth and breadth they might seek for understanding their environment before acting. According to game theory, these variables include but are not limited to the completeness of information, the level of certainty, rationality, given the level of environmental complexity, level of common knowledge required to act, and time (Tadelis, 2013). These variables constitute a continuum where the personality and experience of the decision-maker determines when they might feel comfortable to act represented by the blue box depicted in figure 4. Some decision-makers might prefer to gain more information before acting. Other decision-makers might be comfortable with incomplete information if they feel that they have adequate rationality to address that lack of information. These variables therefore influence a rational actor’s decision to select a given planning domain. As rational actors evaluate their environment, they may examine (either consciously or unconsciously) how complete their information might be, the level of clarity that they have obtained, the level of rationality useful in addressing threats and opportunities. Rational actors may also evaluate the level of environmental complexity i.e. is the environment static or extremely dynamic? Rational actors also must address the level of common knowledge required for acting. For example, “Is the game being played more like checkers or is it a complex multiple role-playing game?” Finally, all decision-makers must be cognizant of the amount of time they have available in which to reach a decision yet still address the threats and opportunities in their environment before a decision is made for them. As depicted in figure 4, decision-makers operating within the factory might accept incomplete information and rationality as long as they feel more certain of their understanding of their environment that is more complete. Additionally, the decision-maker might feel that the actions required are common knowledge and time is short. All of these variables combine to place the decision in the red zone indicating the least complete information, certainty, and rationality given a level of environmental complexity, level of common knowledge required, and time. The resulting decision occurs within the red ring almost bordering on the yellow ring with the blue band of acceptance placed on the red region of the continuum. FIGURE 4 The anatomy of a decision applied to the planning domains. These figures figure depict variables influencing decisions made by rational actors as they decide which planning domain in which to operate given levels of information available, certainty, rationality, given levels of environmental complexity, level of common knowledge required, and time. Source: Richard McConnell Page 329 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 As also depicted in figure 4, decision-makers operating with in the laboratory planning domain might decide that they have a median or higher amount of information, certainty, rationality, and common knowledge which helps them to deal with a higher environmental complexity. Additionally, the decision maker may require more time in order to engage in the combination of critical and creative thinking required to operate within the laboratory planning domain. Of note, although the laboratory planning domain is a combination of critical and creative thinking, it emphasizes critical over creative reasoning. The resulting decision occurs within the yellow ring almost bordering on the green bull’s-eye with the blue band of acceptance placed on the yellow region of the continuum. Finally depicted in figure 4 is a portrayal of decision-makers operating within the Art Institute planning domain. These rational actors might decide that they have more information but less certainty and rationality as well as low environmental complexity and common knowledge levels but they have more time. The Art Institute planning domain combines critical and creative thinking but emphasizes the creative aspect. The resulting decision occurs within the yellow ring almost bordering on the green bull’s-eye with the blue band of acceptance placed on the yellow region of the continuum. The influence of these variables depicted in game theory provides useful theoretical lenses for investigators to view the participants in this study. Additionally, how participants in this study balanced these variables with unfolding realities during planning and execution was an important consideration. Military doctrine contains descriptions of a process used to address expected and unexpected variations of the plan that may unfold during implementation known as decisions during execution or the Rapid Decision and Synchronization Process (RDSP -- see figure 5 below) (Department of the Army, 2012c). Figure 5 depicts a methodology whereby published orders begin execution and if nothing changes, there is a straight line between the start of the operation and the envisioned and state. However, in military operations, this is seldom the case. Execution decisions dealing with opportunities and threats depict decisions that decision makers anticipated. Adjustment decisions dealing with threats and opportunities depict decisions that decision makers did not anticipate but became necessary based on variances dictated by the operational environment. The ability of planners and commanders to visualize is vital to this process. The rational actors within this examination of participant capabilities to visualize threats and opportunities within their environments will have to balance the above-described variables in order to propose sound options for decision-making. Therefore, a mixed methods approach is appropriate to provide rich descriptions of the environment under study. FIGURE 5 Decisions during execution/RDSP. This figure depicts the methodology used by military planners to identify both anticipat- ed and unanticipated threats and opportunities to enable decisions by commanders. Source: ADRP 5-0: The Operations Process. Page 330 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 METHODOLOGY AND DESIGN Introduction The design of the instruments used in this study were intended to facilitate understanding the level of proficiency demonstrated by participants to visualize their environment and apply that visualization to an effective wargaming step of the MDMP. What follows is a description of the methodology and design of this study applied to reach such an understanding of participant capabilities. Method and Design The method for this study was a mixed methods examination of student capabilities and faculty perceptions of how 16 person groups perform the wargaming step of MDMP with or without the playing Kriegspiel prior. The test group (n: 32) played Kriegspiel prior to the wargaming step of MDMP after completing an informed consent form (see appendix C); the control group (n: 79) did not play Kriegspiel but also completed an informed consent form for data collection during the wargaming step of MDMP (see appendix D). The test group consisted of two 16-person groups from two different 64-person student teams within the Command and General Staff College (CGSC). A 16-person staff group is a diverse set of students. The staff group will always include members from different branches of the US Army accounting for officers from conventional operations, force sustainment, operational support, and health service career fields. Additionally, staff groups will consist of at least one student from the US Air Force, at least one student from either the US Navy or Marine Corps, and at least one international military student. The Command and General Staff College also makes deliberate efforts to ensure that each staff group has at least one female officer, and one officer from a minority ethnicity. As such, staff groups are as representative as possible based on student assignments and attendance. The control group consisted of the balance of the 64-person student teaching team who did not play Kreigspeil. Faculty members not affiliated with grading the students completing the informed consent form distributed these forms. For example, a faculty member from Teaching Team One performed recruiting duties for Teaching Team Four and vice versa. Independent faculty members not involved with grading of participating students accomplished all data collection and held their data until faculty entered all grades for the US Army Doctrine Block (C400) of the Core Phase of the Command and General Staff Officer’s Course. After faculty completed grading for C400, then the independent faculty observers entered their data into an online survey and data collection system called Verint ®. Test and control group subjects were administered a post wargame visualization quiz designed to capture student visualization and description skills (See appendix A). Faculty observers had access to an additional online rubric, which enabled observers capturing faculty perceptions of the quality of student wargaming sessions. Faculty observers witnessed the wargaming sessions, recorded their perceptions of the quality of the wargaming sessions, and entered those perceptions using an online survey instrument (See appendix B). The identities were protected of all participants who completed the visualization quiz. The information regarding the nature, purpose, and design of the study was provided to all student participants both verbally and in the form of an informed consent form provided before any observations took place (See appendixes C and D). Kriegspiel The board game Kriegspiel dates back to the early 1800s in Prussia where faculty used it to teach members of the staff. Much of the game centers on a situation for each side of experiencing the ‘fog of war’. Two sides play against each other aided by running umpires that carry messages back and forth between the head umpire and player. Players see their unit represented on a map with small blocks of different shapes and coloring representing their role. During turns, the players write orders for their units or updates for other friendly players. Every two minutes or so their running umpire provides an update and collects new messages. The head umpire adjudicates movement and combat, provides outbound message traffic, and informs each participants running umpire of what they can see and what their force is doing. This manner of play forces the players sitting in seclusion from the rest of the players to deal with anticipating future requirements and analyzing how to accomplish their mission. Explicitly commanders learned that that orders are not executed immediately because there is a delay for the travel of the message and then a subsequent delay in the formation they command to respond to the order. Further, they will also gain an appreciation for concise mission-type orders. Regarding arbitration, there are two forms. First, the original form is strict adjudication. For strict adjudication, combat is resolved by looking up force ratios, rolling a die, consulting a table, and accounting for each loss on a register. In the second type, accounting is simplified using ratios, tables, and accounting to keep the game moving at a brisk pace. Potential Limitations to a Quantitative Survey Based Study Limitations to this study included the willingness of test subjects to participate in the study and although some participants elected not to participate, the response rate for the visualization quiz was 86%. Researchers mitigated this limitation by attempting to access student intrinsic motivations to help improve the quality of the military profession to which many of them have dedicated Page 331 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 on average a decade of their professional lives. Additionally, faculty perceptions of the quality of wargaming sessions may not have equated to actual quality. Researchers mitigated this limitation by the addition of experienced faculty observers whose observations provided meaningful comparisons. TABLE 1 VISUALIZATION QUIZ RESULTS, PART 1 COMPARISON BETWEEN RATERS 1 AND 2 Page 332 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Significance to Scholarship, Leadership, and Practice Many art forms are difficult to teach and wargaming is no exception. Effective wargaming includes creativity, critical thinking, innovation, intuition, and the ability to synthesize multiple conflicting inputs. Although academic instruction regarding wargaming can be useful, there is no substitute for experience tempered by mentorship and coaching. As many crafts use apprenticeship style coaching approaches, so also the mastering of effective wargaming can benefit from such an approach. As the complexities of the world situation continue to increase, anticipation of the unforeseen through effective visualization becomes ever more challenging. If the introduction of simple paper and pencil role-playing games can improve how staff officers perform the wargaming step of MDMP, then military leaders may have a useful and cost-effective tool for leader development of this vital skill RESULTS Introduction The results of this study generally addressed the problem of improving wargaming quality within the military profession. Researchers in this study were able to support the hypothesis in that generally participants who played Kriegspiel displayed statistically significant increased capabilities to visualize especially in regards to displaying the ability to visualize themselves. Researchers were able to generally support research questions R1-4 because analysis of the data provided descriptions of the differences between the test and control groups’ ability to visualize and describe as well as faculty observer perceptions of wargame quality. In general, test group participants visualized themselves more accurately than control group participants did. Additionally, faculty observations of wargame results supported the test group visualization capabilities by suggesting that test group participant Statistically significant differences in question scoring for questions 1-5 using Friedman’s two-way analysis of variance by ranks. Grader 1 judged respondents more strictly overall in all of these cases. Rater 1 found statistically significant differences in responses to question 5 according to the Mann- Whitney-Wilcoxon test. Test group respondents scored better. Rater 1 profiles Question Category N Mean Rank No answer (%) Attempt (%) Partially correct (%) Correct (%) Q1A Test 32 61.27 44 28 22 6 Control 77 52.40 55 34 6 5 Q2A Test 32 59.84 66 13 19 3 Control 77 52.99 74 22 3 1 Q3A Test 32 56.17 69 16 13 3 Control 77 54.51 75 8 0 17 Q4A Test 32 57.97 75 19 3 3 Control 77 53.77 83 12 0 5 Q5A* Test 32 64.16 16 31 47 6 Control 77 51.19 27 47 14 12 Q6A Control 77 55.95 45 1 0 53 Test 32 52.72 44 3 13 41 *Statistically significant difference according to Mann-Whitney-Wilcoxon test, α = 0.05 Rater 2 profiles** Question Category N Mean Rank No answer (%) Attempt (%) Partially correct (%) Correct (%) Q1B Test 32 56.38 47 19 28 6 Control 77 54.43 48 25 19 8 Q2B Control 77 55.74 53 9 31 6 Test 32 53.22 56 16 19 9 Q3B Control 77 56.40 57 18 5 19 Test 32 51.64 63 16 19 3 Q4B Control 77 55.95 71 13 10 5 Test 32 52.70 78 6 13 3 Q5B Test 32 59.06 19 25 50 6 Control 77 53.31 21 39 30 10 Q6B Control 77 56.08 47 0 0 53 Test 32 52.39 47 3 9 41 **No statistically significant difference according to Mann-Whitney-Wilcoxon test, α = 0.05 Note: Statistically significant differences are differences that would be difficult to explain by chance alone when taking into account sample sizes and response variability. TABLE 2 VISUALIZATION QUIZ RESULTS, PART 2 COMPARISON BETWEEN RATERS 1 AND 2 Page 333 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 wargames were significantly more effective in identifying emerging opportunities during their wargames. What follows is a detailed description of the results determined using the data collection instruments of a visualization quiz and a faculty wargame observer record (see appendixes A and B). Student Visualization Quiz Results The student visualization quiz was administered by faculty members not involved in teaching C400 to the student participants. The students were provided a visual display of the sketches depicted in appendix A using a PowerPoint presentation and two 39 by 69 inch video screens at the front of the classroom. Each visualization quiz session was administered by staff group consisting of 16 student participants at a time. The results of the visualization quiz were scored by two different faculty members who were investigators in the study. The scoring process was qualitative requiring scores/raters to use their judgment. The methodology for scoring each answer follows. No answer or wrong answer = 0, OK answer = 1, Good answer = 2, and Perfect answer = 3. Researchers employed a statistical research software called International Business Machines Statistical Package for the Social Sciences (IBMSPSS) to measure statistical significance of the results of the student visualization quiz. Researchers used two statistical tests using IBMSPSS to evaluate these results. Specifically, the Mann-Whitney-Wilcoxon Test combined with the Friedman test. Table 1 depicts those results as produced by IBMSPSS. Researchers chose the Mann-Whitney-Wilcoxon test for analyzing the results of the student visualization quiz because of the numerous data points, two different raters, and the potential for outliers that this test takes into account. Researchers chose the Friedman test to simply compare rater one and rater two to ascertain the level of rigor each rater applied to determining the results. As depicted above, the Friedman test was used to establish differences between the control and test groups that may not have been statistically significant yet were notable as well as establishing differences between how rater one and rater two scored the visualization quiz. Notably, as indicated in the IBMSPSS report above, “statistically significant differences in scoring for questions 3, 5, 6, and 7 using Friedman’s two-way analysis by ranks. Rater one judged respondents more strictly overall in all cases.” This finding is significant because although rater one found statistically significant increases of the test group over the control and rater 2 did not, overall rater one graded more strictly. Such a finding may suggest that rater one’s determination of statistical significance was not a result of a bias to grade leniently. The Mann-Whitney-Wilcoxon test returned results in part one of the student visualization quiz (see appendix A) that showed statistical significance for question six: what friendly elements are operating on the near side of the river? The finding of statistical significance through the Mann-Whitney-Wilcoxon test indicates that these results would be “difficult to explain by chance alone when taking into account sample size and response variability” as indicated in the IBMSPSS report above) (Hogg & Craig, 1978) . An additional interpretation of this finding of statistical significance is connected to the a=0.05 value used in the Mann- Whitney-Wilcoxon test which would indicate with a 5% significance level that this result has a one in 20 probability that this finding is a result of chance (“Statistical significance Mcgraw-Hill Concise Dictionary of Modern Medicine,” 2002). These findings partially support this hypothesis because it addresses the student participant’s ability to visualize themselves in the context of their operational environment. Specifically, the test group demonstrated a statistically significant increase of visualization skills over the control group for visualizing their own situation in the context of their operational environment. This finding is also significant because it TABLE 3 Part 1 Similarities and differences rater one and rater two: statistical significance versus notable. Page 334 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 addresses research question R1: What is the level of student effectiveness at visualizing and describing key aspects of an operation if they have played Kriegspiel prior to the wargaming step of MDMP? Researchers addressed findings regarding student participant’s ability to anticipate threats and opportunities in the discussion of faculty wargame observer results later in the section. These findings of statistical significance using the Mann-Whitney-Wilcoxon test continued as a trend for rater one in part two of the student visualization quiz depicted in table 2 below. As depicted above, rater one graded the student visualization quiz returning results of statistical significance using the Mann -Whitney-Wilcoxon test specifically in regards to question 5A: what is the size, composition, and scheme of maneuver of the friendly main effort? This finding is significant because it partially supports the hypothesis that student participants who play Kriegspiel (the test group) prior to the wargaming step of the MDMP will be more effective at visualizing their environment and thus will anticipate threats and opportunities more readily than student participants who do not (the control group). These findings partially support this hypothesis because it addresses the student participant’s ability to visualize themselves in the context of their operational environment. Researchers determined findings regarding student participant’s ability to anticipate threats and opportunities in the discussion of faculty wargame observer results later in the section. This finding is also significant because it addresses research questions R1 and 2: regarding the level of student effectiveness at visualizing and describing key aspects of an operation if they had or had not played Kriegspiel prior to the wargaming step of MDMP. See the discussion of faculty wargame observer results later in the section for findings regarding student participant’s ability to anticipate threats and opportunities. Prior to discussing the game theory variables results, it is important to analyze some of the differences and similarities between rater one and rater two especially in regards to differences between the test and control groups that were not statistically significant but were notable. Results depicted in table 3 below. Table 3 depicts the results of the student visualization quiz by rater for part one showing the results for the test group alongside the results for the control. The row that is colored beige for question six shows that that result is statistically significant as determined through the Mann-Whitney-Wilcoxon test, which was discussed in the previous section. Although not statistically significant, there are other questions that show notable increases between the test and control groups. For example, questions four and five show a mean increase between the test and control groups of 9.58 and 5.46 respectively. Question 4, “What enablers does the enemy have to support their operations? Question 5: “What is the enemy unit size and location of its reserve across the river?” Although not statistically significant, these results are notable and partially supports the hypothesis as well as research questions R1 and 2. When compared to rater two, questions five and six show notable increase between the test and control of 4.64 and 9.09 respectively. This finding is significant because it partially supports the hypothesis as well as research questions R1 and 2. Additionally, the mean increase of 9.09 for question six supports the findings determined for rater one that found the increase between the test and the control group statistically significant using the Mann-Whitney-Wilcoxon test. Researchers observed similar trends in the findings in part two of the student visualization quiz depicted in table 4 below. Table 4 depicts the results of the student visualization quiz by rater for part two showing the results for the test group alongside the results for the control. Question 5A for rater one is depicted in beige indicating statistical significance using the Mann- Whitney-Wilcoxon test as discussed in the previous section. Although not statistically significant, there are some notable increases between the test and control groups for questions 1A and 2A of 8.87 and 6.85 respectively. Question 1A: what is the designation of enemy higher headquarters? Question 2A: what types of units does the enemy have across the river? Although not statistically significant, these results are notable because they support the hypothesis and research questions R1 and 2. For rater two there is one TABLE 4 PART 2 SIMILARITIES AND DIFFERENCES RATER ONE AND RATER TWO: STATISTICAL SIGNIFICANCE VERSUS NOTABLE. Page 335 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 notable result for question 5B: what is the size, composition, and scheme of maneuver of the friendly main effort? This finding is significant not only because it is notable and supports the hypothesis as well as research questions R1 and 2 but also that it partially replicates the finding determined by rater one. Rater one findings returned the result of statistical significance for question 5A. Although rater two did not return results of statistical significance, a mean increase between the test and control of 5.75 is notable and suggests a similar trend of the test group being better able to visualize themselves in the context of their operational environment. These findings for part one and two of the student visualization quiz may suggest that the test group exhibited more confidence in their responses. The next section describing the findings in the visualization quiz using game theory variables will discuss how students rated their experience answering the questions in the visualization quiz. Game Theory Variables Visualization Quiz Results As discussed earlier, each student participant in the visualization quiz, were given an opportunity to rate their experience using game theory variables (see appendix A). Those variables were information, certainty, rationality, environmental complexity, common knowledge, and time. Researchers employed a Likert scale of 1 to 6 ranging from a low level to a high level to the students and asked them to assess the decisions they made in forming their answers. Information, certainty, rationality, common knowledge, and time all our variables that suggest a high number might indicate increased confidence in participant decisions. For example, for the first variable information, if a student felt they had a low amount of information to make their decision they might have less confidence than a participant who had a score of six. However, environmental complexity has an inverse relationship i.e. participants who felt the environmental complexity was low might be an indication of an increase in confidence. During analysis of the data, investigators discovered that the game theory variable of environmental complexity may have been a confounding variable. It appeared that participants who rated the other game theory of variables high would continue that trend when they came to environmental complexity. Therefore, the decision was made to exclude this category as it appeared to skew the numbers as a confounding variable. Additionally, one subject appeared to select the highest number of 6 for confidence level using the game theory variables, but did not answer any of the questions. Therefore, researchers excluded this sample to prevent skewing of the numbers as a confounding variable. What follows is a brief summary of the analysis of the Game theory variables section of the visualization quiz drawn from detailed statistical reports (See appendix E). Using the 1-tailed Mann-Whitney Test, where H0: T = C and H: T > C (all p values are adjusted for ties) STATISTICALLY SIGNIFICANT: 1. The score for the Common Knowledge variable in Part 2 was higher for the Test Group than for the Control Group (W=3559.5, p=0.0376). THOUGH NOT STATISTICALLY SIGNIFICANT, NOTABLE RESULTS 1. The score for the Information variable in Part 1 was higher for the Test Group than for the Control Group (W=3456, p=0.1944). 2. The score for the Certainty variable in Part 1 was higher for the Test Group than for the Control Group (W=3685, p=0.2030). 3. The score for the Rationality variable in Part 1 was higher for the Test Group than for the Control Group (W=3661, p=0.1029). 4. The score for the Common Knowledge variable in Part 1 was higher for the Test Group than for the Control Group (W=3709, p=0.1835). As described above, test group participants rated their experience in part 2 of the visualization quiz as an experience drawing on common knowledge to a statistically significant level over their control group peers. Additionally, though not statistically significant, it is notable that for part 1 that test group participants scored higher than the control in the game theory variables of information, certainty, rationality, and common knowledge. Analysis of these results suggest that test group participants believed they had enough information, a higher level of certainty, confidence in their level of rationality, and saw these questions as drawing on common knowledge to a notably higher level than their control group peers. What follows is an evaluation of these results. Evaluation of Student Visualization Quiz Results The quantitative and qualitative analysis of the data derived from student visualization quiz partially supports the hypothesis: student participants who play Kriegspiel (the test group) prior to the wargaming step of the MDMP will be more effective at visualizing their environment and thus will anticipate threats and opportunities more readily than student participants who do not (the control group). The portion of the hypothesis supported is the part discussing participant capability to visualize their environment. The student visualization quiz returned results that support the claim that test group participants were better than control group participants were at visualizing their environment. The Mann-Whitney-Wilcoxon test established statistical significance for test group participants increased capability over the control group at seeing themselves in the context of their operational environment. Additionally, although not statistically significant, there were findings that were notable. Analysis of this data suggests that test group participants were better than the control group at visualizing certain aspects of the enemy such as types Page 336 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 of units the enemy may have which partially supports the hypothesis regarding anticipating threats. The visualization quiz results also supported research questions R1 and R2, which focused on determining student levels of effectiveness at visualizing and describing operations whether they had played or not played Kriegspiel. The game theory variable portion of the student visualization quiz also indicated a statistical significance to the increase of the test group over the control in confidence level for their decisions using the game theory variable of Common knowledge. Although not statistically significant, it was notable that test group participants scored their experience higher than control for information, certainty, rationality, and common knowledge. This portion of the study focused on the visualization capabilities and confidence level of student participants who played Kriegspiel or did not. The analysis of the data suggests that test group participants were more effective at seeing themselves in the context of their operational environment to a statistical significance and had a notable increase over the control group in visualizing certain aspects about the enemy while feeling more confident than the control group answering the questions. These results not only suggest that visualization practice provided to the test group through playing Kreigspiel may have improved their performance but also their confidence in their ability to visualize. Such results might imply that visualization is a skill that can improve through practice. The next section will provide descriptions if this student capability to visualize as indicated by the visualization quiz resulted in a more effective wargaming step of the MDMP. What follows is the faculty observations of the level of quality of the wargames conducted as part of the MDMP by the test and control groups. Faculty Wargame Observer Results During the C400 lesson block at the U.S. Army Command and General Staff College (CGSC) students were expected to execute a wargaming step during the Military Decision Making Process (MDMP). The test group played Kriegspiel the week prior to the wargaming step. Faculty members, who were not the instructors of the groups that they observed, utilized a quantitative instrument to evaluate the quality of the wargame that they observed (see appendix B). This quantitative instrument was designed to assist observers in counting student discoveries from different perspectives and applying them to their plans. Scores for test and control group were subjected to the Friedman test to determine statistical significance of the outcomes between the test and the control groups. The Friedman test was selected because this quantitative instrument contained seven datasets with a reduced chance for outliers that might skew the data. What follows is three specific instances that the Friedman test returned statistically significant results. FRIEDMAN TEST: C10 VERSUS C11 BLOCKED BY C12 Q37 ALL Question 3: Did wargamers integrate multiple WFF perspectives while addressing threats and opportunities? S = 17.92 DF = 7 P = 0.012 S = 26.22 DF = 7 P = 0.000 (adjusted for ties) For question three of the faculty observer wargaming survey, the Friedman test returned statistically significant results that the test group was better at integrating multiple war fighting functions while addressing threats and opportunities. As indicated above, the Friedman test returned results of P= 0.012 and P=0.000 both of which are less than a=0.05 indicating statistical significance. This finding is significant because it partially supports the hypothesis in regards to participant’s ability to anticipate threats and opportunities. Additionally, this finding addressed research questions R3 and 4. However, during analysis of the data it was discovered that one 16-member student group observer applied a different protocol to capturing data observed. This was control group number three or C3. Therefore, investigators decided to repeat the Friedman test while excluding C3. FRIEDMAN TEST: C10 VERSUS C11 BLOCKED BY C12 Q37 W/OUT C3 Question 3: Did wargamers integrate multiple WFF perspectives while addressing threats and opportunities? S = 14.14 DF = 6 P = 0.028 S = 21.50 DF = 6 P = 0.001 (adjusted for ties) For question three of the faculty observer wargame survey without C3, the Friedman test returned results of P=0.028 and P=0.001 both of which are less than a=0.05 indicating statistical significance. This finding is significant because it partially supports the hypothesis in regards to test group participant ability to anticipate threats and opportunities. These findings also addressed research questions R3 and 4. Additionally, these findings are significant because they generally correlate with findings determined from the student visualization quiz. Analysis of the data collected from the student visualization quiz indicated that test group participants were better than the control group at seeing themselves in the context of their operational environment to a level of statistical significance. The visualization quiz also returned results that suggest that test group participants were better than control group participants were at visualizing certain aspects about the enemy. These visualization quiz findings generally correlate with the faculty observer wargaming survey findings that test group subjects were better than the control group at addressing threats and opportunities. The analysis of the data suggests that addressing threats and opportunities might also require coordination between different perspectives. What follows is an additional dataset regarding coordination that returned statistical significance through the Friedman test. Friedman Test: C10 versus C11 blocked by C12 Q27A w/out C3 Question 4: How difficult was coordination among wargamers when one or more WFFs were integrated during the wargame? Page 337 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 S = 14.04 DF = 6 P = 0.029 S = 16.96 DF = 6 P = 0.009 (adjusted for ties) For question four of the faculty observer wargaming survey, the Friedman test returned results of P=0.029 and P=0.009 both of which are less than a= 0.005 indicating statistical significance. This finding is significant because it generally supports the hypothesis but specifically addresses research questions R3 and 4. In this case, test group participants experienced less difficulty coordinating one or more war fighting functions during the wargame than control group participants. Evaluation of Faculty Wargame Observer Results The faculty wargame Observer survey partially supported the hypothesis especially in regards to the aspects of anticipating threats and opportunities. This instrument also addressed research questions R3 and 4. Additionally, the faculty wargame Observer survey demonstrated a general correlation to analysis of data collected during the student visualization quiz. Specifically, the visualization quiz showed that test group participants exceeded control group participant capability to see themselves in the context of their operational environment to a level of statistical significance. The visualization quiz also had, although not statistically significant, notable improvements between the test and the control group at visualizing certain aspects of the enemy. There is a general correlation between the visualization quiz results and the faculty wargame observer results. That general correlation manifested itself in demonstrated ability of test group to see oneself and see the enemy effectively may make test group participants more able to address threats and opportunities. Variations in Instrumentation During Study As described earlier, researchers designed both data collection instruments for delivery using a survey program called Verint ®. Researchers experienced difficulty using this survey system for both instruments requiring adjustments to the data collection plan. For the visualization quiz, difficulty creating a technological solution that displayed an image for 60 seconds and then displayed the questions was not possible so researchers employed an analog solution. Researchers displayed the image using a PowerPoint slide displayed for the group for 60 seconds and provided paper questions to the participants. The instruments were then manually scored by two different scorers and the results were entered into excel spreadsheets for further analysis. For the Faculty observer wargame report, although the faculty observers were able to enter their results into Verint ®, the system could not display test and control groups as different data sets. However, Verint ® was able to provide and excel spreadsheet containing the faculty observation reports that were then used for analysis purposes. For future research efforts, which may include larger sampling groups, solving these technological challenges will be essential since manual entry introduces the possibility of transposition errors and slows data analysis especially for larger sampling populations. IMPLICATIONS, RECOMMENDATIONS, CONCLUSIONS Introduction The purpose of this study was to address an historical challenge within the military institution of practicing an effective wargaming step of the MDMP by observing to see if there might be a correlation between playing a simple role-playing game and the effect it might have on planner’s wargaming efforts. Analysis of the data collected in this study suggests that there is a correlation between playing simple role-playing games such as Kriegspiel, which would then make planners more effective at Course of Action Analysis (wargaming) during the Military Decision Making Process (MDMP). Specifically, participants who played Kriegspiel demonstrated a statistically significant increased capability to see themselves in the context of their operational environment while addressing threats and opportunities and integrating those discoveries across war fighting functions. What follows are some general observations regarding the implications, recommendations, and conclusions emerging from this study. Implications The literature review of this research report established that Combat Training Center reports from the last 20 years establish wargaming as an ongoing challenge for units to practice effectively. Units either skip the wargame, turned the combined arms rehearsal into a wargame, or performed a “wargame” that looked more like the first iteration of the rehearsal. This trend may be a result of a cultural norm that emerged in the last 15 years because of the urgency of ongoing conflicts resulting in directed courses of action. As a possible result, many young leaders have not had effective wargaming modeled for them. Consequently, many of these leaders attend Combat Training Centers and inadequately prepared to visualize their operational environment, address emerging threats and opportunities, and integrate those discoveries into their operational plans. Analysis of the data in this study suggests that commanders at Battalion, brigade, and division might be able to improve subordinate leaders visualization capabilities by adopting leader development programs that include simple role playing board games such as Kriegspiel or others like it. As discussed in the literature review section of this research report-using figure 1, the three planning domains, perhaps playing simple role playing board games might assist leaders in improving their ability to balance the art of command and the science of control by improving their ability to operate in all three planning domains. The military institution has struggled with wargaming for at least the last 20 years. The implication of this challenge is that military leaders and planners may be deficient at visualizing, describing, and directing because they fail to see themselves effectively in the context their operational environment and the threats and opportunities they Page 338 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 might face. Simple role playing board games might be a cost effective and simple way for commanders and planners to address this institutional challenge. RECOMMENDATIONS For Commanders and military planners The chief recommendation is for commanders and planners is to wargame. Take the time to analyze deliberately courses of action (COA) by “wargaming” as part of the MDMP process. All too often, planners skip the wargaming step entirely, and operations over the last ten years have only contributed to the atrophy of this skill in military planning. Wargaming provides commanders and staffs a method to analyze and compare courses of action against one another while testing the validity of the courses of action against an uncooperative and thinking enemy. This “test” helps commanders and staffs to identify gaps in planning, synchronize COA events in time and space, identify previously undiscovered threats and opportunities, and ultimately identify and think through potential decisions commanders may be required to make in the execution of the fight. If planners skip or water down the wargaming step, then the gaps and synchronization will only become evident during rehearsals or worse, in execution. Wargame regardless of the number of COAs: When the commander gives a directed COA to staffs there is only one option for execution, therefore in a time constrained environment, the perceived need to wargame only one COA may seem like a waste of time. After all the sixth step of MDMP is “Course of Action Comparison” so why wargame if there is only one COA to compare? If COAs are perfectly developed with no gaps in understanding or synchronization then the wargaming step would seem to be a waste of time. However, no matter how skilled planners are, COAs can always be refined, and staffs must test them for the reasons identified in the paragraph above. Again, if staffs fail to wargame then the gaps in understanding and synchronization will only become evident when it may be too late. Consider role-playing games as staff and officer professional development activities in order to increase commander and staff visualization abilities. This study used Kriegspiel and found it to have a correlation with increased visualization particularly with planners abilities to better understand and visualize their own units on the battlefield. Other games and techniques may be useful to facilitate this end. Other options include GO ™ (Ancient Chinese strategy game), Stratego ™, Hunabi ™, and simple visualization exercises of having subordinates draw out their understanding of the operation on a white board or even in the dirt. This capability will help staff officers envision operations and enable commander and unit understanding. For researchers and scholars Analysis of the results of this study suggest that further study of the effect of simple role playing board games on the wargaming step of MDMP is warranted. Therefore, the investigators of this study recommend an ongoing longitudinal examination of this phenomenon at CGSC to see if researchers can replicate or expand upon these results. This study has yielded several lessons that would be useful if applied to further study. What follows is a discussion of some of those recommended improvements to the research protocol of this effort. Regarding scoring of the visualization quiz, the two raters that scored the visualization quiz used the following scoring protocol. A wrong answer or no answer both = 0, OK = 1, Good = 2, Perfect = 3. Researchers recommended in a subsequent study of this phenomenon, the following scoring protocol for the visualization quiz. No answer = 0, a wrong answer = 1, OK = 2, Good = 3, Perfect = 4. Researchers recommended this approach because a failure to try may be indicative of a lack of confidence whereas attempting an answer although failing to answer it correctly is not the same as refusing to attempt to answer it. Researchers deemed the above scoring protocol better because it was more representative of the reality of the phenomenon under investigation. Scoring the game theory variables: As discussed in the finding section the game theory, researchers excluded the variable of environmental complexity because it was a confounding variable. If researchers decided to use the variable of environmental complexity in the future, they should inform participants its inverse relationship to the other variables. Additionally, researchers recommended improvements the game theory variable section of this study by replacing the one-word descriptors with a question. For example, instead of just giving information as a category and a scale of 1 to 6 as possibilities, a question such as, “Did you feel you had enough information to make a decision? Rate your level of information 1 to 6, six being the most information possible for warranting a decision.” Such a change to the game theory variable section of this protocol might assist participants in understanding the purpose of this section and prevent the occurrence of confounding variables. Faculty observer wargaming instrument: As described in the finding section of this research report one of the control groups was excluded from the data due to an alternative protocol. This protocol assisted control group number three (C3) in visualizing, describing, and directing the wargame by recording discoveries made by war fighting function, recording the difficulty, and categorizing the solutions using a table (see table 6 below). Applying this protocol to further studies of wargaming might improve wargaming for test and control groups alike thus making improving commander and plane are visualization skills as a form of leader development. For question three of the faculty observer wargame survey without C3, the Friedman test returned results of P=0.028 and P=0.001 both of which are less than a=0.05 indicating statistical significance. This finding is significant because it partially supports the hypothesis in regards to test group participant ability to anticipate threats and opportunities. These findings also addressed Page 339 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 research questions R3 and 4. Additionally, these findings are significant because they generally correlate with findings determined from the student visualization quiz. Analysis of the data collected from the student visualization quiz indicated that test group participants were better than the control group at seeing themselves in the context of their operational environment to a level of statistical significance. The visualization quiz also returned results that suggest that test group participants were better than control group participants were at visualizing certain aspects about the enemy. These visualization quiz findings generally correlate with the faculty observer wargaming survey findings that test group subjects were better than the control group at addressing threats and opportunities. The analysis of the data suggests that addressing threats and opportunities might also require coordination between different perspectives. What follows is an additional dataset regarding coordination that returned statistical significance through the Friedman test. Friedman Test: C10 versus C11 blocked by C12 Q27A w/out C3 Question 4: How difficult was coordination among wargamers when one or more WFFs were integrated during the wargame? S = 14.04 DF = 6 P = 0.029 S = 16.96 DF = 6 P = 0.009 (adjusted for ties) For question four of the faculty observer wargaming survey, the Friedman test returned results of P=0.029 and P=0.009 both of which are less than a= 0.005 indicating statistical significance. This finding is significant because it generally supports the hypothesis but specifically addresses research questions R3 and 4. In this case, test group participants experienced less difficulty coordinating one or more war fighting functions during the wargame than control group participants. Evaluation of Faculty Wargame Observer Results The faculty wargame Observer survey partially supported the hypothesis especially in regards to the aspects of anticipating threats and opportunities. This instrument also addressed research questions R3 and 4. Additionally, the faculty wargame Observer survey demonstrated a general correlation to analysis of data collected during the student visualization quiz. Specifically, the visualization quiz showed that test group participants exceeded control group participant capability to see themselves in the context of their operational environment to a level of statistical significance. The visualization quiz also had, although not statistically significant, notable improvements between the test and the control group at visualizing certain aspects of the enemy. There is a general correlation between the visualization quiz results and the faculty wargame observer results. That general correlation manifested itself in demonstrated ability of test group to see oneself and see the enemy effectively may make test group participants more able to address threats and opportunities. Variations in Instrumentation During Study As described earlier, researchers designed both data collection instruments for delivery using a survey program called Verint ®. Researchers experienced difficulty using this survey system for both instruments requiring adjustments to the data collection plan. For the visualization quiz, difficulty creating a technological solution that displayed an image for 60 seconds and then displayed the questions was not possible so researchers employed an analog solution. Researchers displayed the image using a TABLE 6 RECOMMENDED CHANGE TO FACULTY WARGAME OBSERVER PROTOCOL Page 340 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 PowerPoint slide displayed for the group for 60 seconds and provided paper questions to the participants. The instruments were then manually scored by two different scorers and the results were entered into excel spreadsheets for further analysis. For the Faculty observer wargame report, although the faculty observers were able to enter their results into Verint ®, the system could not display test and control groups as different data sets. However, Verint ® was able to provide and excel spreadsheet containing the faculty observation reports that were then used for analysis purposes. For future research efforts, which may include larger sampling groups, solving these technological challenges will be essential since manual entry introduces the possibility of transposition errors and slows data analysis especially for larger sampling populations. IMPLICATIONS, RECOMMENDATIONS, CONCLUSIONS Introduction The purpose of this study was to address an historical challenge within the military institution of practicing an effective wargaming step of the MDMP by observing to see if there might be a correlation between playing a simple role-playing game and the effect it might have on planner’s wargaming efforts. Analysis of the data collected in this study suggests that there is a correlation between playing simple role-playing games such as Kriegspiel, which would then make planners more effective at Course of Action Analysis (wargaming) during the Military Decision Making Process (MDMP). Specifically, participants who played Kriegspiel demonstrated a statistically significant increased capability to see themselves in the context of their operational environment while addressing threats and opportunities and integrating those discoveries across war fighting functions. What follows are some general observations regarding the implications, recommendations, and conclusions emerging from this study. Implications The literature review of this research report established that Combat Training Center reports from the last 20 years establish wargaming as an ongoing challenge for units to practice effectively. Units either skip the wargame, turned the combined arms rehearsal into a wargame, or performed a “wargame” that looked more like the first iteration of the rehearsal. This trend may be a result of a cultural norm that emerged in the last 15 years because of the urgency of ongoing conflicts resulting in directed courses of action. As a possible result, many young leaders have not had effective wargaming modeled for them. Consequently, many of these leaders attend Combat Training Centers and inadequately prepared to visualize their operational environment, address emerging threats and opportunities, and integrate those discoveries into their operational plans. Analysis of the data in this study suggests that commanders at Battalion, brigade, and division might be able to improve subordinate leaders visualization capabilities by adopting leader development programs that include simple role playing board games such as Kriegspiel or others like it. As discussed in the literature review section of this research report-using figure 1, the three planning domains, perhaps playing simple role playing board FIGURE 6 Decisions during execution/RDSP and cognitive planning domains. This figure depicts the RDSP model with the cognitive planning domains of the factory, laboratory, and art institute with planning horizons and planning processes such as ADM and MDMP. Source: Typhanie Dial & Richard McConnell. Page 341 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 games might assist leaders in improving their ability to balance the art of command and the science of control by improving their ability to operate in all three planning domains. The military institution has struggled with wargaming for at least the last 20 years. The implication of this challenge is that military leaders and planners may be deficient at visualizing, describing, and directing because they fail to see themselves effectively in the context their operational environment and the threats and opportunities they might face. Simple role playing board games might be a cost effective and simple way for commanders and planners to address this institutional challenge. RECOMMENDATIONS For Commanders and military planners The chief recommendation is for commanders and planners is to wargame. Take the time to analyze deliberately courses of action (COA) by “wargaming” as part of the MDMP process. All too often, planners skip the wargaming step entirely, and operations over the last ten years have only contributed to the atrophy of this skill in military planning. Wargaming provides commanders and staffs a method to analyze and compare courses of action against one another while testing the validity of the courses of action against an uncooperative and thinking enemy. This “test” helps commanders and staffs to identify gaps in planning, synchronize COA events in time and space, identify previously undiscovered threats and opportunities, and ultimately identify and think through potential decisions commanders may be required to make in the execution of the fight. If planners skip or water down the wargaming step, then the gaps and synchronization will only become evident during rehearsals or worse, in execution. Wargame regardless of the number of COAs: When the commander gives a directed COA to staffs there is only one option for execution, therefore in a time constrained environment, the perceived need to wargame only one COA may seem like a waste of time. After all the sixth step of MDMP is “Course of Action Comparison” so why wargame if there is only one COA to compare? If COAs are perfectly developed with no gaps in understanding or synchronization then the wargaming step would seem to be a waste of time. However, no matter how skilled planners are, COAs can always be refined, and staffs must test them for the reasons identified in the paragraph above. Again, if staffs fail to wargame then the gaps in understanding and synchronization will only become evident when it may be too late. Consider role-playing games as staff and officer professional development activities in order to increase commander and staff visualization abilities. This study used Kriegspiel and found it to have a correlation with increased visualization particularly with planners abilities to better understand and visualize their own units on the battlefield. Other games and techniques may be useful to facilitate this end. Other options include GO ™ (Ancient Chinese strategy game), Stratego ™, Hunabi ™, and simple FIGURE 7 Cognitive planning domains and the culture between commanders, planners, and subordinate units. This figure depicts a potential cultural perspective of how different groups might interact given the cognitive planning domains, planning horizons, and the perspectives of each group. Source: Typhanie Dial & Richard McConnell Page 342 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 visualization exercises of having subordinates draw out their understanding of the operation on a white board or even in the dirt. This capability will help staff officers envision operations and enable commander and unit understanding. For researchers and scholars Analysis of the results of this study suggest that further study of the effect of simple role playing board games on the wargaming step of MDMP is warranted. Therefore, the investigators of this study recommend an ongoing longitudinal examination of this phenomenon at CGSC to see if researchers can replicate or expand upon these results. This study has yielded several lessons that would be useful if applied to further study. What follows is a discussion of some of those recommended improvements to the research protocol of this effort. Regarding scoring of the visualization quiz, the two raters that scored the visualization quiz used the following scoring protocol. A wrong answer or no answer both = 0, OK = 1, Good = 2, Perfect = 3. Researchers recommended in a subsequent study of this phenomenon, the following scoring protocol for the visualization quiz. No answer = 0, a wrong answer = 1, OK = 2, Good = 3, Perfect = 4. Researchers recommended this approach because a failure to try may be indicative of a lack of confidence whereas attempting an answer although failing to answer it correctly is not the same as refusing to attempt to answer it. Researchers deemed the above scoring protocol better because it was more representative of the reality of the phenomenon under investigation. Scoring the game theory variables: As discussed in the finding section the game theory, researchers excluded the variable of environmental complexity because it was a confounding variable. If researchers decided to use the variable of environmental complexity in the future, they should inform participants its inverse relationship to the other variables. Additionally, researchers recommended improvements the game theory variable section of this study by replacing the one-word descriptors with a question. For example, instead of just giving information as a category and a scale of 1 to 6 as possibilities, a question such as, “Did you feel you had enough information to make a decision? Rate your level of information 1 to 6, six being the most information possible for warranting a decision.” Such a change to the game theory variable section of this protocol might assist participants in understanding the purpose of this section and prevent the occurrence of confounding variables. Faculty observer wargaming instrument: As described in the finding section of this research report one of the control groups was excluded from the data due to an alternative protocol. This protocol assisted control group number three (C3) in visualizing, describing, and directing the wargame by recording discoveries made by war fighting function, recording the difficulty, and categorizing the solutions using a table (see table 6 below). Applying this protocol to further studies of wargaming might improve FIGURE 8 Game theory variables and their relation to sense making for decisions. This figure depicts the game theory variables in the form of a sense-making model supporting decision makers. Source: Typhanie Dial & Richard McConnell Page 343 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 wargaming for test and control groups alike thus making improving commander and plane are visualization skills as a form of leader development. Proposed further research: subsequent research studies could be conducted building on this research applying visualization skills to the RDSP model (discussed in the literature review section of this report) applying the cognitive planning domains from this research from a process perspective in light of planning horizons. Planning horizons indicate time frames necessary for addressing certain problems. For Example, current operations (CUOPs) could describe short term such as the next 24 hours, future operations (FUOPs) might depicts mid-term such as the next week or more, and plans could describe the next month to six months. Figure 6 below could serve as a theoretical model enabling such a study. Researchers proposed further investigation: further examination of the cognitive planning domains from perhaps a cultural or ethnographic perspective. How do planners and commanders interact between the culture created between the factory, laboratory, and art institute? Figure 7 below could serve as a theoretical model to enable such a study. Proposed further research: the investigators in this study recommend further study focused on the potential effects of the game theory variables on decision makers as they make sense of their environments. Figure 8 below could serve as a theoretical model for a possible grounded theory study investigating the anatomy of a decision as decision makers over time gain understanding of their environment achieving whatever level of clarity they require to make a decision. Such a study could provide descriptions of levels of decision maker comfort to act while balancing threats and opportunities to achieve desired outcomes. Kreigspiel and Leader Development A new Kriegsspiel game is difficult to acquire as no one-publisher markets a whole package. Translated instructions and digital maps are available from a firm in the United Kingdom called Two Lardies. In addition to the materials they sell, Verdy’s Free Kriegspiel is a valuable resource. Pieces can be printed on paper and cut out but will not be durable. A more practical solution is to contract the print job to a professional to have cork-backed plastic printed pieces for each side made. These will be a lot closer to the original wooden blocks. Kriegsspiel can be useful to create an experiential learning event for small to medium sized groups. Cadets in a ROTC program or officers on the battalion or brigade staff can learn to play. The U.S. Army Command and General Staff College uses students in the role of running umpire which correlates closely to the role the Executive Officer (XO) plays in wargaming. Researchers noted that learning also occurs in this position since they see more of the battle as it is unfolding. In these educational or training cases, an after action review is essential to ensure the players depart understanding the right lessons. To learn the role of head umpire, it is best to at least to do it as on the job learning. Start with serving a few games as a running umpire, sit with the head umpire for a couple, and then, under the supervision of the experienced umpire, be the head umpire. In a unit, the commander could serve as the senior mentor/Observer Trainer and could have subordinate commanders play against each other or even have the staff play against the commanders and provide feedback. This could be an effective way for commanders to pass on to the next generation of leaders the ability to use the art of command and the science of control. CONCLUSIONS The wargaming step of MDMP is an historical challenge for the military institution that potentially results in commanders and planners failing to balance effectively the art of command with the science of control. Such failures can result in an inability to effectively visualize, describe, and direct while appreciating the context of their operational environment with potential threats and opportunities. Improving wargaming across the military institution can help address such problems. The world is becoming a more complex place in the rate of change is only expected to continue to accelerate. Therefore, the ability of commanders and planners to develop plans enabled by effective visualization is only likely to increase going forward. Analysis of the data of this study suggests a simple and cost-effective solution: simple role playing board games. Leaders in our military institution used similar devices in the past two instruct subordinates on how to visualize. It is the conclusion of the primary investigator of this study that such endeavors constitute a vital need in our military institution. Commanders could periodically encourage subordinate planners and commanders to place such simple role-playing board games while the senior commander serves as their coach and mentor. Such endeavors could be a valuable way for that commander to improve their subordinate commanders and planner’s ability to visualize, anticipate, integrate discoveries, while balancing the art of command with the science of control. SUMMARY The initial problem examined for this research was the ongoing challenge over the last 20 years facing commanders and planners to produce effective wargames that uncovered flaws in thinking and planning shortfalls in the operational plans they produce. The analysis of the data in this research report suggests that simple role-playing games might assist in addressing this problem. The hypothesis that student participants who play Kriegspiel (the test group) prior to the wargaming step of the MDMP will be more effective at visualizing their environment and thus will anticipate threats and opportunities more readily than student participants who do not (the control group) was partially supported by both data collection instruments. The student visualization quiz partially supported the hypothesis by establishing that the test group was more effective than the control at visualizing themselves within the context of their operational environment that was statistically significant. The student visualization quiz also Page 344 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 returned notable increases of the test group over the control at visualizing certain enemy capabilities. Both of these findings partially supported the hypothesis. The student visualization quiz also addressed research questions R1 and 2 regarding the level of student effectiveness at visualizing and describing key aspects of an operation if they had or had not played Kriegspiel. The faculty wargaming observer instrument returned statistically significant increases of the test group over the control at recognizing threats and opportunities as well as integrating R3 and 4 regarding how faculty members described the effectiveness of wargaming sessions performed by students who played or did not play Kriegspiel prior to the wargaming step of MDMP. For some military professionals, these outcomes may not be surprising. Some senior leaders may recall playing military focused board games as younger leaders and may wonder why these practices have become the exception not the rule in operational units. Perhaps some military leaders believe we have replaced these practices with complex simulations that underpin leaders and planners ability to visualize. Observations at the combat training centers suggest that wargaming and the visualization that it enables is an ongoing challenge. If a senior leader/commander could improve the visualization/wargaming skills of their subordinates using a simple role playing board game why would they not do it? The conclusions of this study support the notion that simple role playing board games do improve visualization skills vital to wargaming thus improving the quality of the process and its outcomes. The choice to use them is up to the organizational leaders of our military institution. REFERENCES Augustijnen, M., Schnitzer, G., & Esbroeck, R. (2011). A model of executive coaching: a qualitative study. International Coaching Psychology Review, 6(2), 150– 164. Center for Army Lessons Learned. (1997). (FOUO) NTC Trends Fort Irwin, CA: Combined Arms Center. (NTC Trends No. 97–3) (p. 50). Fort Irwin, CA: Combined Arms Center. Center for Army Lessons Learned. (1999). (FOUO) NTC Rotation Report (Unit trends during exercises No. Rotation 00-01 No. 0–1) (p. 415). Fort Irwin, CA: Combined Arms Center. Center for Army Lessons Learned. (2004). (FOUO) NTC Rotation Report (Unit trends during exercises No. Rotation 05-01 No. 5–1) (p. 673). Fort Irwin, CA: Combined Arms Center. Center for Army Lessons Learned. (2007). (FOUO) CTC Trends (Unit trends during exercises No. 8–3 No. 8–3) (p. 56). Fort Leavenworth, KS: Combined Arms Center. Center for Army Lessons Learned. (2008). (FOUO) CTC Trends (Unit trends during exercises No. 09–18 No. 09 –18) (p. 118). Fort Leavenworth, KS: Combined Arms Center. Cilliers, F. (2011). Positive psychology leadership coaching experiences in a financial organization. South African Journal of Industrial Psychology, 37(1), 14. Combined Arms Center. (1999). (FOUO) NTC Trends Compendium (Unit trends during exercises No. 99–1 No. 99–1) (p. 130). Fort Irwin, CA. Department of the Army. (2012a). ADP 5-0: The Operations Process. Washington, D.C.: U.S. Government Printing Office. Department of the Army. (2012b). ADRP 3-0 Unified Land Operations. Washington, D.C.: U.S. Government Printing Office. Department of the Army. (2012c). ADRP 5-0: The Operations Process. Washington, D.C.: U.S. Government Printing Office. Department of the Army. (2012d). ADRP 6-0 Mission Command. Washington, D.C.: U.S. Government Printing Office. Department of the Army. (2014). Field Manual 6-0 Commander and staff organization and operations. Washington D.C.: U.S. Government Printing Office. Elston, F., & Boniwell, I. (2011). A grounded theory study of the value derived by women in financial services through a coaching intervention to help them identify their strengths and practice using them in the workplace. International Coaching Psychology Review, 6(1), 16–32. Fallesen, J. J. (1995). Overview of practical thinking instruction for battle command (Research report No. 1685). Fort Leavenworth, KS: Armyb Research Institute. Gladwell, M. (2007). Blink: The Power of Thinking Without Thinking (1 edition). New York: Back Bay Books. Hogg, R. V., & Craig, A. T. (1978). Introduction to mathematical statistics (4th ed.). New York, N.Y., United States: Macmillan Publishing Co., INc. Holt, S., Bjorklund, R., & Green, V. (2009). Leadership and culture: Examining the relationship between cultural background and leadership perceptions. Journal of Global Business Issues, 3(2), 149–164. Lewin, K. (1943). Defining the “field at a given time.” Psychological Review, 50(3), 292–310. https:// doi.org/10.1037/h0062738 Statistical significance Mcgraw-Hill Concise Dictionary of Modern Medicine. (2002). Retrieved April 28, 2017, from http://medical-dictionary.thefreedictionary.com/ statistical+significance Tadelis, S. (2013). Game theory: An introduction (1St Edition edition). Princeton ; Oxford: Princeton University Press. Page 345 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 APPENDIX A: STUDENT VISUALIZATION QUIZ. Questionnaire Pre-Brief: Thank you for participating in this study regarding the effect of wargaming on individual capabilities to visualize. Your participation is voluntary and you can decline to answer any question. You can also withdraw from the study at any time. I am very interested in understanding your capability to visualize, so please be detailed in your answers. The survey system administering this questionnaire will ensure your anonymity by removing all personal information so I as the primary investigator conducting the study will not be able to identify you. Continuing with this questionnaire constitutes your consent to participate. Visquiz Questions: Have you played Kriegspiel: Yes or No Do you regularly table top role playing games such as Dungeons and Dragons, Mutants and Masterminds, or others: Yes or No. Have you attended any of the Simulation Department Table Top Game Brown bags such as Yom Kipur, Triumph and Tragedy, Paths of Glory, other: Yes or No? Display slide one for 60 seconds: 1. What is the designation of enemy higher headquarters? 2. What types of units as the enemy have across the river? 3. What size enemy element is operating in the vicinity of Ft Leavenworth and Leavenworth? 4. What enablers does the enemy have to support their operations? 5. What is the enemy unit size and location of its reserve across the river? 6. What friendly elements are operating on the near side of the river? 7. What type of friendly units are operating on the near side of river? 8. What is the size and composition of friendly units operating on the near side of the river? 9. What are the major roadways transacting the operational environment? Slide 1: Assess the decisions you made in forming your answers: Certainty: In regards to decision making, certainty can assist a decision maker to address an emerging thereat or opportunity gaining an advantage or seizing the initiative over an opponent before the opponent can act (Department of the Army, Page 346 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 2012b; Tadelis, 2013). Common Knowledge: In relation to decision making, common knowledge is the level to which decisions might be able to be easily or more difficulty made based on the commonness of the knowledge required to reach a reasonable decision (Tadelis, 2013). Environmental Complexity: In regards to decision making, decisions in static environments may require one time only solutions whereas decisions in dynamic environments may require multiple decisions requiring understanding of constant environmental change being caused by multiple drivers and resisters (Lewin, 1943; Tadelis, 2013). Information: The information continuum in regards to game players and decision makers describes the level of completeness might be perceived gamer players and decision makers to act with a level of confidence that they have enough information to pursue reasonable actions (Tadelis, 2013). Rationality: In relation to decision making, rationality is the level to which decisions might be able to be logically reasoned (Tadelis, 2013). Time: In regards to decision making, each decision maker engages in a process balancing all if the above variables given what they think is adequate time to make an infomed decision (See figure 2) (Tadelis, 2013). Place and X on the continuum below regarding the level of information, certainty, rationality, environmental complexity, common knowledge, and time you thought you had enabling your decision. Display slide two for 60 seconds: 1. What is the enemy unit size and location of its division reserve? 2. How far would the enemy division reserve have to travel in order to cross the bridge over the Missouri River? 3. What is the distance to cross the operational environment east to west? 4. What is the distance to cross the operational environment south to north? 5. What is the size, composition, and scheme of maneuver of the friendly main effort? 6. What is the mission of the friendly cavalry squadron? Page 347 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Slide 2: Assess the decisions you made in forming your answers: Place and X on the continuum below regarding the level of information, certainty, rationality, environmental complexity, common knowledge, and time you thought you had enabling your decision. Questionnaire Out-Brief: Thank you very much for your time. As a reminder, the survey system administering this questionnaire will ensure your anonymity by removing all personal information so I as the primary investigator conducting the study will not be able to identify you. I will attempt to maintain your confidentiality to the greatest extent possible, but I cannot guarantee your confidentiality especially if you voluntarily share your questionnaire content with others. Page 348 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 APPENDIX B: FACULTY OBSERVER WARGAMING SURVEY. Questionnaire Pre-Brief: Thank you for participating as a faculty observer in this study regarding the effect of wargaming on individual capabilities to visualize. Your participation is voluntary and you can decline to answer any question. You can also withdraw from the study at any time. I am very interested in understanding the capabilities of the group you observed to visualize resulting in an effective wargame, so please be detailed in your answers. The survey system administering this questionnaire will ensure your anonymity by removing all personal information so I as the primary investigator conducting the study will not be able to identify you. Continuing with this questionnaire constitutes your consent to participate. 1. The group I observed played Kriegspiel before the Wargaming step of the military Decision Making Process (MDMP) o YES o NO Evaluate the level of coordination/cooperation across War Fighting Functions (WFF) i.e. integration of WFFs. 2. Question: Did wargamers discover threats and opportunities during the wargame by War Fighting Functions (WFFs)? Evaluate the number of discoveries *integrated across War Fighting Functions (Wffs) *Integration is the process whereby wargamers address threats and opportunities by including perspectives of multiple WFFs. Threats and opportunities Additional threats discovered No additional threats discovered Additional opportunities discovered No additional opportunities discovered Mission Command o o o o Protection o o o o Maneuver o o o o Intelligence o o o o Fires o o o o Sustainment o o o o Page 349 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 3. Question: Did wargamers integrate multiple WFF perspectives while addressing threats and opportunities? Evaluate the level of difficulty encountered by wargamers when one or more WFFs were integrated during the wargame. 4. Question: How difficult was coordination among wargamers when one or more WFFs were integrated during the wargame? Effect of integration of War Fighting Functions (WFF) One additional threat was discovered Two additional threats were discovered Three or more additional threats were discovered One additional opportunit y was discovered Two additional opportuniti es were discovered Three or more additional opportuni ties were discovere d 1 WFF Integrated o o o o o o 2 WFFs integrated o o o o o o 3 WFFs ointegrated o o o o o o 4 WFFs integrated o o o o o o 5 Wffs integrated o o o o o o All 6 Wffs integrated o o o o o o Coordination across War Fighting Functions Not difficult Somewhat difficult Moderately difficult Very difficult Extremely difficult Did not integrate this many WFFs When only 1 WFF Integrated o o o o o o When 2 WFFs integrated o o o o o o When 3 WFFs integrated o o o o o o When 4 WFFs integrated o o o o o o When 5 Wffs integrated o o o o o o When all 6 Wffs integrated o o o o o o Page 350 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Evaluate the level of creative discoveries made by wargamers i.e. creative anticipation of knowledge gaps. 5. Question: Did the group deprive the enemy of options, discover planning shortfalls, or discover branches as a result of the wargame? Select the type and quantity of creative discoveries/actions Definition of creative discovery: Use of Art of Command to discover Exceptional information. Art of Command: The creative and skillful exercise of authority through timely decision making and leadership (ADRP 6-0, Para 2-28) Exceptional Information: Information that would have answered one of the CCIR if the requirement for it had been foreseen and stated as one of the CCIR Evaluate level of wargaming effectiveness Level of creativity during wargame ○ Reflex (Instinctive response to stimulus) ○ Problem solving(Creativity constrained by reality determining practical solutions) ○ Creativity (Rubbing two ideas together to create something unique) ○ Imagination (Completely unrestrained creativity resulting in something novel) Level of creativity before Kriegspiel (For groups who played prior to Wargaming) ○ Reflex (Instinctive response to stimulus) ○ Problem solving(Creativity constrained by reality determining practical solutions) ○ Creativity (Rubbing two ideas together to create something unique) ○ Imagination (Completely unrestrained creativity resulting in something novel) Questionnaire Out-Brief: Thank you very much for your time. As a reminder, the survey system administering this questionnaire will ensure your anonymity by removing all personal information so I as the primary investigator conducting the study will not be able to identify you. I will attempt to maintain your confidentiality to the greatest extent possible, but I cannot guarantee your confidentiality especially if you voluntarily share your questionnaire content with others. Creative discovery during wargame Scale 1 Adaptively planned using discovery? None One Two Three Four or more Yes No Deprived enemy of options o o o o o o o Discovered planning shortfalls o o o o o o o Discovered braches to plans o o o o o o o Other o o o o o o o Page 351 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 APPENDIX C: WARGAMING STUDY PARTICIPANT INFORMED CONSENT (KRIEGSPIEL). The Effect of Simple Role Playing Games on the Wargaming step of MDMP Purpose You are invited to participate in a research study examining influences on effective wargaming during the Military Decision Making Process (MDMP). The purpose of this mixed methods study is to describe the potential effects of simple role playing games on the wargaming step of MDMP. The individuals conducting this study will use qualitative surveys to gather data regarding student visualization capabilities as quantitative surveys to gather data from faculty observers regarding wargame quality. There is no deception in this study. We are interested in determining potential effects of simple roleplaying games on participant wargaming capabilities. Participation requirements. I am asking you to participate in playing Kriegspiel, a 19th century Prussian wargame and to consent to the presence of independent faculty observers during the wargaming step of MDMP. The Kriegspiel wargame will take approximately 2 hours to complete and will require some participants to role play 19th century military commanders, some messengers, and some observers. After the wargaming step of MDMP, an automated email will be sent to you requesting your participation in an online visualization quiz which will take approximately 15 minutes. Research personnel. The following person will be the lead investigator conducting the research for this study and may be contacted at any time: Richard A. McConnell, D.M., richard.a.mcconnell4.civ@mail.mil, 684-4766. Potential risks/discomfort. There are no known risks in this study. You may withdraw at any time either during the Kriegspiel session or during the qualitative survey following the wargaming step of MDMP and you may choose not to answer any questions you feel uncomfortable answering. I will securely safeguard paper copies of this informed consent form for a period not less than three years after the publication of this study. I will store all identifiable data in a secure office in a locked cabinet. After three years, I will destroy all identifiable data. Potential benefit There are no direct benefits to you to and I will offer no compensation or incentives for your participation in this study. The results of this study might eventually benefit people who serve as commanders and planners. Anonymity/Confidentiality. All data obtained about you, as an individual, will be considered privileged and held in confidence; you will not be identified in any presentation of the resultso. Complete confidentiality cannot be promised to participants, particularly to subjects who participate in the Kriegspiel board game. However, all student participants in the qualitative survey will have their identities protected.. All data collected in this study are confidential and are coded so that your name is not associated with them. Additionally, the coded data will be available only to the lead investigator. This form will be stored by the lead investigator for three years. Right to withdraw. Participation in this study is voluntary. You have the right withdraw from the study at any time without penalty. You have the right to decide to not answer questions during the qualitative survey if you do not feel comfortable answering them. If you withdraw from the study, I will not use any data collected from you and you will suffer no penalties whatsoever from your withdrawal. I will be happy to answer any question that may arise about this study. Page 352 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Contacts for Additional Assistance. Please direct your questions or comments about this study to Richard A. McConnell, D.M., richard.a.mcconnell4.civ@mail.mil, 684-4766. If you have any questions or concerns about the conduct of this study, please contact the CAC-E HPA, (Bobby Murray, bobbie.j.murray6.civ@mail.mil, 684-7311). Signatures I have read the above description of The Effect of Simple Role Playing Games on the Wargaming step of MDMP, and understand the conditions of my participation. My signature indicates that I agree to participate in the study. Participants printed name:______________________________________________________ Participants signature: _________________________________________________________ Date:___________________________ Lead Researchers name: Richard A. McConnell Researcher’s signature:________________________________________________________ Date:___________________________ Page 353 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 APPENDIX D: WARGAMING STUDY PARTICIPANT INFORMED CONSENT (NON-KRIEGSPIEL). The Effect of Simple Role Playing Games on the Wargaming step of MDMP Purpose You are invited to participate in a research study examining influences on effective wargaming during the Military Decision Making Process (MDMP). The purpose of this mixed methods study is to describe the potential effects of simple role playing games on the wargaming step of MDMP.. The individuals conducting this study will use qualitative surveys to gather data regarding student visualization capabilities as quantitative surveys to gather data from faculty observers regarding wargame quality. There is no deception in this study. We are interested in determining potential effects of simple roleplaying games on participant wargaming capabilities. Participation requirements. I am asking you to consent to the presence of an independent faculty observer during the wargaming step of the MDMP. Research personnel. The following person will be the lead investigator conducting the research for this study and may be contacted at any time: Richard A. McConnell, D.M., richard.a.mcconnell4.civ@mail.mil, 684-4766. Potential risks/discomfort. There are no known risks in this study. You may withdraw at any time. I will securely safeguard paper copies of this informed consent form for a period not less than three years after the publication of this study. I will store all identifiable data in a secure office in a locked cabinet. After three years, I will destroy all identifiable data. Potential benefit There are no direct benefits to you to and I will offer no compensation or incentives for your participation in this study. The results of this study might eventually benefit people who serve as commanders and planners. Anonymity/Confidentiality. All data obtained about you, as an individual, will be considered privileged and held in confidence; you will not be identified in any presentation of the results. All student participants in the qualitative survey will have their identities protected.. All data collected in this study are confidential and are coded so that your name is not associated with them. Additionally, the coded data will be available only to the lead investigator. This form will be stored by the lead investigator for three years and then destroyed. Right to withdraw. Participation in this study is voluntary. You have the right withdraw from the study at any time without penalty. You have the right to decide to not answer questions during the qualitative survey if you do not feel comfortable answering them. If you withdraw from the study, I will not use any data collected from you and you will suffer no penalties whatsoever from your withdrawal. I will be happy to answer any question that may arise about this study. Contacts for Additional Assistance. Please direct your questions or comments about this study to Richard A. McConnell, D.M., richard.a.mcconnell4.civ@mail.mil, 684-4766. If you have any questions or concerns about the conduct of this study, please contact the CAC-E HPA, (Bobby Murray, bobbie.j.murray6.civ@mail.mil, 684-7311). Page 354 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Signatures I have read the above description of The Effect of Simple Role Playing Games on the Wargaming step of MDMP, and understand the conditions of my participation. My signature indicates that I agree to participate in the study. Participants printed name:______________________________________________________ Participants signature: _________________________________________________________ Date:___________________________ Lead Researchers name: Richard A. McConnell Researcher’s signature:________________________________________________________ Date:___________________________ Page 355 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 A P P E N D IX E : F A C U L T Y O B S E R V E R W A R G A M E IN S T R U M E N T S C O R IN G M E T H O D . Q u e stio n 2 : D id w a rg a m e rs d is c o v e r th re a ts a n d o p p o rtu n itie s d u rin g th e w a rg a m e b y W a r F ig h tin g F u n c tio n s (W F F s )? Th re ats an d o p p o rtu n itie s A d d itio n al th re ats d isco ve re d N o ad d itio n al th re ats d isco ve re d A d d itio n al o p p o rtu n itie s d isco ve re d N o ad d itio n al o p p o rtu n itie s d isco ve re d M issio n C o m m an d 1 0 1 0 P ro te ctio n 1 0 1 0 M an e u ve r 1 0 1 0 In te llige n ce 1 0 1 0 Fire s 1 0 1 0 Su stain m e n t 1 0 1 0 0 Q u e stio n 3 : D id w a rg a m e rs in te g ra te m u ltip le W F F p e rs p e c tiv e s w h ile a d d re s s in g th re a ts a n d o p p o rtu n itie s ? Effe ct o f in te gratio n o f W ar Figh tin g Fu n ctio n s (W FF) O n e ad d itio n al th re at w as d isco ve re d Tw o ad d itio n al th re ats w e re d isco ve re d Th re e o r m o re ad d itio n al th re ats w e re d isco ve re d O n e ad d itio n al o p p o rtu n ity w as d isco ve re d Tw o ad d itio n al o p p o rtu n itie s w e re d isco ve re d Th re e o r m o re ad d itio n al o p p o rtu n itie s w e re d isco ve re d 1 W FF In te grate d 1 2 3 1 2 3 2 W FFs in te grate d 2 3 4 2 3 4 3 W FFs o in te grate d 3 4 5 3 4 5 4 W FFs in te grate d 4 5 6 4 5 6 5 W ffs in te grate d 5 6 7 5 6 7 A ll 6 W ffs in te grate d 6 7 8 6 7 8 Q u e stio n 4 : H o w d iffic u lt w a s c o o rd in a tio n a m o n g w a rg a m e rs w h e n o n e o r m o re W F F s w e re in te g ra te d d u rin g th e w a rg a m e ? C o o rd in atio n acro ss W ar Figh tin g Fu n ctio n s (W FFs) N o t d ifficu lt 5 So m e w h at d ifficu lt 4 M o d e rate ly d ifficu lt 3 V e ry d ifficu lt 2 Extre m e ly d ifficu lt 1 D id n o t in te grate th is m an y W FFs 0 W h e n o n ly 1 W FF In te grate d 5 4 3 2 1 0 W h e n 2 W FFs in te grate d 5 4 3 2 1 0 W h e n 3 W FFs in te grate d 5 4 3 2 1 0 W h e n 4 W FFs in te grate d 5 4 3 2 1 0 W h e n 5 W ffs in te grate d 5 4 3 2 1 0 W h e n all 6 W ffs in te grate d 5 4 3 2 1 0 Q u e stio n 5 : D id th e g ro u p d e p riv e th e e n e m y o f o p tio n s , d is c o v e r p la n n in g s h o rtfa lls , o r d is c o v e r b ra n c h e s a s a re s u lt o f th e w a rg a m e ? N o n e O n e Tw o Th re e Fo u r o r m o re Y e s N o D e p rive d e n e m y o f o p tio n s 0 2 3 4 5 1 0 D isco ve re d p lan n in g sh o rtfalls 0 2 3 4 5 1 0 D isco ve re d b rach e s to p lan s 0 2 3 4 5 1 0 O th e r - N o So lu tio n /Ign o re d th e P ro b le m 0 1 1 1 1 1 0 C re ative d isco ve ry d u rin g w argam e Scale 1 A d ap tive ly p lan n e d u sin g d isco ve ry? Page 356 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 APPENDIX F: VISUALIZATION QUIZ STATISTICAL RESULTS; MANN WHITNEY WILCOXON AND FRIEDMAN TESTS. Game Theory Variable Statistical Analysis Using the 1-tailed Mann-Whitney Test, where H0: T = C and H: T > C Part 1 Information – potential significance N Median Control 70 3.0000 Test 31 3.0000 Point estimate for ETA1-ETA2 is -0.0000 95.0 Percent CI for ETA1-ETA2 is (-1.0001,0.0004) W = 3456.0 Test of ETA1 = ETA2 vs ETA1 < ETA2 The test is significant at 0.1944 (adjusted for ties) Certainty – potential significance N Median Control 73 3.0000 Test 30 3.0000 Point estimate for ETA1-ETA2 is 0.0000 95.0 Percent CI for ETA1-ETA2 is (-0.9998,0.0001) W = 3685.0 Test of ETA1 = ETA2 vs ETA1 < ETA2 The test is significant at 0.2030 (adjusted for ties) Rationality – potential significance N Median Control 73 4.0000 Test 31 4.0000 Point estimate for ETA1-ETA2 is 0.0000 95.0 Percent CI for ETA1-ETA2 is (-1.0002,0.0003) W = 3661.0 Test of ETA1 = ETA2 vs ETA1 < ETA2 The test is significant at 0.1029 (adjusted for ties) Common Knowledge – potential significance N Median Control 73 4.0000 Test 31 4.0000 Point estimate for ETA1-ETA2 is -0.0000 95.0 Percent CI for ETA1-ETA2 is (-1.0000,0.0002) W = 3709.0 Test of ETA1 = ETA2 vs ETA1 < ETA2 The test is significant at 0.1835 (adjusted for ties) Time – no significance Page 357 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Part 2 Information – no significance Certainty – no significance Rationality – no significance Common Knowledge – statistically significant N Median Control 73 4.0000 Test 30 4.0000 Point estimate for ETA1-ETA2 is -0.0000 95.0 Percent CI for ETA1-ETA2 is (-1.0003,0.0003) W = 3559.5 Test of ETA1 = ETA2 vs ETA1 < ETA2 The test is significant at 0.0376 (adjusted for ties) Time – potential significance difference that T ≠ C (with Mann-Whitney 2-tailed and Kruskal-Wallis Tests) MANN-WHITNEY 2-TAILED TEST N Median Control 73 2.0000 Test 30 2.0000 Point estimate for ETA1-ETA2 is -0.0000 95.0 Percent CI for ETA1-ETA2 is (-0.0002,0.9998) W = 3959.5 Test of ETA1 = ETA2 vs ETA1 not = ETA2 The test is significant at 0.2202 (adjusted for ties) KRUSKAL-WALLIS TEST N Median Ave Rank Z Control 73 2.000 54.2 1.19 Test 30 2.000 46.5 -1.19 Overall 103 52.0 H = 1.41 DF = 1 P = 0.235 H = 1.51 DF = 1 P = 0.219 (adjusted for ties) Page 358 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 APPENDIX G: THE MILITARY DECISION MAKING PROCESS (MDMP) The military decision making process is one of the Army’s three planning methodologies, that enables detailed planning, usually conducted at the battalion level (500+ Soldier organization) or higher. The military decision making process (MDMP) is an iterative planning methodology to understand the situation and mission, develop a course of action, and produce an operation plan or order. The MDMP helps leaders apply thoroughness, clarity, sound judgment, logic, and professional knowledge to understand situations, develop options to solve problems, and reach decisions. This process helps commanders, staffs, and others think critically and creatively while planning (Department of the Army, 2014; Department of the Army, 2012a). FIGURE 9 Adapted from Field Manual 6-0, the MDMP process for detailed planning. Step 4 of the process is Course of Action Analysis or Wargaming. Page 359 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 The MDMP consists of seven steps. Those steps are Receipt of Mission, Mission Analysis, Course of Action Development, Course of Action Analysis, Course of Action Comparison, Course of Action Approval, and Orders Production, Dissemination, and Transition. Each step of the MDMP has various inputs, a step to conduct, and outputs. The outputs lead to an increased understanding of the situation, facilitating the next step of the MDMP. Commanders and staffs generally perform these steps sequentially; however, they may revisit several steps in an iterative fashion as they learn more about the situation before producing the plan. (Field Manual 6-0) The staff’s effort during the MDMP focuses on helping the commander understand the situation, make decisions, and synchronize those decisions into a fully developed plan or order. The products the staff develops during mission analysis help commanders understand the situation and develop the commander’s visualization. During course of action (COA) development and COA comparison, the staff provides recommendations to support the commander in selecting a COA. After the commander makes a decision, the staff prepares the plan or order that reflects the commander’s intent. (Field Manual 6-0)