Simulations – Bridging From Thwarted Innovation to Disruptive Technology Developments in Business Simulation and Experiential Learning, Volume 33, 2006 SIMULATIONS – BRIDGING FROM THWARTED INNOVATION TO DISRUPTIVE TECHNOLOGY John Kenworthy Corporate Edge john.kenworthy@celsim.com INTRODUCTION Gartner Research dubbed simulation the new "killer application" in e-learning (Lundy et al., 2002) but even assuming the best estimates for the adoption of simulations, they represent a tiny proportion of the annual spend in training and education. Considerable research has been done to evaluate the effectiveness of simulations and, by and large, the results suggest that simulations are effective but there are doubts about even the most fundamental claims of the efficacy of simulations (Feinstein and Cannon, 2002) partly because there isn’t a clear, acceptable methodology, partly because there is no real agreement on definitions, and partly because there is little agreement on what should be evaluated. Burns et al. (1990) consider the multi-fold problem with evaluating experiential pedagogies stating that there is firstly a need to compare the efficacy to ‘traditional’ approaches, and there is a need to compare alternative experiential pedagogies competing to achieve the same learning. Not surprisingly, they note a paucity of solid empirical evidence regarding the relative effectiveness of experiential techniques. Other authors (e.g. Pierfy, 1977) note two particular problems with respect to evaluating simulations or experiential techniques: the first being the conceptual problems pertaining to definitions, domain boundaries and the theoretical basis which underpin and frame pedagogical research. The second fundamental problem is that there remain significant methodological difficulties including experimental design, constraints within the organisations and institutions, time considerations and ethical questions associated with any comparative study. This paper does not intend to argue in favor of one approach, method or definition over another but to consider why simulations have not yet emerged as training and education’s “killer application” and how it may be possible to bridge from being a thwarted innovation (Zemsky & Massy, 2004) to a disruptive technology (Christensen, 1997). It is important to clarify first, what the author means by simulations, as these come in many guises and the term is used ubiquitously in the field of education and management learning. Summers (Summers, 2004) suggests three basic types of business simulations currently in use – computer- based simulations, board games and behavioral simulations. MacDonald et al. (1977) define that within a simulation, the learner may only tinker within the parameters, not the central working of the system. Whilst board games and behavioral simulations as described by Summers, do simulate real situations, users are an integral part of the simulation and can change the central working of the system. Such ‘simulations’ are not the focus of this paper. The simulations considered here are computer-based virtual realities representing real situations and, drawing also on Barr and Tagg (1995), are classic examples of the Revelatory paradigm. To ensure clarity, the author uses the term “Vebeat” an acronym for ‘Virtual Environments for Business Education and Training’. Vebeats may be enhanced with [video] game-like characteristics, utilize agent-based behavioral elements and increasingly include implementations of artificial intelligence (AI). They may be stand-alone environments with a user interacting directly with the computer or in cooperation or competition with other human users. Such Vebeats have become widely accepted pedagogical techniques, in part because participants are more actively involved in the learning process and receive immediate feedback on the results of their actions (Brenenstuhl & Catalanello, 1979). This supports Senge’s (1990) view that human beings learn best through first-hand experience, particularly when feedback from actions is rapid and unambiguous. Garris et al. (2002) support this view and propose that the game cycle is iterative, in that game play involves repeated judgment-behaviour-feedback loops and user reactions to this lead to greater persistence or intensity of effort because it is enjoyable, interesting and builds confidence. Support for simulations in greater learning in and of itself is one aspect, but Swanson and Holton (1999) also argue that learning activities that recreate work situations, such as simulation enhanced learning activities, foster better transfer of learning. A recent study by Kenworthy (Kenworthy, 2005) shows clear empirical evidence supporting the effectiveness of Vebeats over the use of case studies in a strategy management programme across all four levels of Kirkpatrick’s ubiquitous framework. Vebeats also bring a different value proposition to the market of business education and management training… they are more effective! The efficacy of simulations and games has been substantially questioned and tested for decades and proponents of simulations and games, Hoberman and Mallick (1992) and Greber (1994) suggest an impressive number of benefits of training using simulations including: 1) Improved transfer of learning to the work venue; 2) Well-suited for teaching participants how to respond to change; 3) Relatively risk-free environment in 149 mailto:JOHN.KENWORTHY@CELSIM.COM Developments in Business Simulation and Experiential Learning, Volume 33, 2006 which to try new behaviours; 4) Higher participant involvement and motivation; 5) Ability to manipulate several variables at once; and 6) Potential for immediate feedback. Researchers have identified benefits that are unique to simulation techniques: improved ability to teach teamwork (Keys et al., 1994); unique contribution to long-term strategy making (Gopinath & Sawyer, 1999); demonstrate the complexities of dynamic business systems (Romme, 2003); and positive relationship between business game experiences and outcomes such as income and organizational position (J Wolfe & Roberts, 1993). It has also been proposed that simulations seem well suited to promoting what Argyris and Schon (1978) have termed ‘double-loop learning, a learning climate that supports valid information, free and informed choice, and internal commitment: and strategic knowledge that requires applying learned principles to different contexts or deriving new principles from general or novel situations (Garris et al., 2002). So, if Vebeats deliver more value in terms of greater enjoyment, learning, transfer and business impact than more traditional pedagogies, why are they, after nearly 50 years, still a niche product? Is this an example of thwarted innovation that lives up to a promise of a better future but fails to deliver to enough customers to be successful? A recent article in The Wall Street Journal (Totty, 2005, Page R6), states that companies in the U.S. “spend about $60 billion a year on training their employees, but there's a good chance much of that is wasted”. Totty goes on to cite several examples of major US corporations using computer gaming in training employees effectively. According to Gartner Research, simulations may be the ‘killer application’ for e-learning (Lundy et al., 2002) and in 2005, annual spending for training worldwide is over $100 billion, and e-learning content accounts for only a few little of that and when it comes to potential growth in the e- learning content market, Lundy suggests that much of the expected growth for e-learning will be driven by simulations (Boehle, 2005). Brandon-Hall estimate that simulations accounted for 2.9% of the e-learning industry in 2002 and predicted that they would represent 7.3% of the e-learning market in 2006 (Adkins, 2002) and Summers (Summers, 2004) drawing from a number of sources, estimates the size of the worldwide market for business simulations between $623 and $712 million. Estimating the number of business simulations in use by companies and academia is particularly difficult as there is little consistency in definition of business simulations, though Faria and Wellington (Faria & Wellington, 2004) report a thorough analysis of the academic market showing more than 52% of respondents had used a business game with expectations that use would increase. The ASTD State of the Industry Report (Suqrue & Kim, 2004; Thompson et al., 2002) support the figures and show that these are all very positive trends, but realistically, the current use of business simulations represents a tiny proportion of overall spending on corporate training – perhaps 1% or 2%. By way of contrast, computer simulation games for entertainment have a market size of $7.3 billion (ESA, 2005) in the US alone – ten times greater than the estimated worldwide market for ‘serious games’. It is possible that the intended market is policed by gatekeepers who are preventing access to potential learners, Naish (2005), for example, suggests that serious games can overcome the poor motivation to participate in e-learning but that bringing these into the learning space remains difficult, particularly since the budget-holders may not have played any such games. Clearly, a dominant design has not yet emerged, and this may be holding back the greater adoption of the technology in ways that parallel the adoption of e-learning (Zemsky & Massy, 2004). There are doubts about the efficacy of simulations and since the early days of simulation and gaming as a method to teach, there have been calls for hard evidence that support the teaching effectiveness of simulations (Hays & Singer, 1989). In spite of the extensive literature, many of the claims and counterclaims for the teaching power of business games and simulations rest on anecdotal evidence or inadequate or poorly implemented research. These research defects, according to Keys and Wolfe (1990), have clouded the business gaming literature and hampered the creation of a cumulative stream of research. Gredler (1996) notes, as did Pierfy (1977) that a major design weakness of most studies evaluating simulation based training is that they are compared to regular classroom instruction even when the instructional goals for each differ. Vebeats are benefiting from new computing technologies increasingly readily available and the development of sophisticated decision trees, agent-enabled simulations, artificial intelligence, natural language processing, voice recognition and haptic devices are transforming Vebeats from roots in models made from formulae (Summers, 2004). Are Vebeats mired in arguments about efficacy or is it time for a new mindset? Rather than striving for incremental change, improving particular aspects of the technology or integrating straight-forwardly with existing technologies (like the Internet), we might consider business simulations to be a disruptive technology that is going to fundamentally change the way people learn about business and management. DYNAMICS OF INNOVATION The development of simulation based learning is a story of radical technological innovation in that as an invading technology it has the potential to deliver dramatically better performance or lower costs in what has been a stable industry. The important word is potential. When a new technology first emerges, it is often inferior and clumsy compared to its predecessor. In the early days, it is the new technology’s promise rather than its actual performance that attracts adherents. In The Innovator’s Dilemma, Christensen (Christensen, 1997) states that disruptive technologies bring 150 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 Disrupt existing competitors, rather than leap ahead with a better product in an existing market. a very different value proposition to the market compared to sustaining technologies. The latter foster improved product performance and although disruptive technologies frequently are regarded as under-performing established products in mainstream markets, they have other features that certain (niche) customers value. Christensen notes that disruptive products are usually cheaper, simpler, smaller and easier to use. They move up-market when they have sufficient functionality to unseat incumbent, sustaining technologies – which often provide more features than customers need or want. The simulation and gaming literature is replete with contentions about the efficacy of the medium as a teaching pedagogy and there are repeated calls for a clear methodology and research direction suggesting that there are doubts about the medium in part, because it is disrupting existing competitors – the more traditional methods. However, there are some indications that it is a leap ahead technologically, or at least becoming so. It enables a larger population of less skilled or less wealthy people to do something more simply and conveniently that historically could only be done by experts or the wealthy, in inconvenient centralized settings. Incremental change is the most common way in which technology develops, either through improvement to particular aspects of a technology or to the integration of that technology with other technologies in a relatively straightforward manner. Incremental change is, in this sense, evolutionary, focusing on making the key technology drivers better. Are the changes and improvements in simulations and games for education and training only driven by the increasing ease of producing more sophisticated products because the underlying technology, personal computers and the Internet have changed? Or are the newer simulations, Vebeats as they have been termed here, an example of a disruptive technology that now possess the functionality necessary to unseat incumbent technologies such as lectures, case studies and the predominantly text-based and PowerPoint™ e-learning prevalent in companies, universities and colleges the world over (Zemsky & Massy, 2004). Vebeats distributed via the Internet or by CD ROM certainly offer the opportunity to be a more convenient delivery mechanism over attending business school or a classroom training session – and through reduced traveling costs and time spent in the learning environment would provide greater access to those on more restrictive budgets. There are high fixed costs associated with developing Vebeats (Summers, 2004) but a different business model might allow this attribute to work well in favor of Vebeats unseating older technologies in use. It targets a set of less demanding customers who would be delighted to have a simple product. Learners can use Vebeats that teach exactly what they need to learn, they can be modularized and easy to navigate without the necessity of undertaking a full curriculum of study which includes some aspects of what they need to know and learn how to do. DISRUPTIVE ATTRIBUTES Disruptive technologies have certain attributes that distinguish them from new technologies brought about by incremental change, according to Christensen (Christensen, 1997). Vebeats currently share some of these attributes but not all. Disruptive technologies have attributes that: It minimizes infrastructure barriers with most new technologies being plug-compatible in existing systems. The major changes seen in recent years in Vebeats are a Disruptive Technology test question Possible responses for Academia Possible responses for Corporate “Where is the application that a simpler, more convenient technology would take root before it matures enough to become relevant to our current best customers?” Replacing assignment writing or projects or even exams. Enabling small businesses to have training for very small groups in specific task areas “Who might be some unsophisticated customers that would be delighted to have a crappy product?” Colleges in poor areas or countries, or have never experienced using simulations Holders of very small training budgets wanting to fulfill compliance quotas. “Do we think that our best customers [existing] would want the product? If so, it’s probably not a disruption.” Not all, it threatens the realm of experts. Not all, it threatens HRD and their position as experts and organizers. “Will it strengthen our existing The business model changes to reflect fewer resources in delivery, business model? If so, it probably is not a disruption.” decentralization, less time in delivery and travel. Focuses attention on applying knowledge not simply gaining knowledge ‘just-in-case’. 151 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 4. The disruptive technology is developed to the point where it can now enter the original markets, challenge and disrupt the incumbent, sustaining technology, based on new performance criteria. There are over 1.5 billion consumers with mobile phones (Economist.com, 2005) and the OECD outlook shows that there is almost one mobile phone for every inhabitant in the developed world and globally, there are more mobile phones than fixed line phones in 2005 (OECD, 2005). result of work in computational science and the massive growth in information technology. Few Vebeats require new IT infrastructure (exceptions being high-end virtual reality) and few demand the computational power of the more ubiquitous video-games that work happily on home computers. It targets a trajectory of improvement that has not yet overshot what customers can absorb. Most of the education and training industry’s traditional products are generalist in nature combining a number of skills areas and/or knowledge acquisition, whereas many of the newer Vebeats teach specific knowledge addressing a particular task, such as reading financial statements or using a particular sales methodology. Such Vebeats provide customers with smaller chunks of knowledge and skills. The sustaining technology, as in this example the plain old telephone system or POTS, may not be totally replaced; however, mobile telephone technology has substantially affected the market, radically changing the original telco business model and consumer habits. DISRUPTIVE POTENTIAL OF VEBEATS It builds an outcomes purpose or brand. Outcomes from a Vebeat are likely to be clear because the purpose must be designed and programmed into the software. Artificial intelligence technologies may begin to cause this to change but it is unlikely that even the most sophisticated AI would be able to go off on the tangents that the human mind is capable of. However, are Vebeats, particularly commercial Vebeats, designed with outcomes first and foremost or are they designed to demonstrate the sophistication of the technology? Using Christensen’s (1997) litmus test for disruptive technologies, it is suggested here that, at a minimum, for Vebeats to be a disruptive technology they must: • Compete on a different performance metric(s) • Change the business model of an organisation • Incubate fringe markets Competing on different performance metrics. Vebeats fall at the first hurdle to date. There has been a great concentration on comparing simulations and evaluating the effectiveness against other incumbent technologies, case studies, lectures, outdoor games and so on. The evidence suggests that Vebeats are as good as, if not better than case studies across different levels of evaluation, learning, transfer and business impact. In addition, Vebeats gain much better scores from users in enjoyment – which may be one of the key intrinsic motivator’s for learning. As has been suggested, perhaps it is time for a new performance metric that establishes a common instrument to measure internal and external validity, though there seems to be little motivation to make this happen (Gosen & Washbush, 2004), or perhaps it is time to change mindset and consider other performance criteria that are intuitively more appealing and much easier to measure. We could, for example, borrow criteria from e-learning, accessibility and cost-saving, or borrow from the video-game industry and perform on fun and enjoyment, or from the mobile phone industry and perform on freedom from fixed locations and choice of what is accessed. Combine these and you have a powerful set of performance metrics such as – freedom to choose to enjoy learning what you need when you need it wherever you need it by yourself or together with other like- minded people. So do Vebeats have the potential to be a disruptive technology? Christensen (Christensen, 1997) presents a series of questions to help test the disruptive potential of an innovative idea shown in the table below together with possible example answers for the academic and corporate markets: It seems that Vebeats have some potential to be a disruptive technology, or rather, as Christensen suggests, enable a disruptive business model to take root. DISRUPTIVE PROCESS The disruptive process can be characterized in four steps as follows: 1. The sustaining technology is mature and performing well and is being improved according to customer feedback and demands. For example, the fixed-line telephony service in the early 80’s 2. The disruptive technology is developed but doesn’t meet the performance requirements associated with the sustaining technology and its market. The early cell-phones with limited network availability and frequent dropped calls and low quality transmission. Changing the business models of organizations. We have seen that Vebeats certainly have the potential to change the business model but more frequently many Vebeats require a seminar presentation either due to the fact that the Vebeat still requires an expert to run it, or the Vebeat package may not be a complete learning solution in 3. Alternative markets are found or created where the price and performance metrics of the disruptive technology match the customer requirements. Short-Message-Service on mobile phones. 152 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 So how could these impediments be overcome? and of itself by design, or, more frequently, because the sales process is necessarily consultative and the initial cost of development is extremely high and is built into the selling price. Fragmentation of the industry with decreasing barriers to entry, high selling costs, high delivery costs and little real product differentiation combine to cause most players to adopt an existing (traditional) business model - simulations are used as add-ons to existing development programs to add-value, at a price. If we consider how other technologies have overcome the impediments of diffusion, there may be some good indicators of what needs to happen in this Vebeat industry. Typically, innovations adoption follows the classic S-curve. Adoption processes of a new technology usually start slowly because of the need for experimentation, accelerating once a dominant design emerges. The actors at various stages of adoption differ markedly and are usually referred to as innovators, early adopters, early majority, late majority and diehards. The mobile phone phenomenon has been used above as an example and I shall continue with this comparative disruptive technology. Incubating fringe markets – in some ways, this is where Vebeats do operate – it tends to be driven by a few believers. Individuals within organizations and academia who have an interest in simulations and either develop one themselves or buy-in one that they have used for themselves successfully in the past. In the early days of mobile telephony, there were numerous problems. Readers of a certain age, may recall that cell phones were rather large and cumbersome objects – originally, many came attached to a motor vehicle as the source of power, later a briefcase sized battery. Now they fit in a small pocket, have the computational power of a laptop computer, battery life extending to days and are capable of video conferencing or replaying a movie. The same mobile phone can access networks across the world with few limitations. Part of the phenomenal success of mobile telephony is the result of deregulation across the world allowing new companies to compete in a market-driven but regulated environment. The devices and networks that support them would not be as useful without industry standards and robust platforms. The Vebeat industry could benefit from both establishing standards and providing strong, adaptable platforms on which to build simulations. I will consider these two aspects first and then discuss how these might be achieved. Are we, as suggested by Brown (2000) only in the gradual development phase of this transformative medium and yet to experience its explosive impact? Experiencing technology enhanced learning in its early forms and have much to learn about its inherent capabilities and the creation of a new learning ecology? What are Vebeats relative advantages and are they really compelling enough to force a reconceptualisation of the teaching and learning transaction? Perhaps, because of our inability to come to grips with this learning ecology, business education and training is largely unchanged by simulation and game technology that has otherwise changed society through entertainment, video games, the media and massive, multi-player simulations across the Internet. Are the impediments to the diffusion of this technology too great? OVERCOMING IMPEDIMENTS TO THE DIFFUSION OF THE NEW TECHNOLOGY Standards: Feinstein and Cannon (Feinstein & Cannon, 2003) suggest a hermeneutical approach to the external validation of simulation models, this might be extended to provide an expert judgment on the internal validity as well as the external validity of a simulation model and whether the fidelity and realism in the simulation either is appropriate, or for whom it is appropriate. The approach accepts that there is judgmental bias but with transparency and good governance, such a body equates to the ITU (International Telecoms Union) in the telecommunications industry. Summers (2004) suggests a number of impediments to the diffusion of simulation technology including cultural obstacles that simulations place more responsibility on the learner, and that if learners are self-motivated, this threatens human resource departments (and presumably academic teachers). He further identifies that people learn differently and the new technologies don’t address the necessary diversity of learning styles. Development costs for simulations can be very high and these costs are passed onto a relatively small number of customers – keeping selling prices high. Much of the time, those potential customers in the market don’t know what they are buying, they are unable to try simulations before buying and the current business model for most corporate market simulation developers includes expensive consultative selling. In the academic market, the predominant business model uses publishers as distribution and sales channels. This suggests that the potential customer is not sure of the inherent advantages associated with simulations and the fragmentation of the industry means there are no formal standards for what a simulation is, let alone that it should deliver on the objectives claimed by the developer. Platform: The high cost of developing simulations is frequently cited as the reason for high prices and hence, poor take-up. The industry is mostly made-up from bootstrapping companies who provide their own funds with few venture capitalists becoming involved. There are currently a number of proprietary platforms developed and a small number of developers license or sell their platforms to others to develop simulations. Industry consolidation may take place and may help this become a reality, though without a recognized set of standards, which platform, or few platforms that become the de-facto standard is pure conjecture. In mobile telephony, the standards that moved the industry from small, niche low quality to the mass market today came with the change from analog to digital 153 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 and were established essentially in geographic regions, the GSM standard in Europe, CDMA in the US and PCS in Japan. For 3G telephony, the ITU approved two standards – perhaps by the time of 4G, there will be one standard. Such platforms take time to develop and agree, though the benefits of reducing development costs and enabling cross- functionality could be immeasurable. How could simulation technology diffuse and genuinely become the ‘killer application’ of both e-learning and management development? Again we can take lessons from the mobile telephony industry to help identify how these technologies have diffused and the importance of particular segments of the early adopter market to the future success of technological innovation Following Gilbert and Kendal (Gilbert & Kendall, 2002) the market segmentation model is used to help identify segments within the corporate and academic markets for simulations over the backdrop of a product life cycle S-curve. Relationships are proposed among the intention to use Vebeats, specific requirements and influencing variables resulting in the identification of four early user segments: Techno Toys: Mostly innovators, filling needs for hands-on knowledge about technological developments, intend to stretch existing boundaries and establish if it works. Influenced by technical and/or academic reports. First in market and tend to develop and/or research simulation technologies. Training Professionals: Early adopters of simulation products to create new value related to training work, including new revenue opportunities, differentiation, acceleration of training, higher selling prices. Influenced by employers and clients – particularly in competitive sales pitch or need to demonstrate ROI of training or find products that users will enjoy and actually use on previously installed learning management systems. Sophisticates: use of simulations demonstrates sophistication and willingness to embrace new technology particularly when superior to known, close competitors. Influenced by mass media and trade journal reports. Socialites: use simulation products because major competitors, friends and children are using them, wish to keep up with trend. Influenced by friends, peers, and family. As the product matures, indications in the trade media suggest that it will become mainstream, and most likely, will be web-based – this makes up the early and late majority segment: Lifestylers: Partly overlapping previous categories, simulations provide convenience related to learning and development, particularly on-demand specific knowledge chunks. Influenced by free trials. The last two segments are unlikely to adopt: Misers: Members of this segment are unwilling to pay for simulations (or any training). Laggards: The last to know about and adopt new technologies. The figure below shows how each of these segments is influenced and influences another segment. The limited data we have on the actual use of simulations, games and the newer technologies including Vebeats gives us an indication that we are still, after 50+ years in the very low market penetration stage of the product lifecycle. Perhaps as much as 1% of the corporate training market and 2% of the academic market. It is predominantly the techno toys and some training professionals who are actively using, developing and promoting the use of Vebeats and a few trade journals and mass media are picking up on the potential of these technologies to change the way training and education are delivered. The October 2004 special report from Chief Learning Officer on simulations (Summerfield, 2004) suggests that the combination of high-cost, lack of organizational infrastructure, scarcity of quality products and services, and management-level comprehension deficiency have all contributed to the limited use of simulations and Vebeats. In other words, the techno toys and training professionals are not communicating the message to the right people. These are the very people who have the knowledge and experience and expertise to ensure that Vebeats are seriously considered for corporate learning Adoption Curve Lifestylers Misers & Laggards Socialites Techno Toys Technical Reports Employers & Clients Mass Media Friends & Family Free trials Sophisticates Training Pros Time M ar ke t Adoption Curve Lifestylers Misers & Laggards Socialites Techno Toys Technical Reports Employers & Clients Mass Media Friends & Family Free trials Sophisticates Training Pros Time M ar ke t 154 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 Competition Competitive against others Cooperative with others User group size Individual Team-based Visual Fidelity Textual High-end graphics Realism Simple Complex Usage Environment Stand-alone Blended Purpose Education Training and development and business school education. They also have the professional and academic objectivism that could assess and insist on quality standards helping to organize a coherent framework for platform development to reduce development costs ands reduce the fragmentation of the industry. Wolfe and Roge (1997) describe a variety of games suitable for different purposes allowing a potential user to select from the list, which goes some way to help overcome the difficulties above, that potential users don’t know what they are buying – and it must be possible to go further and provide quality and efficacy judgment on particular products. However, this is certainly not the first call for the establishment of standards in this field, Gosen and Washbush (2004) lament the failure of several initiatives to develop and validate a suitable and acceptable instrument to measure learning effectiveness in business simulations and games. BUILDING A STANDARD EVALUATION FRAMEWORK It is well recognized that establishing robust measures for learning effectiveness is difficult, and assessing the external validity of a business simulation, its fidelity, realism and appropriateness is perhaps, asking too much. Pragmatically, is it possible to create a framework that assesses whether or not a particular product, or method, appears to be efficacious in the environment for which it is designed (Gosen & Washbush, 2004) and what might such a standards framework include? Reeves (1997) provides a useful framework for conceptualizing the important factors for measurement in computer-based education. Adapting this to computer-based simulations and games and with a particular intention to focus on newer technologies, called Vebeats here, may provide a basis for discussion. Reeves considers fourteen dimensions to provide improved criteria for understanding, describing, and evaluating Computer-Based Education (CBE) that may be used to compare one form of CBE with another to overcome the noted difficulties in empirical evaluation to measure effectiveness of one method over another. The figure below, shows a summary of the pedagogical dimensions Reeves identifies: In light of more recent research in the field of Vebeats, we might consider adding or substituting other dimensions: Such dichotomous scales could include an empirical scale and be useful for an overall understanding of how a particular Vebeat is designed; it’s validity both internal and external. However, it still reflects traditional foundationalist views that force a dichotomy between each factor, though Epistomology Objectivism Constructivism Pedagogical Philosophy Instructivist Constructivist Underlying Psychology Behavioural Cognitive Goal Orientation Sharply-focussed Unfocussed Experiential Value Abstract Concrete Teacher Role Didatic Facilitative Program Flexibility Teacher-proof Easily modifiable Value of errors Errorless Learning Learning from Experience Motivation Extrinsic Intrinsic Accommodation of Individual Differences Non-existent Multi-faceted Learner Control Non-existent Unrestricted User Activity Mathemagenic Generative Cooperative Learning Unsupported Integral Cultural Sensitivity Non-existent Integral 155 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 inevitably must recognize that judgmental bias exists. Such a framework may be helpful in the research community or for reviewers to be able to compare one product with another or against a benchmark. It is deliberate that the framework is not suggesting what is good and what is bad, nor does it intend to suggest that one a product with a profile towards the right is in any way superior to one with a left profile. It may help researchers gain a clearer understanding of what attributes of a particular simulation or game are causing learning of a particular aspect of knowledge or developing a particular skill set. The following figure is an example of how this framework might be used comparing two very different simulations, a commercial airlines flight simulator used for training airline pilots, and Imparta’s Strategy CoPilot™ - a business simulation or Vebeat used to develop strategic management knowledge and skills. The choice of position on each scale is the author’s own, though influenced by discussions with colleagues and users of the simulations, and the purpose is to illustrate the concept. Clearly the two simulations have very different learning purposes, but to serve their respective purposes well, they share some similarities and some marked differences. Realism, for example, in a commercial airlines flight simulator is as close to real as it is possible to be on the basis that trainee pilots need to practice in a completely realistic way but that doesn’t place lives at risk. Interestingly, the US Air Force has been reported to be using video game simulations that are less realistic and much less expensive in training pilots in the early stages of training. To take the other example, Strategy CoPilot™, the simulation has a certain amount of realism within it, utilizing realistic situations with video-based characters that the user interacts with – however, time is accelerated within the simulation and users benefit from an improbably useful personal digital assistant that neatly keeps and provides access to critical information and the appropriate business models. Too much realism in such soft-skills training or education and users might get lost in the plethora of other events and not focus on what is being learnt. Teacher role in the above examples are also different, the didactic role with a CAS flight simulator is to tell trainee pilots how to fly the aircraft, the precision of pressing the right button at the right moment and applying just the right amount of throttle, flaps etc is critical, and it must be right – otherwise the aircraft falls from the virtual sky and crashes – and as someone who has crash-landed a 747 simulator, I can assure you that these beasts can be broken and you’ll be relieved to know that I won’t be at the front of your next flight. Such a framework may be useful in clarifying the key attributes of any simulation product and be beneficial for comparable research in learning effectiveness and in Objectivism Constructivism Epistomology Instructivist Constructivist Behavioural Cognitive Sharply-focussed Unfocussed Abstract Concrete Didatic Facilitative Teacher-proof Easily modifiable Errorless Learning Learning from Experience Extrinsic Intrinsic Non-existent Multi-faceted Non-existent Unrestricted Mathemagenic Generative Unsupported Integral Non-existent Integral Competitive against others Cooperative with others Individual Team-based Textual High-end graphics Simple Complex Stand-alone Blended Education Training Strategy CoPilot CAS Flight Simulator Purpose Value of errors Teacher Role Cultural Sensitivity Competition User group size Visual Fidelity Usage Environment Realism Learner Control Accommodation of Individual Differences User Activity Cooperative Learning Experiential Value Underlying Psychology Program Flexibility Motivation Pedagogical Philosophy Goal Orientation 156 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 describing the attributes of a simulation learning product to potential buyers. From this, the research community may be able to establish appropriate standards for particular learning objectives and be able to suggest how a particular simulation can be improved to achieve particular learning objectives. Using such a framework may also clarify the attributes where simulations outperform alternative methods and enable the simulation community to change the performance metric. It is possible that simulations provide a more concrete learning from experience that is intrinsically motivating. This would compare favorably in comparison with classroom learning, which is usually more abstract, and more intrinsically motivating than non-simulation based e- learning. If the simulation community can establish a framework to assess diverse simulation products, it mat then be able to communicate with the mass media and potential customers with useful, judgmental opinions about the quality of the product – perhaps in a way that has been successful for video-games and may be catching on in the e-learning community (see tmreview.com or eqcheck.com for examples). According to the CLO article cited above, potential simulation users want to know, the cost and quality of products and what they achieve. Beyond a description of the product, from agreed standards and means of assessment, it may be possible to provide simple answers to these questions with a scoring system such as the example below: Ease of installation 9/10 Ease of Navigation 7/10 Production quality 5/10 Instructional Value of content 5/10 Value for Money 6/10 Overall Value 5/10 (numbers shown are completely arbitrary and bear no relation to any product living or dead) By doing this, we can begin to address part of the issue for the lack of real growth in the use of simulations in education and training, and only by understanding what it is about simulations, the attributes they have over other methods, can we establish a performance metric that is sufficiently different, knowing that there are benefits to the education and training community, its students, users and paymasters. If we know what attributes are important, we can then guide development or emergence of the platform(s) that most easily, effectively and usefully provide the benefits. CONCLUDING THOUGHTS This paper began in an attempt to answer why simulations are not yet the “killer application” for education and training that some have been suggesting it will be (by now) and why the market in terms of dollars and users is still such a small amount compared to the whole education and training industry. Estimates of actual usage are particularly difficult to know with any real certainty, but it is safe to suggest that there is more classroom-based training and education than simulation-based. It may be that there are gatekeepers preventing simulation accessing the training and education market because they threaten HRD and teachers or it may be that the innovators and early adopters of the technologies are not communicating the right messages to the potential market. The industry is very fragmented and we are yet to see a dominant platform or player emerge and perhaps this is because the simulation advocate community has been reluctant to abandon performance metrics and evaluation that compare it with more traditional pedagogies, replete with semantic debate and a desire to make absolutely sure that simulations really are effective. I have gone on to suggest that we can learn from other disruptive technologies and consider a mindset shift, develop a framework to aid research, communicate with the potential market, compete on new performance metrics, change the business model and guide the development of a platform on which simulations can be built. It is likely that without the establishment of standards that the industry will continue to fragment until a small group gets together and defines standards through success, will it be a Microsoft or a Linux? REFERENCES Adkins, S. (2002). The 2002-2010 us market for e-learning simulation: The shape of the next generation e-learning market.: Brandon-Hall. Argyris, C., & Schon, D. (1978). Organizational learning: A theory action perspective. Reading, MA: Addison- Wesley. Barr, R. B., & Tagg, J. (1995). From teaching to learning: A new paradigm for undergraduate education. Change, 13-25. Boehle, S. (2005). Simulations: The next generation of e- learning, trainingmag.com: VNU. Brenenstuhl, D. C., & Catalanello, R. F. (1979). The impact of three pedagogue techniques on learning. Journal of Experiential Learning and Simulation, 1, 211-225. Burns, A. V., Gentry, J. W., & Wolfe, J. (Eds.). (1990). A cornucopia of considerations in evaluating effectiveness of experiential pedagogies. London: Kogan Page. Christensen, C. M. (1997). The innovator's dilemma: When new technologies cause great firms to fail. Boston, MA.: Harvard Business School Press. Economist.com. (2005). Mobile telecommunications. July 7th 2005. from http://www.economist.com/research/Backgrounders/dis playBackgrounder.cfm?bg=1258181 157 http://www.economist.com/research/Backgrounders/displayBackgrounder.cfm?bg=1258181 http://www.economist.com/research/Backgrounders/displayBackgrounder.cfm?bg=1258181 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 158 ESA. (2005). Facts and research. from http://www.theesa.com/facts/top_10_facts.php Faria, A. J., & Wellington, W. J. (2004). A survey of simulation game users, former-users and never-users. Simulation & Gaming, 35(2), 178-207. Feinstein, A. H., & Cannon, H. M. (2003). A hermeneutical approach to external validation of simulation models. Simulation & Gaming, 34(2), 186-197. Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441-467. Geber, B. (1994). Let the games begin. Training, 31(4), 10- 15. Gilbert, A. L., & Kendall, J. D. (2002). A marketing model for mobile wireless services. Paper presented at the 36th Hawaii International Conference on System Sciences, Hawaii. Gopinath, C., & Sawyer, J. E. (1999). Exploring the learning from an enterprise simulation. Journal of Management Development, 18(5), 477-489. Gosen, J., & Washbush, J. (2004). A review of scholarship on assessing experiential learning effectiveness. Simulation & Gaming, 35(2), 270-293. Gredler, M. E. (1996). Educational games and simulations: A technology in search of a (research) paradigm. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan. Hays, R. T., & Singer, M. J. (1989). Simulation fidelity in training systems design: Bridging the gap between reality and training. New York: Springer-Verlag. Hoberman, S., & Mailick, S. (1992). Experiential management development. New York: Quorum. Kenworthy, J. M. (2005). Evaluation of the effectiveness of using computer-based training simulations to develop managerial competency. Unpublished DBA, Henley Management College, Henley-on-Thames. Keys, J. B., Wells, R. A., & Edge, A. G. (1994). The multinational management game: A simuworld. The Journal of Management Development, 13(8), 26-37. Lundy, J., Logan, D., & Harris, K. (2002). Simulation may be your e-learning 'killer application': Gartner. MacDonald, B., Atkin, R., Jenkins, D., & Kemmis, S. (1977). Computer assisted learning: Its educational potential. In R. Hooper (Ed.), The national development programme in computer assisted learning. The final report of the director (pp. Chapter 3.). London: Council for Educational Technology. Naish, R. (2005). Why games are so serious. e-learning age, April 2005, 16-17. OECD. (2005). OECD communications outlook 2005 (Vol. 2005): OECD. Pierfy, D. A. (1977). Comparative simulation game research: Stumbling blocks and steppingstones. Simulation and Gaming, 8(2), 255-268. Reeves, T. (1997, 21 Nov 1997). Evaluating what really matters in computer-based education. from http://www.educationau.edu.au/archives/cp/reeves.htm Romme, A. G. L. (2003). Learning outcomes of microworlds for management education. Management Learning, 34(1), 51-61. Senge, P. (1990). The fifth discipline. New York: Doubleday Currency. Summerfield, B. (2004). Business intelligence: The growth of simulations. Chief Learning officer - Special report, October 2004, 20-22. Summers, G. J. (2004). Today's business simulation industry. Simulation & Gaming, 35(2), 208-241. Suqrue, B., & Kim, K. (2004). 2004 state of the industry report: ASTD. Swanson, R. A., & Holton, E. F., III. (1999). Results: How to assess performance, learning in the art of systematic change. San Francisco: Berret-Koehler Publications, Inc. Thompson, C., Koon, E., Woodwell, W. H. J., & Beauvais, J. (2002). Training for the next economy: An ASTD state of the industry report on trends in employer- provided training in the united states. Alexandria, VA.: ASTD. Totty, M. (2005, April 25 2005). Business solutions: Better training through gaming. The Wall Street Journal, p. Page R6. Wolfe, J., & Roberts, C. R. (1993). A further study of the external validity of business games: Five year peer group indicators. Simulation & Gaming, 24, 21-33. Wolfe, J., & Roge, J., N. (1997). Computerized general management games: The effectiveness of business games in strategic management course work. Simulation & Gaming, 28(4), 423-441. Zemsky, R., & Massy, W. F. (2004). Thwarted innovation: What happened to e-learning and why (A Learning Alliance Report): Learning Alliance. http://www.theesa.com/facts/top_10_facts.php http://www.educationau.edu.au/archives/cp/reeves.htm Table of Contents Volume 33, 2006 Learning Assurance Using Business Simulations Applications To Executive Management Education Team Teaching In An Integrated Business Course Using Critical Problem Based Learning Factors In An Integrated Undergraduate Business Curriculum: A Business Course Success Personality Type And Strategic Planning Business Games As Strategic Management Laboratories The Relationship Between Students' Success On A Simulation Exercise And Their Perception Of Its Effectiveness As A PBL Problem Forecasting Accuracy And Learning: The Key To Measuring Simulation Performance The Business Strategy Game: A Performance Review Of The New Online Edition Using The Socratic Method And Bloom's Taxonomy Of The Cognitive Domain To Enhance Online Discussion, Critical Thinking, And Student Learning Using Negotiation Exercises To Promote Critical Thinking Skills Effective Leadership Experiences For Management Majors In A Futures Class The Role Of Learning Versus Performance Orientations When Reacting To Negative Outcomes In Simulation Games Is Pay Inversion Ethical? A Three-Part Exercise Simulations And Experiential Exercises - Do They Result In Learning? Have We Figured It Out Yet? Examining Program Management In Business Simulations: Student And Faculty Views Validating Business Simulations: Do Simulations Exhibit Natural Market Structures? Characterizing Business Games Used In Distance Education Utilizing Games In A Graduate Level Instructional Game Course Employment Interview Preparation: Assessing The Writing-To-Learn Approach Simulations - Bridging From Thwarted Innovation To Disruptive Technology Creating An Authentic Cultural Lens Using Case Dialogue Learning By Fire: Reflections Of A First Time Online Instructor An International Internship With A Service-Learning Focus Learner Participation In The Online Learning Experience: Help Or Hindrance? Any Given Sunday: Intervention In Pursuit Of Simulation Team Parity Beginning With The End: Creating An Experiential Exercise From Assessment Criteria Simulating Life Cycles: Life Span As The Measure Of Performance In Business Gaming Simulations It's Puzzling: Communications, Competition, And Cooperation Balanced Scorecard Implementation For Strategy Management: Variation Of Manager Opinion In Real And Simulated Companies Cases And Business Games: The Perfect Match! Three-Attribute Interrelationships For Industry-Level Demand Equations Using A Web-Based Module To Teach Information Literacy Decision Support System For Demand Forecasting In Business Games The Invalidity Of Profit=F(Market Share) PIMS Validation Of Marketing Games Online Market Test Laboratory With The MINSIM* Program The Gas Mileage Game - A Policy Simulation Delivered Cost And Differentiation Applied To Threshold 3rd Ed. The Effect Of Team-Leadership Modes On Team Performance: A Preliminary Study The Design And Use Of A Macroeconomics Simulation Using Maple Software: A Pilot Study The Instructor's Toolbox: A Meaning-Centered Framework For The Social Construction Of Experiential Learning Incorporating Strategic Product-Mix Decisions Into Simulation Games: Modeling The 'Profitable-Product Death Spiral' Group Composition And Groupthink In A Business Game A Direct Approach To Teaching Business Ethics A Decision Support System For Planning Sales, Production, And Plant Addition With Manager: A Computer Simulation Polish - American Entrepreneurial Business Cooperation Workshop Utilizing The Income/Outcome Simulation Student Leader Training Exercise Student Preference To Mode Of Learning In Hong Kong Experiential Learning For Technology-Based And Management Programme In Hong Kong: A China Study Tour A Price Game With Product Differentiation In The Classroom Discrete Event Modeling In A New Transportation Simulation Supply-Side Modeling In A Total Enterprise Simulation The Quality Game Towards A Massive Multiplayer Online Business Simulation Making The Connection: Improving Virtual Team Performance Through Behavioral Assessment Profiling And Behavioral Cues Individual Learning Producing A Learning Organization: 'Playing Dice With Polar Bears' Narratology and Ludology: Competing Paradigms or Complementary Theories in Simulation Framework For Evaluating Internet Research Using Children's Games To Illustrate Strategy Concepts: Is Less Better?