A Framework for Evaluating Internet Research Developments in Business Simulation and Experiential Learning, Volume 33, 2006 A FRAMEWORK FOR EVALUATING INTERNET RESEARCH J. Alexander Smith St Louis University Smith82@slu.edu ABSTRACT Though the Internet is an interesting phenomenon, there is relatively little research that addresses the added-value of using it. Much of the literature is focused on attempts to apply the technology to existing methodologies. With this focus on Internet application, there is relatively little progress being made toward a general theory of the Internet. This paper presents a basic framework for understanding the Internet and how it relates to current methods and thoughts. INTRODUCTION The growing acceptance of the Internet has sparked increasing research about the Internet and its capabilities. Much of this research has gone towards understanding the limits of the Internet and how to apply it to existing methodologies, creating a process of often using the Internet without fully understanding the value of its use. This is especially true in the area of education. The last decade has shown an increase in research on how to use the Internet for instruction, how to evaluate this Internet instruction, and how to adapt traditional (in class) teaching methods to the Internet. Additional research has compared the efficacy of Internet education versus the more traditional methods. Little has been done to research the added benefits (if any) that occur from using the Internet for teaching. This lack of research is due mainly to the lack of a unifying direction, or framework for understanding how the Internet interacts with educational practices. Smith (2005) proposed one possible framework. This paper takes Smith’s basic idea and builds upon it, creating a more formalized framework for future evaluation of the Internet. NEED FOR UNIFYING DIRECTION FOR RESEARCH AND APPLICATION The Internet has received both praise and alarm. For many, it is said that the Internet has the power to change the way that education and business are done (Ash and Burn, 2003; Folkers, 2005). Others see it as an over-hyped tool that adds little to the overall value of education in its current form (without major additional investments) (Gladieux, 2000). Good or bad, it is here to stay and thus merits further investigation. Many researchers have taken it upon themselves to examine the internet, but the findings of this study show that the vast majority of articles look at it as a tool to enhance current processes. This research fails to examine the added benefit of the Internet. As a tool, the Internet has limited value for further academic research. There is no need to examine in detail the abilities of the telephone or fax machine, and the same can be said about the Internet, if it is only another tool. But unlike other tools, the Internet appears to provide more than simple facilitation of accepted methods. It is from the possibility of additional value that the Internet becomes a phenomenon worthy of study. The previous discussion raises a pertinent question, “Is the Internet a tool for facilitation or a new paradigm for thinking?” This question is the basis of a paper presented by Smith (2005). He suggests an evolutionary process where the Internet is first used to facilitate delivery of existing pedagogy. It then evolves into pedagogy itself; the message is the medium. The final stage is the development of new paradigms. Smith suggests that it is impossible to have a new paradigm for the Internet without first going through the first two stages. He further suggests that it is a lack of understanding of this process that has researchers continually evaluating the facilitation process and ignoring the true value of the Internet, the potential paradigm shift. This process is similar to one proposed by Haeckel (1998, 1985) who suggested that the use of new technology follows an ‘S’ curve. In the first stage, new technology is used to facilitate existing practices to increase efficiency and convenience. The second stage looks at issues of effectiveness. That is, it answers old questions that were previously unanswerable, or ignored. The final stage is the creation of previously unthinkable new concepts (i.e. paradigm change). Using these ideas as stepping stones, this paper examines current research about online education, to formally develop a general framework. METHODOLOGY A literature search is used to develop the framework. It was initially undertaken using the ABI/Inform database available through Proquest.com. At the time, there were over 16000 “scholarly” articles in the database that use the keyword ‘Internet’ in the title or abstract. Additional keywords were then added until a reasonable number of articles were presented. As such, the final keywords used (Internet, online, education and 373 mailto:Smith82@slu.edu Developments in Business Simulation and Experiential Learning, Volume 33, 2006 Figure 1 learning) were all required to be present in the title or abstract and presented a list of 94 articles for review. These articles were then scanned for characteristics and attributes related to the Internet and its abilities. During this process additional issues of related interest were unveiled and thus added to the general framework for consideration. After the various Internet characteristics were discovered, they were then grouped together for ease of reference to form different categories. These categories were then applied to the basic idea presented by Smith (2005). From this process, the framework in figure 1 was designed. FRAMEWORK The framework has 3 basic parts, the inputs (1), processes (2), and outputs (3) with the processes being split into two sections, Tools (4) and New Methods (5). The arrows connecting the different parts and sections represent the interactions between parts. The following will further expand each part of the framework. INPUTS Inputs pertain to the items that are necessary for any action to occur. As the point of this discussion is about understanding the Internet, it is not surprising that one of the main inputs discovered for this process was the physical Internet. This would include all items necessary for the Internet to exist and run. These items include both hardware (cables, servers, etc) and software (web browser, HTML code, etc). This also includes issues of bandwidth and speed (Liaw, Pearce, et al, 2002; Aft 2000). Without these items, there would be no Internet and thus a necessary requirement for any Internet activity. Additionally, these inputs do not include the actions taken with this technology. This is such that a web browser is considered an input, but the websites created are applications of the technology and thus not an input. These applications of technology are considered a combination of inputs and further explanation will occur under the ‘processes’ section below. A second specific input was also specified. Current methods are thoughts that exist independent of other inputs. Such items may be pedagogies or theories. Some of which may include multiple choice exams for education or even the 4Ps (price, product, placement, promotion) of marketing. Individually these methods can be of interest for further research but have little relation to the Internet specifically. As such, it is only when current methods are combined with the Internet that they become of special interest to Internet researchers. It is this combination of inputs that creates the applications for the Internet; these applications are defined as tools (4) in regards to the framework. PROCESSES Processes deal with the creation of new activities that would not exist without the interaction of various inputs. For our framework, two necessary inputs were presented; physical Internet and current methods. As this research has focused on the use of the Internet, it is not surprising that these inputs are a necessary requirement for the process stage of the framework. The processes are split into two sections, tools (4) and new methods (5). The first section, tools, requires the combination of different inputs. It is in the combining of these different inputs that change occurs. It is these changes Tools New Methods OutcomesInputs Processes Current Methods Physical Internet Paradigm Added Learning 54 3 21 374 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 that that have received the majority of research over the years. ‘Tools’ pertains to the Internet (input #1) being used to facilitate current methods (input #2). This is in fact a process of efficiency where current methods become more efficient. Various applications have been created by the Internet to develop increased efficiencies. In education, examples include the more efficient distribution of course materials (Shrivastava, 1999) and instructor-to-student communication (Beer, Green, et al., 2002). In marketing, examples include more efficient method for customer service or opinion polling (Kaynama and Keesling, 2000). This increase in efficiency is normally translated into lower overall costs (McFadzean, 2001; Burke and Slavin, 2000; Seay, Rudolph, et al, 2003). The second section of the ‘processes’ are new methods. These are new methods that did not exist prior to the combination of inputs. They take the facilitation process offered by the ‘tools’ and then create something new that wasn’t feasible before. Generally this entails the combination of different tools whose inputs were previously unable (or unwilling) to combine before. In other words, the facilitation of old methods are enhanced by adding another old method that couldn’t be combined before the Internet was available. For example, customer service has been available for years, but the addition of various tools facilitated by the Internet provides an additional level of interaction not seen before (Kaynama and Keesling, 2000). The combination of these would create a new method. In education distance learning has existed for a long time, as has real-time learning (classroom) but until the Internet they could not be both real-time and interactive distance education (Evans and Haase, 2001). For some, this is the final limit or ‘Holy Grail’ of the Internet. It has allowed us to take actions that were previously unavailable. But this short-sighted view ignores the fact that just because it can be done, doesn’t mean that it should be done. Spending large sums of money on new machinery that does something new is worthless unless there is some additional benefit to the purchase. What then is the value of these new methods? OUTPUTS Outputs (3) deal with the final result of the processes (2). It answers the question, “what’s the value?” It is this part of the framework where the added-value of using the Internet is explored and explained. This is also the part of the framework that is lacking the most in the literature. This lack is not necessarily from oversight, as previous authors have raised the question “what’s the value?” (Windschitl, 1998; Owston, 1997; Mergendoller, 1996) As new methods (5) are created, the outputs (3) are the eventual gain that is received from these methods. But what are they and how are they measured? It has been suggested that the internet has the potential of increasing levels of learning (Poon, Low, et al, 2004), or changing the prevailing marketing theories (Sharma and Sheth, 2004). For some, it is hoped that the Internet will provide the output of a paradigm shift or new pedagogies (Eastman and Swift, 2001; Volery and Lord, 2000). Yet these suggestions lack the ability to measure the resultant outcomes and thus make further research on outputs difficult to pursue. Additionally, it is difficult to expand further on the outputs as there is relatively little written on the subject beyond the need for more research to be done. For education, it parallels the discussion on experiential learning. Many instructors feel that experiential learning is extremely effective and the best way to teach, but it is difficult to measure or even identify this additional learning (Gentry and Burns, 1997). Similarly the benefits of using the Internet are, so far, hard to identify or measure. Perhaps the answer to understanding (and measuring) the outputs is the evaluation of the entire process of development as represented by the framework as a whole. DISCUSSION AND CONCLUSIONS The framework provides a solid foundation and direction for future research. Both the individual parts and their relevant connections should be more fully studied. The literature search showed that online education research focuses on the connection between Inputs and Tools. These articles range from a focus on how to use a website for course delivery (Hornyak et al., 2001) to explaining the use of e-mail attachments for enhanced classroom instruction (Cabaniss and Portis, 2004). Other research has shown how to use the Internet for class lectures (Pillutla, 2000) or how to convert simulations and other teaching aides to the Internet (Palia, Keong, and Roussos, 2000; Lai and Siau, 2003). Other articles have looked at the Internet as a tool for course administration and class management (Kumar, Merriman, et al., 2001). Though most of the literature focused on the Internet as a tool, this is not to say that research on the topic has ignored the connection between tools and new methods completely. In one attempt to study the potential outputs, Arbaugh (2000) examined the additional learning that was occurring from the Internet teaching environment by examining test scores between traditional and Internet classes. Some research examined a hybrid use of Internet and traditional classroom usage, the results of these studies (Thoennessen et al. 1999) show the possibility of some increased learning, or in terms of the framework, the results examine the existence of different outputs. Eastman and Swift (2001) tried to show that the classic paradigm of ‘Chalk and Talk’ (lecture based teaching) is slowly being replaced by a more interactive nature, or a ‘Show and Go’ process. Though a promising first step in evaluation, many of these articles are lacking for a clear definition for study, or are methodologically flawed (Gosen, 2003). These flaws may not be corrected through use of the framework, but it 375 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 may allow for a more solid base to springboard future attempts at measuring these concepts. Beer, Martin, Sharon Green, Gillian Armitt, Johanna Bruggen, et al. (2002), “The provision of education and training for health care professionals through the medium of the Internet” Campus - Wide Information Systems, 19 (4), 135. This framework also takes steps in answering some of the questions that were left unanswered by Smith (2005). It does not explicitly answer the questions “At what point does the Internet cease to be just a more effective delivery system?” and “When does the medium become the message?”, but it does provide a direction for study. From a general overview of the framework it is possible to see that that the Internet is still only a collection of tools in the new methods section (5) and ceases to be so before the output section (3). As such it is easy to see that further research into the connection between these two parts will lead to a solid answer to the first question. A similar process can be applied to the second question as well. Burke, Jacqueline and Nathan Slavin (2000), “Just-in time accounting education” The CPA Journal, 70 (4), 46. Cabaniss, Roy F. & Ron Portis (2004) “How to Receive and Process Attachments while greatly Reducing the Risk of Viruses and Trojans.” Developments in Business Simulation and Experiential Learning, 31, 51-57. Reprinted in The Bernie Keys Library, Hugh M. Cannon (ed). [Available from http://www.absel.org] Eastman, Jacqueline K. and Cathy O. Swift (2001), “New horizons in distance education: The online learner- centered marketing class” Journal of Marketing Education, 23 (1), 25. As much as the framework might help in guiding future research, in its current form, this framework is limited in its generalizability. It was created using literature on Internet based education and thus there are no claims that it will work on other areas of Internet interest. But the general nature of the attributes found in this review could be applied to any application of the Internet. Furthermore, Sharma and Sheth (2004) used a similar process when identifying potential paradigm changes for marketing theory. They looked at the specific attributes of the Internet and what it allowed them to do that was previously impossible. They then extrapolated potential value from these attributes, a process similar to the transition of new methods (5) to outputs (3) described in the framework. Evans, Joel R. and Ilene M. Haase (2001), “Online business education in the twenty-first century: An analysis of potential target markets” Internet Research, 11 (3), 246. Folkers, Deidre A. (2005) “Competing in the Marketspace: Incorporating Online Education into Higher Education – An Organizational Perspective” Information Resources Management Journal, 18 (1) 61 Gentry, James W. & Alvin C. Burns. (1997) “Thoughts About The Measurement Of Learning: The Case For Guided Learning And Associated Measurement Issues.” Development in Business Simulation and Experiential Learning, 24, 241-246. Reprinted in The Bernie Keys Library, Hugh M. Cannon (ed). [Available from http://www.absel.org] Another limitation is the lack of connection to global issues. The Internet is a global medium whether it is used for local or long-distance interactions. Consideration of these different distances needs to be taken into account. Furthermore laws and customs are not represented and will likely need to be added for future use of this framework. Gladieux, Lawrence E. (2000) “Global On-line Learning: Hope or Hype?” Higher Education in Europe, 25 (3) 351-353 Haeckel, Stephen H. (1985) in In Marketing in an Electronic Age, R. Buzzell (ed.), Cambridge, MA: Harvard Business School Press, p 320 Despite these limitations, this paper presents a basic framework for understanding the Internet and how it relates to current methods and thoughts. It shows the interactions that occur in developing new uses for the Internet. Though the Internet is an interesting phenomenon, there is relatively little research that addresses the added-value of using it. This framework highlights this fact and provides direction for future research. Haeckel, Stephen H. (1998) “About the nature and future of Interactive Marketing” Journal of Interactive Marketing, 12(1) 63-71 Hornyak, Martin J., Fred R. Blass and Eric S. Holt (2001) “Integrating Management Curriculum: Linking Courses with Frontpage.” Developments in Business Simulation and Experiential Learning, 28, 106-110. Reprinted in The Bernie Keys Library, Hugh M. Cannon (ed). [Available from http://www.absel.org] REFERENCES Kaynama, Shohreh A. and Garland Keesling (2000), “Development of a Web-based Internet marketing course” Journal of Marketing Education, 22 (2), 84. Aft, Larry (2000), “www.industrial_engineering.edu.” IIE Solutions, 32 (5), 32. Arbaugh, J. Ben (2000) “Virtual Classroom versus Physical Classroom, An Exploratory Comparison Of Class Discussion Patterns And Student Learning In An Asynchronous Internet-Based MBA Course.” Journal of Management Education, 24: 207-227. Kumar, M. S. Vijay, Jeff Merriman and Philip Long (2001), “Building 'open' frameworks for education” EDUCAUSE Review, 36(6), 80. Ash, C. G. and J. M. Burn (2005) “A strategic framework for the management of ERP enabled e-business change” European Journal of Operational Research, 146 (2) 374 376 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 377 Lai,Yuen Poh & Tan Long Siau (2003) “Ebiz Game: A Scalable Online Business Simulation Game For Entrepreneurship Training.” Development in Business Simulation and Experiential Learning, 30, 270-278. Reprinted in The Bernie Keys Library, Hugh M. Cannon (ed). [Available from http://www.absel.org] Liaw, Siaw-Teng, Chris Pearce and Mike Keppell (2002), “Developing a Web-based learning network for continuing medical education” Journal of Workplace Learning, 14 (3), 98. McFadzean, Elspeth (2001), “Supporting virtual learning groups. Part 1: A pedagogical perspective” Team Performance Management, 7 (3/4), 53. Mergendoller, John R. (1996) “Moving From Technological Possibility To Richer Student Learning: Revitalized Infrastructure And Reconstructed Pedagogy.” Educational Researcher, 25 (8) 43-46 Owston, Ronald D. (1997) “The World Wide Web: A Technology to Enhance Teaching and Learning?” Educational Researcher, 26, (2) 27-33 Palia, Aspy P., Mak Wai Keong and Dean S. Roussos (2000) “Facilitating Learning In The New Millennium With The Complete Online Decision Entry, System (Codes).” Development in Business Simulation and Experiential Learning, 27,250-251. Reprinted in The Bernie Keys Library, Hugh M. Cannon (ed). [Available from http://www.absel.org] Pillutla, Sharma (2000) “Creating a Comprehensive Web- Enhanced Classroom.”; Development in Business Simulation and Experiential Learning, 27, 210-216. Reprinted in The Bernie Keys Library, Hugh M. Cannon (ed). [Available from http://www.absel.org] Seay, Robert A., Holly Rudolph and Margaret N. Boldt (2003), “Online CPE-getting easier all the time” Journal of Accountancy, 196 (1), 63. Sharma, Arun and Jagdish N. Sheth (2004) “Web-based marketing: The coming revolution in marketing thought and strategy.” Journal of Business Research; 57 (7) 696-703 Shrivastava, Paul (1999), “Management classes as online learning communities” Journal of Management Education, 23 (6), 691. Smith, J. Alexander (2005), “Evaluating the Direction of Research in Online Education: Are we Going Anywhere?” Developments in Business Simulation and Experiential Learning, 32, forthcoming. Reprinted in The Bernie Keys Library, Hugh M. Cannon (ed). [Available from http://www.absel.org] Thoennessen, M. and E. Kashy, Y. Tsai and N. E. Davis (1999) “Impacts Of Asynchronous Learning Networks In Large Lecture Classes” Group Decision and Negotiation, 8, 371-384. Volery, Thierry and Deborah Lord (2000), “Critical success factors in online education” The International Journal of Educational Management, 14 (5), 216. Windschitl, Mark (1998) “The WWW and Classroom Research: What Path Should We Take?” Educational Researcher, 27, (1) 28-33 Table of Contents Volume 33, 2006 Learning Assurance Using Business Simulations Applications To Executive Management Education Team Teaching In An Integrated Business Course Using Critical Problem Based Learning Factors In An Integrated Undergraduate Business Curriculum: A Business Course Success Personality Type And Strategic Planning Business Games As Strategic Management Laboratories The Relationship Between Students' Success On A Simulation Exercise And Their Perception Of Its Effectiveness As A PBL Problem Forecasting Accuracy And Learning: The Key To Measuring Simulation Performance The Business Strategy Game: A Performance Review Of The New Online Edition Using The Socratic Method And Bloom's Taxonomy Of The Cognitive Domain To Enhance Online Discussion, Critical Thinking, And Student Learning Using Negotiation Exercises To Promote Critical Thinking Skills Effective Leadership Experiences For Management Majors In A Futures Class The Role Of Learning Versus Performance Orientations When Reacting To Negative Outcomes In Simulation Games Is Pay Inversion Ethical? A Three-Part Exercise Simulations And Experiential Exercises - Do They Result In Learning? Have We Figured It Out Yet? Examining Program Management In Business Simulations: Student And Faculty Views Validating Business Simulations: Do Simulations Exhibit Natural Market Structures? Characterizing Business Games Used In Distance Education Utilizing Games In A Graduate Level Instructional Game Course Employment Interview Preparation: Assessing The Writing-To-Learn Approach Simulations - Bridging From Thwarted Innovation To Disruptive Technology Creating An Authentic Cultural Lens Using Case Dialogue Learning By Fire: Reflections Of A First Time Online Instructor An International Internship With A Service-Learning Focus Learner Participation In The Online Learning Experience: Help Or Hindrance? Any Given Sunday: Intervention In Pursuit Of Simulation Team Parity Beginning With The End: Creating An Experiential Exercise From Assessment Criteria Simulating Life Cycles: Life Span As The Measure Of Performance In Business Gaming Simulations It's Puzzling: Communications, Competition, And Cooperation Balanced Scorecard Implementation For Strategy Management: Variation Of Manager Opinion In Real And Simulated Companies Cases And Business Games: The Perfect Match! Three-Attribute Interrelationships For Industry-Level Demand Equations Using A Web-Based Module To Teach Information Literacy Decision Support System For Demand Forecasting In Business Games The Invalidity Of Profit=F(Market Share) PIMS Validation Of Marketing Games Online Market Test Laboratory With The MINSIM* Program The Gas Mileage Game - A Policy Simulation Delivered Cost And Differentiation Applied To Threshold 3rd Ed. The Effect Of Team-Leadership Modes On Team Performance: A Preliminary Study The Design And Use Of A Macroeconomics Simulation Using Maple Software: A Pilot Study The Instructor's Toolbox: A Meaning-Centered Framework For The Social Construction Of Experiential Learning Incorporating Strategic Product-Mix Decisions Into Simulation Games: Modeling The 'Profitable-Product Death Spiral' Group Composition And Groupthink In A Business Game A Direct Approach To Teaching Business Ethics A Decision Support System For Planning Sales, Production, And Plant Addition With Manager: A Computer Simulation Polish - American Entrepreneurial Business Cooperation Workshop Utilizing The Income/Outcome Simulation Student Leader Training Exercise Student Preference To Mode Of Learning In Hong Kong Experiential Learning For Technology-Based And Management Programme In Hong Kong: A China Study Tour A Price Game With Product Differentiation In The Classroom Discrete Event Modeling In A New Transportation Simulation Supply-Side Modeling In A Total Enterprise Simulation The Quality Game Towards A Massive Multiplayer Online Business Simulation Making The Connection: Improving Virtual Team Performance Through Behavioral Assessment Profiling And Behavioral Cues Individual Learning Producing A Learning Organization: 'Playing Dice With Polar Bears' Narratology and Ludology: Competing Paradigms or Complementary Theories in Simulation Framework For Evaluating Internet Research Using Children's Games To Illustrate Strategy Concepts: Is Less Better?