Indian Journal of Educational Technology Volume 3, Issue 2, July 2021 1 Research Article Rethinking Program Evaluation in Educational Technology David Devraj Kumar Professor of Science Education, College of Education Florida Atlantic University, United States of America (U.S.A.), Email: david@fau.edu Abstract Ideas for rethinking program evaluation in educational technology in order to make informed decisions to reform teaching and learning are explored. The complexity of program evaluation when dealing with complex educational technology systems involving various subject disciplines, grade levels and the degree (e.g., how, where) to which it overlaps and/or integrates with each individual discipline and grade demands more comprehensive approaches to evaluation. A comprehensive evaluation model placed in real-world supportive and contextual factors with insights from classic evaluation methods is suitable for gaining a detailed layout of educational technology programs. Keywords: Program Evaluation, Educational Technology, Context, Comprehensive, Teaching , Learning Introduction Program Evaluation in educational technology is an essential area of inquiry with considerable impact on almost every aspect of education, especially teaching and learning in this twenty-first century. Reforming educational technology remains a priority in the United States, Canada, India, Europe, Australia, and many countries worldwide because of its significant role in developing a technology-literate workforce. An apparent ongoing rush to implement educational technology is evident in almost every sector of education from K-12 to post-secondary levels. However, adequate program evaluation methods suitable for evaluating educational technology are still lacking. This lack of sufficient program evaluation methods originates primarily due to a lack of a clear understanding of educational technology, and a scarcity of strategies to fully integrate educational technology in education. Often, conventional program evaluation methods developed for single discipline evaluations are arbitrarily applied to evaluate educational technology because of a push for educational technology integration in teaching and learning from businesses especially computer and software industries. In this context, further discussion will address how to think about developing program evaluation methods for educational technology. Before proceeding further, it is necessary to define operationally the terms program evaluation and educational technology in the context of this paper.. Program Evaluation Program evaluation is a complex, but systematic method of gathering and analyzing qualitative and quantitative Indian Journal of Educational Technology Volume 3, Issue 2, July 2021 2 information (data) to determine the value (efficiency and effectiveness) of a group of related educational activities aimed at achieving the intended educational outcome (CDC Program Evaluation, 2018, Stufflebeam and Shinkfield, 2007, Weiss, 1997; Fitzpatrick, Sanders, and Worthen, 2004; Altschuld and Kumar, 2002). Often mistakenly used interchangeably with classroom testing, the term evaluation is defined and used in many different ways in education. In earlier days, quantitative methods were dominant in the evaluation scene. However, as Weiss (1993) said, compared to previous days, “evaluations are making effective use of a wider array of methods and techniques. Perhaps the most notable difference from earlier days is the more frequent use of qualitative methods of study. Evaluators today engage in intensive interviewing, observation, review of documents, and other such techniques. They often spend enough time on-site to observe changes in the environment, program, and participants and to develop insights about conditions associated with beneficial change. They have more to say about “how” of programs and the “why” of consequences” (p. 108). The earlier days Weiss was referring to was a period when the “effectiveness of social science in general, and evaluation in particular, for guiding the improvement of anti- poverty policies and programs’’ during the “war on poverty” in the sixties and seventies in the United States (p. 107). In this twenty-first century, what we as a society across the world encounter is somewhat similar in nature, not hunger, but an apathy for education. Educational Technology Educational technology refers to a family of technological (software, hardware and associated technologies) tools mostly based on computer technology devoted to promoting more engaging, interactive and individualized learning experiences. Laptops, desktops, multimedia, interactive media, the Internet, World Wide Web, whiteboards, iCloud, videoconferencing, cellphones, iPads, virtual/augmented reality, robotics, artificial intelligence, and software systems are examples of educational technology (International Society of Technology in Education, ISTE, 2021; Lazaro, 2020). The ISTE has created standards for students, educators and education leaders. As Handal and Herrington (2003) argued, how to use educational technology in teaching and learning should be one of the highest priorities in its implementation. Following discussion will explore ways to think about program evaluation in helping stakeholders of education understand educational technology in order to make informed decisions to improve in teaching and learning. Program Evaluation in Educational Technology As a primer to exploring program evaluation in educational technology, an understanding of the ways in which educational environments (e.g., classrooms) implementing educational technology differ from traditional educational environments is necessary. The goal for using educational technology in classrooms is to emphasize student-centered Indian Journal of Educational Technology Volume 3, Issue 2, July 2021 3 approaches to higher order thinking and problem solving skills with real-world connections (Tata Trusts, 2019). On the other hand, traditional classrooms tend to be textbook-centered and teacher-centered, where lower level (e.g., memory recall) skills are often the focus. Since there are considerable differences between educational technology classrooms and traditional classroom environments it is obvious that there is a need for developing program evaluation methods tailored to educational technology with respect to teaching and learning, and it is not an easy task. Because, if not carefully developed and implemented, a program evaluation plan purposed for evaluating educational technology programs might be skewed to evaluating the technology and not the educational processes of teaching and learning and thereby considerably deviating from evaluating the education program that is implementing educational technology in its entirety. It should be emphasized that educational technology and program evaluation are complex in their own terms; therefore program evaluation in educational technology is a much more complex and challenging task. Insights from Classic Program Evalu- ations Insights from classic program evaluation models generally suitable for single discipline evaluation might, in sum, help in realizing the challenges in developing evaluation models for complex educational technology environments. The Context, Input, Process, Product (CIPP) model (Stufflebeam, 1983) is suitable for evaluating program components such as the plan, implementation, and outcomes. Pre- and post-tests, survey instruments, and interviews are part of the evaluation methods. The CIPP model is simplified by Shavelson, McDonnell, and Oakes (1989) in their approach to evaluating the state of US science and mathematics education by looking at Input – student backgrounds and teacher quality, Process – the quality of the curriculum and teaching, and Output – achievements and attitudes. According to Odden (1990), some of the limitations of this model are the loosely defined input-process-output connection, absence of process variables, and a lack of socio-demographic indicators that are more valid and reliable. Therefore, while dealing with program evaluation in science education, Altschuld and Kumar (1995) noted that “mere input-output approaches may not be sufficient to determine the success of an educational reform. Rather, process variables and gaining an understanding of the interactions amongst variables are essential for assessing the nature and effectiveness of reforms” in education (p. 14). Also, “carefully evaluating development, studying process variables, evaluating outcomes along the way rather than just at the end of product [project] development, and analyzing supportive and contextual variables generates a comprehensive understanding of the overall effectiveness of science education programs and, to a degree, the interface between levels” (p. 13). This is the rationale for the contextual program evaluation model for science education developed by Altschuld and Indian Journal of Educational Technology Volume 3, Issue 2, July 2021 4 Kumar (1995) and field-tested (Kumar and Altschuld, 1999). The guiding principles of the contextual program evaluation model developed by Altschuld and Kumar (1995) based science education program evaluation model with adaptations to educational technology programs follow. Formative evaluation of the development of an educational technology product or program should be the main focus. The evaluation framework should emphasize the process of gaining knowledge and understanding in educational technology. The significance of the context in which educational technology programs and their support system exist should be recognized. The evaluation should take into consideration the interface between the micro and macro levels, and the subject disciplines that are addressed with educational technology, without which much of the program level data may not be interpretable. The model should also take into consideration the current societal press for accountability. Overall, the model emphasizes that the development of the program (and/ or product) is placed in real-world supportive and contextual factors. The supportive factors are learning environment, teacher preparation, instructional materials, administrative support, fixed facilities, and community involvement (Exline and Tonelson, 1987). The contextual factors are characteristics of students, parents, the nature of the community, features of the program, specific school, school district, and society (Field and Hill, 1988). It could be argued that since the context of educational technology itself is rich enough to complement any subject discipline it is aligned with in the teaching and learning process, the context stands to provide invaluable insights into education programs in terms of curriculum, instruction, teachers, students and testing useful for making informed evaluative decisions, and should be an integral part of program evaluation in educational technology (Altschuld and Kumar, 1995; Fitzpatrick, Sanders, and Worthen, 2011; Field and Hill, 1988; Kumar and Altschuld, 2003; Altschuld and Kumar, 2010). In program evaluation, the context refers to the culture, environment or climate in which the program is conceptualized, developed and implemented and a supportive context is essential to program or project success (Kumar and Altschuld, 2003). “If a context is not supportive of change, if policies are not there to foster and reinforce change, if resources in the form of time and training besides finances are not provided, if the environment does not afford the opportunity to try out ideas and to learn from failures, and if other aspects of a conducive, open atmosphere are not present, the probability of institutionalizing successful new programs will be extremely low” (Kumar and Altschuld, 2003, p. 605-606). (See Kumar and Altschuld (2003) and Altschuld and Kumar (1995) for details and discussions about the context based evaluation model.) This is the condition of many well intentioned educational technology programs. Indian Journal of Educational Technology Volume 3, Issue 2, July 2021 5 Complexity of Educational Technology and the Complexity of Evaluating Educational Technology Programs The complexity of educational technology creates a need for more comprehensive approaches to program evaluation methods. Ideally, when distinct disciplines such as science, language, social studies, mathematics, and arts are represented individually or in some combination using educational technology, the resultant teaching and learning process is complex. The role and nature of educational technology will change depending upon the degree (e.g., how, where) to which it overlaps and/or integrates with each individual discipline. Most approaches to program evaluation in educational technology ignore this basic difference in overlaps and integration and end up adapting evaluation methods that are developed for single discipline educational settings. Therefore it is necessary that program evaluation should be made comprehensive enough to include multiple methods and through their use should enable it to capture an in- depth picture of large-scale educational technology programs and projects in teaching and learning. From this perspective, it is worth reviewing how three different approaches to evaluating an interactive multimedia-based science teacher education project in sum provided a comprehensive view of the project, which is impossible to obtain otherwise. This example is used due to the availability of published research and reports on the project in addition to familiarity with the project. Sample Comprehensive Evaluation A review of three different approaches used in the evaluation of the project “Improving science education: A collaborative approach to the preparation of elementary school teachers,” and the findings from the three different approaches may help to gain an understanding of a comprehensive approach to program evaluation. The reason for using this example is that this project has developed interactive multimedia technology-based cases of effective and ineffective elementary school science teaching strategies suitable for science methods courses taught at Vanderbilt University. As discussed earlier this project integrated educational technology with science and science teaching methods. Science educators from the college of education, science faculty from the college of arts and sciences, and teachers from grades 4-7 from the local school district have collaborated. The three evaluation methods used and their respective findings follow (Kumar and Altschuld, 2003). 1. Traditional evaluation conducted by project staff (Barron et al., 1999) Method - Onsite observations, tests, surveys and follow-up interviews. Findings – Significant gains in teacher competency (e.g., section of learning e x p e r i e n c e s and materials which stimulate student curiosity and scientific investigations). student behavior (e.g., student involvement in lesson) and other activities (e.g., hands-on discovery). Indian Journal of Educational Technology Volume 3, Issue 2, July 2021 6 2. Traditional evaluation conducted by an external agency (U.S. Congress, Office of Technology Assessment, 1995) Method - Observation of science methods class, review of project publications and software and interviews of project investigators. Findings – Strong faculty development and research effort, theoretical foundation, incentives for faculty involvement, “expensive infrastructure.” 3. Context evaluation conducted by external evaluators (Kumar and Altschuld, 1999) Methods - Onsite and in-school observations, document reviews and semi- s t r u c t u r e d interviews of faculty, staff and administrators on university campus, school administrator interviews, university teacher education students and graduates’ interviews. Findings - Strong administrative support, technical support, conducive organizational environment, a critical mass of interest by project participants in the project, mutual permeation in science and education departments, methods students perception of project benefits to understanding pedagogy on campus and in teaching situations in schools. It is obvious that the findings of the three distinct evaluations summarized above have generated a comprehensive picture of the project with a comprehensive evaluative outcome that is not possible to obtain by a single evaluation alone. Also, this evaluation was for evaluating an undergraduate elementary teacher preparation program involving interactive multimedia technology in science education methods, and should not be applied without needed adaptations to evaluating educational technology programs that involve other subject disciplines in other grade levels. Final Thoughts In terms of program evaluation in educational technology, an argument could be made that models for evaluating educational technology in a single discipline like science have limited fit to the spectrum of disciplines in education programs. Obviously when it comes to evaluating educational technology with reference to a wide variety of subject disciplines, there is no one universally accepted model of program evaluation. As factors embedded in the context of the educational setting prescribe the adaptation and implementation of educational innovations, program evaluation of “educational innovations should involve not only whether change occurred but also issues such as why change did or did not occur as a result of a program and its meaning to the participants. These issues can be addressed by expanding evaluation plans to include the context in which an innovation is embedded” (Altschuld, Kumar, Smith and Goodway, 1999, p. 66). Indian Journal of Educational Technology Volume 3, Issue 2, July 2021 7 Therefore, a compelling need for developing comprehensive program evaluation models quite suitable for context-specific educational technology applications will continue to exist as education reform becomes more innovative with the development of newer technologies. Rethinking existing program evaluation models for educational technology in teaching and learning in science, languages, social studies, mathematics, and computer science, and conceptualizing and developing models for individual educational technology situations has merit, and program evaluation for educational technology remains a fertile field for research and development. (Acknowledgments: Editorial assistance provided by Jessica McNair, Florida Atlantic University is thankfully acknowledged.) References Altschuld, J. W. and Kumar, D. D. (2010). Needs assessment: An overview. (ISBN-13: 978- 1412975841; ISBN-10: 1412975840). CA: Sage Publications. Altschuld, J. W. and Kumar, D. D. (eds.) (2002). Evaluation of science and technology education at the dawn of a new millennium. mill ennium (ISBN-13: 978-0306467493; ISBN-10: 0306467496). New York: Kluwer Academic/Plenum Publishers. Altschuld, J. W., Kumar, D. D., Smith, W. D., and Goodway, J. D. (1999). The changing countenance of context-sensitive evaluations: Case illustrations. Family and Community Health, 22(1), 66-79. Altschuld, J. W. and Kumar, D. D. (1995). Program evaluation in science education: The model perspective. New Directions for Program Evaluation, 65, 5-17. Barron, L. C., Joesten, M. D., Goldman, E. S., Hofwolt, C. A., Bibring, J. B., Holladay, W. G., and Sherwood, R. D. (1993). Improving science education: A collaborative approach to the preparation of elementary school teachers. A final report to the National Science Foundation under grant number TPE-8950310. Nashville, TN: Vanderbilt University. CDC Program Evaluation. (2018). https://www.cdc.gov/eval/index.htm. (Retrieved 3-9- 2021) Exline, J. D. and Tonelson, S. W. (1987). Virginia’s program assessment model resource guide. National Science Teachers Association Science Education Suppliers. Field, S. L. and Hill, D. D. (1988). Contextual appraisal: A framework for meaningful evaluation of special education programs. Remedial and Special Education, 9(4), 22-30. Handal, B., & Herrington, A. (2003). Re-examining categories of computer-based learning in mathematics education. Contemporary Issues in Technology and Teacher Education [Online serial], 3(3). 275-287. https://citejournal.org/issue-3-03/volume-3/ mathematics/re-examining-categories-of-computer-based-learning-in- mathematics-education (Retrieved 3-9-2021) International Society for Technology in Education. (2021). ISTE standards.standards. Portland, OR: Author. Retrieved March 9, 2021 from https://www.iste.org/standards. Indian Journal of Educational Technology Volume 3, Issue 2, July 2021 8 Kumar, D. D. and Altschuld, J. W. (2003). The need for comprehensive evaluation in science education. In Kumar, D. D. and Altschuld, J. W. (eds.), Science education policy: A symposium. The Review of Policy Research, 20(4), 561-645. Kumar, D. D. and Altschuld, J. W. (1999). Evaluation of interactive media in science education. Journal of Science Education and Technology, 8(1), 55-65. Lazaro, H. (2020). What is edtech and why should it matter to you? General Assembly Blog. Retrieved March 9, 2021 from https://generalassemb.ly/blog/what-is-edtech/. Odden, A. (1990). Educational indicators in the United States: The need for analysis. Educational Researcher, 19(5), 24-29. Shavelson, R. J., McDonnell, L. M., and Oakes, J. (eds.). Indicators for monitoring mathematics and science education. (ISBN-13: 978-0833009722; ISBN-10: 0833009729). Santa Monica, CA: Rand. Stufflebeam, D. L. (1983). The CIPP model for program evaluation. In G. F. Madaus, M. Scriven, and D. L. Stufflebeam (Eds.), Evaluation models (pp. 117-141). (ISBN-10: 0898381320; ISBN-13: 978-0898381320). New York, NY: Springer. Stufflebeam, D. J. and Shinkfield, A. J. (2007). Evaluation theory, models and applications (ISBN-13: 978-0787977658; ISBN-10: 1118870328). San Francisco, CA: Jossey-Bass. Tata Trusts. (2019). Technology in education. India: Author. Retrieved March 9, 2021 from https://www.tatatrusts.org/our-work/education/deepening-learning/ technology-in-education. U.S. Congress, Office of Technology Assessment. (1995). Teachers and technology: Making the connection, OTA-HER-616. Washington, DC: US Government Printing Office. Retrieved March 9, 2021 from https://www.princeton.edu/~ota/disk1/1995/9541/ 9541.PDF Weiss, C. H. (1997). Evaluation, second edition. (ISBN-13: 978-0133097252; ISBN-10: 0133097250). NJ: Prentice Hall. Weiss, C. H. (1993). Politics and evaluation: A reprise with mellower overtones. Evaluation Practice, 14(1), 107-109.