Australasian Journal of Educational Technology, 2021, 37(6). 156 Instructional design and instructional effectiveness in virtual classrooms: Research trends and challenges Baris Cukurbasi Manisa Celal Bayar University, Turkey Mubin Kiyici Sakarya University, Turkey This literature review aimed to examine the status, trends and tendencies in publications about virtual classrooms, instructional design in the virtual classroom and instructional effectiveness in the virtual classroom, as indexed in the ERIC database. For this review, we examined 2680 publications indexed in ERIC between 1994 and 2018. We collected data with data mining and utilised several Python libraries as resources for developing the analysis plan. The results of the analysis are presented in visual form. For each of the three subject matter areas examined in this study, we present the themes in the publications by years, titles, abstracts and ERIC descriptors. We report in detail the trends and challenges that emerged from the analysis. The results show that the words “learning”, “online”, and “environment” were prominent in each of the three subject matter areas. The primary topics addressed were literacy and curriculum development, and researchers examined the roles of instructors, learners and managers. Implications for practice or policy: • This review will be a useful resource for scientists, researchers and policymakers who conduct studies on the virtual classroom. • Instruction should be planned according to differences in instructor, learner and manager roles, as revealed by the studies. • Practitioners who want to teach in virtual classrooms can also use it as a guide. Keywords: distance education, online learning, virtual classroom, instructional design, instructional effectiveness, data mining, trend analysis Introduction The impact of technology on learning environments is growing day by day as learners of the 21st century make multidimensional technology knowledge an integral part of their daily lives. This is reflected by transformations in student profiles, teaching and learning cultures and learning environments. The introduction of innovative technological tools, platforms and educational models has diversified alternative modalities for the provision of education, further expanding the theoretical and semantic framework of open and distance learning. Virtual classroom (VC) environments as components of open and distance education have also been updated as a result of this diversification and expansion. According to Ay (2014), learning will be moved to VC environments parallel to rapid advances in technology. It is important to increase the number of these environments using flexible methods so that educational activities can be delivered everywhere and for everybody equally (Fermin-Gonzalez, 2019). Consequently, issues including the extent to which VCs differentiate learning in terms of time and space relationships, how they can be designed to the extent of the possibilities provided by new technologies, which principles can the teaching design be based on, which teaching materials can support classrooms and which functions VCs need to overcome the physical and space-bound limitations of education have been examined increasingly in scientific research. Despite the growing number of studies on these topics, there are no comprehensive studies that examine the research on these subjects systematically and present general tendencies. Furthermore, no systematic reviews are available about instructional design in VC (ID-in-VC) and instructional effectiveness in VC (IE-in-VC). In this context, we believe that research can help to provide a general understanding of the sub-topics and trends related to VC, ID-in-VC and IE-in-VC by presenting the current situation. Accordingly, we examined the status of publications about VC, ID-in-VC and IE-in-VC indexed in the ERIC database, as well as trends and challenges in the context of several Australasian Journal of Educational Technology, 2021, 37(6). 157 variables. We report these in light of the findings obtained in the study. Based on the research purpose, we addressed the following questions: 1. What are the research trends by years of publication? 2. What kind of research trends emerge according to the words used in the titles of the publications? 3. What kind of research trends emerge according to the words used in the abstracts of the publications? 4. What are the trends in research subjects (the information provided by the “descriptor” in ERIC) of published research? VCs The term virtual refers to the simulation of real objects (Ebbers et al., 2003). VCs are one group of the most well-known systems simulated in virtual learning environments as real classrooms equipped with a lot of learning support tools (Kultawanich et al., 2015). VCs have been in use for more than 3 decades and they are constantly updated in light of technological developments. Especially with e-learning technologies, it is possible to set up web-based VCs enabling teacher-student and student-student communication, allowing for the distribution of course materials and providing online student assessment tools (Lan et al., 2014). Content and/or screen sharing, audio, chat, drawing and pointer tools, polls, instant feedback and breakout rooms are common features of VCs (Christopher, 2014). Rather than the use of sophisticated technology, how well the virtual and physical tools in the environment enable and facilitate an interesting learning activity in VCs is more important (Christopher, 2014). Moreover, students’ learning styles come to the fore in virtual learning environments as they can be analysed to provide highly effective alternatives for teaching, enhancing the usefulness of VC education (Palloff & Pratt, 2013). For this reason, VCs should be designed according to the learning objectives and student characteristics to ensure their effective use for teaching and learning. The introduction of innovative technologies leading to the development of VCs, the increase in opportunities for distance university education and the diversification of student profiles have all challenged those involved in VC processes, especially curriculum developers, instructional designers, web developers and teachers (Fermin-Gonzalez, 2019). In this context, online learning has created new issues and problems in requiring academics, leading instructors and institutions to be more flexible in addressing them (Palloff & Pratt, 2007). Therefore, many studies have been conducted and published, examining variables for enhancing the success of teaching in VCs. However, there are no studies specific to the VC environment that have systematically examined this research area as a whole. A limited number of studies, which have systematically investigated research on VCs in the literature, have focused only on one specific aspect of VCs. One of those was conducted by Fermin-Gonzalez (2019), who systematically examined 89 publications about VCs in Web of Science and Scopus published in the years between 2007 and 2017. He reported that 51 publications included in the study focused on disability while 38 focused on other human differences. He also reported that the publications included in the study investigated e-learning methodologies used in VCs, the conceptualisation of VCs and the methodologies used for defining human differences. Moore-Adams et al. (2016) systematically examined publications about online teaching for training K -12 teachers. They analysed 26 articles published in peer-reviewed journals since 2004, as determined by the EbscoHost Web online search tool, and interpreted their findings within the technological pedagogical content knowledge theoretical framework. Arnesen et al. (2019) analysed 356 articles published since 1994 on K-12 online learning in the context of the topics, authorship, journals, citations and methodology. They concluded that while theoretical articles were more common in the earlier years, the number of data-based articles has gradually increased. Research method In this research, data mining was used as a method of big data analysis. Data mining is an analytic method for extracting meaningful information and new patterns from large unstructured data sets, as well as for transforming data into information (He et al., 2013). Proposed by Chen and Zhang (2014) and Gandomi and Haider (2015), two different road maps are available for use in data mining. In general, the major Australasian Journal of Educational Technology, 2021, 37(6). 158 processes in these roadmaps can be summarised as obtaining data from different sources, processing the data for analysis, performing data analysis and presenting a visualisation of the data. This study aimed to examine the status, trends and tendencies in publications about VCs, ID-in-VC and IE- in-VC indexed in ERIC. For this purpose, the indexed publications were downloaded as separate documents for each year in XML format from the download page of the ERIC website (https://www.eric.ed.gov) and saved. After saving, the ERIC data was converted to Pandas data frame data type by using The ElementTree XML API (https://docs.python.org/3/library/xml.etree.elementtree.html) and Pandas (McKinney, 2010) libraries to process the data more easily later in the software. Then, column names were standardised by using Pandas libraries, irrelevant items were excluded, columns were renamed and only the columns relevant for the analysis were selected. After these processes, the file was saved on a computer in a format that facilitates reading and writing. An Internet-based digital library, ERIC, was used as the data source. The rationale for using it is the availability of more than 1000 journals indexed in it and a large set of studies about VCs. When the studies about VC published before 1 January 2019 were searched in the literature, 902 publications were found in Web of Science and 2680 publications were found in ERIC. Therefore, the publications indexed in ERIC constituted the data of the research in order to better reveal the current situation regarding studies on VCs. The publications indexed in ERIC in January 2019 and published in the period between 1965 and 2018 were included in the study in the form of a separate XML file for each year. Using these criteria, the research comprised 1,738,046 publications published within that period. The number of studies published each year and the total number of studies included in the research are shown in Figure 1. Figure 1. Total and annual numbers of publications indexed in ERIC in the period between 1965 and 2018 In order to determine the sample size and to group the publications in line with the purpose of the study, the Thesaurus section of ERIC and the descriptors assigned to publications by ERIC were used. The study sample was determined and grouped using the following descriptors: “virtual classroom” for VC, “Virtual Classroom” + “Instructional Design” for ID-in-VC and “Virtual Classroom” + “Instructional Effectiveness” for IE-in-VC. The numbers and the publication date intervals of the publications included in the database of this study are presented in Table 1. Table 1 Numbers and publication date intervals of the publications VC ID-in-VC IE-in-VC Year range Publication count Year range Publication count Year range Publication count 1994–2018 2680 1999–2018 322 1999–2018 313 https://www.eric.ed.gov/ https://docs.python.org/3/library/xml.etree.elementtree.html Australasian Journal of Educational Technology, 2021, 37(6). 159 The distribution of these publications by type is presented in Figure 2. Figure 2. Distribution of publications by types Data analysis In the process of reviewing the selected publications, firstly, the obtained data were listed in a Microsoft Excel format that facilitates reading and writing. Then, the list of words was created. A Python library (at https://github.com/amueller/word_cloud) was used for presenting the list of words in word cloud format. In addition, line charts were created according to years of publication. The results of each set of analyses were examined comparatively. In this context, the following information about the publications was systematically examined: • years • title • abstract • ERIC descriptors. The title and abstract of the study were analysed as they reflect the spirit of the study. The most important findings and most important words can be found in the abstract; by analysing the abstract important keywords can be revealed furthermore content of the studies can be understood by reading abstract. In addition, ERIC descriptors show that discipline, method, school level of the study; furthermore, sometimes ERIC descriptors show the survey or scale used in the study. Generally, ERIC descriptors represent the research topics of the study. Because of these reasons, the idea of analysing the title, abstract and ERIC descriptors emerged. To make the results understandable, word clouds were used instead of frequency tables. Some results were expressed in frequencies (f) and some were expressed as the number of publications (N). The results of the analysis were interpreted comparatively across the VC, ID-in-VC and IE-in-VC groups. https://github.com/amueller/word_cloud Australasian Journal of Educational Technology, 2021, 37(6). 160 Before analysing the title and abstract sections of the publications, punctuation marks were removed from the text, all letters in the text were converted into lowercase letters, repetitive and irrelevant words were excluded from the text and plural words were converted into singular ones by using the NLTK library (Bird et al., 2009) of the Python system. This processing allowed us to detect the most commonly used 100 words. These words were sorted according to their frequency of use and analysed in detail across the three data groups. The first 10 of these 100 words are listed in the Results and discussion section. All of these 100 words were analysed in the word clouds and interpretations. Results and discussion Distribution of the publications by year The distribution of the publications in ERIC about VC, ID-in-VC and IE-in-VC by year is presented as charts in Figure 3. Figure 3. Distribution of the publications by year Figure 3 shows that publications about VCs have been indexed in ERIC since 1994 and those about ID-in- VC and IE-in-VC since 1999. The highest number of publications were indexed in ERIC in 2009, including all papers in the three groups. More than 55% of the publications have been indexed since 2010. However, the number of publications substantially decreased in 2017 and 2018. The lowest number of indexed studies was in 2018, when the studies were reviewed from 2008 until 2018. The number of published studies about VCs might have decreased possibly because of the developments in educational technologies with their growing popularity, including the Internet of Things, massive open online courses, gamification, adaptive learning technologies, mobile learning, flipped classroom models, robotics, mixed reality and makerspaces (Alexander et al., 2019). Examining the distribution of the VC studies by years shows that 50.03% (N = 1341) of these studies were published between 2008 and 2013. Although the number of indexed studies per year has ranged between 22 and 319 since 2007, only 78 publications were indexed in 2018. The number of indexed publications increased continuously from 2004 to 2009; however, their numbers have started to fluctuate since 2010. Australasian Journal of Educational Technology, 2021, 37(6). 161 We found that 69.25% (N = 223) of published studies about ID-in-VC were indexed in the years between 2008 and 2012. The number of ID-in-VC studies followed a course in the range from 35 to 67 per year in the years from 2008 until 2013. However, only a total of 59 publications have been indexed in since 2013. Regarding IE-in-VC studies, 62.30% (N = 195) of the publications were published between 2008 and 2012. The number of IE-in-VC studies ranged from 15 to 67 per year starting from the year 2007. However, the total number of publications about IE-in-VC studies indexed was 33 starting from 2016. A detailed analysis revealed 73 studies were indexed in ERIC, where the descriptors of VC, ID and IE were used together. Furthermore, 75.34% (N = 55) of these publications were indexed between 2009 and 2013. No such publications were indexed in ERIC after 2015. Analysis of the publication titles After analysing the titles of the studies indexed in ERIC about VC, ID-in-VC and IE-in-VC, a total of 100 words were identified as the most commonly used words in the publication titles. Of them, the top 10 words and their frequencies of use are presented in Table 2. Table 2 The 10 most commonly used words in the publication titles Table 2 shows that the most commonly used words in the publication titles are “learning”, “online” and “environment”. They constitute more than 25% of the total frequency value found as a result of the analysis. The 10 most commonly used words were among the 100 most commonly used words in each of the three groups of subjects. Of these, “world”, “design” and “e-learning” were used in the VC, ID-in-VC and IE- in-VC groups respectively. The word clouds created by the analysis of the titles of the published publications about VC, ID-in-VC and IE-in-VC are presented in Figure 4. Top 10 Words f Top 10 Words f Top 10 Words f Learning 991 Learning 143 Learning 137 Online 573 Online 92 Online 84 Environment 462 Environment 79 Environment 61 School 294 Design 54 Teaching 32 Classroom 208 Teaching 31 Classroom 26 Teacher 208 Development 29 School 23 Teaching 206 Classroom 29 Teacher 18 Technology 163 Teacher 29 E-Learning 18 Distance 144 Technology 26 Development 17 World 131 Distance 23 Technology 17 fTotal 3380 fTotal 535 fTotal 433 Percentage 43.09 Percentage 46.32 Percentage 39.76 VC (fVC = 7844) ID-in-VC (fIDinVC = 1155) IE-in-VC (fIEinVC = 1089) Australasian Journal of Educational Technology, 2021, 37(6). 162 Figure 4. Analysis results of the publication titles presented as word clouds Figure 4 shows that the words “learning”, “online” and “environment” are the most notable ones for the three groups examined in this research. In addition, the words “school”, “classroom” and “teacher” stood out for VC; the words “design”, “teaching” and “development” stood out for D-in-VC; and the words “teaching”, “classroom” and “school” stood out for IE-in-VC. When the word clouds in Figure 4 were analysed to compare the common words in the three groups, the following points emerged: • The top 10 most commonly used words in the VC and ID-in-VC groups were the same and the words “design” and “development” in the IE-in-VC group replaced the words “school” and “world” in the other two groups. • The words “K-12”, “learner” and “teacher” were used in each of the three groups of publications. Besides these words, the word “educator” was used in the ID-in-VC group and the word “instructor” was used in the VC and IE-in-VC groups. • The words “engineering”, “language” and “science” were used in each of the three groups of publications. However, the word “mathematics” was used only in the ID-in-VC and IE-in-VC groups. The topics of ID or IE are examined in studies about VCs and mathematics in the literature (Adnan & Boz, 2015; Chang et al., 2019; Krishnan, 2016). • One of the commonly used words is “chemistry” in the studies about IE-in-VC. Generally, there are studies on IE-in-VC in the chemistry publications included in the data set (Herga et al., 2016; Lengyel, 2020; Limniou et al., 2008; Tuysuz, 2010). This finding shows that chemistry is a point of focus in the IE-in-VC studies, unlike the other subject areas. Figure 5 shows the distribution of the number of common and non-common words in each data group after comparing the 100 most commonly used words across the three publication groups. Australasian Journal of Educational Technology, 2021, 37(6). 163 Figure 5. Distribution of the common and non-common words in the publication titles Figure 5 shows that 14 words for VC (f = 478; 6.09 %), 20 words for ID-in-VC (f = 120; 10.39%) and 25 words for IE-in-VC (f = 108; 9.92%) were unique in the publication titles. This finding notably indicates the degree of distinctiveness of each of the examined publications. The three most commonly used words in each cluster area in Figure 5 are presented in Figure 6. Figure 6. The three most used common and non-common words in the publication titles Figure 6 shows the most common and non-common words, revealing that unique words were more frequently used in the studies about ID-in-VC. In general, the figure shows that the publication titles differed according to each of the three subject areas examined in this study. The examination of the unique words in the publication titles revealed the following points: Australasian Journal of Educational Technology, 2021, 37(6). 164 • The words “challenge”, “team”, “charter”, “academic”, “knowledge”, “policy” and “work” were most commonly used in the VC studies; the words “implementation”, “examining”, “pedagogical”, “age”, “convention”, “association”, “educator”, “curriculum”, “integration” and “creating” were most commonly used in the ID-in-VC studies; and the words “effectiveness”, “asynchronous”, “outcome”, “process”, “feedback”, “chemistry”, “enhancing”, “satisfaction”, “assessing” and “cognitive” were most commonly used in the IE-in-VC studies. • The word “leadership” was used only in the titles of the VC studies. • Names of the variables such as “influence”, “evaluating”, “thinking” and “characteristics” were used in the titles of the ID-in-VC studies. • The word “attitude” was used in the titles of the IE-in-VC studies. • The comparison of the titles of VC and ID-in-VC studies revealed 30 different words. Of these, the ones that came to the fore were “implementation”, “educator”, “curriculum”, “integration”, “mathematics”, “strategy”, “influence”, “enhance”, “thinking” and “multi-user”. Similarly, the examination of IE-in-VC studies revealed 25 different words, including primarily the words “effectiveness”, “asynchronous”, “outcome”, “enhance”, “feedback”, “chemistry”, “satisfaction”, “assessing”, “attitude” and “mathematics”. When the ID-in-VC and IE-in-VC groups were compared, 35 different words were identified. Of these words, the words “impact”, “skill”, “effectiveness”, “application”, “asynchronous”, “instructor”, “collaboration” and “chemistry” were primarily used in the IE-in-VC studies but not in the ID-in-VC studies, whereas the words “implementation”, “pedagogical”, “professional”, “curriculum”, “educator”, “integration”, “management”, “exploring”, “faculty” and “thinking” were used primarily in the ID-in-VC studies but not in the IE-in-VC studies. Analysis of the abstracts of publications The abstracts of the studies about VCs, ID-in-VC and IE-in-VC underwent the content analysis, revealing the most commonly used 100 words. Table 3 lists the 10 most commonly used words and their frequencies in the abstracts of the three groups of studies. Table 3 The 10 most commonly used words in the abstracts of studies Top 10 Words f Top 10 Words f Top 10 Words f Learning 6063 Learning 971 Learning 833 Online 3677 Online 603 Online 424 School 2114 Design 340 Environment 290 Environment 2086 Environment 305 Teaching 206 Technology 1569 Teacher 206 Teacher 166 Teacher 1490 Technology 205 School 161 Teaching 1393 Teaching 194 Technology 136 Development 1005 Development 181 Experience 133 Classroom 999 Instructional 163 Design 128 Experience 988 Classroom 137 Classroom 125 fToplam 21384 fToplam 3305 fToplam 2602 Percentage 34.93 Percentage 36.89 Percentage 33.95 VC (fVC = 61213) ID-in-VC (fIDinVC = 8958) IE-in-VC (fIEinVC = 7664) Australasian Journal of Educational Technology, 2021, 37(6). 165 Table 3 shows that the most commonly used words in the abstracts of each of the three groups of studies are “learning” and “online”. These are followed by “school”, “environment”, “design” and “teaching” with varying frequencies. In the three groups of studies, nine of the 10 most commonly used words were the same, with varying frequencies. The words “experience” and “instructional” were not common to the VC and ID-in-VC groups, the words “development” and “design” were not common to the VC and IE-in-VC groups and the words “instructional” and “experience” were not common to the ID-in-VC and IE-in-VC groups. The comparison of the content analysis results of the three groups of studies is presented as word clouds in Figure 7. Figure 7. Word clouds presenting the content analysis results of the abstracts Figure 7 shows that the words “learning” and “online” stand out in the three data groups. The display size of the words in the word clouds reveals that these two words were used more commonly than the other words. They are followed by the word “environment” for the three groups of publications. Besides these, the words “school” and “technology” are prominent for the VC studies, the word “design” is prominent for the ID-in-VC studies and the word “teaching” is prominent for the IE-in-VC studies. When Figure 7 was analysed, the following were found after comparing the common words in each of the three publication groups: • Of the 100 most commonly used words, 67 were common for each of the three publication groups. When these words were ranked in the decreasing order of frequency, the first 41 words were among those common to each of the three groups. • The words “learning”, “online”, “design”, “environment”, “teacher”, “technology”, “teaching”, “development”, “instructional” and “classroom” were common in the three-word clouds. • The examination of the common words between the VC and ID-in-VC groups revealed that the words “digital”, “professional”, “educator”, “team”, “management”, “theory” and “pedagogical” were primarily used aside from the common words between the VC and IE-in-VC pair and the ID- in-VC and IE-in-VC pair. • When the common words between the VC and IE-in-VC groups were examined, the words “academic”, “participant”, “performance”, “perception”, “individual”, “help”, “feedback” and “effect” stood out, aside from the common words between the VC and ID-in-VC pair and the ID- in-VC and IE-in-VC pair. • When the common words between the ID-in-VC and IE-in-VC groups were examined, the words “3D”, “web-based”, “designed”, “delivery” and “curriculum” stood out, aside from the common words between the VC and IE-in-VC pair and the VC and ID-in-VC pair. Australasian Journal of Educational Technology, 2021, 37(6). 166 In addition to these findings, the distribution of the number of common and non-common words in each data group after the comparison of the most commonly used 100 words in the abstracts is given in Figure 8. Figure 8. Distribution of the common and non-common words in abstracts As seen in Figure 8, 8 words (f = 2644; 4.32%) in abstracts about VCs, 15 words (f = 607; 6.78%) in abstracts about ID-in-VC and 16 words (f = 737; 9.62%) in abstracts about IE-in-VC were determined to be unique. In addition, examining the chart shows that the number of unique words used in the publications about IE-in-VC is higher. The three most commonly used words are written in each cluster area on the diagram in Figure 8 and are presented in Figure 9. Figure 9. The three most frequently used common and non-common words in abstracts Australasian Journal of Educational Technology, 2021, 37(6). 167 Figure 9 shows that the publications differ according to their subject areas. Accordingly, the following findings were obtained as a result of the analysis of the words that were unique in the publication groups: • The words “schools”, “presence”, “K-12”, “Access”, “VLE”, “collaboration”, “impact” and “change” were used differently in VC studies; the words “implementation”, “blended”, “material”, “specific”, “technical”, “medium”, “relationship”, “literacy”, “designing” and “multimedia” were used differently in ID-in-VC studies; and the words “laboratory”, “effectiveness”, “provided”, “better”, “control”, “simulation”, “task” and “difference” were used differently in IE-in-VC studies. • The word “literacy” is included only in the abstracts of the publications about ID-in-VC, and there are studies on literacy in this area unlike other subject areas. • Words such as “presence”, “collaboration”, “impact” and “change” used in the abstracts of studies on VC give information about the variables included in these studies, unlike other subject areas. • Words used in the studies on IE-in-VC such as “simulation”, “effectiveness”, “laboratory” and “control” different from other data groups provide evidence of experimental studies on this subject. • There are 20 non-common words between VC and ID-in-VC. When these unique words are examined, variables such as “academic”, “performance”, “presence”, “perception”, “collaboration”, “impact” and “effect” stand out in VC studies and concepts such as “3D”, “web- based”, “blended”, “material”, “curriculum”, “medium”, “literacy” and “methodology” stand out in ID-in-VC studies. • The number of non-common words between VC and IE-in-VC is 21. When these non-common words are examined, concepts such as “professional”, “educator”, “team”, “management”, “presence”, “collaboration”, “theory” and “impact” stand out in VC studies and “laboratory”, “effectiveness”, “3D”, “simulation”, “task”, “experimental”, “success”, “curriculum” and “synchronous” stand out in IE-in-VC. • The difference in the number of unique words between ID-in-VC and IE-in-VC is 28. When these words are examined, concepts such as “pedagogical”, “management”, “professional”, “implementation”, “theory”, “blended”, “literacy”, “educator”, “designing”, “methodology” and “learners” stand out in ID-in-VC studies and “effectiveness”, “perception”, “feedback”, “performance”, “laboratory”, “participant”, “simulation”, “outcome”, “task”, “experimental”, “success” and “benefit” stand out in IE-in-VC studies. The non-common words used in the titles of the publications actually represent the subject areas. These words give information about the studies’ different points of focus. The analysis of the descriptors assigned to publications The content of descriptors assigned by ERIC to VC, ID-in-VC and IE-in-VC related to the studies indexed in ERIC were analysed and all descriptors were sorted according to their frequency of appointment. It was determined that 2560 (fVC = 37627) different descriptors were assigned to studies on VC, 1107 different descriptors were assigned to studies on ID-in-VC (fIDinVC = 6006) and 1129 different descriptors were assigned to studies on IE-in-VC (fIEinVC = 5622). When all the descriptors assigned were compared, it was seen that those assigned to VC publications covered those assigned to other subjects. The most commonly assigned 10 descriptors and information about their frequency in the three publication groups are given in Table 4. Australasian Journal of Educational Technology, 2021, 37(6). 168 Table 4 The 10 most commonly assigned descriptors in the publications Table 4 shows that “educational technology”, “electronic learning”, “online courses”, “distance education”, “teaching methods' and “web-based instruction” are among the 10 most commonly assigned descriptors for all three publication groups. The descriptor “instructional design”, which represents the ID in ID-in-VC, is the most commonly assigned descriptor for studies on ID-in-VC, the 10th most commonly assigned descriptor for studies on IE-in-VC and the 14th most commonly assigned descriptor for studies on VC. Similarly, the descriptor “instructional effectiveness”, which represents the IE in IE-in-VC, is the most commonly assigned descriptor for studies on IE-in-VC, the 15th most commonly assigned descriptor for studies on VC and the 11th most commonly assigned descriptor for studies on ID-in-VC. The representation of the findings on the descriptors as a word cloud is given in Figure 10. Top 10 Subject f Top 10 Subject f Top 10 Subject f Educational Technology 968 Instructional Design 322 Instructional Effectiveness 313 Electronic Learning 892 Educational Technology 166 Electronic Learning 146 Online Courses 853 Electronic Learning 155 Educational Technology 125 Distance Education 763 Online Courses 133 Distance Education 116 Teaching Methods 608 Distance Education 131 Online Courses 106 Web Based Instruction 518 Web Based Instruction 112 Web Based Instruction 101 Computer Mediated Communication 482 Teaching Methods 88 Student Attitudes 96 Internet 459 Computer Mediated Communication 85 Teaching Methods 88 Higher Education 440 Computer Assisted Instruction 82 Computer Simulation 78 Student Attitudes 413 Computer Simulation 80 Instructional Design 73 fToplam 6396 fToplam 1354 fToplam 1242 Percentage 17 Percentage 22.54 Percentage 22.09 VC ID-in-VC IE-in-VC Australasian Journal of Educational Technology, 2021, 37(6). 169 Figure 10. The representation of descriptors as word clouds Comparing the word clouds in Figure 10 shows that the descriptors “educational technology”, “electronic learning” and “online courses” stood out in studies on VC, “instructional design” stood out in studies on ID-in-VC and “instructional effectiveness” stood out in studies on IE-in-VC specifically. The findings obtained as a result of the analysis of descriptors were grouped and compared and the distribution of the common and non-common descriptors was assessed and shown in the diagrams in Figure 11. Figure 11. The distribution of the descriptors assigned to the publications Australasian Journal of Educational Technology, 2021, 37(6). 170 Figure 11 shows that all the descriptors assigned to publications about VC include those assigned to studies about ID-in-VC and IE-in-VC, 1165 (f = 2797) of 2560 different descriptors assigned to VC were assigned only to publications about VC. When ID-in-VC and IE-in-VC were compared, we found that 842 descriptors were common (fIDinVC = 5675, fIEinVC = 5245), 266 unique descriptors were assigned to publications about ID-in-VC (fIDinVC = 331) and 288 unique descriptors were assigned to publications about IE-in-VC (fIEinVC = 377). The three most commonly assigned descriptors in the diagrams in Figure 10 are given in Figure 12. Figure 12. The three most commonly assigned descriptors in publications The following findings were obtained when descriptors in Figure 12 were examined: • The descriptors assigned to studies on VC include those assigned to studies on ID-in-VC and IE- in-VC. Additionally, 1165 non-common descriptors were also assigned to VC studies. • The descriptors “enrollment”, “graduation rate”, “federal legislation”, “governance” and “private schools” are most commonly assigned only to studies on VC. In addition, “computer attitudes”, “technology”, “writing”, “reading achievement”, “parent attitudes” and “teacher participation” are among other descriptors assigned only to publications about VC. • “Information systems”, “college environment”, “elementary school teachers”, “administrators”, “media specialist”, “futures (of society)”, “literacy education”, “teaching conditions”, “student teachers and profiles” stand out as descriptors that were assigned to studies on ID-in-VC but are different from those assigned to studies on IE-in-VC. • Similarly, “achievement gains”, “secondary school curriculum”, “scientific literacy”, “genetics”, “transcripts (written records)”, “student evaluation of teachers” performances”, “self- management”, “longitudinal studies”, “portfolio assessment” and “manipulative materials” stand out as descriptors that were assigned to studies on IE-in-VC but are different from those assigned to studies on ID-in-VC. Australasian Journal of Educational Technology, 2021, 37(6). 171 • The descriptors “scientific literacy”, “literacy education”, “emergent literacy”, “cultural literacy” and “critical literacy” were assigned to studies on VC related to literacy, whereas only “literacy education” was assigned to studies on ID-in-VC and “scientific literacy” was assigned to studies on IE-in-VC. • Examining the disciplines of the descriptors assigned to publications about ID-in-VC and IE-in- VC shows that “literacy”, “mathematics”, “science”, “art”, “engineering”, “biology”, “psychology”, “language”, “political science” were common disciplines. Furthermore, the disciplines of “surgery”, “history”, “coding”, “anthropology”, “archaeology”, “astronomy”, “linguistic”, “medical”, “military” and “theology” were unique to studies on ID-in-VC and the disciplines of “genetics”, “sport”, “geography”, “anatomy”, “nursing”, “dance education”, “radiology”, “tourism”, “aerospace”, “statistics”, “law”, “chemistry” and “business” were unique to IE-in-VC. In addition, “STEM education” was one of the descriptors assigned to the studies on ID-in-VC. • Examining the descriptors assigned to publications about ID-in-VC and IE-in-VC in detail shows descriptors related to program development and roles included “administrator roles”, “curriculum evaluation”, “role”, “leadership role”, “faculty”, “English curriculum”, “faculty-college relationship” and “elementary school curriculum” for ID-in-VC and IE-in-VC and “portfolio assessment”, “educational improvement”, “accreditation (institutions)”, “technical assistance”, “teaching style”, “sense of community”, “curriculum enrichment”, “educational legislation”, “college curriculum” and “college role” for IE-in-VC. As a result of the analysis of descriptors assigned by ERIC, Arnesen et al. (2019) found that the descriptor “VC” was assigned seven times and the descriptor “IE” was assigned four times in the 20 most cited publications they reviewed. Also, similar to our research results, they observed that descriptors such as “distance education”, “educational technology” and “online learning” were frequently assigned. Our study found that descriptors assigned by ERIC included “gender”, “intermode”, “generational”, “cultural”, “age”, “racial”, “individual” and “generational differences”. In addition, the descriptors assigned to studies related to curriculum included “college”, “core”, “elementary school”, “English”, “experimental”, “honors”, “national”, “science”, “teacher education”, “student-centered”, “unified studies curriculum”, “curriculum design”, “development”, “enrichment”, “evaluation”, “guide” and “implementation”. Moreover, the descriptors related to disabilities included “mild intellectual disability”, “moderate intellectual disability”, “disabilities”, “disability identification”, “learning disabilities”, “physical disabilities” and “severe disabilities”. Therefore, the results of our study are consistent with the findings available in the literature. By 2020, educational activities had largely been moved to online learning environments due to the COVID- 19 pandemic, which has greatly affected educational environments. VCs have been one of the online learning applications that are used in this process. Hence, the use of VCs and other online learning environments has become widespread significantly. VCs allow learners to attend classes from anywhere in the world and provide learning experiences similar to a real classroom environment (Ebbers et al., 2003). Well-designed VC applications, especially at the K-12 level, provide students with access to learning opportunities whenever and wherever they want as an alternative to face-to-face learning environments (Dipietro, 2010). In addition, the use of VCs at higher education level has positive effects (Kelleci & Aksoy, 2021). At this point, the importance of good design of teaching and the effectiveness of teaching in VC applications becomes more evident. Daher and Awawdeh Shahbari (2020) stated that ID-in-VC, teacher- student interaction and student characteristics are effective on middle school students’ virtual identities. They concluded that besides encouraging learning in ID-in-VC, the interaction should also be taken into account. In another study, Abrami et al. (2011) emphasised that the ID-in-VC will activate the social environment in the classroom and the interaction of the student with their teacher and peers. Therefore, emphasising an issue that is important for instructional effectiveness, such as interaction in ID-in-VC, shows that the ID-in-VC and IE-in-VC cover each other, which we examined in the context of the research. This situation is similar to our research findings. Conclusion, limitations and recommendations For our study, we systematically compared and examined all publications about VC, ID-in-VC and IE-in- VC indexed in ERIC and reported the trends and challenges on the basis of the findings. In conclusion, we observed that there were significant changes in the number of publications related to VCs over the years. It Australasian Journal of Educational Technology, 2021, 37(6). 172 is noteworthy that although the end of the first decade of the 2000s was the period when the number of publications reached its peak and the VC study area was the most popular, the trend has been falling ever since. As the number of scientific publications published on VCs has tended to decrease significantly after 2016, interest in VCs seems to be gradually decreasing. However, this does not necessarily mean that the subject of VCs has lost its importance or has been discredited. VCs are directly or indirectly affected by developments in many areas of society, especially technology. In particular, issues such as the continuing development in distance education technologies may provide a momentum for more research in the area. As a matter of fact, as stated under the title of “Emerging technologies and practices” in the Horizon Report 2020 (Brown et al., 2020), studies on ID-in-VC and IE-in-VC will gain importance. However, they estimate that the importance and the number of publications about VCs will increase due to the fact that most of the learning activities around the world are performed using open and distributed learning due to the COVID- 19 pandemic. The prevalence of “educator” in the titles of studies on ID-in-VC and “instructor” in the titles of studies on IE-in-VC provides an important clue as to how the role of the teacher differs in ID and IE. Unlike ID and IE, the word “leadership” was used in the titles of publications about VCs. Analysing the words used in the titles usually gives clues about the subjects and tendencies of studies that differ from others. In particular, ID-in-VC publications are easily distinguished by their titles. In general, the concepts “design” and “instruction” stand out for ID-in-VC and “teaching” and “experience” stand out for IE-in-VC. The most important concepts for VC, in general, were “learning”, “online” and “environment”. In addition, these publications in three data sets were defined by ERIC mostly as “educational technology”, “electronic learning” and “online courses”. The descriptors assigned to publications give information about the subject of the study. In particular, the descriptors assigned to studies on ID-in-VC and IE-in-VC provide significant information about the similarities and differences between the two subjects. In this context, many aspects of literacy were addressed within the context of VC and only studies on literacy education were carried out in ID-in-VC. More studies were carried out on scientific literacy in the context of IE-in-VC. In addition, there were publications about STEM education and administrator roles in ID-in-VC and about portfolio assessment and educational improvement in IE-in-VC. Also, many aspects of curriculum development were addressed within the context of ID-in-VC and IE-in-VC. This study has two limitations. Firstly, the research was carried out with data mining on articles indexed in ERIC database. Since more publications are covered compared to other databases, ERIC was preferred. Similar work could be performed in other databases for comparison. Secondly, it analysed only the years, title, abstract and ERIC descriptor sections the of publications. Declaration of conflicting interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Funding This study was not funded by any organisation or institution. References Abrami, P. C., Bernard, R. M., Bures, E. M., Borokhovski, E., & Tamim, R. M. (2011). Interaction in distance education and online learning: using evidence and theory to improve practice. Journal of Computing in Higher Education, 23, 82–103. https://doi.org/10.1007/s12528-011-9043-x Adnan, M., & Boz, B. (2015). Faculty members” perspectives on teaching mathematics online: Does prior online learning experience count? Turkish Online Journal of Qualitative Inquiry, 6(1), 21–38. https://doi.org/10.17569/tojqi.60223 https://doi.org/10.1007/s12528-011-9043-x https://doi.org/10.17569/tojqi.60223 Australasian Journal of Educational Technology, 2021, 37(6). 173 Alexander, B., Ashford-Rowe, K., Barajas-Murphy, N., Dobbin, G., Knott, J., McCormack, M., Pomerantz, J., & Weber, N. (2019). EDUCAUSE Horizon Report: 2019 Higher education edition. EDUCAUSE. https://library.educause.edu/resources/2019/4/2019-horizon-report Arnesen, K. T., Hveem, J., Short, C. R., West, R. E., & Barbour, M. K. (2019). K-12 online learning journal articles: trends from two decades of scholarship. Distance Education, 40(1), 32–53. https://doi.org/10.1080/01587919.2018.1553566 Ay, D. (2014). Eğitim sektörü raporu. Türkiye Sınai Kalkınma Bankası Ekonomik Araştırmalar [Education industry report]. TSKB Economic Research. http://www.tskb.com.tr/i/content/731_1_Egitim_Sektoru_Raporu_Haziran_2014.pdf Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python. O’Reilly Media Inc. http://www.datascienceassn.org/sites/default/files/Natural%20Language%20Processing%20with%20P ython.pdf Brown, M., McCormack, M., Reeves, J., Brooks, D. C., Grajek, S., Alexander, B., Bali, M., Bulger, S., Dark, S., Engelbert, N., Gannon, K., Gauthier, A., Gibson, R., Lundin, B., Veletsianos, G., & Weber, N. (2020). 2020 EDUCAUSE Horizon Report: Teaching and learning edition. EDUCAUSE. https://library.educause.edu/-/media/files/library/2020/3/2020horizonreport.pdf Chang, F., Luo, M., Walton, G., Aguilar, L., & Bailenson, J. (2019). Stereotype threat in virtual learning environments: Effects of avatar gender and sexist behavior on women's math learning outcomes. Cyberpsychology, Behavior and Social Networking, 22(10), 634–640. https://doi.org/10.1089/cyber.2019.0106 Chen, C. P., & Zhang, C. Y. (2014). Data-intensive applications, challenges, techniques and technologies: A survey on big data. Information Sciences, 275, 314–347. https://doi.org/10.1016/j.ins.2014.01.015 Christopher, D. (2014). The successful virtual classroom: How to design and facilitate interactive and engaging live online learning. AMACOM. Daher, W., & Awawdeh Shahbari, J. (2020). Secondary students’ identities in the virtual classroom. Sustainability, 12(11), 1–13. https://doi.org/10.3390/su12114407 Dipietro, M. (2010). Virtual school pedagogy: The instructional practices of K-12 virtual school teachers. Journal of Educational Computing Research, 42(3), 327–354. https://doi.org/10.2190/EC.42.3.e Ebbers, M., Balagué, D., Ganguly, B., Noyes, D., & Salm, P. (2003). Using IBM Lotus Virtual Classroom: A best practices guide to e-learning (2nd ed.). Redbooks. http://www.redbooks.ibm.com/redbooks/pdfs/sg246842.pdf Gandomi, A. & Haider, M. (2015). Beyond the hype: Big data concepts, methods and analytics. International Journal of Information Management, 35(2), 137–144. https://doi.org/10.1016/j.ijinfomgt.2014.10.007 He, W., Zha, S., & Li, L. (2013). Social media competitive analysis and text mining: A case study in the pizza industry. International Journal of Information Management, 33(3), 464–472. https://doi.org/10.1016/j.ijinfomgt.2013.01.001 Herga, N. R., Cagran, B., & Dinevski, D. (2016). Virtual laboratory in the role of dynamic visualisation for better understanding of chemistry in primary school. Eurasia Journal of Mathematics, Science & Technology Education, 12(3), 593–608. https://doi.org/10.12973/eurasia.2016.1224a Kelleci, Ö., & Aksoy, N. C. (2021). Using game-based virtual classroom simulation in teacher training: User experience research. Simulation & Gaming, 52(2), 204–225. https://doi.org/10.1177/1046878120962152 Krishnan, S. (2016). Students” perceptions of learning mode in mathematics. The Malaysian Online Journal of Educational Sciences, 4(2), 32–41. https://mojes.um.edu.my/article/view/12647 Kultawanich, K., Koraneekij, P., & Na-Songkhla, J. (2015). A proposed model of connectivism learning using cloud-based virtual classroom to enhance information literacy and information literacy self- efficacy for undergraduate students. Procedia – Social and Behavioral Sciences, 191(2), 87–92. https://doi.org/10.1016/j.sbspro.2015.04.394 Lan, R. W., Yen, N. Y., Li, F. W., & Wah, B. (2014). Recent development in multimedia learning technologies. World Wide Web, 17, 189–198. https://doi.org/10.1007/s11280-013-0206-8 Lengyel, P. S. (2020). Can the game-based learning come? Virtual classroom in higher education of 21st century. International Journal of Emerging Technologies in Learning, 15(2), 112–126. https://doi.org/10.3991/ijet.v15i02.11521 Limniou, M., Roberts, D., & Papadopoulos, N. (2008). Full immersive virtual environment CAVE in chemistry education. Computers & Education, 51(2), 584–593. https://doi.org/10.1016/j.compedu.2007.06.014 https://library.educause.edu/resources/2019/4/2019-horizon-report https://doi.org/10.1080/01587919.2018.1553566 http://www.tskb.com.tr/i/content/731_1_Egitim_Sektoru_Raporu_Haziran_2014.pdf http://www.datascienceassn.org/sites/default/files/Natural%20Language%20Processing%20with%20Python.pdf http://www.datascienceassn.org/sites/default/files/Natural%20Language%20Processing%20with%20Python.pdf https://library.educause.edu/-/media/files/library/2020/3/2020horizonreport.pdf https://doi.org/10.1089/cyber.2019.0106 https://doi.org/10.1016/j.ins.2014.01.015 https://doi.org/10.3390/su12114407 https://doi.org/10.2190/EC.42.3.e http://www.redbooks.ibm.com/redbooks/pdfs/sg246842.pdf https://doi.org/10.1016/j.ijinfomgt.2014.10.007 https://doi.org/10.1016/j.ijinfomgt.2013.01.001 https://doi.org/10.12973/eurasia.2016.1224a https://doi.org/10.1177/1046878120962152 https://mojes.um.edu.my/article/view/12647 https://doi.org/10.1016/j.sbspro.2015.04.394 https://doi.org/10.1007/s11280-013-0206-8 https://doi.org/10.3991/ijet.v15i02.11521 https://doi.org/10.1016/j.compedu.2007.06.014 Australasian Journal of Educational Technology, 2021, 37(6). 174 McKinney, W. (2010). Data structures for statistical computing in Python. In Proceedings of the 9th Python in Science Conference (pp. 56–61). SciPy.org. https://doi.org/10.25080/Majora-92bf1922-00a Moore-Adams, B. L., Jones, W. M., & Cohen, J. (2016). Learning to teach online: A systematic review of the literature on K-12 teacher preparation for teaching online. Distance Education, 37(3), 333–348. https://doi.org/10.1080/01587919.2016.1232158 Palloff, R. M., & Pratt, K. (2007). Building online learning communities: Effective strategies for the virtual classroom (2nd ed.). Jossey-Bass. Palloff, R. M., & Pratt, K. (2013). Lessons from the virtual classroom: The realities of online teaching (2nd ed.). Jossey-Bass. Tuysuz, C. (2010). The effect of the virtual laboratory on students’ achievement and attitude in chemistry. International Online Journal of Educational Sciences, 2(1), 37–53. http://mts.iojes.net//userfiles/Article/IOJES_167.pdf Corresponding author: Baris Cukurbasi, baris.cukurbasi@cbu.edu.tr Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Cukurbasi, B., & Kiyici, M. (2021). Instructional design and instructional effectiveness in virtual classrooms: Research trends and challenges. Australasian Journal of Educational Technology, 37(6), 156-174. https://doi.org/10.14742/ajet.6882 https://doi.org/10.25080/Majora-92bf1922-00a https://doi.org/10.1080/01587919.2016.1232158 http://mts.iojes.net/userfiles/Article/IOJES_167.pdf mailto:baris.cukurbasi@cbu.edu.tr https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.6882 Introduction VCs Research method Data analysis Distribution of the publications by year Analysis of the publication titles Analysis of the abstracts of publications The analysis of the descriptors assigned to publications Conclusion, limitations and recommendations Declaration of conflicting interests Funding References