International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol 17 No 05 (2023) Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and Programming: A Bibliometric Analysis and Overview https://doi.org/10.3991/ijim.v17i05.36503 Agariadne Dwinggo Samala1(), Usmeldi1, Taali1, Yose Indarta1,2, Apdoludin3, Muhammad Hakiki3, Kelvin Leong4 1 Universitas Negeri Padang, West Sumatera, Indonesia 2 Indonesian National Police Headquarters, Jakarta, Indonesia 3 Universitas Muhammadiyah Muara Bungo, Jambi, Indonesia 4 University of Chester, Chester, United Kingdom agariadne@ft.unp.ac.id Abstract—Blended learning, also known as mixed-mode instruction, com- bines in-person and online instruction. Blended learning is widely used in school and university subjects. This research aims to determine how blended learning has been applied to algorithms and programming courses over the last 20 years. This study analyzes the quality and quantity of scientific publications using bib- liometric techniques and then provides an overview of how blended learning is used and its impact. For this analysis and review, this study conducted a biblio- metric analysis of articles published in the last 20 years (2000–2021) and then presented the 10 most cited articles. We established the following criteria for ar- ticles: 1) sourced from the Scopus database, 2) concerned about blended learning in algorithms and programming, and 3) publication is limited to articles published in indexed international journals and proceedings. The VOSviewer and MS-Ex- cel applications help with data presentation in this method. We collected 240 ar- ticles that met these criteria from the Scopus database, which contained 297 arti- cles published between 2000 and 2021. The most-cited article received 52 cita- tions, while the least received only 3. The top 10 most cited articles are from the following countries: 1) Norway, 2) Serbia, and 3) Saudi Arabia. We divided the articles into categories based on the Sustainable Development Goals (SDGs). The findings of this study can be used as a reference for state-of-the-art and novelty, as well as for the dissemination of scientific references related to the use of blended learning for introductory algorithms and programming. Keywords—blended learning, programming, bibliometric, most-cited 1 Introduction The excessive papers published in journals in internet databases have led to an over- abundance of information, making it difficult for practitioners and researchers to iJIM ‒ Vol. 17, No. 05, 2023 57 https://doi.org/10.3991/ijim.v17i05.36503 Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… quickly find relevant references to their topics of interest [1]. From bibliometric analy- sis, we can see trends and patterns of scientific development in disciplines [2]. This information will be used to conduct further research so that re-examining does not oc- cur. Therefore, it is very necessary to filter papers concerned with a research topic. So this helps make it easier for practitioners and researchers to access articles that are use- ful in their research fields efficiently. In addition, bibliometrics can also be used to assess the productivity of authors, institutions, and even a country. The two primary goals of the bibliometric analysis are (1) research performance evaluation of individual and institutional research and publication trends and (2) map- ping, which seeks to expose the structure and development of a studied topic [3]. Each uses a distinct method. We employ precise quantitative calculations such as the overall number of citations, the average number of citations, and the collaboration index to evaluate research performance. [4] Like mapping analysis, keyword analysis can ex- amine the relationship between one topic and another in a specific field [5]. This study was undertaken following one of the primary goals of bibliometric anal- ysis. We can determine the current state of knowledge and the direction of future re- search. In addition, we may follow the citation chain from one paper to the next. Bibli- ometric analysis can help us decide which publications are influential, who the influen- tial authors on the issue are, the progression of a topic (based on citations between pa- pers), previously covered topics, and even identify potential collaborators whom we will invite to be co-authors [6]. 2 Methodology This study includes quantitative research. This study examines the trends in blended learning research on the introduction of algorithms and programming over the last 20 years using bibliometric studies from the Scopus database. Before that, we established several criteria for collecting article data: 1) Publications are sourced solely from the Scopus database; 2) there is concern about blended learning in algorithms and program- ming; and 3) publications are limited to indexed international journal articles and pro- ceedings. On October 24th, 2021, the investigation began with an online search. Visu- alization software VOSviewer and MS-Excel were used for data analysis. The VOSviewer tool was used to create and visualize a structure map of the researchers' and co-authors' source coupling networks [6], [7]. Figure 1 shows a more detailed research procedure: Fig. 1. Bibliometric analysis procedures 58 http://www.i-jim.org Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… Fig. 2. Detailed procedure of bibliometric analysis (cont.) 3 Results and discussion 3.1 Data selection By using Publish or Perish (Pop) with keywords "BLENDED LEARNING" and "PROGRAMMING", we found 319 articles. Then, we selected based on the range of publication years (2000-2021). As a result of this selection, we found 297 articles con- cerning blended learning for introductory algorithms and programming from 2000- 2021. Finally, we re-selected by only taking articles published in international journals and proceedings indexed by Scopus. We collected 240 articles that met these criteria. Now, let's look at Figure 3 below. Based on the data in Figure 4, it can be seen that as many as 240 articles met the criteria (journals and proceedings only). The bars and graphs show increased publica- tions in 2010, from 1 article in 2009 to 10 articles in 2010. Article data for the last 2 years includes as many as 37 articles in 2020 and 36 articles in 2021. So, it can be iJIM ‒ Vol. 17, No. 05, 2023 59 Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… interpreted that research on blended learning in algorithms and programming courses has increased significantly every year. The articles consist of 56% published in Scopus- indexed journals and 44% published in Scopus-indexed proceedings. Fig. 3. Article publication data from 2000 - 2021 Fig. 4. Article publication data from 2000 – 2021 (journals and proceedings only) 60 http://www.i-jim.org Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… Fig. 5. Distribution of articles by document type It can be said that these results do not reach hundreds in a year or are relatively few. We only need to underline that these data have very specific selection criteria and focus, which only discuss the study of "blended learning" for "programming." If we exclude the keyword "programming," we will get very different results because blended learn- ing has been widely applied in various courses at universities around the world. 3.2 Classification of articles by SDGs According to the Sustainable Development Goals (SDGs). The 240 articles are clas- sified as follows (See Figure 6): Fig. 6. Classification of articles by SGDs In Figure 6, it can be seen that the dominance of articles on blended learning did contribute to improving the quality of education based on the SGDs, namely 51% of the number of articles that had an impact on 04, "Quality Education". 3.3 Co-authorship analysis Co-authorship analysis aims to see the relationship between one article and another and see collaboration between researchers. Figure 7 shows that 7 researchers own the iJIM ‒ Vol. 17, No. 05, 2023 61 Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… widest network: Fraga, Francisco J; Goya, Denise Hideko; Josko, João Marcelo Borovina; Kobayashi, Guiou; Pimentel, Edson Pinheiro; Savegnago, Heitor Rodrigues; and Zampirolli, Francisco A. Fig. 7. Co-authorship analysis: the most extensive network 3.4 Top 25 most-cited articles In Table 1, there are 25 articles concerned with blended learning in algorithms and programming courses with the highest number of citations. S. Hadjerrouit (52) is from 62 http://www.i-jim.org Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… Norway, followed by S. DJenic (43) from Serbia, followed by A. Alammary (41) from Saudi Arabia, and followed by J. Galvez (41) from Spain. Meanwhile, the lowest quote was obtained by Y. Murai (3) from Canada. The article with the most citations (52) was published in 2008, CPY (3.71), and the lowest citation (3) in 2020, CPY (1.50). Meanwhile, based on CPY, the highest CPY (13.67) was published in 2019 by A. Alammary, and the lowest CPY (0.45) was published in 2011 by Mohorovicic. Table 1. Top 25 most-cited articles No Authors Cited Year CPY Type Country 1 S. Hadjerrouit 52* 2008 3.71 Article Norway 2 S. Djenic 43 2011 3.91 Article Serbia 3 A. Alammary 41 2019 13.67* Article Saudi Arabia 4 J. Gálvez 41 2009 3.15 Article Spain 5 J. van Niekerk 37 2016 6.17 Article South Africa 6 O. Deperlioglu 37 2013 4.11 Article Turkey 7 T.B. Bati 26 2014 3.25 Article Ethiopia 8 A. El-Zein 19 2009 1.46 Article Australia 9 F. Lazarinis 13 2019 4.33 Article Greece 10 W. Farag 13 2012 1.30 Proceedings United States 11 F.A. Zampirolli 11 2018 2.75 Article Brazil 12 S. Thamarai Selvi 11 2012 1.10 Proceedings India 13 A. Al-Azawei 10 2016 1.67 Article Iraq 14 M. Yağci 9 2016 1.50 Article Turkey iJIM ‒ Vol. 17, No. 05, 2023 63 Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… No Authors Cited Year CPY Type Country 15 V. Del Fatto 9 2016 1.50 Article Italy 16 K. Georgouli 9 2010 0.75 Proceedings Greece 17 F.L. Wang 9 2008 0.64 Proceedings Hong Kong 18 M. Demaidi 8 2019 2.67 Article Palestine 19 E. Albrecht 8 2018 2.00 Proceedings Germany 20 A. Anggrawan 7 2018 1.75 Proceedings Indonesia 21 C. Huda 6 2018 1.50 Proceedings Indonesia 22 S. Mohorovičič 5 2011 0.45 Article Croatia 23 Y. Shi 4 2019 1.33 Proceedings China 24 V. Gupta 3 2020 1.50 Proceedings India 25 Y. Murai 3 2020 1.50 Article Canada Note: CPY = Cites Per Year; * = the highest 3.5 Top 10 authors: most-cited articles The top 10 authors with the most citations are S. Hadjerrouit, S. Djenic, A. Alammary, J. Gálvez, J. van Niekerk, O. Deperlioglu, T.B. Bati, A. El-Zein, F. Lazarinis, and W. Farag. All of them come from different countries. However, apart from the top 10 authors, it is known from Table 1 or Figure 9 that 3 countries have 2 articles: Greece, India, and Indonesia (2). 64 http://www.i-jim.org Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… Table 2. Top 10 authors: most-cited articles Author Year Title Publisher S. Hadjerrouit 2008 [8] Informatics in Education S. Djenic 2011 [9] IEEE Transactions on Education A. Alammary 2019 [10] PLoS ONE J. Gálvez 2009 [11] Knowledge-Based Systems J. van Niekerk 2016 [12] Computers and Education O. Deperlioglu 2013 [13] Computer Applications in Engineering Education T.B. Bati 2014 [14] Computer Science Education A. El-Zein 2009 [15] Advances in Engineering Education F. Lazarinis 2019 [16] Education and Information Technologies W. Farag 2012 [17] Informatics in Education Fig. 8. Top 10 authors: most-cited articles iJIM ‒ Vol. 17, No. 05, 2023 65 Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… Fig. 9. Top 10 authors: most-cited articles by countries 3.6 Blended learning for algorithm and programming teaching We have reviewed 10 articles written by the top 10 authors and sorted them by year. Table 3. Top 10 most-articles: overview Author Year Overview S. Hadjerrouit 2008 It is concluded that combining face-to-face learning with information technology cannot provide effective and efficient learning solutions if appropriate learning theories and pedagogical strategies are not used. This research presents a blended learning model for Java programming. Blended learning relies heavily on online resources. Lectures and self-paced online resources are used to deliver course content. Students work on coding exercises individually or in small groups in labs. Online collaborative activities have been added for students to discuss their programming solutions. J. Gálvez 2009 OOPS (Object-Oriented Programming System) learning tools and a testing system used (SIETTE). OOPS is a problem-solving technique that allows students to complete Object Oriented Programming exercises. Web-based scoring system for learning strategies that assess students formatively.. A. El-Zein 2009 A WebCT-based Self-Practice Online Tool (SPOT) was used to support student programming learning. S. Djenic 2011 Applied programming learning through the use of the Internet and multimedia in an integrated learning environment. Students practice coding in the lab. Additional practical exercises and knowledge assessments were given regularly via the Internet. W. Farag 2012 Programming integrated learning environment was used. Blended learning has the potential to improve learning outcomes significantly. 66 http://www.i-jim.org Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… Author Year Overview O. Deperlioglu 2013 Blending can improve learning and flexibility by integrating face-to-face and online learning. The use of the blended e-learning model in data structures and algorithms was explained in this study. LMS is the technology used. Students practice coding both in-person and online. Class time is spent explaining complex concepts and answering questions from students. T.B. Bati 2014 This study contributes to theory by presenting a design of a blended learning solution for large class programming instruction based on constructivist learning theory and free and open source technologies. Theories and practices from education, blended learning, and computer science education were used to design the course and blended learning environment. The emphasis was on the problematic area of large-class programming instruction. Most lecture time is spent on content delivery, live coding, and problem-solving. Online resources are being added to allow students to study at their own pace. J. van Niekerk 2016 Some pedagogical principles must still be followed when developing e-learning content. This study found that e-learning material that adheres to brain- compatible education principles shows promise in teaching students fundamental programming logic. A. Alammary 2019 The following models were identified: flipped model, mixed model, flex model, supplemental model, and online-practicing model. Their classification has been made based on whether the content delivery and practical activities take place face-to-face or online. All of these models appear to have the potential to improve the learning experience of novice programmers. As a result, a monitoring strat- egy should be implemented to ensure that students complete the online work. In labs, pair programming with questions from the online resource is used. F. Lazarinis 2019 In a blended learning approach, the course was designed as a Moodle course that was realized in discrete runs with manageable groups. The educational materials included learning objects with specific objectives, video content, and try-out activities. The activity was implemented with additional face-to-face support to increase active participation and improve completion rates. 3.7 Discussion Based on the data presented, it is clear that the trend of using blended learning in algorithm and programming learning has increased significantly. Although the data does not detail blended learning, we have an overview of the 10 authors who have re- ceived the most citations on the subject. Then, from the articles, we discovered an ex- citing summary by Alammary (2019), who identified 5 blended models used in algo- rithms and programming courses, as shown in Table. 3. Incorporating digital technol- ogy and online components into learning can give students more autonomy in managing their learning, better access to learning resources, and more flexibility in planning and managing their studies. Blended learning approaches have also improved students' sat- isfaction and learning outcomes. This is because they have understood the learning flow from the start. Various learning models have advantages, disadvantages, and chal- lenges. However, a good learning model must include the following components: (1) iJIM ‒ Vol. 17, No. 05, 2023 67 Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… syntax, (2) social system, (3) reaction principle, (4) support system, and (5) instruc- tional impact and nurturing effect. 4 Conclusions Education has a significant impact on a nation's quality of life. Education will deter- mine where this country goes in the future and whether it becomes a great country that is civilized, intelligent, and ready to adapt to changing times. Implementing blended learning is nothing more than the goal of improving the learning process, which impacts the quality of education itself. As we discussed earlier, all the analyzed data contributed to Sustainable Development Goal 4, namely education quality. According to the anal- ysis and summary, the 10 most cited authors are from 10 countries: Norway, Serbia, Saudi Arabia, Spain, South Africa, Turkey, Ethiopia, Australia, Greece, and the United States. The Scopus database contained most citations with publication dates ranging from 2000 to 2021; S. Hadjerrouit in 2008. We must decide which learning media to use during the learning process when using blended learning. Web-based LMS, multi- media, e-books, smart devices, the Internet, and other technologies are widely used to support blended programming. Online assessments must support the blended learning approach, and lecturers must monitor students' online activities either regularly or au- tomatically through the technology used to ensure students do not neglect their work while studying online. Finally, the blended learning model's implementation must con- sider student characteristics, pedagogical strategies, supporting resources, facility con- ditions, and infrastructure. 5 References [1] N. Donthu, S. Kumar, D. Mukherjee, N. Pandey, and W. M. Lim, "How to conduct a bibli- ometric analysis: An overview and guidelines," J Bus Res, vol. 133, pp. 285–296, Sep. 2021. https://doi.org/10.1016/J.JBUSRES.2021.04.070 [2] M. K. Linnenluecke, M. Marrone, and A. K. Singh, "Conducting systematic literature re- views and bibliometric analyses," Australian Journal of Management, vol. 45, no. 2, pp. 175–194, May 2020. https://doi.org/10.1177/0312896219877678 [3] J. A. Moral-Muñoz, E. Herrera-Viedma, A. Santisteban-Espejo, and M. J. Cobo, "Software tools for conducting bibliometric analysis in science: An up-to-date review," Profesional de la información, vol. 29, no. 1, pp. 1699–2407, Jan. 2020. https://doi.org/10.3145/EPI.2020. ENE.03 [4] R. Pranckutė, "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today's Academic World," Publications 2021, Vol. 9, Page 12, vol. 9, no. 1, p. 12, Mar. 2021. https://doi.org/10.3390/PUBLICATIONS9010012 [5] N. J. van Eck and L. Waltman, "Citation-based clustering of publications using CitNetEx- plorer and VOSviewer," Scientometrics, vol. 111, no. 2, pp. 1053–1070, May 2017. https://doi.org/10.1007/s11192-017-2300-7 [6] N. J. van Eck and L. Waltman, "Software survey: VOSviewer, a computer program for bib- liometric mapping," Scientometrics, vol. 84, no. 2, pp. 523–538, Dec. 2010. https://doi.org/ 10.1007/s11192-009-0146-3 68 http://www.i-jim.org https://doi.org/10.1016/J.JBUSRES.2021.04.070 https://doi.org/10.1177/0312896219877678 https://doi.org/10.3145/EPI.2020.ENE.03 https://doi.org/10.3145/EPI.2020.ENE.03 https://doi.org/10.3390/PUBLICATIONS9010012 https://doi.org/10.1007/s11192-017-2300-7 https://doi.org/10.1007/s11192-009-0146-3 https://doi.org/10.1007/s11192-009-0146-3 Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… [7] L. Waltman, "A review of the literature on citation impact indicators," J Informetr, vol. 10, no. 2, pp. 365–391, May 2016. https://doi.org/10.1016/J.JOI.2016.02.007 [8] S. Hadjerrouit, "Towards a blended learning model for teaching and learning computer pro- gramming: A case study," Informatics in Education, vol. 7, no. 2, pp. 181–210, 2008. https://doi.org/10.15388/infedu.2008.12 [9] S. Djenic, R. Krneta, and J. Mitic, "Blended learning of programming in the internet age," IEEE Transactions on Education, vol. 54, no. 2, pp. 247–254, 2011. https://doi.org/10.1109/ TE.2010.2050066 [10] A. Alammary, "Blended learning models for introductory programming courses: A system- atic review," PLoS One, vol. 14, no. 9, pp. 1–26, 2019. https://doi.org/10.1371/journal.pone. 0221765 [11] J. Gálvez, E. Guzmán, and R. Conejo, "A blended E-learning experience in a course of ob- ject-oriented programming fundamentals," Knowl Based Syst, vol. 22, no. 4, pp. 279–286, May 2009. https://doi.org/10.1016/J.KNOSYS.2009.01.004 [12] J. van Niekerk and P. Webb, "The effectiveness of brain-compatible blended learning mate- rial in the teaching of programming logic," Comput Educ, vol. 103, pp. 16–27, Dec. 2016. https://doi.org/10.1016/J.COMPEDU.2016.09.008 [13] O. Deperlioglu and U. Kose, "The effectiveness and experiences of blended learning ap- proaches to computer programming education," Computer Applications in Engineering Ed- ucation, vol. 21, no. 2, pp. 328–342, Jun. 2013. https://doi.org/10.1002/CAE.20476 [14] T. B. Bati and H. Gelderblom, "Blended learning of programming in large classes: a reflec- tion of students' experience from an Ethiopian University." [15] A. El-Zein and T. Langrish, "Blended Teaching and Learning of Computer Programming Skills in Engineering Curricula." https://eric.ed.gov/?id=EJ1076062 [16] F. Lazarinis, C. v. Karachristos, E. C. Stavropoulos, and V. S. Verykios, "A blended learning course for playfully teaching programming concepts to school teachers," Education and In- formation Technologies 2018 24:2, vol. 24, no. 2, pp. 1237–1249, Oct. 2018. https://doi.org/ 10.1007/S10639-018-9823-2 [17] W. Farag, "Comparing achievement of intended learning outcomes in online programming classes with blended offerings," SIGITE'12 - Proceedings of the ACM Special Interest Group for Information Technology Education Conference, pp. 25–30, 2012. https://doi.org/ 10.1145/2380552.2380561 6 Authors Agariadne Dwinggo Samala is a doctoral candidate in the Technology and Voca- tional Education field specializing in Informatics and Computer Engineering Educa- tion. He is now a lecturer and researcher at the Faculty of Engineering, Universitas Negeri Padang, West Sumatera, Indonesia. His research has extensively covered tech- nology-enhanced learning (TEL), digital learning, blended learning, flexible learning, virtual learning environments, and TVET (email: agariadne@ft.unp.ac.id). Usmeldi is a professor at the Faculty of Engineering, Universitas Negeri Padang, West Sumatera, Indonesia (email: usmeldy@yahoo.co.id). Taali is a senior lecturer at the Faculty of Engineering, Universitas Negeri Padang, West Sumatera, Indonesia (email: taaliftunp@gmail.com). iJIM ‒ Vol. 17, No. 05, 2023 69 https://doi.org/10.1016/J.JOI.2016.02.007 https://doi.org/10.15388/infedu.2008.12 https://doi.org/10.1109/TE.2010.2050066 https://doi.org/10.1109/TE.2010.2050066 https://doi.org/10.1371/journal.pone.0221765 https://doi.org/10.1371/journal.pone.0221765 https://doi.org/10.1016/J.KNOSYS.2009.01.004 https://doi.org/10.1016/J.COMPEDU.2016.09.008 https://doi.org/10.1002/CAE.20476 https://eric.ed.gov/?id=EJ1076062 https://doi.org/10.1007/S10639-018-9823-2 https://doi.org/10.1007/S10639-018-9823-2 https://doi.org/10.1145/2380552.2380561 https://doi.org/10.1145/2380552.2380561 Paper—Top 10 Most-Cited Articles Concerning Blended Learning for Introductory Algorithms and… Yose Indarta is a police officer in the National Police Headquarters, Jakarta. In 2009, he acquired his master's degree in Vocational Education at Padang State Univer- sity. In 2022, he also received a Master's in Smart Strategic Studies from STIN, Jakarta. He is a doctoral candidate in technology and vocational education. He is an active lec- turer in his organization and is interested in educational technology, virtual learning environments, and TVET (email: yose_11@yahoo.co.id). Apdoludin is a lecturer at Universitas Muhammadiyah Muaro Bungo, Jambi, Indo- nesia. His research interests are digital learning, learning models, learning media (email: apdoludinstkipmb@gmail.com). Muhammad Hakiki is a lecturer at Universitas Muhammadiyah Muaro Bungo, Jambi, Indonesia. His research interests are digital learning, learning models, learning media and TVET (email: qiqi.lubis7@gmail.com). Kelvin Leong is a Certified Data Scientist. He holds a Ph.D. in Computer Science from The Hong Kong Polytechnic University. He has published peer-reviewed journal papers, encyclopaedia entries, book chapters and conference papers on different fields including Accounting, Criminology, Computer Sciences, Education, Engineering and Management. He had also been invited to serve as guest expert reviewer for several reputable journals, including International Journal of Information Technology and De- cision Making, IEEE Transactions on Knowledge, and Data Engineering, etc. In Janu- ary 2019, he joined the University of Chester as Professor of Financial Technology and Data Analytics (email: k.leong@chester.ac.uk). Article submitted 2022-10-31. Resubmitted 2023-01-13. Final acceptance 2023-01-13. Final version pub- lished as submitted by the authors. 70 http://www.i-jim.org