Çekiç, A., & Bakla, A. (2021). A review of digital formative assessment tools: Features and future directions. International Online Journal of Education and Teaching (IOJET), 8(3). 1459-1485. Received : 27.01.2021 Revised version received : 15.04.2021 Accepted : 17.04.2021 A REVIEW OF DIGITAL FORMATIVE ASSESSMENT TOOLS: FEATURES AND FUTURE DIRECTIONS Research article Correspondence: Ahmet Çekiç https://orcid.org/0000-0002-7292-3301 Sivas Cumhuriyet University acekic@cumhuriyet.edu.tr Arif Bakla https://orcid.org/0000-0001-5412-4330 Yıldırım Beyazıt University arifbakla@gmail.com Biodata(s): Ahmet Çekiç is a lecturer at the ELT Department, Sivas Cumhuriyet University, Turkey. He holds a PhD degree in ELT. His research interests include CALL, vocabulary instruction and teacher education. Arif Bakla is a lecturer at Ankara Yıldırım Beyazıt University, Turkey. He holds a PhD degree in ELT. Among his research interests are CALL, LMSs, feedback in L2 writing and formative assessment. Copyright © 2014 by International Online Journal of Education and Teaching (IOJET). ISSN: 2148-225X. Material published and so copyrighted may not be published elsewhere without written permission of IOJET. https://orcid.org/0000-0002-7292-3301 mailto:acekic@cumhuriyet.edu.tr https://orcid.org/0000-0001-5412-4330 mailto:arifbakla@gmail.com http://orcid.org/xxxx http://orcid.org/xxxx Çekiç & Bakla 1460 A REVIEW OF DIGITAL FORMATIVE ASSESSMENT TOOLS: FEATURES AND FUTURE DIRECTIONS Ahmet Çekiç acekic@cumhuriyet.edu.tr Arif Bakla arifbakla@gmail.com Abstract The Internet and the software stores for mobile devices come with a huge number of digital tools for any task, and those intended for digital formative assessment (DFA) have burgeoned exponentially in the last decade. These tools vary in terms of their functionality, pedagogical quality, cost, operating systems and so forth. Teachers and learners need guidance on how to choose the most effective digital formative software to make the most out of it. This study provides an in-depth critical review of the features of most popular formative assessment tools available on the Internet. It aims to unearth what current DFA tools can do and what further developments are needed for more effective use. The tools for analysis were sampled using frequency of mentions in educational technology websites and blogs and two scholarly databases (Web of Science and Scopus). After identifying the most frequently recommended, reviewed and researched formative assessment tools, the researchers inspected 14 tools in terms of various issues, ranging from platforms and devices used, item-types offered by the software, features related with monitoring student performance and providing feedback (through student/instructor dashboards), grading, scoring of open-response items and collaborative responses. The results indicated that all closed-ended items were common to all the tools examined and they were automatically scored, while only a few of them offered underdeveloped methods of grading open-ended items. All the tools provided learner analytics with diverse forms of data and different mechanisms for feedback, yet the most common form of data were immediate answers and numerical scores. It was also clear that popularity did not necessarily mean offering more functionalities and better tools. Based on the status of the tools, avenues for further research are discussed. Keywords: Digital formative assessment, mobile learning, feedback, distance education and online learning, improving classroom teaching 1.1. Introduction Teachers assess levels of learning to assign grades to their students for summative purposes so that they receive documents showing their achievement. As a common practice at schools, summative assessment refers to the practice of assessing learning at the end of the learning process, usually for making decisions regarding success or failure (often called “assessment of learning”). However, this is not the only type of assessment used in classes. Vonderwell and Boboc (2013) remark that educators need to use, not only traditional assessment methods but also alternative ones. Likewise, they have to use not only summative but also formative assessment. Teachers are to monitor students’ performance to pinpoint incomplete or missing knowledge/skills and try to fill the gaps in these. The purpose of this practice, which is known as formative assessment, is to inform and shape instructional processes, engage learners in learning activities and make a positive impact on affective dimensions of education mailto:acekic@cumhuriyet.edu.tr mailto:arifbakla@gmail.com International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1461 (assessment for learning/assessment as learning). Formative assessment can be carried out not only through traditional methods but also through technology thanks to recent developments in technology making it possible to directly merge instruction with assessment (Walter, Way, Dolan, & Nichols, 2010). Technology, as Robertson, Humphrey and Steele (2019) claim, can be used to facilitate formative assessment, besides its widely recognized function of providing learners with learning materials. Digital formative assessment (DFA), which is also referred as ‘online formative assessment’ or ‘web- based formative assessment’, is the outcome of the research into formative assessment and computer-assisted assessment in the last two decades (McLaughlin & Yan, 2017). Elmahdi, Al-Hattami and Fawzi (2018) claim that using DFA tools to assess the gaps in knowledge and skills is an interesting and useful method. Even though empirical research has proved positive impacts of formative assessment on instructional processes, research into this promising area has received much less attention than it deserves in education and language instruction (Abedi, 2010; Bailey, 2017). For example, in a recent study on formative assessment (Tsulaia & Adamia, 2020), the majority of the participant lecturers reported that they did not use formative assessment tools in their teaching. Two major reasons could account for this: more time is need for formative assessment, and crowded classes make it difficult for teachers to provide personalized feedback for each student (Buchanan, 2000; Hatziapostolou & Paraskakis, 2010), particularly in e-learning contexts with large numbers of students (Hsu, Chou, & Chang, 2011). The use of digital tools for formative assessment purposes is even less frequent. Using DFA tools could contribute to the process of resolving such issues (Beatty & Gerace, 2009). However, a major problem is the lack of guidance on how to select appropriate tools. Today, most educators have access to abundant digital tools for formative assessment (McLaughlin & Yan, 2017), unlike a decade ago, when Hsu et al. (2011) complained about the lack of practical e-learning solutions for implementing formative assessment. However, despite the current profusion of software programs that are labelled as “educational”, there is not a verified set of criteria or scientific data that indicate how useful these tools are. Therefore, teachers try to evaluate software through trial and error, software reviews and personal methods of evaluation (Robertson et al., 2019). Likewise, our professional field experience tells us that teachers do not know much about DFA tools or how to select an appropriate one when they intend to use a tool for formative assessment. Despite the need for clearly set guidelines for software selection, evaluation of technological tools for assessment has been a neglected issue of research (Robertson et al., 2019). Therefore, there is need for further research into modern formative assessment software (Anders, Lindberg, & Ulf, 2011; Bhagat & Spector, 2017; Blanco & Ginovart, 2012; Sek et al., 2012). Providing teachers and researchers with a review of basic features of DFA tools could help raise their awareness, make more informed decisions, and ultimately help excite more attention in this area. Therefore, it is essential that different aspects of formative assessment be examined in scholarly research to increase teachers’, learners’ and researchers’ awareness of the potential affordances of formative assessment, particularly DFA tools. In line with this need, this study examined 14 DFA tools through the lens of various key issues, including accessibility, monitoring student performance, scoring, feedback processes, platforms and devices used, quizzes and item types that could be created, gamification, practicality and cost-effectiveness. 2. Literature Review The term ‘formative assessment’ was introduced by Scriven (1967), yet it was Bloom (1969), who made it a popular concept in education (as cited in Bailey & Heritage, 2008). It is Çekiç & Bakla 1462 defined as “the process of seeking and interpreting evidence for making substantively grounded decisions or judgements about the product of a learning task in order to decide where the learners are in their learning, where they need to go, and how best to get there” (Colby-Kelly & Turner, 2008, p. 11). As it is clear in this definition, a significant property of formative assessment is that instructors gather data to arrive at decisions about instructional practices and shape teaching and learning processes accordingly (Berridge, Penney, & Wells, 2012; Gikandi, Morrow, & Davis, 2011). According to Tsulaia and Adamia (2020), it functions as a tool for monitoring progress in learning, considering students’ needs and pinpointing problems which could collectively help identify what to do next. It is a collaborative work carried out by the teacher and learners, and learning is a joint effort of these two parties. Chaiyo and Nokham (2017) note that formative assessment is considered as a useful augmentation to the traditional instruction characterised by lecturing. Data from formative assessment processes are used for day-to-day decisions, so non- evaluative feedback is a significant component of formative assessment (Cizek, 2010). In contrast to its summative counterpart, formative assessment is implemented using regularly collected data (Bailey & Heritage, 2008) about learners and learning processes. For this type of assessment to create the desired effect it should employ several strategies: (a) identifying learning objectives and criteria accepted as an indication of success, (b) offering effective in- class activities that are indicative of learning and understanding, (c) giving students some feedback that could help them proceed, (d) teaching learners how to be a source of learning for their peers, and (e) helping them to take the responsibility of their own learning (Black & Wiliam, 2009, p. 8). Using these strategies can ensure ‘assessment for learning’ rather than ‘assessment of learning’. Properly implemented formative assessment can have various affordances for teachers and learners. As various researchers note (e.g., Anwar, 2019; Hussein, 2019), formative assessment functions as a tool to promote a deeper and richer learning experience. It helps shape teaching and learning processes, attain instructional goals and therefore makes a positive impact on student grades obtained as a result of summative assessment (Tsulaia & Adamia, 2020). Formative assessment seems to be an efficient tool for learners to enhance learning gains (William, 2010). DFA helps significantly increase students’ achievement (Bhagat & Spector, 2017; Chaiyo & Nokham, 2017; Elmahdi et al., 2018; McLaughlin & Yan, 2017) through accessible digital technologies that aim to improve learning gains. One of the common characteristics of classroom response systems is that they make it possible to collect real-time assessment data, which teachers use to provide immediate feedIback (Chaiyo & Nokham, 2017; Elmahdi et al., 2018; Hadiri, 2015; Ismail, Ahmad, Mohammad, Fakri, Nor, & Pa, 2019; McLaughlin & Yan, 2017). Such immediate feedback, which is also detailed and constructive, facilitates the individualization of instruction. Robertson et al. (2019) used a modified version of a Learning Object Review Instrument by Akpınar (2008) to select the best formative assessment tool. They used eight criteria to compare three pre-selected tools: “immediacy, elaborative feedback from the instructor, personalized feedback for the student, reusability, accessibility, interface design, interaction, and cost” (p. 3). A team of three faculty members evaluated these tools and ended up choosing Socrative as the tool that gets the highest scores based on these criteria. These criteria helped the researchers identify the focus of the review. According to the results of this study, immediate scores and detailed feedback were considered to be the ultimate advantage of using a digital tool for formative assessment. The tool used in the study provided learners with extensive feedback on their responses, including information about why some answers were wrong and how they could improve. The authors note that the time which teachers saved by using automatic scoring and feedback procedures International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1463 was used for addressing issues regarding poorly answered questions. In formative assessment, the gaps in knowledge and skills are identified using formative data and the instructional focus is shifted towards them by adapting the learning activities to achieve the related objectives. In this way, such an effort gives teachers and learners an idea of what to do next during the instructional process. DFA tools also provide affective benefits, such as increased motivation (Bhagat & Spector, 2017; Faber, Luyten, & Visscher, 2017; Ismail et al., 2019; T.-H. Wang, 2008; Youhasan & Raheem, 2019), higher levels of engagement (Bhagat & Spector, 2017; Elmahdi et al., 2018; Gikandi et al., 2011; Ismail et al., 2019) and more positive attitudes (Bhagat & Spector, 2017). Such affective benefits are the result of interactivity, enjoyment, and competitiveness available in activities supported by these tools. For example, Iaremenko (2017) used Kahoot in an intermediate ESL class and found that the tool helped increase the students’ intrinsic motivation, as it involves elements of competition, mastery, cooperation and purpose. 87% of the participants thought that Kahoot motivated them to learn grammar. Ismail et al. (2019) reported that Kahoot provided a source of motivation and a sense of satisfaction. Robertson et al. (2019) report that even with little effort and time teachers allocate, DFA tools can also help increase the teacher’s presence, which can enhance motivation and engagement. DFA tools can reduce scoring time, opening room for active teaching and learning activities. According to Elmahdi et al. (2018), classroom response tools also support learner-centeredness as teachers can monitor real-time student performances and shape instruction according to learners’ immediate needs (Vonderwell & Boboc, 2013). In short, DFA tools offer extensive benefits in terms of increasing learning gains and improving students’ engagement and motivation. Despite their benefits, there are no empirical studies focussing on the characteristics of commonly used DFA tools. Therefore, the present study aimed to review the most frequently used DFA tools to provide educators and researchers with information regarding fundamental aspects of processes involved in DFA. It aims to provide a critical examination of their features to identify characteristics of formative assessment software, what makes them better tools and further directions for improvement. The following sections explain how the researchers went about collecting and analysing data to review the sampled software. 3. Method This review study aimed to investigate what current DFA tools offer in terms of a set of parameters, including features of quizzes, monitoring, feedback, grading of student responses, elements of gamification, practicality and so forth. It is usually difficult to find teachers and students who trialed most popular DFA tools to survey their perspectives of DFA software, so a viable option could be to review the features of these tools by using their websites and the tools themselves. As there is a huge amount of documentation about DFAs, which offer their research-evidenced affordances for language teaching, we aimed to paint a detailed picture of the current state of DFAs and their features. With these aims in mind, this study sought answers to the following research questions: Q1. What DFA tools are popular among educators and researchers? Q2. How accessible are DFA tools? Q3. What are the common item types used in DFA tools? Q4. How are student responses to different item types graded? Q5. What are the affordances of DFA tools for monitoring student progress and providing feedback? Çekiç & Bakla 1464 Q6. What do the formative assessment tools offer to increase their user-friendliness? Q7. What emergent innovations do the tools feature to take DFA to the next level and what might be possible future directions? The DFA tools were sampled using quantitative data (frequency of mentions in educational websites and the number of research studies on the tools in two scholarly databases). Then the features of the tools were reviewed based on the documentation they provide (e.g., on their websites) and the researchers’ comments on each tool. The next section provides detailed information about the sampling procedure. 3.1.Sampling Procedure A two-step procedure was followed to identify the tools that educators, educational leaders or technologists and other stakeholders of education find useful and worth trying. The researchers initially identified several search terms through a quick overview of the literature and carried out a Google search to find the websites listing commonly recommended formative assessment tools (Table 1). In addition, various free-form searches were used. The results mostly led the researchers to blogs, newsletters, magazine articles and so forth, in which descriptions and short reviews of the most favourite DFA tools were provided. The search was abandoned when the results seemed similar across different blogs or websites and no new tools were offered. 60 websites identified were downloaded using NCapture and were imported into NVivo and examined using word frequency analysis. NVivo functioned only as tool used to count the frequency of mentions in the websites (through word frequency analysis). That is, a word frequency analysis was carried out to obtain a frequency index of mentions. This analysis provided more than 30 DFA tools. Table 1 The search strings used in the Google search Search Term Number of Results Real-time formative assessment tools (without quotation marks) 234 Formative assessment software OR apps OR applications OR programs 251 Digital exit tickets OR slips AND formative assessment 163 Note. These results represent the number of hits provided by the search strings as of September 2020. Several other criteria informed the sampling of the software. First, the tool must allow teachers to create quizzes and monitor learner responses. Second, only free tools or those with free features were included in the sample. We excluded the tools with a different focus and objective such as learning management systems (e.g., Blackboard, Moodle, Edmodo), larger systems with embedded formative assessment (usually for distance learning) like Istation, vocabulary memorization software (e.g., Quizlet, Memrise or Gimkit, which functioned like a flashcard tool in the homework assignment mode) and tools with a narrower scope and functionality (e.g., Answergarden, Polleverywhere and so forth). Finally, software applications International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1465 intended for a single platform (i.e., those for IOS only) were also disregarded. Totally, 14 tools satisfied all these criteria (Figure 1). The researchers also examined website traffic of the selected tools (See Appendix). Figure 1. The frequency of mentions in the 60 educational blogs and websites examined in the study To cross-validate and further support the frequency analysis above, a keyword search for each of the 14 tools was carried out in two major databases (Web of Science and SCOPUS). Initially, each search string included the Boolean operator “AND” and the word “formative” to limit the number of hits, but this search strategy was abandoned as it severely limited the results for most of the tools. To ensure the accuracy of the results for some of the tools, possible spellings (including capitalized spellings for possible case sensitivity) were provided as alternatives using the Boolean operator ‘OR”. The researchers looked for some keywords signaling that the study is related to educational assessment. These were words or phrases like “formative assessment,” “poll/polling/voting”, “gamified/gamification/game-based”, “audience/learner/student response system/tool”, “real- time response”, “e-quizzes”, “exit tickets”, “quizzing” “digital (educational) tools” or the existence of the formative assessment tools (those investigated in the present study). However, when the titles included none of these words, the abstracts were examined. Table 2 presents the number of sources listed in the two databases for each of the tools. The search for Google Forms yielded very high numbers of hits (201 and 522 for Web of Science and SCOPUS, respectively), which were screened by examining the abstracts besides the titles. Second, for the tool Formative, formerly known as GoFormative, the researchers used “GoFormative” as the search with its new name yielded unmanageable number of hits. 0 20 40 60 80 100 120 140 Number of Sources Mentioning the Tool Number of Mentions Çekiç & Bakla 1466 Table 2 Web of Science (WOS) and Scopus Search Results for the Sampled Tools Tool Search String WOS SCOPUS Kahoot Kahoot OR Kahot.it 266 185 Socrative Socrative 119 79 Google Forms (“Google forms” OR "Google Forms" OR "GoogleForms" OR "Googleforms" OR “Google form” OR "Google Form" OR "GoogleForm" OR "Googleform") AND (formative OR assessment OR polling OR voting OR gamified OR gamification OR game- based OR response OR real-time OR realtime OR e-quizzes OR tickets OR quizzing) 41 35 Plickers Plickers 25 28 Nearpod Nearpod 27 17 Quizizz Quizizz OR Quizziz 15 22 Mentimeter Mentimeter 18 12 GoSoapBox Gosoapbox OR GoSoapBox OR "Go Soap Box" 4 4 Formative “Go Formative” OR GoFormative OR Goformative 2 2 Quizalize Quizalize 2 1 Classflow Classflow OR "Class Flow" OR "ClassFlow" OR "Class flow" 0 1 Triventy Triventy 0 0 Edulastic Edulastic 0 0 Yaca Paca "Yaca Paca" OR Yacapaca 0 0 Note. These keywords appeared in the title, abstract, keywords or elsewhere in the manuscript, meaning that some of the studies just mentioned the tools rather than specifically researching them, but they were all related to education and assessment. 3.2.Data Collection and Analysis The features of the tools were examined through the information obtained from the official websites of the software, data from GooglePlay and Appstore (e.g., software descriptions, user reviews etc.), software guides, software reviews on blogs and websites, scholarly publications and first-hand experience obtained through the actual use of the software in the class or specifically for the purposes of this study. The data collected through these sources were imported into NVivo and a predetermined set of codes created based on the research questions were used, but at the same time the authors allowed the codes to evolve as the features of the software were added into these codes. The project also included studies from the literature, which informed the analysis. Then all the information obtained from various sources was fully processed and comparison tables were produced. The next section presents the findings from the analysis. 4. Findings This section reports the findings of the present study based on various parameters. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1467 4.1. Popularity Socrative, Plickers, Kahoot, Google Forms, Quizizz, Nearpod and Formative (GoFormative) were more popular tools. Similarly, Kahoot, Socrative, Google Forms, Nearpod, Plickers, Mentimeter and Quizizz were the most researched tools. Six of the most popular seven tools were common in research and practice. Similarly, four of these tools were among the most frequently visited sites (Socrative, Kahoot, Nearpod and Quizizz). This indicated that both educators and researchers focused on almost the same tools. 4.2. Accessibility All the tools feature a website, which functions as a hub for all versions, and most of the tools also come with mobile versions (IOS and/or Android platforms). As of September 2020, eight of the tools (Google Forms, Socrative, Kahoot, Plickers, Nearpod, Quizizz, Mentimeter and Quizalize) had the Android version. The six of these tools (Socrative, Kahoot, Plickers, Nearpod, Quizizz and Mentimeter) also appeared on the IOS platform besides the Android platform. Learners need digital devices and an internet connection to be able to use 12 of these tools, while they do not need to have access to digital devices or the Internet to use Plickers and Quizalize. For these two tools, using a mobile device the teacher scans students’ printed cards to transfer student performances automatically. The tools examined offer free versions, but users usually have to pay for more sophisticated features. The only tool that is completely free is Google Forms. Free versions of the software come with different constraints across the tools, such as the number of questions in a quiz, rooms, students or question types and so forth. Some tools offer a time-limited trial of the paid version, yet teachers can go on using the free version. Another point related with accessibility is suitability for the handicapped. For instance, Quizalize features a text-to-speech tool, so that students could hear questions and options rather than read them. However, such features are exceptionally rare. 4.3. Item Types The most common item types are multiple-choice and true-false questions, which are featured by all the sampled tools. Other commonly seen item types include multi-select, short- response and open-ended questions (Table 3). Some other rare features include labelled diagrams, Venn diagrams, word search items, timelines, memory games, crosswords and flashcards in Classflow. It features an item type specific to maths and an instant whiteboard. The questions can be easily shared using a unique code. Similarly, Nearpod offers “draw it”, allowing highlight, draw, type, or add visuals (as GoFormative does). Moreover, it features 22 different item types for maths lessons and nine for English language arts. In terms of discussions, GoSoapBox offers a threaded discussion item. In Yacapaca, it is possible to create multiple-choice cloze tests or upload a picture or map and create a “locate position” question. Mentimeter has a word cloud item, which asks learners to respond to a question or comment by using one or two words, which are used to form a word cloud. This can be used for brainstorming or personal response tool. The paid version of Kahoot has a similar feature. Quizalize features unscramble or order letters or words item-type. Çekiç & Bakla 1468 Table 3 Question/Activity Types Featured by the Tools Tool1 M u lt ip le -C h o ic e /T ru e - F a ls e M u lt i- se le c t (C h e c k b o x ) S h o rt -A n sw e r F il l- in - th e -b la n k O p e n -E n d e d D is c u ss io n Im a g e -s u p p o rt e d V id e o -s u p p o rt e d S o rt in g M a tc h in g S e q u e n c in g ( O rd e ri n g ) P o ll in g ( S u rv e y ) Google Forms       Socrative   1 Kahoot        Plickers  2  Nearpod    3    Quizizz    4  Mentimeter    5   Quizalize     Formative        Classflow         Triventy    Gosoapbox      Edulastic         Yacapaca    6  7    Available  Provided in paid versions Notes. (1) Image upload is available for questions but it is a pro feature for options or feedback. (2) The image database can only be used in the paid version. (3) Provided as a collaborative discussion board. Student comments are checked by the teacher. (4) 1000-character limit. (5) 250-character limit. (6) multiple-choice cloze test option. (7) Videos can be embedded into the introductions of questions. 4.4.Grading and Monitoring Statistics about the whole class and individual students are provided in the instructor dashboard, and students also get reports of their performance. On the one hand, closed item types (multiple-choice, true-false and multi-select) are automatically scored and automatic feedback is given. In addition, Quizalize provides an item type allowing teachers to give weighted scores to different options. On the other hand, DFA tools are in their infancy in terms of grading open-ended questions. Two tools provide their own methods of grading these items. First, Formative uses teacher-identified keywords to assign marks to essay-type or short- answer questions. Second, Yacapaca uses artificial intelligence to assess open-ended items and supports peer feedback in the form of paired comparisons of responses. Open-ended questions can be manually marked in Classflow, and in Edulastic the teacher can score responses by using rubrics or rating scales and provide feedback (Table 4 and 5). International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1469 Table 4 Features of the Tools in relation to Scoring and Feedback Tool Feedback Live /Assignment Scoring Open- Ended Questions Aligning Quizzes to Standard Google Forms Detailed written feedback Embedding a YouTube video using the integrated search as feedback Providing a web link Feedback independent of the options    Socrative Detailed written feedback (supported with pictures in the pro version) A web link could be pasted to the feedback   (Short-response questions can be autoscored)  Kahoot Right and wrong answers Both  In ready-made games in Kahoot Studio Plickers Oral feedback based on numeric scores and statistics Live   Nearpod Sending individual student open-ended responses to student devices anonymously for detailed examination for feedback. Peer feedback through open-ended responses or collaboration board Embedding questions in slides and asking them at the right time during the presentations and provide timely feedback No automatic written feedback to options in the multiple-choice test (apart from showing the correct answer) Both   Quizizz Individual students’ global scores/responses, a leaderboard Audio-visual aids in explanations to questions Both  (Automatically marked as correct)  Mentimeter Number/percentage of Correct/incorrect answers for each question, response time, leaderboard, total points Live   Quizalize Results, answer explanations, correct answers Both Not Applicable  Formative Number/percentage of correct/incorrect answers for each question Writing feedback to student responses while viewing the responses Both Scoring based on the existence of pre-determined keywords  Classflow Correct/incorrect responses, answer feedback Both Assigning a numerical score manually  Çekiç & Bakla 1470 Triventy -Correct/incorrect responses, Leaderboard, clue, answer feedback Live   Gosoapbox -Wrong/right responses, answer feedback (detailed explanations) Both   Edulastic -Correct/incorrect responses, answer feedback (including videos), hints -Categorisation of the items based on Bloom’s taxonomy, difficulty level (easy, medium, hard), depth of knowledge (recalling, skill/concept, strategic thinking, extended thinking Both Easy manual grading through numeric scores, rating, rubrics Also feedback (all responses on a single page)  Yacapaca Progress charts, badges, structured peer assessment, correct/incorrect responses, answer feedback Both The use of artificial intelligence to score open-ended responses (up to 100 words Peer feedback through paired comparison of responses to open- ended questions Sharing key objectives with the development team so that they could align the tool to meet them  Available  Not Available International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1471 Table 5 Features of the tools in terms of data in the teacher’s dashboard and options for exporting Tool Learner Analytics Exporting Responses/Results Google Forms Mean, median, range, graphical representation (copiable graphs) Print out, download as CSV, Socrative Number/percentage of correct answers for each student Individual quiz completion levels Item report (the percentage of students who answered an item correctly) E-mailing to the instructor E-mailing to the students Downloading individual student reports/summary reports Downloading an answer key Adding detailed feedback and/or answers in the reports is a pro feature. Kahoot Individual, class or question-based reports Visual representations Information about difficult questions Download as spreadsheets Directly save to Google Drive Plickers Question-based responses and global scores and individual responses, graphs, response counts Print out (including pdf) CSV export Uses color-coding to show responses Nearpod Correct answer ratios Student engagement and participation details Individual student reports (with rates of participation in each activity) Export as pdf or CSV Quizizz Percentage of accuracy, ranks, scores, number of correct/incorrect responses for each item, average time spent for each question, toughest question and the question that took the longest time to answer Print out and download and individual student reports to parents Mentimeter Number/percentage of correct/incorrect answers for each question and total points for the participants PDF or image download of the results in the free version and export to Excel in the paid version Quizalize Percentage of class mastery for individuals, groups and the whole class, colour-coding for higher and lower rates of mastery Export or print (paid) Sharing results others (paid) Formative Individual responses CSV, spreadsheets, Google spreadsheets (a paid feature yet five free exports per month) Classflow Number/percentage of correct/incorrect answers for each question, total scores Download as spreadsheets Triventy Popularity of each answer option Students’ ranks and scores  Gosoapbox Correct/incorrect answers for each question, total scores Download activity and grade reports as spreadsheets Edulastic Correct/incorrect answers for each question, total scores, summary Very advanced statistics in the paid version (e.g., sub-group performance, question analysis, performance by students and many more) Download as spreadsheets Export to Google Classroom Yacapaca Number of attempts, number questions answered, time spent, information about attainment, scores (all these for individuals), average grade for a given date range, printable versions of progress charts for parents, individual responses and scores Download as spreadsheets  Not Available Çekiç & Bakla 1472 4.5. Gamification Tools feature gamified quizzes involving teamwork and competitiveness. For example, in Socrative and Kahoot, the teacher could form teams and each team’s scores are shown live on the screen., which can add excitement and fun. Limited time given to students to answer a question creates a challenge and some excitement. Seeing the results live on the screen even adds to the excitement and fun. Several tools take the lead when it comes to gamification. Mentimeter or Kahoot allows the teacher to play music while playing the game. Mentimeter can give more points for faster answers, which can be used for language-fluency activities. It can also show the leaderboard during or at the end of the quiz. Quizizz automatically generates fun names for the participants and show funny visuals (memes) after each question. Participants see a timer and a leaderboard; both of which introduce more excitement. Another element used to increase participation and engagement is Power-ups, such as Power up Time Freeze (freezes your time but you still get a full score), Power up x2 (doubles the score you get for an answer) or Power up Eraser (omits one wrong answer from a question). Some tools (e.g., Classflow, Nearpod) award badges to motivate students. 4.6. Practicality and Reusability All the sampled tools aimed to offer a user-friendly experience. With various features, DFA tools attempt to make materials production a seamless activity. Importing questions from third- party software (e.g., a spreadsheet) and copying a question and editing it to create the next question are great features for saving time. Each teacher can use a test many times with different classes. The ability to store quizzes for later use is a significant property of tools for reusability. Tools allow teachers to use quizzes or add questions from a database created by their user community. For instance, the search feature in the question bank offered by Kahoot using the question “What is the capital of the USA?” provides 265 results, so it takes less than a minute to prepare a capitals quiz with ten or twenty items. The ‘Teleport’ feature in Quizizz fulfils the same function. Besides this, users could share tests or test items. Two of the tools (i.e., Classflow and Quizalize) feature a marketplace where you can find free or paid materials by users. In some other tools (e.g., Triventy), teachers can copy and customize a publicly shared quiz. Mentimeter allows teachers to add Mentimeter slides directly to Powerpoint presentations (Table 6). Such features obviously add to the reusability of the materials. Table 6 Features of the Tools in terms of User-friendliness and Reusability Tool Duplicating Items/Quizzes Importing Questions/Quizzes/Merging Quizzes Question/Quiz Database Collaboration during quiz/activity creation Google Forms Both   (Within the account but it could be shared with others by the author)  Socrative Both Importing from Excel (provides a template for accurate format) Merging quizzes within an account (a pro feature) A Google spreadsheet with a list of openly shared quizzes (There is no embedded database) Sharing a quiz with a colleague for direct import  International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1473 Kahoot Both Spreadsheets Slides A searchable quiz database in paid versions Plickers Both Copy and paste from different applications and tweak the questions. Sharing packs by joining the Plickers Creator Program and can get cash from Plickers Creator Fund. in paid versions Nearpod Both PPT slides, PDF that will form the base for the quiz items Searchable lesson database  Quizizz Both Import items from a spreadsheet and the teleport feature Item database (Teleport) (It is up to the teacher to make an item private or public) Not directly in the software, (but teachers and students can prepare spreadsheets and share them) Mentimeter Both (Also possible to move questions from one presentation to another within one’s account) In the paid version A database of presentations/quizzes (from the paid accounts if allowed and from the free accounts) Teachers can share presentations with a team (paid) Quizalize Both Excel spreadsheets also from Quizlet A searchable database of questions The links to the quizzes can be shared with colleagues. Formative Both PDF, Docs, Google Docs (20 pages per month for free) A searchable quiz and item bank collaboration in quiz creation Classflow Both Importing questions within the account A searchable database (marketplace) with free and paid lessons and quizzes  Triventy   A database of public quizzes, which could also be customized Inviting others to contribute to your quiz. Teachers and students can co-author quizzes. Gosoapbox     Edulastic Duplicate items Import from pdf A searchable item bank Search using advanced filters, including difficulty, Bloom’s taxonomy, etc. Public sharing of items can be enabled or disabled by the owner of the item Also features a certified item bank Adding co- teachers, who could manage classes and administer tests and see the results Yacapaca  Importing items from your database and that of others A searchable database (materials also come with user reviews and comments). It is possible to add and customize the questions. It is also possible to refine the searches using specific syllabuses or item  Çekiç & Bakla 1474 types. Any question is automatically added to this bank. Tool Duplicating Items/Quizzes Importing questions/quizzes/merging quizzes Question/Quiz Database Collaborating during quiz/activity creation  Available  Not Available Teachers could collaborate with students/colleagues to create quizzes. For example, in Triventy, teachers could ask others to co-author a quiz. Similarly, Google Forms, a truly free tool, allows the teacher to co-author a quiz with colleagues or students. Such features could create a community of practice and increase productivity. Users are allowed to try the tools and learn to create quizzes or activities. Yacapaca is the only tool which allows users to create quizzes only when they have completed their profiles, have trialled using the tool in their classrooms and are a member of a recognized educational institution. Moreover, most of the tools provide written or video tutorials (e.g., Edulastic and Classflow, Yacapaca) on how to create assessment materials. In such tutorials, developer teams explain and demonstrate key processes in their official sites, blogs or video platforms. All the tools provide help services, and some feature a blog for disseminating the tool, interacting with the community of users and providing tips and tricks. Some tools enable students to take quizzes without having to sign up. Socrative, Kahoot and Triventy feature student websites, where learners can easily access to a live quiz by using a teacher-provided code. They can alternatively use a QR code as in Triventy or use the student website. These three tools feature a separate website for learners and allow users to take quizzes easily. This feature is quite significant in an age in which people have to remember so many passwords. 5. Discussion This section provides a discussion of the results based on the research questions. Q1. What DFA tools are popular among educators and researchers? In terms of scholarly research some of these tools were highly popular among researchers. For example, in a recent study, Wang and Tahir (2020) examined 93 studies investigating the effectiveness of Kahoot on learning gains, classroom dynamics, instructors’ and students’ attitudes towards its use. They concluded that it positively affects students’ learning performance, classroom dynamics and its users have had a positive attitude towards its use. Similarly, there are various studies on Google Forms, Socrative, Nearpod and so forth. It appears that there are much fewer or no studies on less popular tools like Gosoapbox, Edulastic, Triventy and so forth (Table 2). All the popular tools have an android or IOS version, which seems to add to the popularity of these tools. However, one could see that popularity does not translate as more useful or sophisticated features. Although they are less popular, some tools offer more sophisticated features with more functionality and item types. Plain interface of the software that offers a user-friendly experience can be the reason for their popularity because as Elmahdi et al. (2018) state, user- friendliness is a reason for preference. Understandably, more functionality usually means a more complicated user interface. Moreover, teachers, who mostly use basic functionalities, do not need others. Therefore, plainness of the design, practicality and functionality seem to contribute to the popularity of a tool. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1475 Q2. How accessible are DFA tools? Accessibility is increased if the tool has a mobile version (besides the web-based version). The greatest strength of most of these tools is their usability on mobile devices. The existence of mobile versions of DFA software increases accessibility because most students possess a mobile digital device. Even if students have access to them, laptops and desktop computers naturally take more space, and it takes more time to turn them on and connect them to the Internet. However, mobile devices are always available; one finger tap away from the student. This makes them ideal for short time use during classes. For instance, the teacher could send the students just a couple of questions and return to the regular class activities. Mobile versions of formative assessment tools become more appropriate for the nature of formative assessment. There seems to be a two-way relationship between popularity and the availability of mobile versions. That is, the more popular a tool gets, the more likely for it to feature a mobile version. Therefore, teachers are recommended to add the availability of the mobile version as a critical criterion for selection. Despite the limitations in their features, free versions are highly functional and are good enough to implement formative assessment because the most basic features, such as creating quizzes, monitoring students’ responses at least at a basic level, are functional for all the tools examined. This makes them appropriate for resource-limited teachers, schools or regions. Some limitations can easily be tolerated, while some others are critical in terms of effective implementation of formative assessment. Therefore, for these tools to reach larger groups of users, it is essential that they offer valuable features free. The development teams of these tools are aware of this necessity. For instance, Pytel (2013) noted that GoSoapBox was looking for a sponsoring institution so that the company could offer the tool free of charge. The need for an internet connection might seem like a severe disadvantage, particularly for the users in technologically less developed or poorer countries. For two of the tools (Quizalize and Plickers), students do not have to possess digital tools, such as computers, mobile phones or tablet computers or an internet connection. Therefore, these tools might be highly appropriate for resource-poor areas where only the main computer or the teacher’s device can connect to the Internet. Another scenario might be in cases where the age of the learners might not be appropriate to use cell phones, or still in other cases students may not be allowed to bring their devices to school or to turn them on at school. The teacher scans students’ responses to the questions projected on the screen through printed cards and the mobile software installed on a mobile phone. In some cases, these tools could be a lifesaver as a single phone owned by the teacher is enough. Q3. What are the common item types and elements used in DFA tools? All the tools examined offer basic question types, such as multiple-choice, true-false, gap- fill and so forth, which necessitate less learner engagement and are cognitively less demanding. This is not a very positive picture as such items fail to activate learners’ higher-end cognitive skills. Despite such a drawback, a quick examination of the quiz databases across tools indicates that these are the most commonly created, used and shared item types across different platforms. 60.5% of the questions created by Yacapaca users, for instance, are multiple-choice items (Grove-Stephensen, 2020). This could also mean that most formative assessment activities (in this platform and most probably in others) could be based on a shallow understanding of topics and issues assessed. To achieve more learner engagement and Çekiç & Bakla 1476 cognitive involvement via multiple-choice items, some tools introduced new features. For instance, in Yacapaca, learners are asked to add a maximum of 25-word explanation for their choice. Several reasons could account for the popularity of close-ended items. First of all, closed- ended items are automatically scored in these systems and such scoring is objective in its nature. However, most of the tools provide no methods for the automatic grading of open- ended responses. Secondly, a quick look at some of the tools’ databases indicates that one could find hundreds of previously authored close-ended quizzes or activities. Finally, the tools working without a digital device and used with cards (e.g., Plickers and Quizalize) offer fewer item types. Short-response or open-ended items, on the other hand, work well when the teacher wants to see what students could remember, rather than select among a list of provided options. They are particularly important to have students work on higher-end cognitive skills. Such features, and particularly open-ended items are critical for higher learner engagement, which can facilitate learning according to theory of the depth of processing (Craik & Lockhart, 1972). Accordingly, Plump and LaRosa (2017) considered the lack of open-ended questions as a limitation of Kahoot. Another disadvantage mentioned is the restrictions on the number of characters. As for other question types, some of the tools offer a lot more functionalities. For instance, Nearpod features a lesson builder for creating slides with embedded interactive features, including quiz questions and discussion or interactive video lessons. For instance, after a few slides of explanation or demonstration, a quick quiz could follow. This tool also integrates with Google Forms and synchronous meeting tools (Microsoft Teams and Zoom), which makes it a good tool for online classes during the pandemics. Similarly, it is possible to launch Google Meet within the application in Edulastic. Quizizz is also different from the rest because it enables teachers to embed quizzes in presentations, which could help make lessons more interactive and promote just-in-time assessment and feedback. In addition to different question types, elements of gamification are a significant property of most DFA tools, which is reflected in the terminology used by the tools. For instance, quizzes or similar activities are called “games” which are “played” rather than “administered”. Some features including but not limited to, playing music during quizzing, timed responses, an element of competition, the inclusion of some game-like item types determine the extent of gamification (e.g., wordsearch or crossword puzzles or memory game in Classflow and Space race in Socrative). Moreover, autogenerating fun names for the learners and showing funny pictures after the questions (Quizizz), showing a leaderboard for the activities and quizzes to motivate students through competition, assigning extra points to the student answering a question first (Triventy), giving higher marks to faster responses, awarding badges (e.g., Classflow, Nearpod) and so forth are examples of gamification. Teachers could use such features to increase learner motivation and engagement. Q4. What are the affordances of DFA tools for monitoring student progress and providing feedback? Feedback is considered as the most critical component of formative assessment (Black & Wiliam, 1998; Jones, 2005), and all the tools address the issue of feedback and provide different mechanisms to attain the goal of giving learners information about how they are doing. Immediate feedback is a common feature of the tools examined. Although their qualities differ greatly, DFA tools allow feedback at several levels: (1) Information about right and International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1477 wrong answers and explanations. (2) An overall score for the quiz. (3) Progress of the class; used for providing general feedback (sometimes in real-time) (e.g., on poorly understood issues) and shape the instruction based on the formative data. All these justify various researchers (Chaiyo & Nokham, 2017; Elmahdi et al., 2018; Hadiri, 2015; Ismail et al., 2019; McLaughlin & Yan, 2017), who underscore real-time data collection in DFA tools. Moreover, the existence of immediate scores and detailed feedback in the tools are considered as two the most important features making the use of DFA advantageous for formative assessment (Robertson et al., 2019). The tools offer teachers a dashboard to monitor learners’ responses to specific tests or test questions. Dashboards offer various statistical measures showing learners’ success in a test or their overall progress, along with activity stream data, such as the number of attempts, response time and so forth. Such information is obviously intended for identifying gaps in attainment and skills. Some of the tools allow exporting of student responses and related statistical data (e.g., scores of individual students, item-based or whole tests statistics). The former could provide the teacher with data for individual feedback, while the latter helps him/her identify gaps in knowledge in a global sense within the whole class. The data sent as a spreadsheet could further be processed using additional statistical analyses or data mining techniques in larger datasets (Table 5). The administration of the quizzes and feedback delivery can be customized. For instance, quizzes can be teacher-paced or self-navigated. Similarly, feedback (the correct option and the mark obtained for closed-response items) could be provided immediately or delayed (e.g., in Plickers the teacher shows or hides the correct answers while playing the game). Both options are useful because our teaching experience tells us that, in some cases, students could get anxious if they immediately see their wrong answers. It is up to the teacher to choose immediate or delayed feedback. Another important aspect of formative assessment is peer feedback, and some tools (e.g., Yacapaca) support teachers in this respect. Q5. How are responses to different item types graded? All the tools are good at automatically scoring closed-ended items, while very few of them offer methods for grading open-ended ones. Automatic scoring of the test items saves time, and all the tools reviewed provide this feature for these item types. It is a feature that apparently forces teachers to create mostly closed-ended items. Short-answer questions can be automatically scored as well. However, automatic scoring in short-response items could go well with questions with a definite answer (e.g., Who invented the telephone?). Therefore, one source of difficulty is that the system would not accept the answer as true when the response is worded differently or there is a spelling mistake unless the teacher specifies all possible answers, which can be challenging. As for open-ended items, no fully reliable methods of grading have been created so far, but there have been significant developments in this area. Several tools have put an effort in developing systems to grade open-ended items. There have been four methods of grading: (1) autoscoring of short-response questions (Socrative), (2) auto-grading based on the existence of a set of pre-determined keywords (Formative), (3) assigning numerical scores manually (Classflow and Edulastic) and (4) the use of artificial intelligence for scoring open-ended items. Each of these methods is valuable in a time when we desperately need ways to deal with open- ended responses. Even being able to assign numerical scores easily (and providing additional feedback besides scores as in Edulastic) is valuable. Obviously, the success of the keyword method or artificial intelligence is open to debate and should be tested empirically, yet they seem to be good starting points for further developments. Moreover, despite the potential usefulness of the available options, they are relatively time-consuming in terms of providing Çekiç & Bakla 1478 feedback, so there seems to be a long way to go for completely automatic assessment of open- ended responses. Q6. What do the formative assessment tools offer to increase their user-friendliness? According to Trumpower and Sarwar (2010), user-friendliness is a critical element for successful formative assessment practices. User-friendliness for the students is even more important because, contrary to assumption, younger generations, who are good at gaming software and social media, are not as much competent in using academic software and producing content by using them (Author 2, 2019). Therefore, developer teams of educational tools put enormous effort into improving user-friendliness. They provide help centres, blogs, written or video tutorials to educate their community of users and pay attention to user experiences and feedback. This is quite important for user-based improvements in the tools. Reusability is a significant feature of online learning materials. It was one of the nine criteria included in the learning object evaluation instrument created by Vargo, Nesbit, Belfer and Archambault (2003) and further enhanced by Nesbit and Li (2004). There are different methods for increasing the reusability of the tests or test items, and reusability can be observed, not only within a single user account but also across different user accounts in the formative assessment tools. For example, question/quiz databases are becoming an emerging industry standard for DFA tools. Similarly, a common feature found in most of the tools is importing into one’s account through spreadsheets or templates. Another closely related issue is collaborative authoring and sharing of the quizzes and items in the tool. Collaboration means less time for materials production and higher quality output. Collaboration among materials authors happens in several ways: 1. joint creation, 2. sharing, and 3. selling. Allowing teachers to sell the materials they developed could increase their motivation to produce higher quality materials, yet it also leads to the commercialization of high-quality education, which runs counter to open-access initiatives in education. It is always good to collaborate to produce the best materials and share them openly in online platforms. Despite their extensive affordances, formative assessment software has already commercialized this area, so further commercialization through inbuilt marketplaces might be questionable. Q7. What emergent innovations do the tools offer to take DFA to the next level and what might be possible future directions? DFA tools examined offer some innovative features and seem to be working hard to improve their systems. One area of hard work is the methods of automatic scoring of open-ended items. Fortunately, some elementary methods have made their way into these tools. Two eye-catching features are the keyword method in Formative and the use of artificial intelligence in Yacapaca. Digital tools’ offering sophisticated features with respect to grading short-response and open- ended items could be a major reason for preference. It seems that tools offering sophisticated ways of grading open-ended items could take the lead as successful DFA tools in the near future. Another major innovation is related to the user-friendliness of the software. Artificial intelligence could also be used during quiz creation (by autosuggesting options based on the correct option or automatically identifying the correct option). Such developments will go on and we will have access to quicker quiz development and grading features. Some other exciting features that could help DFA tools to promote remarkable development in this area include innovative tools that offer a unified experience to learners via integration with some popular distance learning tools, which have lately been highly popular during the Covid-19 days. DFA tools can improve interaction during online classes delivered using software like Zoom or Microsoft Teams. In this respect, some tools (e.g., Nearpod and International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1479 Mentimeter) are highly useful as they allow interactive presentations with embedded questions/quizzes in them. Nearpod allows a quiz created as a ‘time to climb’ activity or presentations (with embedded quizzes and other activities) to be shared in a Zoom Meeting or allows the teacher to share it on Microsoft teams (in a channel or as an assignment). The teacher can see the results instantly and share the responses to open-ended questions with the whole class anonymously. Similarly, a key tool offered by GoSoapbox (Social Question and Answer) could help boost interactivity in the class because students can ask questions, and before sending a question, they can view earlier questions sent by peers and vote for a similar question to make it a more important one. This allows the teacher to re-explain things or slow down, provide additional information. Such features can potentially make assessment process more interactive and increase involvement. Some tools allow teachers to integrate multimedia materials in the questions or responses. 12 of the tools support images in their items. For instance, Kahoot allows teachers to upload an image or use those in its image database or embed a YouTube video in the question. In paid accounts of GoFormative, learners can insert audios or videos to their responses, and this enables them to give multimedia responses. The teacher can provide incomplete charts, graphs, graphic organizers and ask the student to complete them. Similarly, “creative response” is an item type in Classflow, which enables inserting text, photos, shapes and drawing to responses. As the name suggests, this item could help boost students’ creativity. Likewise, students could respond to questions by uploading files in Yacapaca. Other similar features include but are not limited to “draw it” in Nearpod or “locate position” in Yacapaca. 6. Conclusion DFA tools offer valuable components that could facilitate formative assessment, enrich instruction, boost learner engagement and motivation, introduce gamification and make classes more interactive. Different tools offer different features to achieve each of these aims. More importantly, as noted by Walter et al. (2010), features offered by DFA tools help embed assessment into instruction. In this respect, there is no best tool, it is up to educators and researchers to explore item types, scoring and feedback procedures, collaboration and reusability issues and decide on the tool that suits their instructional objectives and unique context. It is also possible to use different tools for different purposes. The present study provided valuable insight into currently used DFA tools. First, scholarly research on DFAs has been carried out using the most popular tools (e.g., Kahoot, Socrative), yet it should be noted that popularity does not mean more functionality. Second, the tools are easily accessible as they offer free versions, and they might be enough for the average user as such versions include the most basic functionalities. Most of the reviewed tools offer mobile versions, which increase accessibility. However, the access to an internet connection and a digital device is a must for most of the tools. Fortunately, two of the tools can be used without an internet connection or digital devices. Third, the tools usually feature basic question types (mostly closed-ended), which necessitate less learner engagement and are cognitively less demanding. There seems to be a long way to go in terms of automatic scoring of open-ended items. Fourth, learner responses could be monitored through dashboards, and such information could be used for formative assessment. Fifth, the tools also offer elements of gamification (e.g., music, puzzles, competition, leaderboards, badges) to increase learners’ motivation and engagement. They also try to increase their user-friendliness for both teachers and students. In short, DFAs offer a lot of features and functionalities, and they seek ways to improve them further. 6.1. Limitations and Further Research Çekiç & Bakla 1480 The present study aimed to see the current global status of DFA tools rather than provide a comprehensive list of the features of each tool. The sampling was based on software reviews or lists of tools provided by educators in blogs. The actual practice and the tools used in the field might differ although, we suspect, the difference might not be so great. Therefore, further research could identify popular tools based on reports from teachers with the first-hand experience of these tools. It should also be noted that although the conclusions are limited with the tools examined, they have the power to represent most formative assessment tools as these popular tools are prototypically representative of others. As current systems are in their infancy in terms of automatic assessment of open-ended responses, future developments in this area could be possible through a collaboration with findings from automatic writing evaluation research. However, there seems to be a long way to go to assess and provide feedback on open- ended responses. Further effort in research and development is needed to improve the automatic scoring of open-ended questions. Moreover, large amounts of data on learning performances can serve as “educational big data,” which can be an invaluable source for the improvement of instruction. Another limitation of this study is that the tools and their features are bound to change too quickly. This study provides a review of the tools and their features as of September 2020. It is the remit of educators to find the best pedagogical tools, methods and strategies. Therefore, teachers and researchers are recommended to use the information provided here as a point of departure rather than as the ultimate guide, considering that the availability of the item types across tools might change over time. However, the important point is that the reader could get an overall idea even if some features change for the better. The tools examined in this study come with a bunch of exciting functionalities which could have an impact on various educational variables and outcomes, such as motivation, engagement, achievement and so forth. However, the direction (positive or negative) or the size of the impact is currently not known for most of these functionalities. Future studies could investigate innovative features offered by these tools. A few particularly important developments include automatic grading of open-ended questions through artificial intelligence, the use of keywords to grade responses to open-ended questions, the use of quiz items or features that aim to help learners acquire higher-end cognitive skills, such as students’ adding explanations to the option of their choice in a multiple-choice test and so forth. Furthermore, although some tools seem more functional and flexible, they are not popular among teachers and researchers. It is beyond the scope of the present study to investigate why certain tools are more popular than others although their alternatives seem as functional as they are. Future research could investigate what features are more important for teachers and students and provide insight into why some tools are more popular than others. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1481 References Abedi, J. (2010). Research and recommendations for formative assessment with English language learners. In H. L. Andrade, & G. J. Cizek (Eds.), Handbook of formative assessment (pp. 181–197). New York, NY: Taylor & Francis. Akpınar, Y. (2008). Validation of a learning object review instrument: Relationship between ratings of learning objects and actual learning outcomes. Interdisciplinary Journal of Knowledge and Learning Objects, 4, 291–302. https://doi.org/10.28945/380 Anders, D. O., Lindberg, J. O., & Ulf, S. (2011). Shared video media and blogging online; Educational technologies for enhancing formative e- assessment? Campus-Wide Information Systems, 28(1), 41-55. https://doi.org/10.1108/10650741111097287 Anwar, S. (2019). Formative assessment tool for active learning. European Journal of Biomedical, 6(1), 480-485. Author 2. (2019). Bailey, A. L. (2017). Theoretical and developmental issues to consider in the assessment of young learners’ English language proficiency. M. K. Wolf & Y. G. Butler (Eds.), English language proficiency assessments for young learners (pp. 25–40). New York, NY: Taylor & Francis. https://doi.org/10.4324/9781315674391-2 Bailey, A. L., & Heritage, M. (2008). Formative assessment for literacy, grades K-6: Building reading and academic language skills across the curriculum: Corwin Press. Beatty, I., & Gerace, W. (2009). Technology-enhanced formative assessment: A research- based pedagogy for teaching science with classroom response technology. Journal of Science Education and Technology, 18(2), 146–162. https://doi.org/10.1007/s10956- 008-9140-4 Berridge, G. G., Penney, S., & Wells, J. A. (2012). eFACT: Formative Assessment of Classroom Teaching for Online Classes. Turkish Online Journal of Distance Education, 13(2), 119-130. Bhagat, K. K., & Spector, J. M. (2017). Formative assessment in complex problem-solving domains: The emerging role of assessment technologies. Journal of Educational Technology & Society, 20(4), 312-317. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: principles, policy & practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102 Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5-31. https://doi.org/10.1007/s11092-008-9068-5 Blanco, M., & Ginovart, M. (2012). On How Moodle Quizzes Can Contribute to the Formative e- Assessment of First-Year Engineering Students in Mathematics Courses. RUSC. Univ. and Know. Soc., 9(1), 166. https://doi.org/10.7238/rusc.v9i1.1277 Buchanan, T. (2000). The efficacy of a World‐Wide Web mediated formative assessment. Journal of Computer Assisted Learning, 16(3), 193-200. https://doi.org/10.1046/j.1365-2729.2000.00132.x Chaiyo, Y., & Nokham, R. (2017). The effect of Kahoot, Quizizz and Google Forms on the student's perception in the classrooms response system. Paper presented at the 2017 https://doi.org/10.28945/380 https://doi.org/10.1108/10650741111097287 https://doi.org/10.4324/9781315674391-2 https://doi.org/10.1007/s10956-008-9140-4 https://doi.org/10.1007/s10956-008-9140-4 https://doi.org/10.1080/0969595980050102 https://doi.org/10.1007/s11092-008-9068-5 https://doi.org/10.7238/rusc.v9i1.1277 https://doi.org/10.1046/j.1365-2729.2000.00132.x Çekiç & Bakla 1482 International Conference on Digital Arts, Media and Technology (ICDAMT). https://doi.org/10.1109/ICDAMT.2017.7904957 Cizek, G. J. (2010). an introduction to formative assessment: History, Characteristics, and Challenges. In H. L. Andrade, & G. J. Cizek (Eds.), Handbook of formative assessment (pp. 3–17). New York, NY: Taylor & Francis. https://doi.org/10.4324/9781315166933-1 Colby-Kelly, C., & Turner, C. E. (2008). AFL Research in the L2 Classroom and Evidence of Usefulness: Taking Formative Assessment to the Next Level. The Canadian Modern Language Review / La revue canadienne des langues vivantes, 64(1), 9-37. https://doi.org/10.3138/cmlr.64.1.009 Craik, F. I., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of verbal learning and verbal behavior, 11(6), 671–684. https://doi.org/10.1016/S0022-5371(72)80001-X Elmahdi, I., Al-Hattami, A., & Fawzi, H. (2018). Using Technology for Formative Assessment to Improve Students' Learning. Turkish Online Journal of Educational Technology-TOJET, 17(2), 182-188. Faber, J. M., Luyten, H., & Visscher, A. J. (2017). The effects of a digital formative assessment tool on mathematics achievement and student motivation: Results of a randomized experiment. Computers & Education, 106, 83-96. https://doi.org/10.1016/j.compedu.2016.12.001 Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher education: A review of the literature. Computers & Education, 57(4), 2333-2351. https://doi.org/10.1016/j.compedu.2011.06.004 Grove-Stephensen, I. (2020). Tutorial 11: Question Types. Retrieved from https://blog.yacapaca.com/2020/05/19/question-types/ Hadiri, Y. (2015). Click it to check it: An instructional design module to assist university faculty in using Socrative as a smart student response system for student assessment. Hatziapostolou, T., & Paraskakis, I. (2010). Enhancing the Impact of Formative Feedback on Student Learning through an Online Feedback System. Electronic Journal of e- Learning, 8(2), 111-Learning, 2010, Vol.2018(2012), p.2111-2122. Hsu, J.-L., Chou, H.-W., & Chang, H.-H. (2011). EduMiner: Using text mining for automatic formative assessment. Expert Systems With Applications, 38(4), 3431-3439. https://doi.org/10.1016/j.eswa.2010.08.129 Hussein, H. J. (2019). The impact of using Socrative based formative assessment to enhance student achievement in a nutrition course: A digital forward assessment. Iaremenko, N. V. (2017). Enhancing English language learners' motivation through online games. Information Technologies and Learning Tools, 59(3), 126–133. https://doi.org/10.33407/itlt.v59i3.1606 Ismail, M. A.-A., Ahmad, A., Mohammad, J. A.-M., Fakri, N. M. R. M., Nor, M. Z. M., & Pa, M. N. M. (2019). Using Kahoot! as a formative assessment tool in medical education: a phenomenological study. BMC medical education, 19(1), 230. https://doi.org/10.1186/s12909-019-1658-z https://doi.org/10.1109/ICDAMT.2017.7904957 https://doi.org/10.4324/9781315166933-1 https://doi.org/10.3138/cmlr.64.1.009 https://doi.org/10.1016/S0022-5371(72)80001-X https://doi.org/10.1016/j.compedu.2016.12.001 https://doi.org/10.1016/j.compedu.2011.06.004 https://blog.yacapaca.com/2020/05/19/question-types/ https://doi.org/10.1016/j.eswa.2010.08.129 https://doi.org/10.33407/itlt.v59i3.1606 https://doi.org/10.1186/s12909-019-1658-z International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1483 Jones, J. (2005). Developing effective formative assessment practices in the primary modern foreign language classroom. Encuentro: revista de investigación e innovación en la clase de idiomas(15), 39-47. McLaughlin, T., & Yan, Z. (2017). Diverse delivery methods and strong psychological benefits: A review of online formative assessment. Journal of Computer Assisted Learning, 33(6), 562-574. https://doi.org/10.1111/jcal.12200 Nesbit, J. C., & Li, J. (2004). Web-based tools for learning object evaluation. Proceedings of the International Conference on Education and Information Systems: Technologies and Applications, 2, 334–339. Plump, C. M., & LaRosa, J. (2017). Using Kahoot! in the classroom to create engagement and active learning: A game-based technology solution for eLearning novices. Management Teaching Review, 2(2), 151-158. https://doi.org/10.1177/2379298116689783 Pytel, J. (2013). We want to make GoSoapBox free to use. Retrieved from https://www.gosoapbox.com/blog/ Robertson, S. N., Humphrey, S. M., & Steele, J. P. (2019). Using Technology Tools for Formative Assessments. Journal of Educators Online, 16(2). https://doi.org/10.9743/JEO.2019.16.2.11 Sek, Y.-W., Law, C.-Y., Liew, T.-H., Bt Hisham, S., Lau, S.-H., & Pee, A. N. B. C. (2012). E- Assessment as a Self-Test Quiz Tool: The Setting Features and Formative Use. Procedia - Social and Behavioral Sciences, 65, 737-742. https://doi.org/10.1016/j.sbspro.2012.11.192 Trumpower, D. L, & Sarwar G. S. (2010). Formative structural assessment: Using concept maps as assessment for learning. In: Proceedings of Fourth International Conference on Concept Mapping (pp. 132–136). Viña del Mar, Chile. Retrieved from http://cmc.ihmc.us/cmc2010Papers/cmc2010-214.pdf Tsulaia, N., & Adamia, Z. (2020). Formative assessment tools for higher education learning environment. 3(1), 86-93. Vargo, J., Nesbit, J. C., Belfer, K., & Archambault, A. (2003). Learning object evaluation: Computer-mediated collaboration and interrater reliability. International Journal of Computers and Applications, 25(3), 1-8. https://doi.org/10.1080/1206212X.2003.11441703 Vonderwell, S. K., & Boboc, M. (2013). Promoting Formative Assessment in Online Teaching and Learning. TechTrends: Linking Research and Practice to Improve Learning, 57(4), 22-27. https://doi.org/10.1007/s11528-013-0673-x Walter, D., Way, R. P., Dolan, & Nichols, P. (2010). Psychometric challenges and opportunities in implementing formative assessment. In H. L. Andrade, & G. J. Cizek (Eds.), Handbook of formative assessment (pp. 297–315). New York, NY: Taylor & Francis. Wang, A. I., & Tahir, R. (2020). The effect of using Kahoot! for learning: A literature review. Computers & Education, 149, 86. https://doi.org/10.1016/j.compedu.2020.103818 https://doi.org/10.1111/jcal.12200 https://doi.org/10.1177/2379298116689783 https://www.gosoapbox.com/blog/ https://doi.org/10.9743/JEO.2019.16.2.11 https://doi.org/10.1016/j.sbspro.2012.11.192 https://doi.org/10.1080/1206212X.2003.11441703 https://doi.org/10.1007/s11528-013-0673-x https://doi.org/10.1016/j.compedu.2020.103818 Çekiç & Bakla 1484 Wang, T.-H. (2008). Web-Based Quiz-Game-Like Formative Assessment: Development and Evaluation. Computers & Education, 51(3), 1247-1263. https://doi.org/10.1016/j.compedu.2007.11.011 William, D. (2010). An integrative summary of the research literature and implications for a new theory of formative assessment. In H. L. Andrade, & G. J. Cizek (Eds.), Handbook of formative assessment (pp. 18–40). New York, NY: Taylor & Francis. Youhasan, P., & Raheem, S. (2019). Technology Enabled Formative Assessment in Medical Education: A Pilot Study through Kahoot. Education in Medicine Journal, 11(3), 23- 29. https://doi.org/10.21315/eimj2019.11.3.3 https://doi.org/10.1016/j.compedu.2007.11.011 https://doi.org/10.21315/eimj2019.11.3.3 International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1459-1485. 1485 Appendix Statistics on Website Traffic of the Software Tool URL Address Global Rank Category Rank Sites Linking Total # of Visits Google forms https://docs.google.com/forms N/A N/A N/A N/A Socrative https://socrative.com/ 8,277 182 1,252 28,650,000 Socrative Student https://b.socrative.com/ N/A N/A 27,050,000 Kahoot https://kahoot.com/ 3,137 180 276 66,100,000 Kahoot.it https://kahoot.it 947 6 418 267,900,000 Nearpod https://nearpod.com/ 3,266 52 504 62,200,000 Quizizz https://quizizz.com/ 970 10 639 344,600,000 Quizalize https://www.quizalize.com/ 62,733 4,221 99 2,030,000 Formative (aka “Go Formative”) https://goformative.com/ 22,638 672 191 14,180,000 Mentimeter https://www.mentimeter.com/ 8,004 288 198 15,300,000 Classflow https://classflow.com/ 252,782 11,456 98 426,000 Triventy http://www.triventy.com/ 695,907 N/A 233 29,300 Gosoapbox https://www.gosoapbox.com/ 1,107,418 N/A 87 24,300 Edulastic https://edulastic.com/ 29,759 920 117 11,960,000 Plickers https://get.plickers.com 56,759 5,024 223 1,030,000 Yacapaca https://yacapaca.com/ 2,742,112 N/A 187 9,600 https://docs.google.com/forms https://socrative.com/ https://b.socrative.com/ https://kahoot.com/ https://kahoot.it/ https://nearpod.com/ https://quizizz.com/ https://www.quizalize.com/ https://goformative.com/ https://www.mentimeter.com/ https://classflow.com/ http://www.triventy.com/ https://www.gosoapbox.com/ https://edulastic.com/ https://get.plickers.com/ https://yacapaca.com/