International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 15, No. 08, 2021 Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map Authoring Tool https://doi.org/10.3991/ijim.v15i08.20489 Aryo Pinandito (*) Hiroshima University, Higashihiroshima, Japan Universitas Brawijaya, Malang, Indonesia aryo@ub.ac.id Didik Dwi Prasetya Hiroshima University, Higashihiroshima, Japan Universitas Negeri Malang, Malang, Indonesia Yusuke Hayashi, Tsukasa Hirashima Hiroshima University, Higashihiroshima, Japan Abstract—Apart from contributing to students' learning outcomes, learning activities with digital concept maps were useful, fun, and engaging. Kit-Build concept map is a learning framework that incorporated concept map recomposi- tion as its essential activity. Students learn through recomposing digital concept maps from a set of teacher's concept map components; hence, a teacher concept map is essential in Kit-Build. In composing a teacher concept map of Kit-Build, teachers should reflect the learning context and strategy, embody their purpose and intention, students' understanding level, and focus questions in the forms of concept maps. Automatic generation approach produces general concept maps that are perceived to be unsuitable in corresponding said reasons; thus, a semi- automatic approach becomes preferred. The Kit-Build concept mapping tool has been extended with a support function that semi-automatically generates con- cept maps with Concept Map Mining approach. The design of the extraction and summarization phase of the concept map generation process, which sug- gests the extracted concepts and proposition triples to the teachers, is presented in this study. However, the accuracy of the support system's suggestions has yet to be discovered before the tool being considered to be used in a real learning environment of EFL reading comprehension with Kit-Build and therefore inves- tigated in this study. The result suggested that the proposed Kit-Build concept map authoring support tool is better used to refine a concept map in more detail. Keywords—Accuracy, concept map, Concept Map Mining, EFL, Kit-Build 1 Introduction A concept map can be identified as a form of boundary objects. A boundary object is a tool, artifact, or scaffold to mediate discussion and negotiation between two or 50 http://www.i-jim.org https://doi.org/10.3991/ijim.v15i08.20489 mailto:aryo@ub.ac.id Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… more different views [1] and could be realized in any form of tangible objects. Elabo- rating concept maps into learning activities helped students depict and explore their understanding [2], and improve their learning achievement [3-5]. Their interaction performance could improve if the activities were supported by a computer-supported concept mapping tool [6]. Thus, learning and assessment through interactive activities [7] could be conducted with more fun and engaging [8,9]. Computer and mobile devices have become an enabler to greater access to learning contents in distance learning. In a situation where learning activities have to be con- ducted remotely, the need for mobile learning becomes more imminent [10]. Several studies in learning strategy have incorporated concept maps and computer-based con- cept mapping as one interactive learning activity [7,11] during learning. Another study also involved artificial intelligence in processing students' concept maps to assess and predict their understanding [12]. Further development of a learning strate- gy with digital concept maps brings up a learning framework called Kit-Build concept map to quickly and easily assess students' understanding [13]. Learning effects have been confirmed in learning activities that use Kit-Build. Stu- dents learn through concept map re-composition of a Kit-Build concept map kit, a set of concept map components of a teacher concept map. Kit-Build has been found use- ful in many trials, classrooms, and subjects, such as university-level math [14], uni- versity-level computer science [15], geography in junior high school [16], and science in elementary school [17]. Kit-Build concept map is also being used to support learning English as a Foreign Language (EFL) reading comprehension [18, 19]. Students use their computer tablets and mobile devices to access the learning contents and recompose concept maps to represent their understanding of the readings. The use of mobile devices in learning EFL could help students improve their understanding more [20]. With the provided concept map analysis tool, Kit-Build concept map framework helped the instructor quickly gain insight into the development of students' knowledge through the re- composition and comparison analysis. In an EFL learning strategy that uses Kit-Build concept map, many readings were used. Thus, many concept maps, which represented the readings, also have to be recomposed by the students for practices. In contrast to composing a traditional open-end concept map, students compose their concept maps by recomposing a concept map kit; hence, the teacher concept map is essential in Kit- Build [17]. Moreover, recomposing a concept map from components could help stu- dents focus more on concepts and ideas represented by the components [21]. The main drawback in learning with Kit-Build is that teachers have to prepare a concept map before using the map to explain the learning subject or be decomposed into a kit for the students to recompose. Composing a good concept map of an English text was difficult and time-consuming [19,22]. Preparing the concept maps has been an obstacle for teachers to adopt Kit-Build as their teaching strategy. Moreover, the concept maps have to adequately represent the text while corresponding to the teach- er's strategy. Concept maps could be automatically generated with various techniques and ap- proaches [23-26]. The generation process could incorporate the Concept Map Mining (CMM) method that involved the Natural Language Processing (NLP) and text min- iJIM ‒ Vol. 15, No. 08, 2021 51 Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… ing. Even though concept maps could be generated automatically from texts, the gen- erated concept maps have issues regarding coverage, accuracy, readability, and suita- bility [27] and yet satisfying to be practically used in supporting learning EFL reading comprehension with Kit-Build. Therefore, semi-automatically generating the concept maps with human interference becomes an option to address the issue. This study presented the design of the generation approach of a concept map au- thoring tool that adopted the CMM approach and text mining to generate concept maps of EFL reading comprehension texts semi-automatically. The text processing of the authoring support tool was implemented and developed using web technology, and therefore the application was generally accessible through web and mobile devic- es. The designed authoring support tool was semi-automatically generating the con- cept maps by suggesting keywords and proposition triples for the teachers to choose, modify, and incorporate the suggestions into their concept maps. However, before the tool is considered to be used practically with Kit-Build, the performance of the au- thoring support system has to be evaluated. Therefore, the accuracy of the suggestions yielded by the system, and how the tool is perceived, need to be discovered. Assessing an education application could depict how the application is used and discover how it further affected the learning [28, 29]. However, before evaluating the learning outcomes, several experts were involved. They used and evaluated the per- formance of the authoring support tool in suggesting concept map components of EFL reading comprehension texts and assisting their concept map composition activity. In order to guide the study, the following research questions were addressed: 1. With the designed extraction approach towards EFL reading comprehension learning strategy with Kit-Build, what is the accuracy of the suggested keywords and proposition triples of the authoring support tool? 2. Will the developed concept map authoring support tool perceived to be useful for assisting teachers in composing concept maps from EFL reading comprehension texts and support their teaching activities with Kit-Build concept map? The results suggested that the authoring support tool provided good accuracy in suggesting the concepts; the proposition triple suggestions were acceptable. The tool was perceived as useful to assist teachers compose concept maps of EFL reading comprehension texts, especially in refining a concept map more detail. The design of semi-automatic generation approach is presented in the Literature Review; the meth- odology and the EFL learning strategy with Kit-Build concept map are presented in Methodology section; the results are discussed in the Result and Discussion section; the remaining sections conclude the results and present the limitation and future work. 2 Literature Review 2.1 Kit-build concept map Kit-Build concept map is a learning framework that incorporated high-directed concept maps in its learning strategy. Concept map re-composition is the key activity 52 http://www.i-jim.org Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… in learning with Kit-Build concept map. Students recomposed a set of concept map components—called a kit—to represent their understanding of a particular learning topic. Using concept maps in digital form to compose and recompose concept maps, Kit-Build offered a quick and easy assessment of students’ understanding by compar- ing students’ concept maps with a respective teacher concept map [17,30]. The com- parison and analysis could be easily performed with the provided authoring tool and analyzer. The general process of learning with Kit-Build is depicted in Fig. 1. Fig. 1. General activities of learning with Kit-Build concept map As previously mentioned, students recomposed their concept map from a Kit-Build kit that is a decomposition of a teacher’s respective concept map. Before students recomposed concept maps, teachers have to compose concept maps and decompose the map into a kit before conducting the actual learning activity with Kit-Build con- cept map. Teachers have to carefully plan and compose their concept maps before- hand because their concept maps held important concepts and ideas of a learning topic. Thus, the concept map in which be decomposed into a kit could help the stu- dents focused on important ideas represented by the kit. Extending the concept map- ping activity [15] and incorporating Kit-Build into collaborative learning [31,32] could also improve their interaction during learning [21,33] and help students to un- derstand and comprehend the learning material. 2.2 Concept’s extraction and prioritization Concept's labels in a concept map are commonly specified from a document key- words as they most likely represent the document's topics or main ideas. There are many ways to determine keywords for a text document, but most existing approaches use manual assignments by experts or defined manually based on the concept map's authors' judgment. However, with the NLP and text mining techniques, it is possible to extract keywords from a text document based on statistical approaches. Most concept labels in a concept map are nouns or noun phrases. Therefore, it is evident that one approach to extracting concepts from a text is by capturing all of the Sun Eastern Sky Western Sky rises from sets in “The sun rises from the eastern sky; hence, sets in the western sky” Sun Eastern Sky Western Sky rises from sets in Sun Eastern Sky Western Sky rises from sets in Sun Eastern Sky Western Sky rises from “The sun rises from the eastern sky; hence, sets in the western sky” decomposition Teacher’s concept map decomposed into a set of concept map components Student received the learning materials and concept map components in class Teacher composed concept maps that represented a learning topic to teach in class Students tried to recompose concept maps according to their understanding of a learning topic with the provided kit 1. 2. 3. 4. “The sun rises from the eastern sky; hence, sets in the western sky” sets in rises from sets in sets in Eastern Sky Western Sky Sun Teacher compared students’ concept map and analyze the difference for learning improvement 5. Sun Eastern Sky Western Sky rises from sets in “The sun rises from the eastern sky; hence, sets in the western sky” Students got feedback from both system and teacher, thus improved their understanding 6. iJIM ‒ Vol. 15, No. 08, 2021 53 Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… nouns and noun-phrases tagged from NLP's annotation process. However, not all nouns or noun-phrases are essential to consider to put into a concept map. Subject to using the concept maps, a concept label should be non-trivial and essential. The relevance of words or keywords to a document can be measured by their fre- quency of appearance [34]. The term Term Frequency-Inverse Document Frequency (TF-IDF) is a popular algorithm in calculating the weight of terms or keywords to a document. The TF-IDF weight (wij) of a term i in a document j can be computed from the number of occurrences of the term in the document (tfij), the number of documents containing the term (dfi), and the total number of documents (N) in a corpus as in (1). 𝑤!" = 𝑡𝑓!" × log # $%! (1) The similarity between two keywords can also be computed by finding the cosine similarity (𝑐𝑜𝑠(𝜃)) between vectors of two keywords (𝑑&000⃗ , 𝑑'0000⃗ ) composed of n unique words from both vectors. The similarity value could be calculated based on the TF- IDF weight of the composing words in each keyword (wij, wik) as in (2). 𝑐𝑜𝑠(𝜃) = $"((((⃗ ∙ $#(((((⃗ ,$"((((⃗ ,∙ ,$#(((((⃗ , = ∑ .!$ .!# % !&' /∑ .!$ (% !&' /∑ .!# (% !&' (2) An unsupervised method for extracting keywords from a text document, namely Rapid Automatic Keyword Extraction (RAKE), measures the importance of a key- word to the document based on its composing words' degree and frequency [35]. The degree of a word is defined as the frequency of words appear in the keyword candi- date list plus its frequency of co-occurrence with other words in the candidate list. The score of a keyword (SR), which composed of n words, can be calculated as a sum of the ratio of each composing word's degree (deg(wi)) and its frequency of appear- ance (freq(wi)) in the candidate list as in (3). Important keywords can be selected from top T scoring keywords from the list or by setting a minimum keyword score. When working with keywords on a single document, RAKE is also more computationally efficient than a graph-based ranking approach, such as TextRank [35]. S0 = ∑ $12(.!) %516(.!) 7 !89 (3) 2.3 Relationship extraction In the extraction process of propositions from a text, a CMM system should identi- fy concepts and relationships (links). Identifying an accurate and meaningful relation- ship between a link and two concepts plays an essential role in forming a concept map proposition. A concept map proposition can be represented as a triple of concept-link- concept or a set of subject-relation-object in open information extraction (Open IE). The Open IE annotation process of Stanford CoreNLP annotates and extracts triples from a text, representing a subject, relation, and relation object [36]. The Open IE annotation corresponds to the open domain relation that captures the relation phrases expressed by the combination of verb-nouns patterns [37] and a natural logic classifier 54 http://www.i-jim.org Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… [36]. The triples resulting from the syntactic relationship extraction process serve potential candidates for concept map propositions. Relationships in a complete sentence were mostly constructed by verbs. However, many sentences use a combination of verb-noun phrases instead of a single verb to depict a meaningful relationship between subjects and objects. Therefore, identifying relationship by relying on a single verb is insufficient to identify one good and mean- ingful relationship. A simple regular expression pattern can be applied to a sentence's POS tags annotation sequence to identify verbs and verb-noun phrases as a potential relationship candidate for propositions in a concept map. The extraction process that uses the pattern to extract relationships from a sentence is called the syntactic rela- tionship extraction [36]. The pattern is given by (4) V | VP | VW * P (4) where V = verb particle? adverb? W = noun | adjective | adverb | pronoun | determiner P = preposition | particle | information marker Furthermore, the pattern could reduce the number of uninformative relationships extracted from a sentence [37] and improve the extraction performance [38]. The computed distance between two keyword vectors represents the similarity lev- el between keywords. The similarity among two keywords or phrases using cosine similarity measure can be determined by (a) calculating the keywords' TF-IDF value with (1), (b) transforming the keywords into vectors space model, and (c) computing the distance of both vectors with cosine similarity measure as in (2). 3 Methodology This study focused on supporting the concept map composing activity of teacher concept map semi-automatically. Support was given in the form of recommendations of concepts and propositions extracted from English texts. The extraction process adopted the CMM approach that involved NLP and text mining techniques. In addi- tion to presenting the design of the extraction process, the accuracy of the yielded recommendations was analyzed and evaluated. The Kit-Build concept map authoring tool was designed to be used on tablet com- puters and implemented in HTML5 and Javascript technology. The support feature extended the current authoring tool and could also be run on the same platforms. Us- ers could use the tool with the new support feature with their personal computers or tablet devices. This study extended the current Kit-Build concept map authoring tool’s functionality by adding a recommendation system as an authoring support feature and enriched the way teachers composed and improved their concept map. The support function recommended keywords and proposition triples while also allowed modifica- tions made to the suggested items. The design and development of the authoring support function are shown in Fig. 2. Before the design and development of the support function were conducted, a prelim- iJIM ‒ Vol. 15, No. 08, 2021 55 Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… inary study regarding Kit-Build concept map framework, CMM, and the Stanford CoreNLP toolkit was carried out to identify and analyze how CMM and the NLP toolkit were able to extract concepts and propositions from English reading texts. How teachers get assistance from the support function was designed following the strategy of learning EFL reading comprehension with Kit-Build concept map. Fig. 2. Research Methodology Review and analysis of the current Kit-Build concept map tool were carried out to identify how the authoring support function could be integrated into the current Kit- Build concept map tool, thus resulting in a general requirements specification. The support system architecture, activities, and text processing were designed according to the requirements specification. After all of the designs were implemented into the target program and prototype, several tests were carried out to ensure the system work as designed, thus yielded the expected outcomes. The support system performance regarding support function accuracy was evaluat- ed using several English reading comprehension materials. Several English teachers evaluated the system and composed concept maps using the support function of the authoring tool. The yielded suggestions of the support function were evaluated and classified by the teachers per their initial concept maps to measure the accuracy of suggested items. Additionally, the evaluators were given a questionnaire and request- ed to evaluate the tool’s support function regarding their perceived usefulness in con- trast to the traditional concept mapping approach with Kit-Build concept map. 3.1 EFL reading comprehension learning strategy The concept mapping strategy that was applied to this study consisted of several steps. First, teachers created a concept map that sophisticatedly represented the read- ing and decomposed the map into a set of concept map components (Kit-Build kit). The map would be used in the next phase of learning with Kit-Build concept map framework, where students reconstructed the map to express their understanding re- garding the reading. One sophisticating concept map should have enough relevant concepts and relationships to represent the content. The map should not be too general to represent the text with merely a small number of concepts and relationships or be START END Expert Teachers Preliminary Review and Analysis Design and Implementation Functional Test Compose Initial Map Classify Items next reading next reading Measurement Analysis Accuracy Measurement General Usefulness Evaluation 56 http://www.i-jim.org Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… overwhelmed with many complicated concepts and unnecessary relationships that might confuse the students. During the comparison analysis phase, students’ concept maps were compared with a teacher concept map. The comparison identifies their misconceptions or misunder- standing regarding the reading pointed by the different and missing parts. The teach- ers were then further explaining the reading and refining their concept map to a more detailed concept map that better cover the missing and different parts. In this learning strategy of EFL reading comprehension with Kit-Build concept map, teachers com- posed their concept map at least twice, i.e., during the initial concept map composing activity and during the refinement of their initial concept map. Therefore, the author- ing support features were expected to support their composing activity in these two situations. 3.2 Support function evaluation To evaluate the support system's performance, three English teachers were selected as expert evaluators based on their expertise in using concept maps in teaching EFL reading comprehension. They were requested to compose their initial concept maps of a sophisticated level based on Mueller's concept map rubric by using the system with support function [39]. Upon reviewing several publicly available concept map evalua- tion rubrics, they had agreed that Mueller's concept map rubric was compatible with the strategy of learning EFL reading comprehension with Kit-Build concept map; hence, used in this research. Fifteen reading comprehension texts of Barron’s TOEFL iBT learning materials [40] were selected to evaluate the accuracy performance of support function. During their initial concept map composing, teachers were requested to use the authoring tool’s support features to get recommendations of keywords and proposition triples. They were given a tutorial and practicing to use the concept map authoring tool. A recommendation system's accuracy can be evaluated from the positive predictive value (PPV) and the true-positive rate (TPR). In information retrieval, PPV and TPR are generally called by precision and recall, respectively. PPV considers both true- positive items (tp) and false-positive items (fp), while TPR considers both true-positive items and false-negative items (fn). PPV or precision was calculated with (5), while TPR or recall was calculated with (6). PPV = :) :);%) (5) TPV = :) :);%% (6) In evaluating the suggested keywords and proposition triples' accuracy, the teach- ers were asked to identify and classify every keyword and triple from the suggestion list for its relevance and appropriateness to their concept maps. The support function accuracy performance of this study was measured by F-measure (F1). F-measure is commonly used to evaluate the performance of information retrieval systems such as search engines, machine learning models, and natural language processing. Both pre- iJIM ‒ Vol. 15, No. 08, 2021 57 Paper—Semi-Automatic Concept Map Generation Approach of Web-Based Kit-Build Concept Map… cision and recall values were considered in calculating the system’s accuracy perfor- mance. The F-measure is formalized in (7). 𝐹9 = 2 ∙ <<= ∙ >