International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 15, No. 07, 2021


Paper—Higher Order Thinking Skills for Improved Learning Outcomes Among Indonesian Students… 

Higher Order Thinking Skills for Improved Learning 

Outcomes Among Indonesian Students: A Blended Web 

Mobile Learning (BWML) Model 

https://doi.org/10.3991/ijim.v15i07.17909 

Bambang Hariadi (), M. J. Dewiyani Sunarto, Tri Sagirani,  

Tan Amelia, Julianto Lemantara 
Dinamika University, Surabaya, Indonesia 

bambang@dinamika.ac.id 

Binar Kurnia Prahani, Budi Jatmiko 
Surabaya State University, Surabaya, Indonesia 

Abstract—This research aims to produce a model that can be used as a 

reference in implementing hybrid learning and Problem Based Learning (PBL) 

in learning with the MoLearn application. This research is a research 

development namely building the right learning model for the MoLearn 

application. The Blended Web Mobile Leaning (BWML) model test was 

conducted regarding its validity and practicality. The results showed that (1) test 

the content validity: the average statistic was 3.72 ralpha = .25 and alpha = .81, 

the construct validity: the average was 3.74, the statistic was ralpha = .20 and 

alpha = .75. (2) the feasibility test (practically used by students) an average of 

3.23 statistics ralpha = .91 and alpha = .99. It can be concluded that the BWML 

model meets the valid requirements (in content and construction), and is 

practical in use by students. This research implies that the quality BWML 

model can be used to improve learning outcomes for high school students based 

on higher-order thinking skills (HOTs). Further research can be focused on 

seeing the effectiveness of the BWML model in improving the learning 

outcomes of HOTs-based high school students. 

Keywords—Higher-order thinking skills, blended web mobile learning, 

learning outcomes. 

1 Introduction 

Important problems are facing the world of education in Indonesia today, namely 

how to seek Higher Order Thinking Skills (HOTs) based learning outcomes 

improvement [1-5]. HOTs-based learning outcomes are student learning outcomes 

that involve high-level cognitive thinking activities from Bloom's taxonomy. Based 

on Bloom's taxonomic hierarchy, HOTs-based learning outcome indicators include 

analyzing, evaluating, and creating [6]. HOTs-based learning outcomes are very 

4 http://www.i-jim.org

https://doi.org/10.3991/ijim.v15i07.17909
mailto:bambang@dinamika.ac.id


Paper—Higher Order Thinking Skills for Improved Learning Outcomes Among Indonesian Students 

important because they provide superior student competencies to compete in the era 

of industrial revolution 4.0 and the demands of 21st-century learning. 

On the other hand, 21stcentury learning emphasizes the existence of learning 
innovations, including critical thinking skills, problem-solving skills, literacy, 

collaboration, decision making, creative thinking, responsibility, and being able to 

learn independently [1] [7-9]. Martin's research results, [10] show that on average 

Indonesian students are only able to recognize some basic facts and are not yet able to 

communicate and connect various topics, especially in applying complex and abstract 

concepts. The survey results show that the average score of student achievement is 

below the average international score. In line with the survey conducted by TIMSS, 

the survey conducted by the Program for International Student Assessment (PISA) the 

average achievement score at the HOTs level in Indonesia is still far below the 

international average. This fact is in line with the results of research [1-3] [5] that the 

learning process is still a teacher center and emphasizes more on the process of 

transferring knowledge. 

The low learning outcomes of HOTs-based high school students are thought to 

have something to do with the learning process and model used. The learning model 

used is the conventional learning model. This model is considered unable to facilitate 

the development of HOTs-based high school student learning outcomes [1-3] [5]. 

The PBL model can improve HOTs [11-20]. However, the PBL model is need to 

improve inquiry orientation, training student discipline, and authentic problems are 

still more challenging [14] [21-23]. Therefore, it is still necessary to improve and 

refine the PBL model to improve the learning outcomes of HOTs-based high school 

students.  

Model Hybrid Learning is learning to provide the contents of a learning model in 

various media to keep up with current learning needs [24]. The application of Hybrid 

Learning can improve HOTs-based high school student learning outcomes but still 

needs improvement [24]. Models Hybrid Learning and Model PBL able to motivate 

students to conduct investigations and problem-solving in real-life situations and 

stimulate students to produce a product in improving learning outcomes based on 

HOTs [25]. Thus, educators need to be continuously encouraged to use e-learning 

technology and facilitate students with technology in improving academic learning 

outcomes [26]. 

To cover the weaknesses in the implementation of the Hybrid Learning Model and 

the PBL Model, it is very necessary to develop an Innovative Learning Model that can 

improve the learning outcomes of HOTs-based high school students. As an alternative 

solution that developing an Innovative Learning Model. This is in line with the 

findings of Hariadi's research [27] which suggests an innovative learning model that 

combines online learning and face-to-face learning. The innovative learning model 

developed is the BWML Model. The BWML model is expected to be able to improve 

the learning outcomes of HOTs-based high school students.  

The BWML Model is a learning model that integrates the Hybrid Learning Model 

with the PBL Model which is supported by the use of the MoLearn application in 

every learning activity. The development of the model is Blended Web Mobile 

learning supported by the latest learning theories (constructivism, learning through 

iJIM ‒ Vol. 15, No. 07, 2021 5



Paper—Higher Order Thinking Skills for Improved Learning Outcomes… 

observation, discovery learning, cognitive processes, metacognition, and scaffolding), 

as well as an empirical foundation from the latest research and scientific publications 

of researchers. 

The MoLearn application is an application for Hybrid Learning that has been 

developed by the Dinamika University research team to improve the learning 

outcomes of HOTs-based high school students. In the MoLearn application, the 

teacher functions as a facilitator, mentor, and consultant so that students who learn to 

use the MoLearn application are required to learn actively. The addition of the 

MoLearn application with the Android version is because Mobile technology is 

considered an effective way to improve student skills such as positive thinking, 

collaborative thinking, communication, and is considered a major part of innovation 

in many fields of e-learning research [28]. Besides, research from Haerazi et al [29] 

states that learning with Mobile Apps can improve critical thinking skills in teaching 

writing skills.  

The birth of the MoLearn application is based on the fact that current students are 

students in the Generation Z era. Among the characteristics of Generation Z are (1) 

Comfortable and highly dependent on technology, this is because Z-Generation grows 

surrounded by technology, (2) Multitasking with a variety of products online and 

sophisticated technology tools, and respect simplicity and interactive design, (3) 

Having a higher social responsibility with more information that can be accessed 

online, (4) Always connected, communicating through social networks, across 

countries and culture that indirectly affects the way of thinking and decision-making 

processes [30]. Educators must start thinking about a learning model that can align 

themselves with the needs of students, who always keep up with the fast development 

of gadgets today [31].  

The description above strengthens the reasons for the need for the BWML model to 

improve learning outcomes for HOTs-based high school students. The main objective 

of this research is to produce a quality BWML model to improve HOTs-based 

learning outcomes. The BWML model has five phases, namely: (1) orientation based 

on IoTs and Big Data, (2) investigation, (3) analyzing, (4) presenting, and (5) 

evaluating. Each phase of the BWML model in the implementation of learning is 

carried out and supported by using the MoLearn application.  

2 Research Methods 

2.1 General background 

The main objective of this research is to produce a quality BWML model (content 

and construct valid, reliable, and feasible/interesting). The main product in this 

research is the BWML model in the form of a BWML model book. Thus, this 

research is included in the type of development research (Research and 

Development). The development of the BWML model adapted Wademan's model 

development research design [8] [32]. The subjects in this study were high school 

students. The samples in this study were experts and students as the BWML user 

6 http://www.i-jim.org



Paper—Higher Order Thinking Skills for Improved Learning Outcomes Among Indonesian Students 

model. Experts perform validation regarding the content and constructs of the draft 

BWML model. The validity of content is the need for intervention and its design is 

based on current knowledge [32]. Construct validity is the fulfillment of a logically 

designed intervention [32]. The results of this expert's assessment are used as a 

reference for revising the draft BWML model. Students as users are taken from six 

high schools in four districts/cities in East Java Province. Students as test subjects use 

the BWML model learning with the MoLearn application, and after that are asked to 

fill out a student response questionnaire related to the application of the BWML 

model. 

2.2 Instrument and procedures of research 

The research instrument was a questionnaire for both experts and students which 

was adopted from Nieveen et al. [32]. The questionnaire for experts consists of two 

parts, namely (1) a questionnaire to measure the validity of content and (2) a 

questionnaire to measure construct validity. Questionnaire for students to measure the 

appropriateness or attractiveness of the BWML model of learning tools and activities. 

Research procedures include: (1) Preliminary research, which is carried out to obtain 

related data: (a) learning outcomes based on higher-order thinking skills, (b) PBL and 

hybrid learning models, (c) factors that support learning, (d) student and teacher 

opinions on learning. (2) Design the draft BWML model, which includes formulating 

the BWML model syntax. (3) Test the validity and feasibility of the draft BWML 

model. (4) Revision of the BWML model according to the test results.  

2.3 Data analysis data 

Analysis uses descriptive statistics, namely the average score of the questionnaire 

that has been filled in by the expert. The criteria for the mean score used the Single 

Measures Interrater Coefficient Correlation (ICC) and Cronbach's coefficient alpha 

[8] [33] as in Table 1.  

Table 1.  Evaluation criteria for the validity of the learning model 

Interval Score Criteria for Assessment Information 

3.30 <P < 4.00 Very valid Can be used without revision 

2.30 <P < 3.30 Valid Can be used with a little revision 

1.80 <P < 2.30 Less valid Can be used with many revisions 

1.00 <P < 1.80 Invalid  Cannot be used without revision and still requires 
consultation 

(Adaptation: Erika et al.) [34] 

This criterion is also used to analyze the quality of the developed BWML model. 

iJIM ‒ Vol. 15, No. 07, 2021 7



Paper—Higher Order Thinking Skills for Improved Learning Outcomes… 

3 Findings and Discussion 

3.1 Rational development of the BWML model 

BWML Model built from several basic theories, namely: (1) constructivism theory, 

(2) learning theory through observation, (3) discovery learning theory, (4) cognitive 

process theory, (5) metacognition theory, and (6) multi representation theory. Each 

phase is implemented and supported using the MoLearn application. Emphasis on the 

implementation of BWML with the percentages: 70% (on the job experience), 20% 

(mentoring and coaching), and 10% (classroom, course, and reading) [24] [35]. The 

internally organized strategy allows students to organize their thought processes, for 

example through investigations (phase 2: investigation). Gagne emphasized the 

importance of the role of cognitive strategies as one of the goals of teaching in 

schools. Learning how to think is also known as higher-order thinking skills. Students' 

knowledge of cognitive strategies in learning and thinking is an important component 

in achieving learning objectives, especially building HOTs-based learning outcomes. 

The hallmark of cognitive learning lies in learning to obtain and use representational 

forms (phase 2: investigation) that represent objects- the object faced, whether the 

object is a person, object, or event. These objects are represented or presented in a 

person through responses, ideas, or symbols that are all mental. The mental activity of 

thinking is exposed to objects that are initiated in consciousness, and physical objects 

as occur in observing, hearing, or feeling. Cognitive learning is closely related to the 

focus of this research, namely understanding the concept, which means that students 

must recall knowledge that has been learned in the past and utilize the potential of the 

environment as a learning resource.  

Learning to think is faced with a problem that must be solved (phase 1: 

orientation), but without going through observation and reorganization in observation. 

The problems faced must be solved by mental operations, especially using certain 

concepts and rules and work methods. The ability of students to solve problems 

through representational work is one of the components of HOTs which is the focus 

of this study. Jonassen [36] and Chi, Glaser & Farr [37] that underlies phase 1: 

orientation. Jean Piaget studied how children think and the processes associated with 

intellectual development that are innately curious and try to understand the world 

around them. The child needs to understand the environment by investigating and 

constructing a theory that explains it (phase 2: investigation). Lev Vygotsky Theories 

[38-39] that the basis of Phase 5: evaluation. Problem learning by John Dewey 

describes a view of education, with the school as a mirror of the larger society and the 

classroom as a laboratory for the investigation and resolution of real-life problems 

(phase 2: investigation) [40-41]. Bruner [42] provides theoretical support to discovery 

learning. When discovery learning is applied to the sciences and social sciences, 

Bruner emphasizes the inductive reasoning and investigative processes that 

characterize the scientific method (phase 3: analyzing).  

8 http://www.i-jim.org



Paper—Higher Order Thinking Skills for Improved Learning Outcomes Among Indonesian Students 

3.2 Syntax formulation for the BWML model 

Syntax is the steps listed in the lesson plan and the steps that must be followed 

when the teacher implements the learning model in the classroom. BWML model has 

five phases, namely: (1) orientation based on IoTs and Big Data, (2) investigation, (3) 

analyzing, (4) presenting, and (5) evaluating. Each phase is implemented and 

supported using the MoLearn application. The BWML model aims to improve HOTs-

based high school student learning outcomes and other goals, namely to generate 

motivation, activity, and student responses in learning. To achieve these goals, Model 

BWML was conducted through a collaborative and cooperative approach to scientific 

work (scientific approach), hybrid learning, application integration MoLearn, social 

interaction through the experience of independent study and group, and the grain 

problem-based contextual IoTs and Big Data. The learning objectives are explicitly 

contained in the syllabus and lesson plans (RPP) which are made by the teacher as 

general guidelines in implementing learning in the classroom. Good learning 

objectives are oriented specifically to students, contain a clear description of the 

assessment situation, and contain levels of performance achieved in the form of 

success criteria in learning. In general, the selection of subject matter must refer to 

basic competencies and predetermined indicators.  

3.3 BWML model quality and advisability assessment 

The results of the quality assessment of the BWML Model are presented in Table 3 

regarding the quality of the model and Table 4 regarding the feasibility 

(attractiveness) of the model. Table 3 shows that the content validity and reliability of 

the BWML Model include: (1) The need for the development of the BWML model, 

(2) the State of the Art of the BWML Model, (3) The thinking framework for the 

formation of the BWML model, and (4) Description of the BWML learning model 

has a score validation mean 3.87, 3.50, 3.83, and 3.67 with very valid criteria where 

rα = .25 and greater than r table, so that each component is declared valid. As for the 

reliability of each component with a value of α = .81, so that each component is 

declared reliable. 

 

 

 

 

 

 

 

 

 

 

 

iJIM ‒ Vol. 15, No. 07, 2021 9



Paper—Higher Order Thinking Skills for Improved Learning Outcomes… 

Table 2.  The results of the analysis of the quality assessment Model BWML 

Component 

Validity and Reliability of BWML Model 

Validity 

Score 
rα Validity α Reliability 

Content Validity      

Development Needs of Learning Model BWML 3.87 

.25 Valid .81 Reliable 

State of the Art of Model BWML 3.50 

Framework for Thinking the Formation of 

Learning Model BWML 
3.83 

Description BWML Learning Model 3.67 

Construct Validity      

Consistency of BWML Learning Model 
development  

4.00 

.20 Valid .75 Reliable Thinking framework for the formation of 
BWML Learning Model 

3.56 

Description of Learning BWML Model 3.67 

 

Table 2 shows that the construct validity and reliability of the BWML Model 

include: (1) Consistency in developing the BWML Learning Model, (2) Framework 

for the formation of the BWML Learning Model, and (3) Learning Description The 

BWML Model has an average validation score of 4.00, 3.56, and 3.67 with very valid 

criteria and rα .20 is greater than r table, so that each component is declared valid. As 

for the reliability of each component in terms of the α value, all of them are at a value 

of .75, so that each component is declared reliable. 

Table 3.  The results of the analysis of the feasibility assessment of the BWML 

Component Validity and Reliability of BWML Model 

Advisability Tool 
Validity 

Score 
rα Validity α Reliability 

components and learning activities 3.42 

.91 Valid .99 Reliable 

Newness/updating of device components and learning 

activities 
3.38 

Ease of understanding learning device components 3.19 

Ease in following the components of the process skills that 

are trained 
3.25 

Elements of attractiveness (fun and fun) in learning activities 3.00 

Teacher guidance during the learning process 3.30 

The relationship between BWML model learning and 

learning outcomes 
3.06 

 

Table 3 shows that after a large class trial was conducted, the feasibility of the 

BWML Model has an average validation score of 3.23 with very valid criteria and rα 

= .91 is greater than r table, so that each component is declared valid. As for the 

reliability of each component in terms of α value, all of them are at α = .99, so that 

each component is declared reliable. Thus it can be said that the BWML Model can be 

said to be feasible and attractive to be applied in senior high school. The social system 

in the BWML Model refers to a learning model based on Vygotsky's constructivist 

[38-39]. For a learning model can still be implemented must be supported by learning 

10 http://www.i-jim.org



Paper—Higher Order Thinking Skills for Improved Learning Outcomes Among Indonesian Students 

tools and the completeness of the facilities used. The facts show that an environment 

that provides a conducive atmosphere for teaching and learning activities will promote 

good instructional delivery and better learning outcomes. The support system for a 

learning model is all the means, materials, and tools for implementing the BWML 

Model using the MoLearn application. The support system in the BWML Model uses 

the MoLearn application, namely: (a) Learning tools refer to the BWML Model, 

namely: syllabus, lesson plans, worksheets, student teaching materials (BAM), HOTs-

based learning outcome evaluation instruments. (b) MoLearn application as the main 

support in learning. (c) Learning media in the form of virtual labs and provided 

computers/laptops, as well as internet networks for access to data literacy. Gadgets 

and games have a positive impact on the world of education [43-49]. The instructional 

impact of the BWML Model using the MoLearn application is that students can 

improve HOTs-based learning outcomes.  

4 Conclusion 

This BWML model aims to improve HOTs learning outcomes and other goals, 

namely to generate motivation, activity, and student responses in learning. To achieve 

these goals, Model BWML was conducted through a collaborative and cooperative 

approach to scientific work (scientific approach), hybrid learning, application 

integration MoLearn, social interaction through the experience of independent study 

and group, and the grain problem-based contextual IoTs and Big Data. The result of 

research shows BWML model is qualified (valid in content and constructs, and 

reliable) by experts. (2) The feasibility test (practically used by students) an average 

of 3.23 with the validity of the statistical aspects in rα = .91 and reliability in α = .99 

This means that students claim that the model is novel and easy to use. This research 

implies that a quality BWML model can be used to improve HOTs-based learning 

outcomes. Further research can be focused on the effectiveness of the BWML model 

to improve HOTs-based learning outcomes. 

5 Acknowledgment 

Our gratitude goes to the President Director of the Education Fund Management 

Institute of the Ministry of Finance of the Republic of Indonesia who has provided 

Productive Innovative Research (RISPRO) funding according to the funding 

agreement letter number: PRJ-37 / LPDP / 2018 dated 22 November 2018. 

6 References 

[1] B. Jatmiko, B. Prahani, Munasir, Z. Supardi, I. Wicaksono, N. Erlina, P. Pandiangan, R. 

Athfal and Zainuddin, “The Comparison of OR-IPA Teaching Model and Problem based 

learning Model Effectiveness to Improve Critical Thinking Skills of Pre-service Physics 

iJIM ‒ Vol. 15, No. 07, 2021 11



Paper—Higher Order Thinking Skills for Improved Learning Outcomes… 

Teachers,” Journal of Baltic Science Education, vol. 17, no. 2, pp. 1-22, 2018. 

http://www.scientiasocialis.lt/jbse/?q=node/660 https://doi.org/10.33225/jbse/18.17.300 

[2] B. Jatmiko, W. Widodo, Martini, M. Budiyanto, I. Wicaksono and P. Pandiangan, 

“Effectiveness of the INQF-based learning on a general physics for improving student's 

learning outcomes,” J. Baltic Sci. Educ., vol. 15, no. 4, pp. 441-451, 2016. 

http://www.scientiasocialis.lt/jbse/?q=node/515 https://doi.org/10.1088/1742-6596/947/1/ 

012013 

[3] S. Prayogi, L. Yuanita and Wasis, “Critical inquiry-based learning: A model of learning to 

promote critical thinking among prospective teachers of physics,” Journal of Turkish 

Science Education, vol. 15, no. 1, pp. 43-56, 2018. https://www.researchgate.net/ 

publication/325019447_Critical_Inquiry_Based_Learning_A_Model_of_Learning_to_Pro

mote_Critical_Thinking_Among_Prospective_Teachers_of_Physic 

[4] J. Siswanto, E. Susantini and B. Jatmiko, “Practicality and effectiveness of the IBMR 

teaching model to improve physics problem-solving skills,” Journal of Baltic Science 

Education, vol. 17, no. 3, pp. 381-394, 2018. http://www.scientiasocialis.lt/jbse/?q= 

node/669 https://doi.org/10.33225/jbse/18.17.381 

[5] Zulkarnaen, Z. Supardi and B. Jatmiko, “Feasibility of creative exploration, creative 

elaboration, creative modeling, practice scientific creativity, discussion, reflection 

(C3PDR) teaching model to improve students' scientific creativity of junior high school,” 

Journal of Baltic Science Education, vol. 16, no. 6, pp. 1020-1034, 2017. http://www.scient 

iasocialis.lt/jbse/?q=node/629 

[6] Anderson and Krathwohl, A taxonomy for learning, teaching and assessing: Revision of 

bloom's taxonomy of educational objectives., New York: McGraw-Hill, 2001.  

[7] P. Griffin and E. Care, Assessment and Teaching of 21st Century Skills: Methods and 

Approach, New York: Springer, 2015.  

[8] P. Pandiangan, M. Sanjaya, I. Gusti and B. Jatmiko, “The Validity and Effectiveness of 

Physics Independent Learning Model to Improve Physics Problem Solving and Self-

directed Learning Skills of Student in Open and Distance Education System,” Journal of 

Baltic Science Education, vol. 16, no. 5, pp. 651-665, 2017. http://www.scientiasocialis. 

lt/jbse/?q=node/601 

[9] I. Wicaksono, Wasis and Madlazim, “The Effectiveness of Virtual Science Teaching 

Model (VS-TM) to Improve Student's Science Creativity and Concept Mastery on Senior 

High School Physics Subject,” Journal of Baltic Science Education, vol. 16, no. 4, pp. 549-

561, 2017. www.scientiasocialis.lt/jbse/?q=node/588 

[10] M. Martin, I. Mullis, P. Foy and G. Stanco, “TIMSS 2011: International Science Report,” 

TIMSS and PIRLS International Study, Boston, 2011. 

[11] M. Arizaga, A. Bahar, C. Maker, R. Zimmermen and R. Pease, “How does Science 

Learning Occur in the Classroom? Student's Perceptions of Science Instruction during the 

Implementation of REAPS Model,” Eurasia Journal of Mathematics, Science & 

Technology Education, vol. 12, no. 3, pp. 431-455, 2016. https://www.ejmste.com/article/ 

how-does-science-learning-occur-in-the-classroom-students-perceptions-of-science-instruc 

tion-during-4499 https://doi.org/10.12973/eurasia.2016.1209a 

[12] L. Benade, Being a Teacher in the 21st Century: A Critical New Zealand Research Study, 

New York: Springer, 2017.  

[13] M. Caesar, R. Jawawi, R. Matzim, M. Shahrill, J. Jaidin and L. Mundia, “The Benefit of 

Adopting a Problem Based Learning Approach on Student's Learning Developments in 

Secondary Geography Lessons,” International Education Studies, vol. 9, no. 2, pp. 52-65, 

2016. https://www.researchgate.net/publication/291998125 https://doi.org/10.5539/ies. 

v9n2p51 

12 http://www.i-jim.org

http://www.scientiasocialis.lt/jbse/?q=node/660
https://doi.org/10.33225/jbse/18.17.300
http://www.scientiasocialis.lt/jbse/?q=node/515
https://doi.org/10.1088/1742-6596/947/1/012013
https://doi.org/10.1088/1742-6596/947/1/012013
https://www.researchgate.net/publication/325019447_Critical_Inquiry_Based_Learning_A_Model_of_Learning_to_Promote_Critical_Thinking_Among_Prospective_Teachers_of_Physic
https://www.researchgate.net/publication/325019447_Critical_Inquiry_Based_Learning_A_Model_of_Learning_to_Promote_Critical_Thinking_Among_Prospective_Teachers_of_Physic
https://www.researchgate.net/publication/325019447_Critical_Inquiry_Based_Learning_A_Model_of_Learning_to_Promote_Critical_Thinking_Among_Prospective_Teachers_of_Physic
http://www.scientiasocialis.lt/jbse/?q=node/669
http://www.scientiasocialis.lt/jbse/?q=node/669
https://doi.org/10.33225/jbse/18.17.381
http://www.scientiasocialis.lt/jbse/?q=node/629
http://www.scientiasocialis.lt/jbse/?q=node/629
http://www.scientiasocialis.lt/jbse/?q=node/601
http://www.scientiasocialis.lt/jbse/?q=node/601
http://www.scientiasocialis.lt/jbse/?q=node/588
https://www.ejmste.com/article/how-does-science-learning-occur-in-the-classroom-students-perceptions-of-science-instruction-during-4499
https://www.ejmste.com/article/how-does-science-learning-occur-in-the-classroom-students-perceptions-of-science-instruction-during-4499
https://www.ejmste.com/article/how-does-science-learning-occur-in-the-classroom-students-perceptions-of-science-instruction-during-4499
https://doi.org/10.12973/eurasia.2016.1209a
https://www.researchgate.net/publication/291998125
https://doi.org/10.5539/ies.v9n2p51
https://doi.org/10.5539/ies.v9n2p51


Paper—Higher Order Thinking Skills for Improved Learning Outcomes Among Indonesian Students 

[14] S. Chakravarthi, “Implementation of PBL Curriculum Involving Multiple Disciplines in 

Undergraduate Medical Education Programme,” International Education Studies, vol. 3, 

no. 1, pp. 165-169, 2010. https://eric.ed.gov/?id=EJ1066088 https://doi.org/10.5539/ies. 

v3n1p165 

[15] A. Efendioglu, “Problem-Based Learning Environment in Basic Computer Course: Pre-

service Teacher's Achievement and Key Factor in Learning,” Journal of International 

Education Research, vol. 3, no. 1, pp. 205-216, 2015. https://clutejournals.com/index. 

php/JIER/article/view/9372 https://doi.org/10.19030/jier.v11i3.9372 

[16] P. Leong, “Promoting Problem-Based Learning Through Collaborative Writing,” The 

English teachers, vol. XXXVII, pp. 49-60, 2017. http://journals.melta.org.my/index.php/ 

tet/article/view/295 

[17] K. Kang, S. Kim, S. Kim, J. Oh and M. Lee, “Comparison of knowledge, confidence in 

skill performance (CSP) and satisfaction in problem-based learning (PBL) and simulation 

with PBL educational modalities in caring for children with bronchiolitis,” Nurse 

Education Today, vol. 35, no. 2, pp. 315-321, 2014. https://pubmed.ncbi.nlm.nih.gov/ 

25456258/ https://doi.org/10.1016/j.nedt.2014.10.006 

[18] L. Kong, B. Qin, Y. Zhou, S. Mou and H. Gao, “The effectiveness of problem-based 

learning on development of nursing students critical thinking: A systematic review and 

meta-analysis,” International Journal of Nursing Studies, vol. 51, no. 3, pp. 458-469, 2014. 

https://pubmed.ncbi.nlm.nih.gov/23850065/ https://doi.org/10.1016/j.ijnurstu.2013.06.009 

[19] J. Lucky, “Motivating and empowering students' language learning in flipped integrated 

English classes,” Flipped Instruction: Breakthroughs in Research and Practice, Vols. -, no. 

-, pp. 189-213, 2017. http://www.scientiasocialis.lt/jbse/?q=node/660 https://doi.org/10. 

4018/978-1-5225-1803-7.ch012 

[20] M. Zabit, “Problem-Based Learning on Students'Critical Thinking Skills in Teaching 

Business Education in Malaysia: A Literature Review,” American Journal of Business 

Education, vol. 3, no. 6, p. 19, 2010. https://www.researchgate.net/publication/288880295 

https://doi.org/10.19030/ajbe.v3i6.436 

[21] O. Ates and A. Eryilmas, “Factors Effecting Performance of Tutors During Problem-Based 

Learning Implementation,” Procedia-Social and Behavioral Science, vol. 2, no. 2, pp. 325-

2329, 2010. https://www.sciencedirect.com/science/article/pii/S1877042810003708 https:// 

doi.org/10.1016/j.sbspro.2010.03.330 

[22] L. Sern, K. Salleh, M. Mohamad and J. Yunos, “Comparison of Example-Based Learning 

and Problem Based Learning in Engineering Domain,” Universal Journal of Educational 

Research, vol. 3, no. 1, pp. 39-45, 2015. https://www.researchgate.net/publication/283 

185096 https://doi.org/10.13189/ujer.2015.030106 

[23] G. Thompson, P. McInerney, D. Manning, N. Mapukata-Sondzaba, S. Chipamaunga and 

T. Maswanganyi, “Reflection of students graduating from a transforming medical 

curriculum in South Africa: a qualitative study,” BMC Medical Education, vol. 12, no. 1, 

p. 49, 2012. https://www.researchgate.net/publication/228086813 https://doi.org/10.11 

86/1472-6920-12-49 

[24] J. Watson, Blended Learning: The Convergence of Online and Face-to-face Education., 

Florida: NACOL, 2008.  

[25] B. Hariadi, MD Sunarto, P. Sudarmaningtyas and B. Jatmiko, “Hybrid Learning by Using 

Brilian Applications as One of the Learning Alternatives to Improve Learning Outcomes in 

College,” International Journal of Emerging Technologies in Learning, vol. 14, no. 10, pp. 

34-45, 2019. https://online-journals.org/index.php/i-jet/article/view/10150 https://doi.org/ 

10.3991/ijet.v14i10.10150 

iJIM ‒ Vol. 15, No. 07, 2021 13

https://eric.ed.gov/?id=EJ1066088
https://doi.org/10.5539/ies.v3n1p165
https://doi.org/10.5539/ies.v3n1p165
https://clutejournals.com/index.php/JIER/article/view/9372
https://clutejournals.com/index.php/JIER/article/view/9372
https://doi.org/10.19030/jier.v11i3.9372
http://journals.melta.org.my/index.php/tet/article/view/295
http://journals.melta.org.my/index.php/tet/article/view/295
https://pubmed.ncbi.nlm.nih.gov/25456258/
https://pubmed.ncbi.nlm.nih.gov/25456258/
https://doi.org/10.1016/j.nedt.2014.10.006
https://pubmed.ncbi.nlm.nih.gov/23850065/
https://doi.org/10.1016/j.ijnurstu.2013.06.009
http://www.scientiasocialis.lt/jbse/?q=node/660
https://doi.org/10.4018/978-1-5225-1803-7.ch012
https://doi.org/10.4018/978-1-5225-1803-7.ch012
https://www.researchgate.net/publication/288880295
https://doi.org/10.19030/ajbe.v3i6.436
https://www.sciencedirect.com/science/article/pii/S1877042810003708
https://doi.org/10.1016/j.sbspro.2010.03.330
https://doi.org/10.1016/j.sbspro.2010.03.330
https://www.researchgate.net/publication/283185096
https://www.researchgate.net/publication/283185096
https://doi.org/10.13189/ujer.2015.030106
https://www.researchgate.net/publication/228086813
https://doi.org/10.1186/1472-6920-12-49
https://doi.org/10.1186/1472-6920-12-49
https://online-journals.org/index.php/i-jet/article/view/10150
https://doi.org/10.3991/ijet.v14i10.10150
https://doi.org/10.3991/ijet.v14i10.10150


Paper—Higher Order Thinking Skills for Improved Learning Outcomes… 

[26] M. Tubagus, S. Muslim and S., “Development of Learning Management System-Based 

Blended Learning Model using Claroline in Higher Education,” International Journal of 

Interactive Mobile Technologies (iJIM), pp. 186-194, 2020. https://doi.org/10.39 

91/ijim.v14i06.13399 

[27] B. Hariadi, “Web-based cooperative learning, learning style, and student's learning 

outcomes,” Educational Horizons (Cakrawala Pendidikan), vol. 34, no. 2, pp. 160-170, 

2015. https://journal.uny.ac.id/index.php/cp/article/view/4821 https://doi.org/10.21831/ 

cp.v2i2.4821 

[28] HF El-Sofan and N. El-Haggar, “The Effectiveness of Using Mobile Learning Techniques 

to Improve Learning Outcomes in Higher Education,” International Journal of Interactive 

Mobile Technologies (iJIM), pp. 4-18, 2020. https://www.researchgate.net/ 

publication/341523798 https://doi.org/10.3991/ijim.v14i08.13125 

[29] Haerazi, IMP Utama and H. Hidayatullah, “Mobile Applications to Improve English 

Writing Skills Viewed from Critical Thinking Ability for Pre-Service Teachers,” 

International Journal of Interactive Mobile Technologies (iJIM), pp. 58-72, 2020. 

https://www.researchgate.net/publication/341191039 https://doi.org/10.3991/ijim.v14i07. 

11900 

[30] JR Nichols, 4 Essential Rules of 21st Century Learning, Teach Thought, 2015.  

[31] MD Sunarto, B. Hariadi, T. Amelia, T. Sagirani and J. Lemantara, “MoLearn, a Web-and 

Android-Based Learning Application as an Alternative for Teaching-Learning Process in 

High Schools,” International Journal of Instruction, vol. 13, no. 1, pp. 53-70, 2020. 

https://www.researchgate.net/publication/338372836 https://doi.org/10.1109/mite.2018.87 

47089 

[32] N. Nieven, S. McKenney and V. Akker, Educational Design Research, Ner York: 

Routledge, 2007.  

[33] N. Malhotra, Review of marketing research: Special issue-marketing legends, New York: 

Emerald Group Publishing Limited, 2011.  

[34] F. Erika, B. Prahani, Z. Supardi and Tukiran, “Development of a Graphic Organized-Based 

Argumentation Learning (GOAL) Model for Improving the Ability to Argue and Self-

Efficacy of Chemistry Teacher Candidates,” World Trans. on Engng. and Technol. Educ., 

vol. 16, no. 2, pp. 179-185, 2018. https://www.researchgate.net/publication/325580448 

[35] B. Woolf, Building intelligent interactive tutors: Student-centered strategies for 

revolutionizing e-learning, MA: Morgan Kaufmann, 2010. https://www.researchgate.net/ 

publication/232322117 

[36] D. Jonassen, “Toward a design theory of problem-solving,” Educational Technology 

Research and Development, vol. 15, no. 4, pp. 441-451, 2000. https://www.researchgate. 

net/publication/226873200 

[37] M. Chi, R. Glaser, and M. Farr, The nature of expertise, New York: Psychology Press, 

2014.  

[38] K. Charmaz, “Grounded theory methods in social justice research,” The Sage Handbook of 

Qualitative Research, vol. 4, pp. 359-380, 2011. https://www.researchgate.net/publication/ 

301202606 

[39] J. Stiglitz, Creating a learning society: A new approach to growth, development, and social 

progress, Columbia: Columbia University Press, 2014.  

[40] J. Loughran, Developing a Pedagogy of Teacher Education: Understanding teaching & 

learning about teaching, New York: Routledge, 2013. https://doi.org/10.4324/978020301 

9672 

[41] V. Helterbran, “Teacher leadership: Overcoming 'I am just a teacher' syndrome,” 

Education, vol. 131, no. 2, p. 363, 2010. https://www.questia.com/read/1G1-251534611/ 

14 http://www.i-jim.org

https://doi.org/10.3991/ijim.v14i06.13399
https://doi.org/10.3991/ijim.v14i06.13399
https://journal.uny.ac.id/index.php/cp/article/view/4821
https://doi.org/10.21831/cp.v2i2.4821
https://doi.org/10.21831/cp.v2i2.4821
https://www.researchgate.net/publication/341523798
https://www.researchgate.net/publication/341523798
https://doi.org/10.3991/ijim.v14i08.13125
https://www.researchgate.net/publication/341191039
https://doi.org/10.3991/ijim.v14i07.11900
https://doi.org/10.3991/ijim.v14i07.11900
https://www.researchgate.net/publication/338372836
https://doi.org/10.1109/mite.2018.8747089
https://doi.org/10.1109/mite.2018.8747089
https://www.researchgate.net/publication/325580448
https://www.researchgate.net/publication/232322117
https://www.researchgate.net/publication/232322117
https://www.researchgate.net/publication/226873200
https://www.researchgate.net/publication/226873200
https://www.researchgate.net/publication/301202606
https://www.researchgate.net/publication/301202606
https://doi.org/10.4324/9780203019672
https://doi.org/10.4324/9780203019672
https://www.questia.com/read/1G1-251534611/teacher-leadership-overcoming-i-am-just-a-teacher


Paper—Higher Order Thinking Skills for Improved Learning Outcomes Among Indonesian Students 

teacher-leadership-overcoming-i-am-just-a-teacher https://doi.org/10.3726/978-1-4539-

1799-2/31 

[42] W. Bruner, “Crack growth and the thermoelastic behavior of rocks,” Journal of 

Geophysical Research: Solid Earth, vol. 84, no. B10, pp. 557-559, 1979. https://agupubs. 

onlinelibrary.wiley.com/doi/abs/10.1029/JB084iB10p05578 https://doi.org/10.1029/jb08 

4ib10p05578 

[43] Alhalafawy, W. S., and Zaki, M. Z. T. The Effect of Mobile Digital Content Applications 

Based on Gamification in the Development of Psychological Well-Being. International 

Journal of Interactive Mobile Technologies, vol. 13 no. 8, 2019. https://online-

journals.org/index.php/i-jim/article/view/10725 https://doi.org/10.3991/ijim.v13i08.10725 

[44] Yunita, A., Nursechafia, Setiawan, E., & Nugroho, H. (2018). The Relationship between 

Mobile Phone Usage in Classroom and Academic Achievement in College Life. 

International Journal of Interactive Mobile Technologies, vol. 12 no. 8, pp. 96-103, 2018. 

https://online-journals.org/index.php/i-jim/article/view/9530 https://doi.org/10.3991/ijim.v 

12i8.9530 

[45] Fawareh, H.M.A. & Jusoh, S. The Use and Effects of Smartphones in Higher Education. 

International Journal of Interactive Mobile Technologies, vol. 11 no. 6, pp. 103-111, 2017. 

https://online-journals.org/index.php/i-jim/article/view/7453 

[46] Kalogiannakis, M., & Papadakis, S. Evaluating pre-service kindergarten teachers' intention 

to adopt and use tablets into teaching practice for natural sciences. International Journal of 

Mobile Learning and Organisation, vol. 13 no. 1, pp. 113-127, 2019. https://www.in 

derscience.com/info/inarticle.php?artid=96479 https://doi.org/10.1504/ijmlo.2019.096479 

[47] Papadakis, S. Evaluating pre-service teachers' acceptance of mobile devices with regards to 

their age and gender: a case study in Greece. International Journal of Mobile Learning and 

Organisation, vol. 12 no. 4, pp. 336-352, 2018. https://www.researchgate.net/publication/ 

328037167 https://doi.org/10.1504/ijmlo.2018.095130 

[48] Papadakis, S. Evaluating a game-development approach to teach introductory 

programming concepts in secondary education. International Journal of Technology 

Enhanced Learning, pp. 12 no. 2, pp. 127-145, 2020. https://www.researchgate.net/publi 

cation/338307449 https://doi.org/10.1504/ijtel.2020.106282 

[49] Kalogiannakis, M., Nirgianaki, G.-M., & Papadakis, St. 2018. Teaching magnetism to 

preschool children: the effectiveness of picture story reading. Early Childhood Education 

Journal, vol. 46 no. 5, pp. 535-546, 2018. https://www.researchgate.net/publication/ 

320594185 https://doi.org/10.1007/s10643-017-0884-4 

7 Authors 

Bambang Hariadi is a Vice Rector in Student Affair. His undergraduate degree in 

educational administration, postgraduate and Ph.D, are consistently pursued in 

educational technology. He is also a researcher and lecturer in Department of Film 

and Television Production at Universitas Dinamika. His research interest in the 

education technology field, such as learning and teaching models, innovation in 

learning (e-learning, blended learning, mobile learning), and learning strategy. (e-

mail: bambang@dinamika.ac.id) 

MJ Dewiyani Sunarto, is a researcher and lecturer in Department of Information 

System, Universitas Dinamika, Surabaya, Indonesia. She graduated with a Doctor in 

Mathematics Education. Her research interest in the education field, such as the 

iJIM ‒ Vol. 15, No. 07, 2021 15

https://www.questia.com/read/1G1-251534611/teacher-leadership-overcoming-i-am-just-a-teacher
https://doi.org/10.3726/978-1-4539-1799-2/31
https://doi.org/10.3726/978-1-4539-1799-2/31
https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/JB084iB10p05578
https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/JB084iB10p05578
https://doi.org/10.1029/jb084ib10p05578
https://doi.org/10.1029/jb084ib10p05578
https://online-journals.org/index.php/i-jim/article/view/10725
https://online-journals.org/index.php/i-jim/article/view/10725
https://doi.org/10.3991/ijim.v13i08.10725
https://online-journals.org/index.php/i-jim/article/view/9530
https://doi.org/10.3991/ijim.v12i8.9530
https://doi.org/10.3991/ijim.v12i8.9530
https://online-journals.org/index.php/i-jim/article/view/7453
https://www.inderscience.com/info/inarticle.php?artid=96479
https://www.inderscience.com/info/inarticle.php?artid=96479
https://doi.org/10.1504/ijmlo.2019.096479
https://www.researchgate.net/publication/328037167
https://www.researchgate.net/publication/328037167
https://doi.org/10.1504/ijmlo.2018.095130
https://www.researchgate.net/publication/338307449
https://www.researchgate.net/publication/338307449
https://doi.org/10.1504/ijtel.2020.106282
https://www.researchgate.net/publication/320594185
https://www.researchgate.net/publication/320594185
https://doi.org/10.1007/s10643-017-0884-4
mailto:bambang@dinamika.ac.id


Paper—Higher Order Thinking Skills for Improved Learning Outcomes… 

utilization of technology in education, the latest learning model, and mathematics 

education. She is currently Head of Centre for Educational Development and 

Instructional Activities at Universitas Dinamika. (e-mail: dewiyani@dinamika.ac.id 

Tri Sagirani, is a researcher and lecturer in Department of Information System, 

Universitas Dinamika . She graduated with a Bachelor of Information System, and 

then completed her Master's degree at the Institute Teknologi Sepuluh Nopember. Her 

research interest in human computer interaction, such as computer for health, 

learning, technology for education, technology for special education, and user 

experience in application computer. She is currently Head of Research and 

Community Service Division at Universitas Dinamika. (e-mail: tris@dinamika.ac.id) 

Tan Amelia is a researcher and lecturer in the Department of Information System, 

Universitas Dinamika. Her research interests include software engineering with a 

particular interest in requirements prioritization. She graduated from the Universitas 

Dinamika with bachelor's degrees in Information systems and then completed her 

Master's degrees in Technology Management from the Institut Teknologi Sepuluh 

Nopember. She is currently pursuing a Ph.D. degree with the Faculty of Computer 

Systems and Software Engineering, University Malaysia Pahang, Kuantan, Malaysia. 

(e-mail: meli@dinamika.ac.id) 

Julianto Lemantara is a researcher and lecturer in the Department of Information 

System, Dinamika University. He graduated from STMIK Surabaya with bachelor's 

degrees in Information systems. He continued his study and completed his Master's 

degrees in Information Technology from Gadjah Mada University. He has research 

interests in the field of information systems, decision support systems, and data 

mining. Until February 2020, he has published his researches in Q2 International 

Journal, International Conference with Scopus Index, and Accreted National Journal 

(email: julianto@dinamika.ac.id) 

Binar Kurnia Prahani is Doctor of Science Education. He is a researcher and 

lecturer in Universitas Negeri Surabaya. His research interest in the education field, 

such as higher order thinking skills, the utilization of technology in education, the 

latest learning model, and science education. (e-mail: binarprahani@unesa.ac.id) 

Budi Jatmiko is a researcher and lecturer in Universitas Negeri Surabaya. He is 

also a professor in Department of physics, Universitas Negeri Surabaya. His research 

interest in the education field, such as higher order thinking skills, the utilization of 

technology in education, the latest learning model, and science education. (e-mail: 

budijatmiko@unesa.ac.id; budi@dinamika.ac.id).  

Article submitted 2020-08-20. Resubmitted 2021-01-10. Final acceptance 2021-01-10. Final version 
published as submitted by the authors. 

16 http://www.i-jim.org

mailto:dewiyani@dinamika.ac.id
mailto:tris@dinamika.ac.id
mailto:meli@dinamika.ac.id
mailto:julianto@dinamika.ac.id
mailto:binarprahani@unesa.ac.id
mailto:budijatmiko@unesa.ac.id
mailto:budi@dinamika.ac.id