Journal of Software Engineering Research and Development, 2023, 11:6, doi: 10.5753/jserd.2023.2646
 This work is licensed under a Creative Commons Attribution 4.0 International License..

Insights from the application of Exploratory Tests in the daily
life of distributed teams: an experience report
Jarbele C. S. Coutinho [ Federal Rural University of the Semi-Arid | jarbele.coutinho@ufersa.edu.br
]
WilkersonL.Andrade [ FederalUniversityofCampinaGrande | wilkerson@computacao.ufcg.edu.br
]
Patrícia D. L. Machado [ Federal University of Campina Grande | patricia@computacao.ufcg.edu.br ]

Abstract
The Exploratory Testing (ET) approach has been adopted in the context of agile development due to the effec-

tiveness of its application. Due to these benefits, the need arose to train agile professionals based on the practical
application of this type of test to contribute to its incorporation into the daily work of teams. In this sense, the ob-
jective of this article is to investigate the contributions and limitations of adopting Problem-Based Learning (PBL)
and Just-in-Time Teaching (JiTT) in ET teaching-learning, and the main aspects that favor or limit the incorporation
of ET into the day-to-day of agile teams. For this, we conducted a course in remote teaching format with agile pro-
fessionals from a software development company, distributed geographically. At the end of the course, data were
collected through an online questionnaire and examined with quantitative and qualitative analysis. Then, the ET ac-
tivities performed by the participants in their daily lives were monitored and a brainstorming session was conducted
to evaluate this experience. Our main findings are that (1) the collaboration between participants and the adoption
of a real problem, along with (2) activities and resources made available before the class, and (3) the existence of
specific tool support for ET sessions optimized learning in the context of remote teaching. Other main results refer
to the planning and registration of ET and the need for guidelines to guide the execution of ET. Therefore, integrat-
ing theory and practice in ET is necessary for a better understanding of the effects of tests in the agile environment.
Additionally, it is necessary to investigate specific approaches and tools that contribute to the execution of the ET
and, consequently, to the incorporation of this test into the daily lives of the teams.

Keywords: Software testing, Exploratory Testing, Testing Education, Testing Learning and Teaching, Active Learning,
JiTT, Just-in-Time Teaching, PBL, Problem Based Learning

1 Introduction

Aligning theory and practice regarding the teaching of Soft-
ware Engineering (SE) is a persistent challenge, both in the
academic context and in the industry (Leite et al., 2020). Pro-
viding and stimulating experiences that contribute to the tech-
nical and non-technical training of students and professionals
in this area requires actions to plan the curriculum and cur-
ricular components, articulate new teaching methodologies,
and include innovative pedagogical elements (Cheiran et al.,
2017).
In this context, the teaching of Software Testing (ST) also

stands out. For Cheiran et al. (2017), ST is one of the areas of
SE that presents challenges for teaching. It may be difficult
and inefficient to teach ST through lectures and lectures. Ad-
ditionally, the simplicity of the criteria is a factor that makes
it possible for ST contents to be part of non-specific subjects,
such as SE (Paschoal and Souza, 2018). Moreover, ST con-
tents may be part of the training provided by companies when
their employees do not know a given ST practice or tech-
nique.
Among the existing ST practices, we have Exploratory

Testing (ET). ET emphasizes the responsibility and freedom
of the tester to explore the system, allowing the tester to ac-
quire knowledge of the program in parallel with the execu-
tion of the tests (Costa et al., 2019; Hendrickson, 2013; Bach,
2003; Whittaker, 2009), as there is no script planning or the

definition of test cases defined in test plans (Hendrickson,
2013). For Bach (2003), ET is learning, designing, and exe-
cuting tests performed simultaneously.
As a way to meet the need for management and measure-

ment of ET, Bach (2003) proposed (1) to divide the testing ac-
tivities into sessions, which would be the basic unit of work,
(2) to stipulate a mission for each session, and (3) adopt time
metrics related to testing activities (Castro, 2018), thus giv-
ing rise to the Session-Based Test Management (SBTM) ap-
proach.
Although the problem associated with ST teaching is being

discussed with greater visibility by the academic and scien-
tific community (Paschoal and de Souza, 2018; Garousi et al.,
2017, 2020; Scatalon et al., 2019; Aniche et al., 2019) and is
producing more specific developments (Cheiran et al., 2017;
de Andrade et al., 2019; Martinez, 2018; Coutinho and Bez-
erra, 2018; Paschoal and Souza, 2018; Paschoal et al., 2017;
Queiroz et al., 2019), few studies investigate the possibilities
of streamlining the teaching and application of ET in practice
(Costa et al., 2019; Ferreira Costa and Oliveira, 2020).

Adopting more dynamic strategies that bring theory and
practice closer together to provide academic-professional
training in the real scenario of the software industry is not
a trivial task, especially when this experience is conducted
with geographically distributed teams that work in an agile
environment.
When conducting experiences like this, some challenges

https://orcid.org/0000-0002-7058-7631
mailto:jarbele.coutinho@ufersa.edu.br
https://orcid.org/0000-0003-0656-6139
mailto:wilkerson@computacao.ufcg.edu.br
mailto:patricia@computacao.ufcg.edu.br


Coutinho et al. 2023

emerge, such as (1) integrating the team that works in a cross-
functional way, due to the adoption of agile practices; (2) cre-
ating conditions for the flow of knowledge to develop, con-
sidering the different ways in which people assimilate infor-
mation; and, (3) dealing with contextual challenges, such as
communication, time, internet connection, among others.
There is a need to investigate ways to conduct ET teach-

ing for agile teams working with Distributed Software De-
velopment (DSD). Therefore, our research question is: How
to encourage practical ET learning with geographically dis-
tributed agile teams, seeking integration among members,
and promoting active learning, in order to encourage their
insertion in the daily work?
In these circumstances, learning in a participatory way,

from real problems and situations, can contribute to learning
evolution. Problem-Based Learning (PBL), is an active learn-
ing approach (Bonwell and Eison, 1991; McConnell, 1996)
that, from problem-solving, enables students to live experi-
ences that portray the reality of the professional context in
the academic environment (Cheiran et al., 2017) and aims to
encourage the collaborative resolution of challenges through
research, reflection and development of solutions.
In an associated way, Just-in-Time Teaching (JiTT) (No-

vak, 2011) also aims to contribute to student learning. Based
on activities carried out before class, JiTT encourages the
development of prior knowledge of students (Novak, 2011;
Martinez, 2018) so that we can further develop discussions
about a given content during class.
This study aims to investigate the contributions and lim-

itations of adopting PBL and JiTT in ET teaching-learning
with agile DSD teams in a remote learning context, in order
to encourage the incorporation of ET practices in the daily
lives of these teams. Thus, it is expected to contribute to the
mitigation of the main challenges - mentioned above - faced
in the execution of courses conducted in a DSD context, and
encourage the adoption of ET in the ST practices developed
by agile teams.
For this, we carried out an ET course with agile profes-

sionals from a software development company, distributed
geographically. At the end of the course, data were collected
through an online questionnaire and examined. Then, the ET
activities performed by the participants in their daily lives
were monitored and a brainstorming session was held to eval-
uate this experience.
It is worth noting that this paper is an extended version

of the award-winning paper “Teaching Exploratory Tests
through PBL and JiTT: an experience report in a context
of distributed teams”, published in the proceedings of the
35th Brazilian Symposium on Software Engineering (SBES
2021), Education Track (Coutinho et al., 2021).
In addition to this introductory section, this paper is struc-

tured as follows: Section 2 discusses an overview of ST
Teaching. Section 3 describes the methodological procedure
used in this study. Section 4 presents the results obtained,
in response to the defined research questions. Section 5 dis-
cusses the perspectives, challenges, and limitations of this
study, based on the results obtained. Section 6 discusses the
threats to the validity of this study. Section 7 exposes the
analysis of some related works. Finally, Section 8 presents
the final considerations and perspectives for future work.

2 Background
In this section, we discuss important aspects related to Teach-
ing Software Testing and Exploratory Testing. We then
present and discuss some relevant concepts about two main
approaches to active methodologies, PBL and JiTT.

2.1 Teaching Software Testing
ST is an essential activity to guarantee the quality of software.
Seeking to meet the need to use teaching methods that make
the learning of this activity more effective, some studies
have been dedicated to investigating systematic approaches
to contribute to the teaching in this area of SE (Paschoal and
de Souza, 2018; Garousi et al., 2017, 2020; Scatalon et al.,
2019; Aniche et al., 2019).
One of the most significant difficulties for teaching ST

is the need to apply the process in practice (Paschoal and
de Souza, 2018; Coutinho and Bezerra, 2018). At university,
sometimes, the teaching of ST is distributed in disciplines in
the SE area and does not provide an opportunity for the ST
learned in depth. This aspect causes students to graduate with
deficiencies in software testing skills (Scatalon et al., 2019).
On the other hand, the industry needs professionals with

formation and more solid training in testing. In practice, test-
ing professionals (test analysts, test engineers, or testers)
have been looking for options to improve the effectiveness
and efficiency of testing (Garousi et al., 2017) both to per-
form a more effective job and to find better positions in their
professional career.
Thus, university graduates and SE professionals self-learn

(self-train) ST through books or online resources or by partic-
ipating in industry training and obtaining certification in the
ST area (Garousi et al., 2020), such as those provided by In-
ternational Software Testing Qualifications Board (ISTQB),
for example.

2.2 Exploratory Testing
One type of testing that has become widespread in the ag-
ile environment is ET. In this method, test professionals can
interact with the system the way they want and explore, with-
out restriction, its functionality (Suranto, 2015). In layman’s
terms, it can be said that ET allows professionals to learn
quickly, adjust their tests, and, in the process, encounter soft-
ware problems that are often not anticipated in test plans or
scripts.
For Bach (2003) ET is the learning, design, and execution

of tests performed simultaneously. Thus, the test professional
adapts to the system being tested, creates, and improves the
tests based on the knowledge acquired during the exploration
of the system, without the aid of instructions about the system
(Castro, 2018).

In ET, test design and execution are performed at the same
time (Whittaker, 2009). However, we can perceive some
disadvantages in the application of this test. For instance,
the lack of preparation, structure, and guidance can lead to
many unproductive hours (Suranto, 2015). Also, we can test
the same functionality more than once while others are not
tested (Castro, 2018), especially when multiple testers or test



Coutinho et al. 2023

teams are involved. Moreover, it can be not easy to track
the progress of testing professionals (Suranto, 2015; Castro,
2018); among others.
To overcome some of these disadvantages and as a way

of meeting the need for ET management and measurement,
Bach (2003) proposed (1) to divide the testing activities into
sessions, which would be the basic unit of work, (2) to stip-
ulate a mission for each session and (3) adopt time metrics
related to testing activities, originating the SBTM strategy.
The SBTM strategy is used to make ET is more effec-

tive and with clearer goals (Castro, 2018). For these reasons,
too, ET has gained greater popularity in the agile industry
(Suranto, 2015; Raappana et al., 2016; Garousi et al., 2017),
requiring testing professionals to display a little knowledge,
experience, and skills with ET. Thus, although Garousi et al.
(2020) highlight that most courses have trained little about
ET, it also recommends more ET coverage in ST education.
Ghazi (2017) highlights that an ET session should start

with a document, called a Charter, which contains the mis-
sion described in a succinct way. The purpose is to ensure
that the tester remains focused only on executing the session
described in the Charter.
Some guidelines are indicated to define the mission, in

the Charter: (i) the mission must not be too specific, nor too
generic; (ii) the mission determines what is to be tested (not
how the test is to be carried out); (iii) at the end of the ET
session, new ideas, opportunities or problems, found by the
tester, can be used to create new missions; (iv) after comple-
tion of the mission, it is important to have an evaluation of
the session in order to discuss the results found.
For Hendrickson (2013) the mission format should be

based on the following premise: define the mission and what
should be explored. The mission of an ET, can be defined
with the estimation of Test Points.A Test Points is related
to each test job performed on the ET mission. Each mission
can contain one or several Test Points that must be investi-
gated during the time of the ET session. It is important to note
that the Test Points list is dynamic, that is, new points can be
added, based on errors found and corrections (Ghazi, 2017);
and, they must be tested according to risk (high, medium or
low), being the most at risk first.

2.3 Active Learning
As a way to streamline teaching and offer students differenti-
ated strategies that lead to effective learning, active method-
ologies emerge as an alternative proposal to traditional
teaching-learning approaches (Bonwell and Eison, 1991; Mc-
Connell, 1996).
Currently, active methodologies are being adopted in

teaching-learning from different areas of knowledge as a way
to improve current techniques and involve students in this
process (Paiva et al., 2016), not limiting their learning only
during class.
Active learning is characterized by stimulating students’

autonomy and continuous participation in the learning pro-
cess (Bonwell and Eison, 1991), through different teaching
approachessuch as Problem Based Learning (PBL), Team-
Based Learning (TBL), the Flipped Classroom, Just-in-Time
Teaching (JiTT), among others. Some other trends in ac-

tive methodologies have emerged such as Peer Instruction
(PI) (Crouch and Mazur, 2001), Design Thinking (Brown
and Katz, 2011), Storytelling (Andrews et al., 2009) and
Maker Culture (Milne et al., 2014), for example. Among
these modalities of active methodologies, PBL and JiTT
were adopted in a complementary way during the course
of Exploratory Tests (ET). As the course was conducted re-
motely, PBL contributed to initiating and motivating partic-
ipants to learn through real-life problems and encouraging
group work skills and autonomous learning (Bonwell and Ei-
son, 1991; Coutinho and Bezerra, 2018; de Andrade et al.,
2019). JiTT influenced active participation in different ac-
tivities before and during classes, encouraging participants
to read the material and perform online tasks. For these rea-
sons, these active methodology modalities were selected and
applied in this study. Next, we describe PBL and JiTT sepa-
rately.

2.3.1 Problem Based Learning - PBL

PBL is a teaching method that is characterized by the use
of problems to initiate and motivate the learning of concepts
and promote skills and attitudes necessary for their solution
(Figuerêdo et al., 2011). In addition, PBL also aims to in-
clude the acquisition of an integrated and structured knowl-
edge base around real-life problems, as well as promoting
group work skills and autonomous learning (Figuerêdo et al.,
2011; de Andrade et al., 2019; Cheiran et al., 2017), through
collaboration and ethics.
PBL is considered a methodology strongly oriented to pro-

cesses and accompanied by instruments that can assess its ef-
fectiveness (Figuerêdo et al., 2011). Therefore, the practical
immersion promoted by PBL requires a teaching plan. This
plan includes well-defined learning objectives, the structur-
ing of a practical environment, the determination of roles for
the subjects involved (teacher and student), and result evalu-
ation strategies (Figuerêdo et al., 2011; Cheiran et al., 2017).
In summary, PBL starts with the proposition of a problem

and ends with the resolution of this problem. For this, some
steps are indicated: (1) clarify terms that are difficult to under-
stand; (2) list the problem(s); (3) discuss the problem(s); (4)
summarize the discussion; (5) formulate learning objectives
based on the problems; (6) seek information; and, (7) inte-
grate the information gathered to resolve the case. To carry
out the PBL steps, the participation of a group of 10 to 12
students is indicated (with the figure of a coordinator and a
secretary), a tutor and the definition of a script, with a de-
scription of the problem and a recommended bibliography
or material support, if necessary.
PBL is suggested as a teaching-learning practice when

there is a need to encourage the participation of students or
professionals in the learning process, placing them as protag-
onists in this process, and consequently removing them from
the condition of receiver of knowledge.

2.3.2 Just-in-Time Teaching - JiTT

JiTT is a pedagogical strategy developed by Novak (2011),
whose essence is to connect activities inside and outside the
class through warm-ups (Martinez, 2018). In this approach,



Coutinho et al. 2023

students are encouraged to read material about the content of
the class and complete a small task online, a few hours before
the class takes place (Martinez, 2018). This activity allows
the teacher to plan the next class or make considerations in
class according to the student’s expectations or doubts (an-
swers).
JiTT also aims to encourage students to participate ac-

tively in different classroom activities, through greater con-
trol over their learning, motivation, and engagement (Novak,
2011). With JiTT, class time is used more effectively because
less time is spent on material that students have learned from
reading, and more time is spent on more difficult subjects
(Martinez, 2018).

In summary, the development of JiTT encompasses three
basic stages, centered on the student: (1) WarmUp exercise,
where the student is encouraged to read support materials and
answer conceptual questions - from there, the teacher pre-
pares the class; (2) class discussions on Reading Tasks (RT),
through the re-presentation of questions and (some) answers
from some students, maintaining anonymity; and, (3) group
activities involving the concepts worked in the TL and in the
class discussion, which can be expository, fixation exercises,
among others.
JiTT is indicated when you want to stimulate, in the stu-

dent or professional, the construction of prior knowledge
about the content that will be discussed in class. And also
to create the habit of studying before class. Other factors
involve oral and written communication skills, maximizing
effectiveness and class time, among other factors. JiTT is
mainly suggested for the execution of short courses or for
content taught in a short class time.

3 Methodology
This research examines the contributions of the use of active
methodologies, PBL and JiTT, used in association, to assist
in the teaching-learning process of ET, during the application
of a course conducted remotely with members of agile teams
distributed geographically. Thus, the research is classified as
an experience report (Wohlin et al. (2012)), as it precisely
describes: the planning, in Section 3.1.1; the execution, in
Section 3.1.2; and, the analysis procedures, in Section 3.1.3,
as a way to contribute with relevant considerations for the
ST teaching area, as well as to allow the replication of this
experience in other SE teaching contexts.
In order to learn more about ET, a bibliographic research

was initially conducted to understand the main approaches
and tools that have been used to support the practice of ET in
the agile environment. This study culminated in a ET course,
aimed at agile professionals, to validate the practical applica-
tion of the SBTM approach. To understand the development
phases of the experiment conducted, Figure 1 illustrates the
activities developed, from planning to evaluation.

3.1 Study Design
3.1.1 Planning

The goals of this experience were defined following the
guidelines of the Goal Question Metric (GQM) paradigm.

Thus, we seek to analyze the PBL and JiTT approach in
teaching Exploratory Tests, with the purpose of realizing
their contributions, concerning collaboration and integration
between participants of a remotely conducted course, from
the researchers’ point of view in the context of geographi-
cally distributed agile teams.
To achieve this goal and conduct this research, we defined

the following Research Question (RQ): “How to encourage
practical ET learning with geographically distributed agile
teams, seeking integration among members, and promoting
active learning, in order to encourage their insertion in the
daily work?”.
Thus, RQ aims to identify the main contributions and lim-

itations of the implementation of active learning PBL associ-
ated with JiTT in an ET course in remote format, about con-
tent learning, integration, the collaboration between partici-
pants, practical activities, and other aspects inherent in solv-
ing problems based on real scenarios. Thus, to answer this
RQ a course on ET was planned and executed (see Section
3.1.2) with an agile DSD team. And, in the end, an online
questionnaire was applied to collect the participants’ feed-
back on the adopted teaching-learning methodology.
As shown in Figure 1, the planning phase of the ET course

consisted of four well-defined steps, described below.

Step 1. Define the Course Plan. In this stage, we defined
the course syllabus, the number of hours to be taught, the date
of the course, the target audience, the objectives to achieve,
the materials needed, and classes to be produced in a de-
tailed manner according to the adopted methodology. It is
important to remark that the definition of this Course Plan
was widely discussed, reviewed, and evaluated by two spe-
cialists in the ST field. Moreover, we defined the tools to be
used in the course, considering the context of remote learn-
ing as the following: Google Meet, for video communica-
tion during classes; Discord, for communication between par-
ticipants during practical activities; Google Drive, for stor-
ing and sharing class materials and resources (in documents,
spreadsheets, and presentations); Google Forms, for the elab-
oration and availability of the evaluation questionnaire, af-
ter the course; and, the Xray Exploratory App1, for ET plan-
ning and execution, only in the last practical activity. It is
important to highlight that the contributions of the XRay Ex-
ploratory App tool were as follows: (i) it has desktop and
mobile versions; (ii) it is possible to integrate it with Jira Soft-
ware - although it was not possible to apply it in this study;
(iii) assists in bug detection, while ET sessions are recorded
in video, audio and/or screen capture format; (iv) ET sessions
are detailed and executed directly in the tool; (v) when clos-
ing the session, a report is automatically attached to the test
run - this strategy provides quick feedback to testers. In sum-
mary, the XRay Exploratory App assists in the test report and
in the documentation produced.

Step 2. Develop class materials. The classes in this course
are intended to train participants on the subject of ET in the
agile context and balance the level of knowledge among all

1Xray Exploratory App: https://www.getxray.app/
exploratory-testing-app/

https://www.getxray.app/exploratory-testing-app/
https://www.getxray.app/exploratory-testing-app/


Coutinho et al. 2023

Figure 1. Development phases of this experience.

Table 1. Relation of questions addressed in the questionnaire with the purpose of each section.

Section Goal Questionnaire Questions Question format
I Identify the profile of the pro-

fessional participating in this re-
search and their experience in the
ST area.

01, 02, 03, 04, and 05 All questions are objective.

II Identify the organizational proce-
dures and practices in relation to
the practice of ST in the Sprints
of the projects developed by the
agile teams, before the course is
offered.

06, 07, 08, 09, 10, 11, and 12 All questions are objective, ex-
cept question 12 - it was subjec-
tive. Questions 09 and 10 follow
the response format based on the
Likert scale.

III Identify the participants’ percep-
tions about the teaching-learning
obtained with the ET course.

13, 14, 15, 16, 17, 18, 19, 20, 21,
and 22

All questions follow the response
format based on the Likert scale,
except for question 22. Question
22 is objective.

IV Identify the contributions of the
PBL and JiTT approach, used
in an associated way, in the ET
course.

23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40,
and 41

All questions follow the response
format based on the Likert scale,
except for question 41. Question
41 is subjective.

agile professionals participating in the course. In this con-
text, the content covered in the class materials was based
on Bach (2003); Castro (2018); Hendrickson (2013); Whit-
taker (2009); Crispin and Gregory (2009); Ghazi (2017) and
on current lectures, conducted by renowned experts in the
field of ET. It is important to highlight that (1) lecture notes
(slides) with theoretical content and practical examples on
ET were prepared and (2) a handout with a detailed synthesis
of the content covered in the course; as well as, we selected
(3) a list of tools that support ET planning and execution, (4)
a list of videos (tutorials and lectures) available on the web,
and (5) a list of technical articles and books on ET in the ag-
ile context. The class material adopted in the ET course can
be accessed at https://bityli.com/pUEhUgFN.

Step 3. Develop practical activities. To exercise and re-
inforce learning about the content taught in each module
of the course, examples and practical activities were pre-
pared, based on the guidelines provided by the PBL and JiTT
methodologies. At this stage, the materials and resources
needed to carry out these activities were defined and elab-
orated, for example, the selection of the web system to be
tested; a guide with basic guidelines for each practical ac-

tivity; templates of the test artifacts (such as Charters, Test
Points, and Session Report) to optimize the time devoted
to each activity; requirements artifacts (such as a System
Requirements Specification Document and a Use Case Di-
agram); and, selecting an installation manual for the Xray
Exploratory App program. Some of these materials and re-
sources needed to be improved during the course, to meet the
doubts and needs of the students, diagnosed in advance (ie,
before the class) through the application of the JiTT method-
ology. It is important to highlight that it was possible to fol-
low all the stages of PBL and JiTT in full (Figuerêdo et al.,
2011; Novak, 2011), even though the course was carried out
in a remote teaching format.

Step 4. Elaborate on the evaluation questionnaire. To
collect information about the experience and learning of
the participants, a questionnaire was created online2, with
objective and subjective questions. A total of 41 (forty-
one) questions were included, distributed between 39 (thirty-
nine) multiple-choice questions and 02 (two) open questions,
whose answer was optional for the participant.

The questionnaire was designed in Google Forms and or-

2Access to the evaluation questionnaire: https://cutt.ly/Ym5vEk2

https://bityli.com/pUEhUgFN
https://cutt.ly/Ym5vEk2


Coutinho et al. 2023

Table 2. Structure of the Exploratory Testing Course

Topics Contents Practical Activity (PA) Course Workload (h)
Module I - Introduction 1.1. What is ET?

1.1.1. ET Characteristics
1.2. What is not ET?
1.2.1. Randomness and Testing
Ad Hoc
1.2.2. Scripted tests
1.3. When to use ET?

PA1 Goal: Understand the
product, create hypotheses,
and plan test scenarios.

02h

Module II - ET in practice 2.1. ET Heuristics
2.2. ET planning
2.3. Writing ET Cases: Charters
2.4. Introduction to SBTM
2.5. Running Tests Based on
Sessions
2.6. Evaluation of a session

PA2 Goal: Investigate
Heuristics, run tests, and log
failures.
PA3 Goal: Apply Task
Breakdown Structure (TBS)
metrics.

04h

Module III - A little more
about ET

3.1. Problems, Challenges, Solu-
tions
3.2. ET good practices
3.3. ET Support Tools

PA4 Goal: Practice using the
XRay Exploratory App tool
through the execution of an
ET Session.

02h

ganized into four sections. So, the first section aimed to
briefly characterize the professional profile of respondents.
The second section sought to identify the organizational pro-
cedures and practices about the ST practice in the Sprints of
the projects developed by the agile teams before the course
was offered. The third section sought to identify the respon-
dent’s perceptions about the teaching-learning obtained dur-
ing the ET course. Finally, the fourth section aimed to iden-
tify the contributions of PBL and JiTT in conducting the ET
course. Table 1 relates the questions addressed in the ques-
tionnaire with the goal of each section (see Section 3.1.1).
It is important to highlight that to answer the questionnaire,

participants should: (1) have participated in all modules of
the course; (2) have carried out the practical activities devel-
oped in each module; and, (3) right at the beginning of the
questionnaire, have agreed to a Free and Informed Consent
Form (FICF) for the research.
In Table 2 the structure of the course is presented, together

with the description of the topics and contents covered in the
syllabus, and the practical activities planned for the end of
each class module. Additionally, the workload defined for
each module of the course is informed.

3.1.2 Execution

The population of this study included twelve professionals
from the software development industry who work with ag-
ile methodologies, in the same organization. Currently, these
professionals work in geographically distributed locations,
due to the Corona Virus Disease (Coronavirus Disease) or
Sars-CoV-2 pandemic, which specifically affects the Brazil-
ian population since February 2020. For this reason, too, the
ET course was conducted in a completely remote teaching
context. It is important to highlight that 50% of the course
participants have already performed ET, even without know-
ing the definition of the practice in detail.
In general, the execution of the experience took place as

internal training with agile teams of that organization and as

planned, in four virtual meetings, with a duration of 02 hours
each meeting, on the dates of 06, 07, 12, and 13 April 2021.
It is important to highlight that Module II was divided into
two meetings, due to the extent of the content taught.
At each meeting, the content was taught and participants

were able to ask questions and resolve their doubts through-
out the class. Then, to exemplify the discussed theory, a
demonstration was made with real examples. And then, par-
ticipants were instructed to exercise the knowledge obtained
through a practical activity based on a real web system. For
this, some guidance on the activity was provided.
Participants were distributed in teams and encouraged to

interact and collaborate, through the dynamics of each activ-
ity. The resolution of a real problem also sought to encourage
participants to research, reflect and develop ET relevant to
the context analyzed in the activity. This strategy was based
on the guidelines provided by the PBL.
At the end of each meeting, class materials and resources

were made available to participants so that they had prior
knowledge of the next content to be discussed in the course.
This strategy, based on JiTT, sought to encourage interaction
between the teacher and the course participants, in addition
to enabling more in-depth discussions during the class and
anticipating feedback on the materials and resources adopted
for the next meeting.
To solve the proposed problem, the participants were mon-

itored for approximately 40 to 50 minutes. The time was stip-
ulated according to the complexity of the activity proposed in
each module. For that the activities were defined in order to
build knowledge about the execution of ET sessions and each
activity involved a practice related to the content studied in
the respective course module (see Table 2). In each activity,
a set of practices was defined that served as guidance for the
execution of the ET sessions (see the course material).
At the end of the course, participants were instructed to

fill out a questionnaire online, whose purpose was to col-
lect information about the experience and learning about ET



Coutinho et al. 2023

through PBL and JiTT practices.

3.1.3 Analysis Procedures

After data collection, through the online questionnaire, indi-
vidual reports were generated, according to the objective of
each section investigated in the questionnaire. It is worth not-
ing that the information in these reports was anonymized to
preserve the identity of the participants.
Thus, to analyze the data extracted from the content of the

responses provided by the participants, a quantitative anal-
ysis was conducted (Wohlin et al., 2012), mainly in the re-
sponses provided through the Likert Scale, with options from
1 to 5 (being: 1 - Totally Disagree; 2 - Partially Disagree; 3 -
Neither Agree nor Disagree; 4 - Partially Agree; 5 - Totally
Agree). In this sense, the answers were analyzed by class: dis-
agreement, indecision, and agreement. Additionally, a qual-
itative analysis was conducted on the answers to the subjec-
tive questions in total, alone two questions - 12 and 41), but
as they were optional or complementary answers to the ob-
jective questions, there was little need to apply this type of
analysis. Thus, when necessary, we synthesize and analyze
the responses from open, axial and selective coding oriented
in the Grounded Theory (Corbin and Strauss (2014)).

3.2 Checking the Use of ET in Practice
After the execution of the ET course, the activities performed
by the participant were monitored during their daily work
with agile development. Based on the guidelines learned in
the course, the ET sessions were planned and executed. Then,
a brainstorming session was conducted with the profession-
als to understand the real advantages and difficulties experi-
enced in this context.

3.2.1 Brainstorming Planning

Brainstorming is a technique used in groups to generate inno-
vative ideas or insights into a particular topic (Bonnardel and
Didier, 2020). Overall, brainstorming should (i) generate as
many ideas as possible, (ii) extend the interpretation of ideas,
(iii) present original ideas, and (iv) perform the combination
and improvement of existing ideas.
To conduct the brainstorming, the main question was de-

fined, that is, a problem to be solved, and a set of activities
to be followed. Thus, the following question was defined:
“What to do to be able to integrate Exploratory Tests, as a
test practice, in the team’s daily life?”. From this main ques-
tion, other specific questions were presented to guide and
contribute to the generation of ideas (see Table 3).
Regarding the set of activities followed in brainstorming,

the following were planned (see Figure 2):

1. Activity 1. Brainstorming in silence. This activity con-
sists of generating ideas, individually, to try to solve the
presented problem. Thus, participants must write their
ideas on Post-its.

2. Activity 2. Sharing Ideas. This activity consists of pre-
senting the ideas that were generated and transcribed in
the Post-its. Other participants are allowed to ask ques-
tions or add any new information or ideas.

Figure 2. Activities performed in brainstorming.

3. Activity 3. Filtering ideas. The objective of this activity
is to discard ideas that are not aligned with the context
of the problem or that generate disagreements.

4. Activity 4. First vote. In this activity, all participants
must select the ideas that best solve the exposed prob-
lem. Only the 6 most-voted ideas are listed for the next
activity.

5. Activity 5. Improvement of Ideas. The objective of this
activity is to improve the most voted ideas, adding im-
portant new information, through more post-its, with de-
tails of artifacts, testing activities, documentation, plat-
forms or TE tools, and team organization, among others.

6. Activity 6. Second vote. Finally, the participants vote for
the second time on the most applicable idea to solve the
presented problem.

3.2.2 Execution of Brainstorming

After the execution of the TE course, the participants were
led to apply TE sessions in projects they develop. In total,
nine TE sessions were held, divided into two specific mo-
ments of the project. Then the brainstorming took place.
The brainstorming took place in a completely remote con-

text, with the participants distributed locally. For this reason,
the online tool Lucidspark 3- was adopted to facilitate the
transcription of ideas and collaboration between physically
distant participants.
After some initial orientations, the participants were led to

the brainstorming activities. In total, the brainstorming lasted
a period of seventy-nine (79) minutes. It is important to note
that the brainstorming lasted longer than anticipated in the
initial planning. Figure 3 illustrates the execution of post-its
brainstorming, presented by the Lucidspark online tool. The
results are discussed in Section 4.2.

4 Results
After the experience was carried out, data were collected and
analyzed. In total, the information provided by the twelve-
course participants was considered, as they all agreed to par-
ticipate in this study, followed the discussions during classes,
and performed all the practical activities provided for at the
end of each module. Thus, these results are discussed in Sec-
tion 4.1. Then, the results of the Brainstorming carried out
with the participants, after ET insertion with agile teams, in
Section 4.2.

3Lucidspark: https://lucidspark.com/pt

https://lucidspark.com/pt


Coutinho et al. 2023

Specific questions
1. How did the Session-based Testing strategy get in the way of ET execution?
2. Does something prevent ET from being routinely practiced by the team? What actually prevents it? (process, tool, team,
project, time, etc.)
3. What can we do to improve the execution of ET?
4. Which requirements strategy or artifact is most useful to assist in the realization of ET?
5. What can be done to make these requirements clearer to the team?
6. What information is important to record/plan before performing the ET, in addition to what was indicated?
7. What information is important to record during the execution of the TE, in addition to those indicated?
8. What information is important to record after performing the ET, in addition to what was indicated?
9. Which practices were most interesting?
10. What practices did you not find interesting?
11. What benefits for the team’s day-to-day activities were observed in the course?
12. In light of what was learned, what was the most difficult thing to implement on a day-to-day basis?
13. Has anything changed in the team’s testing practice after the course? What has changed?
14. What do you see that would change in test practice after the course?
15. Did the ET course influence the incorporation of testing practices? What really influenced you?
16. Is ET useful as a testing practice in the context of remote work? What could be incorporated to contribute to remote
work?

Table 3. Brainstorming specific questions.

Figure 3. Execution of Brainstorming.

4.1 Results of Experiment

Next, sections 4.1.1 to 4.1.4 present the characterization
of the participants, the most common Agile ST practices
adopted by the participants before the course, the perception
of ET after the course, and the main contributions of PBL
and JiTT to the teaching-learning ET.

4.1.1 Characterization of participants

Initially, to characterize the participants’ professional pro-
files, an analysis was made regarding each team member’s
attributions and professional experience.
Considering that the composition of agile teams is multi-

disciplinary, that is, each team member can perform different
functions during the developed software project, among the
participants in this study, we identified different attributions
distributed among the team members (see Table 4), among
them are Developer Back-end and Developer Front-end,
played by 50% of participants; Software Engineer, 41.7%;
Project Manager, 25%; Database Administrator and Tester
or Quality Analyst, 16.7% each; Software Architect, Scrum
Master, Designer, Mobile Developer, and Infrastructure En-
gineer, with 8.3% each. Other attributions such as Analyst

or Business Leader, Analyst or Requirements Engineer, and
Product Owner (PO), among others, were not informed.

Table 4. Assignments of participants in agile teams.

Assignments Answers
(Nº)

Answers
(%)

Database Administrator 2 16.7%
Architect 1 8.30%
Back-end Developer 6 50%
Front-end Developer 6 50%
Designer or Human-
Computer Interaction
Specialist

1 8.30%

Project or Product Manager 3 25%
Scrum Master 1 8.30%
Software Engineer 5 41.70%
Quality Tester or Analyst 2 16.70%
Mobile Developer 1 8.30%
Infrastructure Engineer 1 8.30%

Regarding the level of academic education of the partici-
pants, 58.3% have completed graduation, while 33.3% have
a stricto sensu post-graduation at the master’s level, and only
8.3% are still attending graduation.
Another factor observed was the professional experience

of the participants:

1. Working experience in the industry software: 50% of
them work in this context between 1 and 2 years; 16.7%,
between 3 and 5 years; 25%, between 6 and 10 years;
and, 8.3%, for more than 11 years. None of them re-
ported little experience with software development, that
is, less than 1 year of experience in the market.

2. Working time with agile methodologies: 50% work in
this context between 1 and 2 years; 25%, between 3
and 5 years; 16.7%, between 6 and 10 years; and, 8.3%,
for more than 11 years. None of them reported little ex-



Coutinho et al. 2023

perience (less than 1 year) or no experience with agile
methodologies.

3. Working time with agile ST: 50% perform tests be-
tween 1 and 2 years; 16.7%, between 3 and 5 years;
and, 16.7%, between 6 and 10 years. However, another
16.7% reported not working with testing at all.

4.1.2 Common practices in Agile ST

Additionally, to identify how tests are commonly conducted
by agile teams, an analysis of the main ST organizational
practices performed in the Sprints of the projects was carried
out.
Generally those responsible for testing the software or

the software module developed are the Back-end Devel-
oper (41.70%), the Front-end Developer (25 %), the Prod-
uct Owner (PO) (16.70%), the Project or Product Manager
(41.70%), Scrum Master (8.30%), the Engineer Software
(16.70%), the Tester or Quality Analyst (25%) and, in some
cases, everyone on the team (41.70%).
We also identified that tests are usually performed through-

out the software lifecycle (41.7%). In some phases with more
emphasis such as, during (33.3%) or after coding the soft-
ware (58.30%); and, during (25%) or after the software inte-
gration phase (25%). In other phases the test takes place with
less intensity such as, during (8.3%) or after the software ver-
ification phase (16.70%); during (8.3%) or after the produc-
tion of software documentation (8.30%); or, during (25%) or
after the software maintenance phase (16.70%).
In Agile ST the test types are categorized in Quad-

rants Crispin and Gregory (2009). Considering this cate-
gorization, we notice that the tests performed most fre-
quently by the participants are Unit Tests (50%), Exploratory
Tests (50%), Component/Integration Tests (41.7%), Func-
tional Tests (41.7%), Usability Tests (41.7%), Performance
and Load Tests (41.7%), Simulations (33.3%), Scenarios
(16.7%), User Acceptance Tests (16.7%), Alpha/Beta ( 8.3%)
and Examples (8.3%).
To assess the participants’ perception of the types of tests

performed on their teams, with regard to their professional
activities, we expose the following questions:

• Question 09: “I believe that the software testing strate-
gies adopted so far, and reported above, have been suf-
ficient to detect bugs in the system”.

• Question 10: “I believe we need to extend and improve
the software testing practices used so far to try to ensure
higher quality in whatever product we develop”.

These statements contained multiple-choice items, accord-
ing to the Likert scale, which is detailed in Section 3.1.3.
Table 5 presents the result of the answers to Questions 09
and 10, and highlights the choices of the Likert scale as fol-
lows: (1) Totally Disagree, (2) Partially Disagree, (3) Neither
Agree nor disagree, (4) Partially Agree and (5) Completely
Agree.

Regarding question 09, it was possible to observe a pre-
dominance of responses following agreement, which may
symbolize that the participants consider the ST strategies
used to detect system bugs to be sufficient. Although, in ques-
tion 10, they unanimously agree that the ST practices, un-

Table 5. Results of responses to Questions 9 and 10. ((1) Totally
Disagree, (2) Partially Disagree, (3) Neither Agree nor disagree, (4)
Partially Agree and (5) Completely Agree

)
(1) (2) (3) (4) (5)

Question 09 8.3% 25% 8.3% 50% 8.3%
Question 10 0% 0% 0% 8.3% 91.7%

til then adopted by the teams, need to be improved and ex-
panded. These results indicate that although the participants
consider the agile testing practices adopted by the team to be
sufficient, they also perceive the need to add other ST prac-
tices to try to ensure greater quality in the developed projects.
To understand the main problems related to the execution

of tests in the projects developed by the participants in the
daily work of their teams, we also expose assertions as re-
sponse options, with multiple-choice items, according to the
Likert scale. These assertions can be consulted in the Eval-
uation Questionnaire (see Section 3.1.1) and are described
below. The results of the answers obtained can be seen in
Table 6.

Thus, the assertions ‘a’ to ‘s’, presented below, belong
to question 11 of the questionnaire and comprise the partici-
pants’ perception of the problems encountered in the practice
of ST. These assertions were represented by letters of the al-
phabet so as not to be confused with individual questions or
statements in the questionnaire.

• Assertive a: “A weak relationship between the client
and the project leader”.

• Assertive b: “A weak relationship between the leader
and other team members”.

• Assertive c: “Constantly changing objectives, business
process and/or requirements during Sprint”.

• Assertive d: “Lack of collaboration between test ana-
lysts and developers (programmers)”.

• Assertiva e: “Failure to communicate within the devel-
opment team (programmers) of the project”.

• Assertive f: “Software requirements are purposely ex-
pressed in general terms, omitting specific implementa-
tion details”.

• Assertive g: “Hidden, incomplete or inconsistent re-
quirements”.

• Assertive h: “Sprints too short”.
• Assertive i: “Lack of knowledge about software testing
practices and techniques”.

• Assertive j: “Lack of training on specific software test-
ing practices and techniques”.

• Assertive k: “There is no time to test as it should”.
• Assertive l: “There is no specific professional to run the
tests within the team”.

• Assertive m: “Trainings are time consuming and tiring”.
• Assertive n: “There is too much effort to plan/design the
tests”.

• Assertive o: “There is an effort to run the tests”.
• Assertive p: “Finding defects during production causes
rework, delaying the completion of the Sprint”.

• Assertive q: “Use of traditional testing practices in the
agile environment does not favor the work developed
during the Sprint”.



Coutinho et al. 2023

• Assertive r: “A programmer tests their own code or the
development team tests their own project”.

• Assertive s: “Test cases are written only for valid and
expected inputs”.

The assertions that were most in the agreement were those
referring to the constant change in objectives, business pro-
cess, and/or requirements during the Sprint (Assertive c);
the existence of hidden, incomplete, or inconsistent require-
ments (Assertion g); lack of knowledge about software test-
ing practices and techniques (Assertive i); lack of training in
specific software testing practices and techniques (Assertive
j); insufficient time to perform the tests as it should (As-
sertive k); inexistence of a specific professional, on the team,
to perform the tests (Assertive l); and, the effort to perform
the tests (Assertive o). Most of the problems highlighted are
typical results of teams that work in an agile context Alliance
(2016) and can explain, for example, the need pointed out by
the participants to expand and improve the adopted ST prac-
tices.

Table 6. Results of responses on problems in ST practice. ((1) To-
tally Disagree, (2) Partially Disagree, (3) Neither Agree nor dis-
agree, (4) Partially Agree and (5) Completely Agree

)
(1) (2) (3) (4) (5)

Assertive a 50% 25% 16.7% 0% 8.3%
Assertive b 58.3% 16.7% 8.3% 8.3% 8.3%
Assertive c 16.7% 8.3% 8.3% 50% 16.7%
Assertive d 58.3% 0% 16.7% 16.7% 8.3%
Assertive e 58.3% 16.7% 0% 16.7% 8.3%
Assertive f 25% 16.7% 16.7% 25% 16.7%
Assertive g 33.3% 8.3% 0% 33.3% 25%
Assertive h 33.3% 25% 16.7% 16.7% 8.3%
Assertive i 0% 16.7% 25% 25% 33.3%
Assertive j 0% 8.3% 25% 33.3% 33.3%
Assertive k 0% 25% 8.3% 41.7% 25%
Assertive l 0% 0% 8.3% 25% 66.7%
Assertive m 25% 25% 16.7% 16.7% 16.7%
Assertive n 25% 16.7% 25% 8.3% 25%
Assertive o 25% 8.3% 16.7% 16.7% 33.3%
Assertive p 50% 8.3% 8.3% 25% 8.3%
Assertive q 33.3% 25% 16.7% 16.7% 8.3%
Assertive r 25% 8.3% 33.3% 16.7% 16.7%
Assertive s 25% 16.7% 33.3% 16.7% 8.3%

Some participants reported how the ST process occurs on
your team. Figure 4 highlights some of the reports made.

4.1.3 Perception of ET after the course

We also investigate the learning gained by participants during
the course by analyzing the information collected on some
key topics in the ET content covered.
For this, we exposed the following questions to the partic-

ipants and asked that the answers be assigned according to
the multiple-choice options, according to the Likert scale.

• Question 13. “I have come to understand the importance
of using Heuristics in Exploratory Testing”.

Figure 4. ST process executed in team projects agile described by the par-
ticipants Question 12).

• Question 14. “I was able to understand that a list of
heuristics to be adopted in Exploratory Tests helps in
deciding how to test the functionality/module/system”.

• Question 15. “I have come to understand the usefulness
and importance of Test Letters in Exploratory Tests”.

• Question 16. “I was able to realize that although it is not
necessary to prepare a detailed Test Plan, simple plan-
ning helps with the execution of the Exploratory Test”.

• Question 17. “I managed to learn how to plan the Ex-
ploratory Test”.

• Question 18. “I was able to see that requirements arti-
facts, even if not very detailed, can contribute signifi-
cantly to the planning (Setup) of the Session”.

• Question 19. “I was able to see those test artifacts (Mis-
sion Letter, Test Point, and Session Report) generated
while conducting the SBTM were useful for the execu-
tion of the Exploratory Test”.

• Question 20. “From the explanation about SBTM I was
able to apply this approach with ease in the Practice Ac-
tivity”.

• Question 21. “I was able to understand the importance
of the Alignment Meeting between the team to register
possible failures, create possible formal test cases, cre-
ate new missions, register possible requirements, and
register new test points”.

The results associated with questions 13 to 21 demonstrate
a predominant agreement on the learning of all content and
practices taught during the course. Among the questions pre-
sented in this evaluation criterion, the following stood out
with more emphasis: the importance of simple planning for
the execution of the ET (Question 16); that requirements ar-
tifacts, even less detailed, can contribute to Session Setup
(Question 18); the importance of the Alignment Meeting as
a strategy to register possible failures, create possible for-
mal test cases, create new missions, register possible require-
ments, and register new test points (Question 21); the use-
fulness and importance of simple ET artifacts generated in
conducting the SBTM are useful for the execution of (Ques-
tion 19); the relevance of defining Heuristics in ET (Question
13); among other relevant questions presented in the Table 7.

The performance of practical activities provided partici-
pants with a real experience with challenges common to ET,
such as little domain knowledge and necessary qualities of



Coutinho et al. 2023

Table 7. Results of answers from Questions 13 to 21. ((1) Totally
Disagree, (2) Partially Disagree, (3) Neither Agree nor disagree, (4)
Partially Agree and (5) Completely Agree

)
(1) (2) (3) (4) (5)

Question 13 0% 0% 8.3% 33.3% 58.3%
Question 14 0% 0% 25% 25% 50%
Question 15 0% 0% 16.7% 33.3% 50%
Question 16 0% 0% 0% 25% 75%
Question 17 0% 8.3% 8.3% 50% 33.3%
Question 18 0% 8.3% 8.3% 16.7% 66.7%
Question 19 0% 0% 25% 16.7% 58.3%
Question 20 8.3% 0% 25% 50% 16.7%
Question 21 0% 8.3% 0% 25% 66.7%

testers in the application of ET (91.7%) makes it difficult to
carry out the tests; the absence of an ET plan results in the
same functionality being tested several times or an impor-
tant functionality may not be tested or a serious error may
go undetected (91.7%); the lack of a definition of test cases
makes it difficult to reproduce the tests performed, if neces-
sary, such as in regression tests (75%); an incorrectly inter-
preted output can lead to defects that may remain in the sys-
tem or be eventually detected in future tests (5%); as there
is no detailed test guide or plan, and no more complete arti-
facts are produced other than the crash report, it is difficult
to know what has been and has not been tested (50%); and,
ET is not suitable for real-time systems (8.3%).

4.1.4 Contributions from the PBL and JiTT approach

To identify the contributions of PBL and JiTT in the teaching-
learning process applied in the ET course, an analysis of
the characteristics of these methodologies was carried out.
In this perspective, a set of eighteen questions (23 to 40)
were exposed to the participants to be analyzed and answered
through multiple-choice options, also following the Likert
scale. Questions are listed below:

• Question 23. “The scenario (web system) worked on in
the practical activities represented a real scenario of soft-
ware development.”

• Question 24. “The scenario (web system) worked on in
the practical activities had a high level of complexity.”

• Question 25. “Practicing the theoretical content with a
real web system helped me to better understand the con-
cepts of Exploratory Testing.”

• Question 26. “Through practical activities with a real
web system, the course made it possible to learn, au-
tonomously and independently, the main methods and
techniques of Exploratory Testing.”

• Question 27. “Through practical activities with a real
web system, the course made it possible to work collab-
oratively in groups in order to broaden the discussions
in the team about the theory learned.”

• Question 28. “Through practical activities with a real
web system, the course made it possible to work collab-
oratively in groups in order to deliver the project activ-
ities on time.”

• Question 29. “Through practical activities with a real
web system, the course made it possible to work collab-

oratively in groups in order to deliver the project activ-
ities with quality.”

• Question 30. “Although physically separated, interact-
ing with the team during practical activities was not dif-
ficult.”

• Question 31. “The use of conversation tools (such as
Discord) and collaboration (such as Google Sheets,
Google Drive, and Google Docs) contributed to the
team’s interaction in practical activities, decreasing the
physical distance.”

• Question 32. “I realized that giving my opinion (feed-
back) about the class regarding the approach adopted,
the exposed content, or the supporting material used
(slides, pdf, artifacts, videos, etc. ), contributed to the
organization of the class and the instructor’s practice in
the next class.”

• Question 33. “I realized that giving my opinion (feed-
back) about the class regarding the approach adopted,
the exposed content, or the supporting material used
(slides, pdf, artifacts, videos, etc. ), helped the instructor
to focus on the main difficulties that were expressed by
the participants.”

• Question 34. “I realized that giving my opinion (feed-
back) about the class regarding the approach adopted,
the exposed content, or the supporting material used
(slides, pdf, artifacts, videos, etc. ), maximized effi-
ciency and class time.”

• Question 35. “I realized that the practical activities were
also aimed at stimulating my oral and written commu-
nication, through discussions with the team and elabo-
ration of the test artifacts.”

• Question 36. “I realized that the practical activities were
also aimed at stimulating group work skills, such as dis-
tributing the roles of each member, setting goals, under-
standing objectives, providing collaboration and com-
munication, among other aspects .”

• Question 37. “I collaborated more with my team in prac-
tical activity 2 (investigating heuristics) and 3 (apply-
ing TBS metrics) than in practical activity 1 (creating
hypotheses and planning test scenarios) because I felt
more secure about the web system I was exploring, only
in these activities, as I didn’t know the business scenario
well before.”

• Question 38. “I felt more secure in carrying out the prac-
tical activities, only after the course instructor provided
more specific guidance on the task, as the guidance in
the support material (slides) was not clear enough .”

• Question 39. “I felt more motivated in practical activity
2 (investigating heuristics) and 3 (applying TBS met-
rics) after the course instructor made the test artifact
templates available.”

• Question 40. “I had problems collaborating on practical
activities because I couldn’t understand them.”

To more accurately classify the answers given, we grouped
the questions correlated to the main practices of active
methodologies, in general - perceived in Questions 23 to 25;
PBL, in Questions 26 to 29, and 36; and, JiTT, in Questions
32 to 36. We highlight that Questions 36 to 40, characterized
both practices common to PBL and JiTT. Questions 30 and



Coutinho et al. 2023

31 sought to understand the participants’ perception of the dy-
namics of the course in the remote setting. Table 8 displays
the answers given to the questions.

Table 8. Result of responses from Questions 23 to 40.

(1) (2) (3) (4) (5)
Question 23 0% 0% 8.3% 16.7% 75%
Question 24 25% 25% 33.3% 8.3% 8.3%
Question 25 0% 8.3% 0% 25% 66.7%
Question 26 8.3% 16.7% 0% 33.3% 41.7%
Question 27 0% 0% 0% 25% 75%
Question 28 8.3% 16.7% 16.7% 25% 33.3%
Question 29 0% 0% 16.70% 16.7% 66.7%
Question 30 0% 0% 8.3% 25% 66.7%
Question 31 0% 0% 0% 8.3% 91.2%
Question 32 0% 0% 0% 16.7% 83.3%
Question 33 0% 0% 8.3% 25% 66.7%
Question 34 0% 0% 33.3% 16.7% 50%
Question 35 0% 8.3% 8.3% 8.3% 75%
Question 36 0% 0% 8.3% 16.7% 75%
Question 37 0% 16.7% 8.3% 33.3% 41.7%
Question 38 0% 8.3% 16.7% 25% 50%
Question 39 0% 0% 8.3% 33.3% 41.7%
Question 40 25% 33.3% 0% 25% 16.7%

About general practices guided by active methodologies,
we investigated the participants’ perception of the inclusion
of real practical examples in the activities carried out in the
course. It was possible to observe a predominance in the
follow-up of agreement in the answers to questions 23 and
25, which refer respectively to “the scenario (web system)
worked on in practical activities represented a real scenario
of software development” and “practicing the content theory
with a real web system helped to better understand the con-
cepts of ET”. However, in question 24 we identified a major-
ity of disagreement with the “high level of complexity of the
scenario worked on in practical activities”.
The answers provided in questions 26 to 29 and 36, related

to PBL practices, which deal with the inclusion of real prob-
lems as practical activities in the teaching of content, pro-
vide evidence of the efficiency of using this methodology,
especially about: learning autonomously and independently
about ET (Question 26); collaborative group work to expand
team discussions on the theory learned (Question 27), de-
liver project activities on time (Question 28) and with quality
(Question 29); encouragement of group work skills, such as
distributing roles for each member, setting goals, understand-
ing objectives, providing collaboration and communication,
among other aspects (Question 36).
About statements 32 to 36, which deal with information

about practices common to JiTT, the results provided by
the participants express a majority of agreement on the con-
tributions and feedback given to the class from the access
of the contents and materials of the class. Thus, from the
point of view of the participants, this strategy: contributed
to the organization of the class and the instructor’s practice
in the next class (Question 32); helped the instructor to fo-
cus on the main difficulties that were expressed by the par-
ticipants (Question 33); and, maximized efficiency and class
time (Question 34). Another JiTT characteristic observed

in the questions and which showed a predominance of the
agreement was related to the objective of practical activities,
namely: the encouragement of oral and written communica-
tion, through discussions with the team and preparation of
test artifacts (Question 35) and the encouragement of group
work skills (Question 36).

Otherwise, we investigated how collaboration between the
team in practical activities stimulated the participants’ learn-
ing. Agreement in questions 37, 38, and 39 prevailed, which
referred, respectively, to security in collaborating more in the
final practical activities than in the initial ones, as it is already
better adapted to the business scenario provided as a real ex-
ample in the activity; security in performing the activities af-
ter more specific instructions from the instructor; and, moti-
vation after the instructor provides templates for the test arti-
facts. A positive aspect was the predominant disagreement in
statement 40. A large part of the participants disagreed that
they had “problems in collaborating in practical activities be-
cause they could not understand them”. This result can be
explained by the aspects already confirmed in statements 37
to 39.
Finally, the benefits and difficulties of participating in the

theoretical and practical activities of the course were investi-
gated, given its implementation in a completely remote teach-
ing context. Table 9 presents the main testimonies of the par-
ticipants regarding the perceived benefits and difficulties.
According to the statements reported in the responses, we

identified that the content, the main approaches, and the ET
tools were not known by some of the participants, as well as
the usefulness of this test in agile methodologies. These as-
pects were pointed out as a benefit of the course for the work
developed by the teams. Regarding the reported difficulties,
we found that the practices could have been conducted with
products developed by the teams themselves as a way to fa-
cilitate the understanding of the business scenario, and the
course load was also considered short for the extension of
the content and developed practices.

4.2 Results of ET insertion with agile teams

The six activities planned for brainstorming were conducted.
There was no maximum or minimum limit of ideas to be ex-
pressed by the participants. Thus, participants were encour-
aged to present their ideas within the time interval defined
for each activity (see Section 3.2.1).
In Activity 1, 37 answers were obtained to the specific

questions listed in the preparation of the Brainstorming.
Then, in Activity 2 some questions were asked in order to
clarify doubts related to the exposed ideas. In Activity 3, some
ideas had to be grouped together, and others discarded. Thus,
a total of 29 ideas that were not aligned with the main Brain-
storming context were disregarded. It is important to high-
light that not all the answers obtained were considered viable
ideas to be applied, as some were repeated, complemented
each other, or were outside the research context. In Activity
4, 06 ideas were voted on to be included in the next phase. In
Activity 5, the most voted ideas were briefly discussed and
improved, with the aim of favoring the vote to be carried out
in the next activity. Finally, Activity 6 resulted in a single



Coutinho et al. 2023

Table 9. Benefits and Difficulties of participating in an ET course in remote format.

Benefits
Participant B: “The ET scope and planning to deliver quality software.”
Participant H: “Learn how to document the execution of exploratory tests.”
Participant I: “Learning about the topic. Although the team, which also works together on development, somewhat adopted
what was proposed in the classes, it was clear how we could improve.”
Participant J: “The theoretical content and practical activities were of great importance for a more solid understanding of
the ET... The requirements document should also be detailed enough to enable the planning of the ET by the responsible
team ... In addition, the ET technique presented in the training can contribute a lot to the quality of the developed artifacts.”
Participant K: “Through the course, I had my first contact with ET, and participating in it made me learn a lot... What was
seen seemed very attractive for the context of agile methodology.”
Participant N: “I found the use of heuristics in the tests interesting, I didn’t know about it.”
Difficulties
Participant B: “Not having any type of ST course in my graduation.”
Participant F: “The guidelines of the support material, sometimes it was not clear what should be done, generating doubt
in the group.”
Participant H: “Differentiate what information should be placed in each field of the template provided during ET plan-
ning.”
Participant L: “It was a little difficult to think of scenarios for an initially unknown system. I think it would be more
beneficial if the practice of the course had used a system developed and well-known by the team/class.”
Participant O: “I had difficulty participating due to the course schedule, as it wasn’t my actual work schedule.”

viable idea to implement: the definition of a more viable ap-
proach to be implemented in the daily life of the team.
The ideas presented in the Brainstorming were related to

the implementation of ET in the participants’ daily lives,
from the application of the ET course. Thus, the exposition
of some ideas was crucial to understanding the effectiveness
and usefulness of the practices exercised and the artifacts gen-
erated. We categorize these ideas to explain what actually ap-
plies and what does not apply to the daily lives of agile teams.
The following is a summary of the ideas exposed in the brain-
storming that favored the incorporation of ET in the daily life
of agile teams:

• The registration of Test Points and the Test Report were
considered important for the planning and execution of
the ET.

• Regarding the benefits for the team’s day-to-day activi-
ties, the importance of recording the ET made was high-
lighted, in order to highlight the points that were tested
and their pending issues.

• The registration of TestPoints was seen as a significant
contribution of the TE Sessions because, in practice,
there was an improvement in the activity of registration
of the test to be done.

• Considering the context of distributed work, adopting
files or artifacts with permission for simultaneous col-
laboration, and online tools contributed to the TE per-
formed remotely.

Below is a summary of the ideas exposed in the brainstorm-
ing that stood out as limiting factors to the incorporation of
ET in the daily life of agile teams:

• The minimum and maximum time limit for executing an
ET Session does not apply in practice, as this factor is
relative to the tested functionality. As well as recording
the complete execution of an ET session.

• The unavailability of time to plan ET sessions was de-
fined as a difficulty in implementing TE in practice.

• Having a well-defined TE process could contribute to
the insertion of TE as a test practice in the team’s daily
life.

• The minimum time reserved for an ET Session, in
SBTM, could be smaller for small features.

• The lack of experience with the ET makes the profes-
sional who performs the test, dedicate a significant ef-
fort to the preparation of the TE Session as a whole.

• The absence of a well-defined process that optimizes the
preparation time of an ET Session limits the insertion of
the ET in the team, as well as the organization and com-
mitment of the entire team to find a common moment
to perform the ET.

In a complementary way, some other ideas emerged to fa-
cilitate the insertion of ET in the daily lives of teams, such
as:

• Include a description and examples in the attributes of
each artifact of the ET Session, to facilitate the under-
standing of the artifact, or to exemplify the description
of a Test Point.

• Set the Test Points in advance.
• A document or a Use Case model could contribute as
an artifact of base requirements for the planning of ET
Sessions.

• Providing a brief description of the functionality to be
tested in the ET Session could facilitate the planning
and execution of the test.

• Capturing the print screen or screen recordings that con-
tain the functionality defects identified in the ET Ses-
sion contributes to the record of the test performed.

• After running the test, it would be interesting to record
the test execution step by step, the scenario tested
and any impediments or difficulties encountered by the
tester.



Coutinho et al. 2023

5 Discussion of Results
In this section, we discuss the results obtained and presented
in Section 4, in order to expand the considerations about the
ET course applied with agile professionals 5.1 and the mon-
itoring of the incorporation of ET in the daily work of these
professionals 5.2.

5.1 Overview on using PBL and JiTT
To promote active learning and integration among geographi-
cally distributed participants during an ET course in a remote
learning format, PBL and JiTT approaches have proven to be
useful in stimulating hands-on learning in this context. Ac-
cording to the agreement of information and reports from the
participants, some characteristics of PBL and JiTT stood out,
such as:

1. The use of a real scenario of software development con-
tributed to the practice of the ET concepts covered in the
course. The actual scenario practiced encouraged the
participants to further investigate the possible failures
of the analyzed system from the Heuristics and planned
ET scenarios. This strategy allowed the quick identifi-
cation of some bugs implemented in the system’s func-
tionalities - in total, there were 6 bugs in three different
missions (test scenarios) tested in practice 2, and 4 bugs
in three distinct missions tested in practice 3. We high-
light that each mission was executed in a 30-minute ses-
sion. The low level of complexity of the adopted web
system also contributed to the understanding of its op-
eration, since there were - in the first practical activities
- more detailed requirements or business artifacts.

2. Autonomous learning was stimulated through practical
activities, simulating the participants’ daily situations
through the exploration of the web system; studying or
reading classroom material and resources in advance;
the discussion of content and activities during classes;
the elaboration of questions about the understanding of
practical activities; and, the construction of the gener-
ated ET artifacts.

3. Collaborative work stimulated different learning styles
among the participants, such as: distributing the roles of
each member, setting goals, understanding objectives,
and providing communication. It also contributed to the
expansion of discussions in the team, with the different
points of view of the participants, and with the deliv-
ery of the activity on time - although, in some cases,
additional time to complete the activity was necessary
- and with quality - meeting the requirements of the ac-
tivity. The use of online conversation and collaboration
tools contributed to the team’s interaction, narrowing
the physical distance.

4. Stimulation of additional skills, such as reading materi-
als, using logical reasoning to understand the features of
the web system during practical activities, discussions
between teams, and exploring the system, among others.
Teamwork was also an encouraging practice, although
the participants already act in this way in their daily
work.

5. The motivation. Some clarifications about technical
terms, expressions, or ET artifacts were useful to keep
the participants motivated in carrying out the practical
activities. As well as the availability of ET artifact tem-
plates and the socialization of the generated artifacts at
the end of each practical activity.

6. Feedback provided before, during, and after classes
about the content, materials, resources, and methodol-
ogy used, contributed to the organization and practice of
the instructor in the following class; help the instructor
focus on the main difficulties that were expressed by the
participants; and, maximize the effectiveness and time
of the class.

7. The examples shown, as well as the way to present them,
contributed to improving the understanding of TE, as all
the examples also referred to contexts of real systems.
Exemplifying the theory in this way helped the partici-
pants in the understanding and applicability of ET, es-
pecially during the performance of practical activities.

It is important to highlight that the Likert Scale helped
to identify both the benefits and limitations of the PBL and
JiTT approaches, through assertions that represent their main
characteristics. However, the answers provided by the par-
ticipants pointed to these (Question) characteristics more as
benefits than as limitations to the use of these approaches in
remote learning. Although guidance and some clarifications
were provided during the practical activities, some partici-
pants agreed on the difficulty in collaborating in practical ac-
tivities because they were unable to understand them well.
Perhaps this is justified by the absence of face-to-face con-
tact to facilitate communication.

In summary, it is also important to highlight that: (i) al-
though during the course, the participants were geographi-
cally dispersed, we did not address any specific DSD pro-
cess. The purpose of the course was to use strategies and
tools that would make ET viable in the context of isolated
and remote work; (ii) the use of a specific tool (Xray Ex-
ploratory App) for planning, executing and reporting bugs
(with video recording, capturing and annotating screenshots,
annotations, among other aspects) in ET sessions are bene-
fits not found in other tools that aid the execution of ET; (iii)
the experience of remote learning with geographically dis-
tributed participants is challenging and factors such as stimu-
lating participation, collaboration and attention skills need to
be considered for learning to actually happen; (iv) we noticed:
the interest and engagement of the participants, when practic-
ing the theoretical content through a real problem adopted in
the activities and discussions during the classes, mainly due
to the pre-constructed knowledge through prior access to the
materials of the classroom; the quality of the answers in the
exercises, as a good part of the test artifacts generated were
in accordance with the criteria suggested in the description of
the activities; a motivation to use the Xray Exploratory App
tool, due to the ease in creating, executing and exploring the
ET performed; among other aspects already discussed in this
Section.



Coutinho et al. 2023

5.2 Overview of Insights in Practice
The ET course facilitated the participants’ understanding of
ET concepts and practices. In order to facilitate the incor-
poration of this test practice in the daily lives of the teams,
the participants were motivated to apply ET sessions in their
work context. Then, a brainstorming was conducted to evalu-
ate the execution of the ET. Next, we discuss the information
obtained from brainstorming from the questions listed in Ta-
ble 3.
Therefore, some factors were perceived that favored the

incorporation of ET in the daily lives of the teams, such as:

1. The planning of Test Points, and defining the degree of
importance of each Test Point facilitated the understand-
ing of what is a test priority for the moment.

2. Documenting what was tested also contributed to the
conduct of the alignment meeting with the team, con-
sidering that all the information inherent to the test exe-
cution, such as bugs, suggestions for improvements, or
pending issues, was recorded.

On the other hand, some limitations were also noticed in
the implementation of ET Sessions:

1. The artifacts adopted in the ET Sessions - Charters,
Tests Points and Test Report - proved to be complex (i.e.
difficult to understand) or with underused fields for the
test record. In this case, there is a need to implement
guidelines or examples to clarify the effective use of the
artifact.

2. There was a need for adaptations in the artifacts - of
planning and recording the test - for a more specific
adaptation to the agile context in which the ET was con-
ducted.

3. The orientation of minimum and maximum time for the
realization of the Sessions: sometimes, the minimum
time (30 minutes, indicated by the SBTM) for the ses-
sion was not used because the functionality tested was
very simple, and the session that could have been done
less time needed to be extended to reach the minimum
time. In this case, a new orientation for situations like
this needs to be considered.

4. The absence of a well-defined process or approach that
is compatible with the real work context of agile teams.
What is proposed in the literature, such as the SBTM, is
not always applicable in its entirety in the real context
of agile professionals.

5. Experience of the professional who defines the Test
Points because when this activity is performed by pro-
fessionals who are unaware of the application to be
tested, there is a risk of specifying a Test Point in an
inconsistent or incomplete, generating gaps in the ET
planning and generating difficulties in its subsequent ex-
ecution.

To actually incorporate ET as a testing practice in agile
teams, it is still necessary to define an approach that fits the
context of these professionals, in order to consider practi-
cal application guidelines, more specific tools that consider
the particularities of planning and ET execution, and simple,

clear and effective artifacts to be adopted. For this reason,
there is a need for an approach that fits the needs of profes-
sionals working in agile development and that goes beyond
the concept presented in the literature on ET.

6 Limitations and Threats to Validity
Some potential threats to the validity of this study were per-
ceived, such as threats to internal, external, construct, and
conclusion validity. For this reason, some measures were
taken to minimize them. To mitigate construct validity, the
course material and evaluation questionnaire were iteratively
planned, updated, and validated by the authors. As well as,
elaborated based on works related to the ET area, in the con-
text of Agile ST (Bach, 2003; Hendrickson, 2013; Suranto,
2015; Whittaker, 2009; Castro, 2018). To mitigate internal
validity and ensure the anonymity of responses, participant
identification was optional - via email address. This allowed
the data analysis to be performed in an impersonal way.
Other aspects inherent to the selection of individuals and con-
duct of the experiment also contributed and are detailed in
Sections 3.1.2 and 3.1.3. The external validity was attenu-
ated with the availability of resources and teaching materials
to facilitate the application of the active methodologies men-
tioned. Thus, the results can be valid for other course partic-
ipants either in a remote or face-to-face teaching format. To
mitigate the conclusion validity, only percentages were used
to identify common patterns. Complementarily, the question-
naire validation answers were also discarded, regarding pos-
sible errors, such as answer format, and textual expressions
used in the questions, among others. We tried to reduce bias
using Likert scale data. Thus, all the conclusions we draw in
this study are strictly traceable to the data.

7 Related Works
Although the literature shows interest in ST Teaching and is
seeking strategies to boost practical teaching closer to the real
context of the software industry through approaches based
on active learning (Cheiran et al. (2017); de Andrade et al.
(2019); Martinez (2018); Figuerêdo et al. (2011)) see Sec-
tion 7.1 -, there are still few studies that present results on
the teaching of ET (Costa et al. (2019); Ferreira Costa and
Oliveira (2020)) - see Section 7.2.
Some other studies also have been dedicated to the inves-

tigation of ET in the context of the industry in order to un-
derstand the impact or effectiveness of this testing practice
in real projects (Gebizli and Sozer, 2017; Mårtensson et al.,
2019; Pfahl et al., 2014; Afzal et al., 2015) - see Section 7.3.
A brief discussion of these works in relation to this study is
presented in Section 7.4.

7.1 Teaching-learning of ST using active
methodology

Cheiran et al. (2017) present an account of two experiences
on the teaching of ST using PBL in an undergraduate course
of SE at the Federal University of Pampa (UniPampa). In



Coutinho et al. 2023

total, 51 students participated - 25 in the 1st edition and 26
in the 2nd edition of the course. Data collection took place
through questionnaires. To analyze the collected data, statis-
tical and content analyses were adopted. The results point to
evidence of students’ maturity in the context of the curricular
component and the benefits and problems faced by integrat-
ing PBL and gamification elements.
de Andrade et al. (2019) also conducts a study on ST learn-

ing using PBL practices, with students from Computer Sci-
ence, at the University of São Paulo (USP), and Information
Systems, of the Federal University of Juiz de Fora (UFJF).
The results show that (i) classes with many students should
have fewer presentations; (ii) courses with an average num-
ber of students can choose to keep weekly presentations more
dynamic or with fewer presentations; (iii) the approach PBL
is not as effective for students who have less time for extra
classes. In summary, it was noticed that the successful adop-
tion of an active approach is not directly linked to the infras-
tructural aspects.
Figuerêdo et al. (2011) apply PBL to train test engineers.

For this, an empirical study was carried out with two groups,
where each group was composed of five undergraduate stu-
dents. The group should test a CASE tool to support func-
tional testing using ET. Two evaluations were made with the
participants - one before and one after the execution. Par-
ticipants’ knowledge, grades, and amount of bugs identified
were evaluated. The results obtained highlight that the PBL
provides the engagement of participants and obtaining expe-
rience in scenarios that simulate real ST situations.
Martinez (2018) describes the results of an experience

with JiTT based teaching in a graduate course in ST over two
semesters. The approach adopted was evaluated from the per-
spective of students, through a survey, and of teachers, from
an assessment of strengths and limitations. The results show
that a large majority of students (1) believe that their learn-
ing has improved when they prepare for class by reading the
material in advance and (2) consider JiTT to be an adequate
teaching strategy for the course. Teachers highlighted that
students became more involved and participatory in discus-
sions during class.

7.2 ET teaching
Costa et al. (2019) use gamification as a motivating strategy
in teaching and learning ET. This dynamic consisted of prac-
tical activity to apply ET in the form of a game, which refers
to the “Treasure Hunt”. An experience was carried out with
students from an SE discipline of an undergraduate course
in Computer Science. The results indicate that the qualita-
tive results converged with the quantitative results obtained,
showing that gamification helped in the teaching and learn-
ing process of the students (forehead pains).
In another work, Ferreira Costa and Oliveira (2020) repli-

cate a new experience with the gamification strategy for
teaching ET discussed in Costa et al. (2019), with a group of
undergraduate students in Computer Science and with grad-
uate students in a Computer Technician course. As a result,
students achieved good overall performance. Some reports
highlight that gamification facilitated and significantly con-
tributed to better performance, converging with the quantita-

tive data obtained. This can be evidenced mainly by the fact
that both “runs” of the experience (classes) reached a percent-
age higher than 70% of achievement.

7.3 ET in the context of industry
Gebizli and Sozer (2017) evaluate the impact of the educa-
tion and level of experience of testers on the effectiveness of
the ET. For this, a case study was carried out with 19 industry
professionals, with different educational backgrounds and
levels of experience. A Digital TV system was tested, and
detected failures were categorized according to their sever-
ity. Thus, the effectiveness of the ET was evaluated on two
aspects: criticality of detected faults and efficiency in terms
of the number of faults detected per unit of time. The results
show that ET efficiency is significantly affected by training
and educational experience.
Mårtensson et al. (2019) conducted a study based on inter-

views to understand the success factors in the application of
ET in industry projects. For this, interviews were conducted
with 20 professionals. Finally, a list of key factors that en-
able the efficiency and effectiveness of ET in large-scale soft-
ware systems was presented. The nine factors identified are
grouped into four themes: (i) Testers’ knowledge, experience,
and personality; (ii) Purpose and scope; (iii) Ways of work-
ing; and, (iv) Registration and reporting.
Pfahl et al. (2014) investigated how software engineers un-

derstand and apply the principles of exploratory testing, as
well as the advantages and difficulties they experience. For
this, an online survey was carried out among Estonian and
Finnish software developers and testers. The main results in-
dicate that the majority of testers, developers, and test man-
agers who use ET, (1) apply ET to critical software for us-
ability, performance, security, and security to a high degree;
(2) use ET very flexibly at all kinds of levels, activities, and
phases; (3) they perceive ET as an approach that supports
creativity during testing and is effective and efficient; and
(4) feel that ET is not easy to use and has few support tools.
In addition, there was a need for more support for ET users,
such as guidelines and tools.
Afzal et al. (2015) sought to quantify the effectiveness and

efficiency of ET vs. tests with documented test cases (TCT).
For this, four controlled experiments were carried out, with a
total of 24 professionals and 46 students. Manual functional
tests using ET and TCT were done. The number of defects
identified in the 90-minute test sessions, the difficulty of de-
tection, the severity and types of defects detected, and the
number of false defect reports was measured. The results
show that ET found a significantly higher number of defects.
However, the two testing approaches did not differ signifi-
cantly in terms of the number of false defect reports.

7.4 Discussion of related works
We could not find works that apply JITT and PBL in ET. In
general, the application of JITT or PBL in ST, as reported in
the literature (Cheiran et al., 2017; Figuerêdo et al., 2011;
Martinez, 2018), achieved results that converge with ours
in the sense that the adoption of these methodologies has
provided positive gains, related to motivation, engagement,



Coutinho et al. 2023

collaboration, and content learning. We also emphasize that
most of the works are developed in academic environments
(with undergraduate students), others in practical environ-
ments (with industry professionals). Generally, the types of
tests investigated are different, sometimes a more specific
type of test or in a more general context, such as defect de-
tection only. But, it is not always possible to identify in which
development process the work was applied or which devel-
opment methodology was adopted.
In this way, this paper differs from the others in that it

identifies and discusses the contributions of the integration
of active methodologies, PBL and JiTT, in teaching ET in a
remote learning course with agile professionals from the soft-
ware industry and distributed geographically. Some strate-
gies and guidelines seeking to optimize teaching-learning
with PBL and JiTT, as well as the discussion of some per-
ceived challenges, were also highlighted. Another differen-
tial is the monitoring of ET execution in the daily agile de-
velopment of industry professionals, in order to highlight the
aspects that favor or limit the incorporation of ET in the daily
routine of agile teams.

8 Conclusions

This work investigates the use of the PBL and JiTT method-
ologies to teach ET to a DSD team. Based on a literature re-
view and evaluation of the resources available, we planned
and performed a training course on ET and analyzed the
results obtained. While teaching ST has been challenging,
under the circumstances imposed by social distance, where
each team member is working remotely and isolated, teach-
ing such a subject becomes even more challenging. Next, we
followed the incorporation of ET into the daily lives of the
teams that participated in the course and made an analysis of
the application of this practice in the context of agile devel-
opment. Through brainstorming, ideas were raised about the
characteristics that favored or limited the execution of ET.
The use of these methodologies significantly contributed

to the success of the course. They provided the grounds for
adopting a real problem, assessing the student’s needs with
resources available before the class, adjusting the course to
meet students’ expectations and needs, and promoting collab-
oration. Additionally, the existence of a support tool for ET
was key to optimizing remote learning. These aspects also
favored the application of ET by agile teams in their projects.
So that both the practices and the artifacts were put to good
use in the test execution. However, even with this support
given in the course, some limitations were perceived such
as the absence of more specific support for the planning and
execution of the ET, such as guidelines and tools.
As future works, we intend to (1) propose an approach that

facilitates the implementation of ET, considering the DSD
scenario and the generation of simple and robust ET artifacts,
for the effective insertion of this test practice in the daily
life of agile teams. Next, we also intend to (2) validate the
approach through experiments with professionals from the
software development industry.

Acknowledgements

The authors would like to thank the anonymous reviewers for
their valuable comments. The second and third authors were
supported by CNPq/Brazil (processes CNPq 303773/2021-9, and
311215/2020-3).

References
Afzal, W., Ghazi, A. N., Itkonen, J., Torkar, R., Andrews, A.,
and Bhatti, K. (2015). An experiment on the effectiveness
and efficiency of exploratory testing. Empirical Software
Engineering, 20:844–878.

Alliance, A. (2016). Agile glossary. URL: https://www.
agilealliance.org/agile101/agile-glossary/
Accessed on August 13, 2020.

Andrews, D. H., Hull, T. D., and Donahue, J. A. (2009). Sto-
rytelling as an instructional method: Descriptions and re-
search questions. Technical report, Oak Ridge Inst for Sci-
ence and Education Tn.

Aniche, M., Hermans, F., and van Deursen, A. (2019). Prag-
matic software testing education. In Proceedings of the
50th ACM Technical Symposium on Computer Science
Education, SIGCSE ’19, page 414–420, New York, NY,
USA. ACM.

Bach, J. (2003). Exploratory testing explained. Online:
http://www. satisfice. com/articles/et-article. pdf.

Bonnardel, N. and Didier, J. (2020). Brainstorming variants
to favor creative design. Applied ergonomics, 83:102987.

Bonwell, C. C. and Eison, J. A. (1991). Active Learning:
Creating Excitement in the Classroom. 1991 ASHE-ERIC
Higher Education Reports. ERIC, sl.

Brown, T. and Katz, B. (2011). Change by design. Journal
of product innovation management, 28(3):381–383.

Castro, A. K. S. d. (2018). Testes exploratórios: Característi-
cas, problemas e soluções. B.S. thesis, Universidade Fed-
eral do Rio Grande do Norte.

Cheiran, J. F. P., de M. Rodrigues, E., de S. Carvalho, E. L.,
and da Silva, J. a. P. S. (2017). Problem-based learning
to align theory and practice in software testing teaching.
In Proceedings of the 31st Brazilian Symposium on Soft-
ware Engineering, SBES’17, page 328–337, New York,
NY, USA. ACM.

Corbin, J. and Strauss, A. (2014). Basics of qualita-
tive research: Techniques and procedures for developing
grounded theory. Sage publications.

Costa, I., Oliveira, S., Cardoso, L., Ramos, A., and Sousa,
R. (2019). Uma gamificação para ensino e aprendizagem
de teste exploratório de software: Aplicação em um es-
tudo experimental. XVIII Simpósio Brasileiro de Jogos e
Entretenimento Digital (Education Track–Short Papers),
2019(1):1232–1235.

Coutinho, E. F. and Bezerra, C. I. M. (2018). Uma avali-
ação inicial do jogo para o ensino de testes de software
itestleaening sob a ótica de um software educativo. In Con-
gresso sobre Tecnologias na Educação, volume 3, pages
11–22, Fortaleza,CE. SBC Open Library.

Coutinho, J., Andrade, W., and Machado, P. (2021). Teach-

https://www.agilealliance.org/agile101/agile-glossary/
https://www.agilealliance.org/agile101/agile-glossary/


Coutinho et al. 2023

ing exploratory tests through pbl and jitt: An experience
report in a context of distributed teams. In Proceedings
of the XXXV Brazilian Symposium on Software Engineer-
ing, SBES ’21, page 205–214. Association for Computing
Machinery.

Crispin, L. and Gregory, J. (2009). Agile testing: A practical
guide for testers and agile teams. Pearson Education, sl.

Crouch, C. H. and Mazur, E. (2001). Peer instruction: Ten
years of experience and results. American journal of
physics, 69(9):970–977.

de Andrade, S. a. A., de Oliveira Neves, V., and Delamaro,
M. E. (2019). Software testing education: Dreams and
challenges when bringing academia and industry closer to-
gether. In Proceedings of the XXXIII Brazilian Symposium
on Software Engineering, SBES 2019, page 47–56, New
York, NY, USA. ACM.

Ferreira Costa, I. E. and Oliveira, S. R. B. (2020). The use of
gamification to support the teaching-learning of software
exploratory testing: an experience report based on the ap-
plication of a framework. In 2020 IEEE Frontiers in Ed-
ucation Conference (FIE), pages 1–9, Uppsala, Sweden.
IEEE.

Figuerêdo, C. d. O., dos Santos, S. C., Borba, P., and Alexan-
dre, G. (2011). Using pbl to develop software test en-
gineers. In International Conference on Computers and
Advanced Technology in Education, pages 305–322, Cam-
bridge, United Kingdom. sn.

Garousi, V., Felderer, M., Kuhrmann, M., and Herkiloğlu, K.
(2017). What industry wants from academia in software
testing? hearing practitioners’ opinions. In Proceedings
of the 21st International Conference on Evaluation and
Assessment in Software Engineering, EASE’17, page 65–
69, New York, NY, USA. ACM.

Garousi, V., Rainer, A., Lauvås, P., and Arcuri, A. (2020).
Software-testing education: A systematic literature map-
ping. Journal of Systems and Software, 165:110570.

Gebizli, C. S. and Sozer, H. (2017). Impact of education and
experience level on the effectiveness of exploratory test-
ing: An industrial case study. In 2017 IEEE International
Conference on Software Testing, Verification and Valida-
tion Workshops (ICSTW), pages 23–28.

Ghazi, A. N. (2017). Structuring Exploratory Testing
Through Test Charter Design and Decision Support. PhD
thesis, Blekinge Tekniska Högskola.

Hendrickson, E. (2013). Explore it!: reduce risk and increase
confidence with exploratory testing. Pragmatic Bookshelf,
sl.

Leite, F. T., Coutinho, J. C. S., and de Sousa, R. R. (2020).
An experience report about challenges of software engi-
neering as a second cycle course. In Proceedings of
the 34th Brazilian Symposium on Software Engineering,
SBES ’20, page 824–833, New York, NY, USA. ACM.

Mårtensson, T., Martini, A., Ståhl, D., and Bosch, J. (2019).
Excellence in exploratory testing: Success factors in large-
scale industry projects. In Product-Focused Software Pro-
cess Improvement, pages 299–314. Springer International
Publishing.

Martinez, A. (2018). Use of jitt in a graduate software test-
ing course: an experience report. In 2018 IEEE/ACM 40th

International Conference on Software Engineering: Soft-
ware Engineering Education and Training (ICSE-SEET),
pages 108–115, Gothenburg,Sweden. IEEE.

McConnell, J. J. (1996). Active learning and its use in com-
puter science. In Proceedings of the 1st Conference on
integrating Technology into Computer Science Education,
pages 52–54, Barcelona,Spain. ACM.

Milne, A., Riecke, B., and Antle, A. (2014). Exploring maker
practice: Common attitudes, habits and skills from vancou-
ver’s maker community. Studies, 19(21):23.

Novak, G. M. (2011). Just-in-time teaching. New directions
for teaching and learning, 2011(128):63–73.

Paiva, M. R. F., Parente, J. R. F., Brandão, I. R., and
Queiroz, A. H. B. (2016). Metodologias ativas de ensino-
aprendizagem: revisão integrativa. SANARE-Revista de
Políticas Públicas, 15(2):145–153.

Paschoal, L. N. and de Souza, S. d. R. S. (2018). A survey
on software testing education in brazil. In Proceedings of
the 17th Brazilian Symposium on Software Quality, SBQS,
page 334–343, New York, NY, USA. ACM.

Paschoal, L. N., Silva, L., and Souza, S. (2017). Abordagem
flipped classroom em comparação com o modelo tradi-
cional de ensino: uma investigação empírica no âmbito
de teste de software. In Brazilian Symposium on Com-
puters in Education (Simpósio Brasileiro de Informática
na Educação-SBIE), page 476, Recife,PE. SBC Open Li-
brary.

Paschoal, L. N. and Souza, S. R. (2018). Planejamento e
aplicação de flipped classroom para o ensino de teste de
software. RENOTE, 16(2):606–614.

Pfahl, D., Yin, H., Mäntylä, M. V., and Münch, J. (2014).
How is exploratory testing used? a state-of-the-practice
survey. In Proceedings of the 8th ACM/IEEE Interna-
tional Symposium on Empirical Software Engineering and
Measurement, ESEM ’14, New York, NY, USA. Associa-
tion for Computing Machinery.

Queiroz, R., Pinto, F., and Silva, P. (2019). Islandtest: jogo
educativo para apoiar o processo ensino-aprendizagem de
testes de software. In Anais do XXVII Workshop sobre Ed-
ucação em Computação, pages 533–542, Belém,PA. SBC
Open Library.

Raappana, P., Saukkoriipi, S., Tervonen, I., and Mäntylä,
M. V. (2016). The effect of team exploratory testing–
experience report from f-secure. In 2016 IEEE Ninth Inter-
national Conference on Software Testing, Verification and
Validation Workshops (ICSTW), pages 295–304, Chicago,
IL, USA. IEEE.

Scatalon, L. P., Carver, J. C., Garcia, R. E., and Barbosa, E. F.
(2019). Software testing in introductory programming
courses: A systematic mapping study. In Proceedings of
the 50th ACM Technical Symposium on Computer Science
Education, SIGCSE ’19, page 421–427, New York, NY,
USA. ACM.

Suranto, B. (2015). Exploratory software testing in agile
project. In 2015 International Conference on Computer,
Communications, and Control Technology (I4CT), pages
280–283, Kuching, Malaysia. IEEE.

Whittaker, J. A. (2009). Exploratory software testing: tips,
tricks, tours, and techniques to guide test design. Pearson



Coutinho et al. 2023

Education, sl.
Wohlin, C., Runeson, P., Host, M., Ohlsson, M. C., Regnell,
B., and Wesslén, A. (2012). Experimentation in software
engineering. Springer Science & Business Media, sl.


	Introduction
	Background
	Teaching Software Testing
	Exploratory Testing
	Active Learning
	Problem Based Learning - PBL
	Just-in-Time Teaching - JiTT


	Methodology
	Study Design
	Planning
	Execution
	Analysis Procedures

	Checking the Use of ET in Practice
	Brainstorming Planning
	Execution of Brainstorming


	Results
	Results of Experiment
	Characterization of participants
	Common practices in Agile ST
	Perception of ET after the course
	Contributions from the PBL and JiTT approach

	Results of ET insertion with agile teams

	Discussion of Results
	Overview on using PBL and JiTT
	Overview of Insights in Practice

	Limitations and Threats to Validity
	Related Works
	Teaching-learning of ST using active methodology
	ET teaching
	ET in the context of industry
	Discussion of related works

	Conclusions