DataPDF Available

Abstract

The increasing application of student-centered teaching approaches to solve real problems, driven by the market´s demand for professionals with better skills, has prompted the use of PBL in different areas, including in Computing. However, since this represents a paradigm shift in education, its implementation is not always well understood, which adversely affects its effectiveness. Within this context, this paper puts forward a model for assessing the maturity of teaching processes under the PBL approach, the PBL-Test, with a view to identifying points for improvement. The concept of maturity is defined in terms of teaching processes adhering to PBL principles, taken from an analysis of the following authors: Savery & Duffy (1995), Barrows (2001) Peterson (1997) and Alessio (2004). With a view to validating the applicability of the model, an empirical study was conducted by applying the PBL-Test to three skills in the Computing area. Results showed that although the model has shown it needs further enhancement, it has already been possible to identify improvements in PBL teaching processes that clearly affect the effectiveness of the approach.
PBL-Test: a Model to Evaluate the Maturity of
Teaching Processes in a PBL Approach
Simone C. dos Santos, Caliane O. Figuerêdo
Centro de Informática
Universidade Federal de Pernambuco
Recife, Brazil
scs@cin.ufpe.br, cof@cin.ufpe.br
Fernando Wanderley
Universidade Nova de Lisboa
Lisboa, Portugal
fjaw@gmail.com
Abstract—The increasing application of student-centered
teaching approaches to solve real problems, driven by the
market´s demand for professionals with better skills, has
prompted the use of PBL in different areas, including in
Computing. However, since this represents a paradigm shift in
education, its implementation is not always well understood,
which adversely affects its effectiveness. Within this context, this
paper puts forward a model for assessing the maturity of
teaching processes under the PBL approach, the PBL-Test, with
a view to identifying points for improvement. The concept of
maturity is defined in terms of teaching processes adhering to
PBL principles, taken from an analysis of the following authors:
Savery & Duffy (1995), Barrows (2001) Peterson (1997) and
Alessio (2004). With a view to validating the applicability of the
model, an empirical study was conducted by applying the PBL-
Test to three skills in the Computing area. Results showed that
although the model has shown it needs further enhancement, it
has already been possible to identify improvements in PBL
teaching processes that clearly affect the effectiveness of the
approach.
PBL; Evaluation of Teaching Processes; Process Maturity.
I. INTRODUCTION
Professionals in highly dynamic areas such as Computing,
face the need to be familiar with and know how to apply new
concepts, methods and techniques for specific situations that
may arise in the labor market, if they are to keep up with the
pace of rapid change in this market. In seeking better ways to
formally train these professionals, the instructional teaching
method called Problem-Based Learning (PBL) has been
adopted with very favorable results as in [1, 2, 3, 4]. In general,
PBL makes use of real problems to start, motivate and focus on
the acquisition of knowledge. In addition, it encourages
professionals to add to their skills and it fosters positive
attitudes in the professional environment. It is considered to be
a student-centered educational strategy, which helps to develop
reasoning and communication. These are skills which are
essential for professionals to possess if they are to be
successful in the labor market [1].
PBL is becoming increasingly effective in a variety of
disciplines in higher education [5]. Many studies have shown
the efficiency of the PBL method in helping learners to absorb
the content of learning. In [6], the authors conduct a systematic
mapping of the use of PBL when teaching computing, from
1997 to 2011. They analyzed 52 studies in this field, taken
from sources such as IEEE, Scopus, Science Direct and ACM
databases. This study shows that this method has been adopted
with the objective of improving the effectiveness of teaching
and learning processes.
Despite the potential benefits of the PBL approach, it is
also evident that its use implies a profound educational change
and a new culture of teaching and learning [7, 8, 9, 10]. Thus,
managing a process of teaching and learning based on PBL is a
great challenge, especially with regard to implementing a
methodology which adheres to PBL principles [7]. Thus, if
such adherence is absent or partial, this could have an adverse
effect on the effectiveness of this approach.
Faced with this challenge, this paper seeks to answer two
important questions: "How can it be certified that a process of
teaching and learning is adhering to the principles of PBL
approach?" And "At what level of maturity is the process of
teaching and learning in a formal training course which uses
the PBL approach, in accordance with PBL principles? '.
As to the first question, Oliveira et al. [6] stress that there is
a wide variety of educational methods, referred to as being
PBL. According to Norman and Schmidt [11], "anyone who
knows more than one institution in which PBL is used will
verify that in each of them, the methodology is applied in a
different way." Very often, teaching organizations/ institutions
start to work with PBL without their having the necessary
theoretical foundation on which to bring about educational
change. Therefore, the method ends up being confused with
practical experiments or other methods of active/ collaborative
learning, thus jeopardizing the success and authenticity of the
methodology [3].
Additionally, some intrinsic characteristics of PBL, such as
the unpredictability and flexibility in the practical assimilation
of learning, entail that the teaching coordinators lose control of
the actions and constantly improvise [11]. Experiences
undergone as in [2, 3, 4] show that adopting this method is
made difficult when there a lack of appropriate support from
content, processes and human capital in order to implement
PBL. PBL demands flexibility, continuous assessment and
cooperation. It is against this background that this paper argues
that for PBL to be adopted effectively, it is of fundamental
importance that the learning and teaching processes be in
conformity with the basic principles that guide the practice of
this approach.
As to the second question, in view of the variety of PBL
formats, it is important to check at what level of adherence
these formats are. This visibility enables faults in the process of
teaching and learning to be identified as well as the corrective
actions or improvements needed in this process.
In this context, this article proposes a model for assessing
the maturity of processes of learning in the PBL approach,
deemed a PBL-Test. The concept of maturity is defined, based
on the teaching processes that adhere to PBL principles. These
principles have been set out by authors such as Savery & Duffy
[7], Barrows [8] Peterson [9] and Alesssio [10]. With a view to
validating the applicability of the model, an empirical study
was conducted by applying the PBL-Test to three formal
training courses in Computing. Results showed that although
the process of analysis of the model may already shows signs it
needs to be further improved, the model is a valuable tool for
evaluating the maturity of teaching processes in the PBL
approach, thus demonstrating its applicability.
II. THE PRINCIPLES OF PBL
PBL is based on educational principles and on results from
research in cognitive science, which show that learning is not a
process of passively receiving and accumulating information,
but one in which knowledge is constructed [6].
In [7], Savery & Duffy set out a set of eight instructional
principles for PBL, namely: (SD.1) Anchor all learning on
activities to a larger task or problem; (SD.2) Provide support
for engaging the student on the task or problem; (SD.3) Design
an authentic task; (SD.4) Design the task and the learning
environment to reflect the complexity of the environment for
which the students should have the skills to interact in, at the
end of the learning; ( SD.5) Give the student ownership of the
process used to work out the solution; (SD.6) Design the
learning environment to support students' thinking while
challenging them; (SD.7) Encourage the testing of ideas
against alternative views and contexts; (SD.8) Give
opportunity for reflecting on learning and what has been
learned.
In the review of the literature conducted for this research,
which includes the systematic mapping of the use of PBL in
teaching Computing [6], no records were found of other
authors who have set out principles for PBL, using this
nomenclature for this purpose. However, the authors Barrows
[8], Peterson [9] and Alessio [10] define criteria and
characteristics that are key to PBL, which, for comparison
purposes, in this paper, are also referred to as PBL principles.
These studies were chosen with a view to having practical
guidelines for this teaching approach that, to some extent,
derive from or complement the principles suggested by Savery
& Duffy [7] for PBL.
In Peterson [9], the author draws attention to three
important criteria that foster more effective learning when PBL
is used: (PT.1) Learning takes place in an environment where
students are immersed, in practice, in activities in which they
receive feedback from their fellow students and teachers;
(PT.2) Students receive guidelines and support from their peers
such that this encourages multi-directional teaching involving
other students, teachers and monitors, unlike conventional
teaching which is usually unidirectional (teacher to student);
(PT.3) Learning is functional, with real problems as the
starting point.
To Barrows [8] the features essential to PBL are: (B.1) The
student must take responsibility for his/her own learning; (B.2)
The problems used in PBL should be scantily structured and
allow free research; (B.3) Learning should be integrated into a
wide range of disciplines and subjects; (B.4) Collaboration is
essential for the learner to develop the security needed to be
responsible for his/her own learning; (B.5) What students learn
during their self directed learning should be applied when
resolving the problem through interactive discussions, in order
to promote in-depth understanding; (B.6) A more accurate
analysis is essential about what concepts and lessons were
learned in working with the problem; (B.7) Students should be
able to assess their own learning progress and to evaluate that
of their peers; (B.8) The activities conducted in PBL should be
the same as those performed in the real world; (B.9)
Assessments should measure the learner’s progress as per the
PBL objectives; (B.10) PBL should be the pedagogical basis of
the curriculum and not a part of the didactic curriculum.
The author Alessio [10], in her study on the perceptions of
students and the performance of PBL, stresses three key
characteristics of PBL: (A.1) Learning in context, in which
real-life problems are presented, (A.2) Extending knowledge
through social interaction, in which students work together in
small groups, and (A.3) Metacognitive reasoning and self-
directed learning.
The model for evaluating the maturity of PBL put forward
in this paper is at bottom based on PBL principles. Given the
variety of definitions, the study included a mapping of the
principles and characteristics of PBL previously mentioned
with the aim of identifying the correlation or possible
differences between them (Table 1). This mapping enabled
both the concepts of PBL to be more clearly understood and for
it to be seen that the values and proposals suggested by Savery
& Duffy, compared with those of Peterson, Barrows and
Alessio, express practically the same ideas. To the extent that
there are differences, these complement each other. Thus, it
was concluded that two other principles can be added to the list
of PBL principles described by Savery & Duffy, and which, as
per the study by Santos [12], were described as follows:
Principle 9 - "PBL lays down a process of multi-
directional teaching and learning" (Peterson’s
Principle 2): which foresees collaboration through
social interaction between students, and between
teachers and students, unlike conventional teaching in
which learning is largely unidirectional, from teacher
to student, and;
Principle 10 - "PBL is supported by planning
processes and continuous monitoring" (includes
Barrows’ Principles 7 and 8): an effective PBL
methodology is strongly oriented to process, since it
needs to be planned. This ensures that theory and
practice go hand-in-hand and are aligned to each other,
while learning should be monitored by instruments that
can evaluate its effectiveness [1].
Therefore, the result of this mapping indicates that the PBL
may be defined in 10 basic principles as shown in Table I. The
correlation between the principles/ characteristics of the
authors Savery & Duffy, Barrows, Peterson and Alessio is also
presented in this Table, using the indices of the respective
principles. For example, considering the principle PR1, Savery
& Duffy says "Anchor all learning activities to a larger task or
problem." Peterson reinforces this principle when he affirms
that "the learning is functional - based on solving the real
problem," corroborated by Alessio when he says that "PBL
include learning in context, where real life problems are
presented".
TABLE I. PBL PRINCIPLES
PBL Principles Correlation
PR1. All learning activities are anchored on a task
or a problem;
SD.1, PT.3,
A.1
PR2. The learner should feel he/she owns the
problem, and is responsible for his/her own
learning;
SD.2, B.1
PR3. The problem should be real; SD.3, B.2, B.3,
B8, PT.3
PR4. The task and the learning environment should
reflect the reality of the professional market; SD.4, PT.1
PR5. The learner needs to own the process used so
as to work out the solution to the problem; SD.5, A.3
PR6. The learning environment should stimulate
and at the same time challenge the learner´s
reasoning;
SD.6, PT.1
PR7. The learner should be encouraged to test
his/her ideas against alternative views and contexts; SD.7, B.5
PR8. The learner should have the opportunity and
support to reflect on the content learned and the
learning process;.
SD.8, B.6, A.3
PR9. The learning is collaborative and
multidirectional; B.4, PT.2, A.2
PR10. PBL is supported by planning processes and
continuous monitoring. B.7, B.9
It is important to stress that the mapping undertaken in
Table 1 was based on an interpretive analysis from the meaning
and descriptions of the principles discussed in the studies
mentioned above, which are subjective in nature. However, the
practical and research experiences in PBL of the authors of this
paper have contributed to adding reliability to this process, as
shown by cases in [2, 3, 4].
III. THE PBL-TEST MODEL
Based on the principles set out in Table 1, the PBL-Test
consists of a model for assessing the maturity of the teaching
processes based on PBL. This model includes an assessment
tool and is structured into five maturity levels. Additionally, the
model lays down conditions and steps for its implementation.
A. Definition of the PBL-Test Model
The PBL-Test includes an assessment tool, which consists
of a form which contains questions, comprising: a header
section, comprising data to be collected, viz., name of
participant, training program, the participant's status (student,
professor or tutor), and period of his/her training; a section
consisting of 10 objective and standardized questions, made up
of exclusive alternatives as shown in Table II.
TABLE II. PBL-TEST QUESTIONS.
PR 1 - Problem (s) at the core of the educational proposal.
0.0)The learning activities are carried out regardless of tasks or
problems.
0.5) Not all learning activities are associated with the resolution of a
problem or task.
1.0) All learning activities are initiated, directed and motivated to solve a
specific problem or task.
PR 2 - Learner as the owner of the problem.
0.0) Totally passive posture on the part of the learner about the problem.
0.5) The learner engages with the problem, usually on delivery of partial
results required by the teacher or tutor.
1.0) The learner is fully involved with the problem, demonstrating
engagement in seeking to solve it.
PR 3- Authenticity of the problem or task.
0.0) The learning tasks do not reflect real-world situations.
0.5) The problem or task is real, but the client and his context aren´t.
1.0) The actual learning tasks are defined and monitored and the client
and restrictions are real (scope, delivery time and effort spent).
PR 4 - Authenticity of the learning environment.
0.0) The learning environment is defined by the teacher.
0.5) The learning environment is a simulation of the real world.
1.0) The learning environment is real, with the same challenges as those
in the labor market.
PR 5 - Driving the solving the problem process.
0.0) The solving the problem process is fully driven by the teacher or
tutor, without the learner understanding the process.
0.5) The teacher or tutor defines the solving the problem process, but the
student knows how to apply it and to identify strengths and
improvements.
1.0) The student defines the solving the problem process, and can
describe its steps, strengths and the improvements needed.
PR 6 - Complexity of the problem or task.
0.0) The problems or tasks require little training in the subject matter.
0.5) The complexity of the problem or task doesn´t require much effort
with regard to looking for information or alternative solutions that will
resolve them.
1.0) The complexity of the problem or task stimulates thinking and sets
challenges on how to develop ideas for solving the proposed problem.
PR 7 - Evaluation and analysis of how the problem was resolved
0.0) How to resolve the problem is proposed by one of the team
members, from their knowledge and/ or individual experience.
0.5) Solutions are proposed by one or more apprentices, towards the best
solution.
1.0) The solutions are built from an investigative process and questioning
of ideas among all members of the team, towards the best solution.
PR 8 - Reflexion on the content learned and the learning process.
0.0) The students have no opportunity to reflect on their learning.
0.5) The students have the opportunity to reflect on their learning, but it
is not oriented towards their self-awareness during the learning process.
1.0) The student is encouraged to think, and thus to demonstratie skills of
self-awareness on the content learned and the learning process.
PR 9 - Collaborative and multidirectional learning.
0.0) Learning is unidirectional (teacher/tutor -to-student).
0.5) Learning occurs in groups, but there is little collaboration and
interactivity among peers, teachers and tutors.
1.0) Learning is collaborative and multidirectional, involving discussions
and greater interaction among peers, teachers and tutors.
PR 10 - Continuous Assessment.
0.0) The assessments are not aligned with the educational objectives in
the teaching plan.
0.5) The educational objectives were not clearly defined and evaluations
are applied to award a grade that classifies the learner as pass or fail.
1.0) Assessments are continuous and aligned with educational objectives
planned. They are applied in order to monitor the learning progress and
provide feedback to the learner.
The answers to each question are related to how one PBL
principle was applied. Each question is associated with three
statements that correspond to the following scale of values: 0
(Does not meet the principle), 0.5 (Partially meets it); 1 (Fully
meets it). On this scale, the score for each question is related to
one these values.
The score of each participant is defined as the sum of points
obtained in each question. After collecting the responses of all
the participants, the score for the course/ training program
evaluated is determined. For simplicity, we chose to calculate
this score by using the arithmetic average of the final scores of
all participants, equally. The scale range was from 0 to 10
points, given that each of the 10 questions can obtain the
maximum value of 1 point (Meets the principle completely).
The points calculated are associated with classes of quality
indicators, which determine the maturity levels of the model.
According to [13], a level of maturity can be defined as a
"plateau" of improvement achieved by a given organization.
Therefore, the levels of maturity in the PBL Test model set
levels of evolution for processes, by characterizing stages of
improving the implementation of the PBL teaching and
learning processes in the teaching organization/ institution.
As can be seen in Table 2, the PBL Test model is structured
into five maturity levels.
TABLE III. LEVELS OF MATURITY IN PBL
Category Overall Average % of Principles
Evidenced
Level 0: Insufficient Overall average < 7 < 70%
Level 1: Initial <= 7 overall average < 8 < 80%
Level 2: Satisfactory <= 8 overall average > 9 < 90%
Level 3: Good <= 9 overall average > 10 < 100%
Level 4: Excellent Overall average = 10 = 100%
The PBL Test is used to obtain a certain level of maturity
which is achieved when the score set for the level in question is
reached: Level 0 - Insufficient: indicates that the teaching
process evaluated does not adhere to PBL principles; Level 1 -
Initial: indicates that the teaching process evaluated adheres
weakly to PBL principles; Level 2 - Satisfactory: indicates that
the teaching process adheres to a significant extent to PBL
principles; Level 3 - Good: indicates that the teaching process
evaluated adheres strongly to PBL principles; Level 4 -
Excellent: indicates that the teaching process evaluated is fully
adhering to PBL principles, which may be deemed "authentic"
or "pure" PBL.
The classification of the level of maturity is directly related
to the final score obtained by the arithmetic mean of the final
score given by those who take part in the training program
evaluated. On a scale of 0 to 10, we chose the value 7 as the
average, given that at least the first three principles must be
met in full (tasks anchored on a problem, the learner feels
he/she owns the problem and the problem is real) and none of
the other principles was zero.
This division into levels enables the current state of the
teaching process to be visualized with regard to implementing
and adhering to PBL principles. But we note that it is important
not only to define at what level the teaching process is, but
rather also what must be done to ensure that it evolves over
time.
B. Application of the PBL-Test Model
The conditions for applying the model can be summarized
in the following questions: Who applies the model? The
coordinator or manager of the training program; What is
assessed? The process of PBL teaching and learning; Who does
the evaluation? The coordinator or manager is the person who
should evaluate and consolidate the results obtained from
applying the questionnaire; Who takes part in the evaluation?
Students, teachers and/ or tutors; When should the model be
applied? During each cycle of assessing the training program,
i.e. during the steps of the PDCA cycle (Plan, Do, Check and
Act); How should the model be applied? As a survey that uses
the evaluation questionnaire given in the model. This may be
applied via online Web-format, or face-to-face.
Ideally, we recommend that the PBL-Test be applied over
four steps, as shown in Fig. 1.
Figure 1. Steps of applying the PBL-Test model.
Initially, the participants of the evaluation (Teachers,
Students, and Tutors) should be introduced to the model,
informed what the objective of applying it and given guidance
on how to answer the questions in the evaluation questionnaire.
The next step is to apply the evaluation questionnaire. This
should be completed by the students, teachers and/or tutors of
the training program being evaluated. To reduce the influence
responses that are repeated, we recommend that the order of
the questions be altered each time that data is accessed/
collected. The ideal is that all the actors of the teaching process
evaluated take part in the survey. However, it is known that this
is not always possible. In these cases, we recommend that the
minimum number of respondents should be at least 1 student, 1
teacher 1, and 1 tutor. Where there is no tutor, or the teacher
takes on both roles, the teacher can substitute for the tutor.
With the results at hand, the responses obtained are scored and
the overall average of the training program is calculated.
Finally, the maturity profile of the PBL teaching process of the
training program evaluated is checked, taking the levels of
maturity for the proposed model into account. It is
recommended that the coordinator or manager of the training
program be responsible for the activities of applying the model.
Given that one of the objectives of applying the model is
the continuous improvement of the processes of PBL teaching
and learning, we also recommend that this be inserted as an
activity of the Check phase of the PDCA cycle of Deming or
that it be applied to each evaluation cycle of the training
program. In other words, the teaching process as a whole
should be planned, executed, verified (controlled) and
improved in every cycle. In this context, the model is inserted
as a tool to support managing the PBL teaching process, thus
making a contribution towards defining and monitoring
continuous improvement actions (Act).
IV. APPLYING THE PBL-TEST IN SOFTWARE ENGINEERING
EDUCATION
With the aim of validating the PBL-Test model initially, an
empirical study of an applied nature and quantitative approach
was conducted, which used the data collected from responses
to the questionnaire described in Section 3 and consecutively
an analysis was made of these data based on the criteria
described in Table III (PBL maturity levels). The Ex-Post-
Facto method or "method applied after the occurrence of the
facts” was used as a technical research procedure [14].
To undertake the study, we selected three training
programs, referenced in this article as Course A, Course B and
Course C. All programs selected are higher education ones in
Computing: Course A is at the Masters level, lasts 1 year and
the class has 16 students, 10 teachers and two tutors; Course B
at the undergraduate level, lasts 6 months, has a class of 11
students, 1 teacher and 1 tutor; Course C at the undergraduate
level, lasts 5 months, and there are 10 students, 5 teachers and
1 tutor. These formal training programs were selected because
they use PBL as a teaching approach. Specifically in the case
of Course B, the coordinator was not sure that it should be
included in this context, but the fact of there being strong signs
of using PBL in the teaching-learning process prompted us to
apply the PBL-Test in this program.
For data collection, we made the evaluation questionnaire
proposed in the PBL-Test model available on SurveyMonkey
in September-October 2012, given that this made both the
collection of data and the analysis of results easier. We invited
the students, teachers and tutors of three courses to be
participants, and thereby a sample was obtained comprising 23
volunteer collaborators, distributed as follows: 6 participants
were from Course A (2 students, 1 tutor and 3 teachers); 8 from
Course B (7 students, 1 tutor and 0 teachers), 9 from Course C
(5 students, 2 tutors and 2 teachers).
From the point of view of the principles, Table IV presents
the arithmetic averages of the participants’ evaluations of each
PBL principle, in each of the three training programs.
On analyzing the results, we found that all the courses
evaluated used the PBL approach, since they received scores of
above the average of 7.0. Taking the maturity levels of Table 2
into account, Courses B and C were classified within the level
of adherence to PBL principles in the category of "Initial",
while Course A stood out by being classified in the category of
"Satisfactory".
TABLE IV. AVERAGE OF EVALUATIONS PER PBL PRINCIPLE.
Principle Course A Course B Course C
1 0.9 0.8 0.9
2 0.8 1.0 0.7
3 0.9 0.3 0.6
4 0.8 0.3 0.7
5 0.8 1.0 0.6
6 0.8 1.0 0.7
7 0.7 0.6 0.6
8 1.0 0.9 0.8
9 1.0 0.9 0.9
10 1.0 0.9 0.9
Overall average: 8.6 7.7 7.3
On evaluating the principles most perceived as having been
met, the result gives evidence of the PBL approach by showing
adherence to two of the principles that characterize this
approach: the problem is the core of the pedagogical purpose
and the student takes ownership of the problem. Moreover, we
found that the programs had a very positive concern with the
educational methodology, namely, they sought to maintain:
adherence to the principles that targeted reflection on the
learning and the learning process (Principle 8); collaborative
learning based on multidirectional interactions between
students, teachers and tutors (Principle 9 ); and the process of
continuous assessment (Principle 10).
On the other hand, the reality and authenticity of the
problem (Principle 3) and of the learning environment
(Principle 4) were the aspects that showed the greatest need to
improve in the evaluations of courses B and C. Course A
reinforces its greater maturity by remaining adherent to these
principles. It is worth noting another deficiency in perception
regarding the analysis of alternative solutions (Principle 7),
pointed up by all three training programs, which probably
arises from the effort and length of time this approach requires.
All of these deficiencies represent improvements in PBL
teaching processes that clearly affect the effectiveness of the
approach.
From the point of view of the participants, Figs. 2, 3 and 4
represent the distribution of the sample, by case study, and list
the subjects taking part in the experiment and their respective
responses to each PBL principle. The "PR" index refers to the
principles of Table I, "P" to the participants and their
parameters (P - Professor, T - Tutor or S - Student).
As per Fig. 2, the absence can be observed of non-adherent
responses (0-zero) related to any principles questioned on
Course A, which is thus a behavior that is statistically divergent
from the other training programs. This behavior can be
understood as there was a larger number of mature participants
involved in the assessment, a higher educational level
(Masters) and the predominance of active subjects within the
educational process, such as tutors and teachers (1 tutor and 3
teachers). Another factor that deserves consideration is the total
adherence to all the principles, scored by participant P06. This
reinforces the students´ positive perception of this course,
although in smaller number.
Figure 2. Distribution of Course A.
As per the data reported in Fig. 3, what can be seen is a
statistically consistent and better behaved distribution in
relation to other training courses. Thus, we regarded the
behavior of the participants’ responses as tending to be in
harmony on this training course. Given that that this course had
the largest number of students (7 seven) and only one tutor, it is
worthy of note that the effective presence of students in the
assessment may represent a positive aspect, since the PBL
methodology is based on student-centered processes.
Figure 3. Distribution of Course B.
The data presented in Fig. 4 show the greatest variation in
scores between the other training courses analyzed, thus
demonstrating a behavior that is not as statistically balanced.
This class has a well distributed sample on account of the
presence of all the roles involved in a PBL process, in
acceptable proportions - 5 students, 2 teachers and 2 tutors.
Attention is drawn to participant P09 because of the non-
adherent score (0 - zero) in more than one PBL principle and
this may therefore reduce the maturity average of the group. At
this point, it may make sense to exclude points of dispersion
from the analysis, so as to have a more precise statistical
approach from the results of the questionnaire.
Figure 4. Distribution of Course C.
In general, we note that in the sample collected from the
selected training courses there was a significant variation
between the behavior of the subjects analyzed as to the
composition of the training subjects. In this context, it is
understood that a more rigorous assessment of the influence of
each role of a participant as well as identifying atypical
behaviors in the evaluation may result in an analysis that is able
to verify and validate the maturity of PBL processes more
precisely. This supplies greater consistency to the PBL-Test
model. This is work in progress, which will give rise to future
articles.
V. CONCLUSIONS
This article put forward a model for assessing the maturity
of teaching processes under the PBL (PBL-Test) approach.
This model basically consists of an evaluation questionnaire,
which uses a points scale (1, 0.5 and 0) to calculate the result.
This reflects the PBL maturity (Level 0 to Level 4) of the
course evaluated and is supported by a process of applying a
questionnaire and analyzing the results. To validate the
proposed model, an initial experiment was conducted in three
formal IT skills programs. Although this sample is a starting
point for further analysis, the results demonstrate the
applicability and potential contribution of the model for
improving the quality and the PBL teaching management
process, to the extent that it points out the principles that are or
are not met in the teaching process. It also indicates the state of
adherence of the teaching process as a whole. In general, some
shortcomings were found in the PBL processes on the courses
evaluated and in the influence of their participants and we
acknowledge that the PBL-test model needs to be evolved.
Finally, it is important to emphasize that the proposed PBL-
Test aims to be an evaluation instrument the maturity of PBL
according to their principles, but does not evaluate the
effectiveness of PBL approach with regard to the development
of learning and skills in the students involved, a central theme
work on authentic assessment, discussed by the authors of this
article in [15].
REFERENCES
[1] G. X. L Tai and M. C.Yuen, “Authentic assessment strategies in
problem based learning”. In ICT: Providing choices for learners and
learning. Proceedings ascilite, Singapore, 2007.
[2] S. C. Santos; M. C. M. Batista; A. P. C. Cavalcanti; J. Albuquerque; S.
R. L Meira. “Applying PBL in Software Engineering Education”.
CSEET 2009, Hyderabad, Índia, 2009.
[3] C. O. Figuêredo, S. C. Santos., G. H. S. Alexandre, and P. H. M. Borba,
“Using PBL to develop Software Test Engineering”, CATE,
Cambridge, UK, 2011.
[4] S. C. Santos and A. Pinto, “Assessing PBL with Software Factory and
Agile Processes”, CATE, Naples, Italy, 2012.
[5] L. R. C. Ribeiro, M.G. Mizukami, “An experiment with PBL in higher
education as appraised by the teacher and students”. Interface -
Comunic., Saúde, Educ., v.9, n.17, p.357-68, August 2005.
[6] A. M. C. A Oliveira, S. C. Santos and V. C. Garcia. “PBL in Teaching
Computing: An overview of the Last 15 Years”, unpublished.
[7] Savery, J.R.; DUFFY, T. M. Problem Based Learning: An instructional
model and its constructivist framework. Educational Technology, 1995,
35, p. 31-38.
[8] Barrows, H. S. Problem-Based Learning (PBL), 2001.
[9] Peterson, M., Skills to Enhance Problem-based Learning. Med Educ
Online [serial online] 2.3, 1997.
[10] Alessio H. “Student Perception about Performance in Problem Based
Learning”, Journal of Scholarship of Teaching and Learning, Vol. 4., N.
1, pp. 25 – 36..
[11] G.R. Norman; H.G. Schmidt. “Effectiveness of problem-based learning
curricula: theory, practice and paper darts”. Medical Education,
Plymouth, v.34, s.n., 721-728, 2000.
[12] S.C. Santos., F. S. F. Soares. and F. Wanderley, “Towards Maturity in
Problem Based Learning Applied to Software Engineering Education”,
unpublished.
[13] C. A. Caram; R. C. Vasques. “Maturity Level – How to understanding
and Applying it”, (In portuguese: Nível de Maturidade – Saiba
Compreendê-lo e Utilizá-lo). ISD Brasil, Agosto, 2004.
[14] L. Cohen, L. Manion and K. Morrison. “Research Methods in
Education”. Companion, 6th. edition.
[15] S. C. Santos; F. S. F. Soares. “Authentic Assessment in Software
Engineering Education Based on PBL Principles: A Case Study in the
Telecom Market”. ICSE 2013, California, EUA, 2013.

File (1)

Content uploaded by Fernando Wanderley
Author content
... From this motivation, this paper describes a case study of an Enterprise Management Systems (EMS) course, part of an undergraduate course in Information Systems (IS). In order to align the purpose of this course to labour market demands, we chose to adopt the Problem-Based Learning approach (PBL), considering its increasing popularity in this area [3][4][5][6][7][8][9][10][11]. ...
... The PBL method is based on principles [15]. In [9], based on four key studies, ten essential PBL principles were defined that guide an effective PBL approach. The 10 PBL principles are: 1. ...
... The elements refer to: (1) Problem, an essential aspect in learning in this approach, reflects realism and complexity similar to real contexts; (2) Environment, related to the definition of an authentic learning environment that reflects the actual context of the professional market; (3) Human Capital, with evidence to the roles and responsibilities of the pedagogical team in the planning, execution and follow-up of the process; (4) Content, as an essential part to support the theoretical basis of the problem solving process, consistent with the context of the problem; And (5) Processes, for the adequacy of evaluation processes inherent to the learning format in PBL. The Table 1 summarizes the PBL principles presented in [9] to the elements of xPBL. ...
Chapter
The continuous advancement of Information Technology and the range of industries and services dependent on technology have required profound changes in the education of software professionals. In fact, the education of these professionals must include diverse skills (technical and non-technical), in order to enable them to solve real problems that impact the lives of companies and people. In this scenario, active learning approaches can make a lot of difference, when applied effectively, with well-defined educational goals and continuous follow-up and feedbacks. One of these approaches that are working well in Computer Education is the Problem-Based Learning (PBL) approach. PBL uses real problems as an instrument to develop skills such as holistic knowledge, business understanding, task management and group work, essential in the software professional. In this context, this paper describes a case of an undergraduate course in Information Systems, conducted in the PBL approach. In order to guarantee the application of PBL in an effective way, a Framework for PBL application in the teaching of Computing, described by Santos and Rodrigues (2016) was used. This framework systematizes the application of PBL in the four stages Plan, Do, Check and Act (based on the management cycle of Deming), which are repeated in learning cycles aligned to educational objectives. As the main results of this experience, the following stand out: a proposal for applying PBL in a managed way, based on a Framework for Computer Education; benefits of using the Framework; possibilities for improvements in this approach.
... The xPBL methodology [11] aims to align methods and management tools for PBL approach, so that the principles that characterize can be guaranteed during its adoption in educational computing. In [16], the authors set out 10 principles (see Table I) from a thorough study of principles and features about PBL found in Savery & Duffy (1995) [15], Barrows (2001) [17], Peterson (1997) [18] and Alessio (2004) [14]. To ensure the principles of PBL, the xPBL is based on five elements: (1) Problems involving the capture of real problems, and complexity as relevant aspect; (2) Learning Environment, to reflect the characteristics of the professional environment of software; (3) Content, flexible and aligned to problem solving; (4) Human Capital, involving teachers and tutors as experienced professionals and PBL mentors, students as the development team, and actual customers as stakeholders and; (5) Assessment Processes based on different dimensions and continuous feedback. ...
... Table I shows the mapping between the ten principles and elements of xPBL. Four of the principles defined in [16] are related to the element "Problem" of the xPBL. These principles reinforce that in the PBL, all tasks and activities are anchored in problems, not simply content exposure. ...
... Fig. 3 (T1 class) and Fig. 4 (T2 class) show the students evaluations and their all grades, considering the four aspects mentioned above, into a 0 to 10 scale. It was also carried out an evaluation with our students on PBL approach applied in the discipline and how they felt about the same, according to the proposal described in [16]. With the collection of data from this survey, it was able to conclude that the manner in which the model was applied satisfied the goal of instigating learning through the integration of students into a real problem that captivated engagement of same in the task to be accomplished. ...
... The xPBL methodology [11] aims to align methods and management tools for PBL approach, so that the principles that characterize can be guaranteed during its adoption in educational computing. In [16], the authors set out 10 principles (see Table I) from a thorough study of principles and features about PBL found in Savery & Duffy (1995) [15], Barrows (2001) [17], Peterson (1997) [18] and Alessio (2004) [14]. To ensure the principles of PBL, the xPBL is based on five elements: (1) Problems involving the capture of real problems, and complexity as relevant aspect; (2) Learning Environment, to reflect the characteristics of the professional environment of software; (3) Content, flexible and aligned to problem solving; (4) Human Capital, involving teachers and tutors as experienced professionals and PBL mentors, students as the development team, and actual customers as stakeholders and; (5) Assessment Processes based on different dimensions and continuous feedback. ...
... Table I shows the mapping between the ten principles and elements of xPBL. Four of the principles defined in [16] are related to the element "Problem" of the xPBL. These principles reinforce that in the PBL, all tasks and activities are anchored in problems, not simply content exposure. ...
... Fig. 3 (T1 class) and Fig. 4 (T2 class) show the students evaluations and their all grades, considering the four aspects mentioned above, into a 0 to 10 scale. It was also carried out an evaluation with our students on PBL approach applied in the discipline and how they felt about the same, according to the proposal described in [16]. With the collection of data from this survey, it was able to conclude that the manner in which the model was applied satisfied the goal of instigating learning through the integration of students into a real problem that captivated engagement of same in the task to be accomplished. ...
Article
Full-text available
Tutorial Education Program (TEP) is an initiative from Brazilian government to develop education quality in different areas of knowledge, as its main aim to decrease students’ evasion and promotion of academic success. PET-Informática group from Federal University of Pernambuco (UFPE) has invested in these activities, in particular, to teach programming, one of the disciplines in which students have lower grades in technology courses. In this context, this paper describes an experience of an extension course in Python, using the active learning approach Problem-Based Learning (PBL). This course, ministered in two weeks, made possible the construction of a PBL model to teach Python, based in the xPBL methodology described by Santos & Rodrigues (2016) and the lessons learned with this practical experiment. As the main contribution, this experience stands out the possibility to apply this course to different audiences with different levels of education, and also the dissemination of PBL approach in teaching programming.
... PBL follows some principles such as (i) an authentic learning environment, simulating the situation found in the professional environment, (ii) the use of real problems as a learning object, and (iii) the monitoring of evaluation by continuous feedback [42]. Santos and colleagues [48] defined ten principles for the teaching of computing that founded a methodology called xPBL [49], as shown in Fig. 2. ...
... To overcome these challenges of adopting the PBL, it is necessary to plan all the processes and resources before implementation, assisting in the correct use of the method, in the alignment of theory and practice, and respecting the principles of this approach [48]. When planning to change a teaching and learning process, it is important to carry out a diagnosis to characterize the respective educational institution that will face the changes. ...
Article
Full-text available
In Computing Higher Education (CHE), the desired transformation of traditional teaching and learning methods, almost always based on the transmission of information and content-based curricula, has been the objective of several educational institutions that wish to combat students’ demotivation and dropout. Among successful approaches, Problem-Based Learning (PBL) stands out as one of the most effective and radical methods regarding pedagogical innovations. While the PBL implementation means a great opportunity to achieve better educational performance, it also represents many challenges that can only be managed if they are first known and understood. In this context, the motivation for this study comes from the following research question: “How to know if an institution at CHE is ready to implement the PBL?”. As a response, an institutional diagnostic model regarding the adoption of PBL is proposed. It conducted an opinion survey in two kinds of educational institutions: technical and academic ones. Thirty-eight technical educational institutions in computing answered this survey, involving 302 participants, and fifteen academic institutions, involving 20 participants. The results showed that the model reached its objective, allowing the identification of favorable, warning, and critical points regarding the adoption of PBL in these institutions. This study is an evolution of the results focusing only on technical institutions published at the CSEDU 2021 conference and conducted by the NEXT Research Group.
... The PBL methodology is based on principles [12]. In a research [16] performed from 4 key studies, it was defined 10 essential principles in PBL, that are a guide to an effective approach. The 10 principles are: ...
... To verify the result of the planning, a checklist was developed adapting an instrument called PBL-Test [16] that evaluates the maturity of the PBL process in the execution of its principles. The checklist consists of 10 questions of multiple choices, referring to the principles, which were filled from the perspective, perception and experience of the author of this research. ...
... The first one advocates the use of the 10 PBL principles defined in ref. [18]. This study proposes a maturity model to evaluate the compliance of the learning process to the PBL principles, called PBL-Test. ...
... Thus, the 10 PBL principles are: (1) Problem(s) at the core of the educational proposal; (2) Learner as the owner the problem; (3) Authenticity of the problem or task; (4)Authenticity of the learning environment; (5) Learner drives the problem solving process; (6) Complexity of the problem or task; (7) Assessment of how the problem was solved; (8) Reflection on the content learned and the learning process; (9) Collaborative and multidirectional learning; and (10) Continuous assessment. In order to evaluate the PBL process maturity, verifying the compliance with its principles as well as the weakness in its execution [18], defined a model, called PBL-Test. The model proposes to identify the level of maturity of the PBL being implemented from the perspective of the human capital, using a multiple-choice questionnaire, aiming to assess the teaching/learning process from the standpoint of the PBL principles. ...
Article
Problem‐Based Learning (PBL) has gained widespread approval as a constructive, student‐centered approach to teaching, and learning. Advocating the use of real‐life problems to initiate and motivate the learning process, PBL is an inductive strategy promoting the understanding of new concepts as well as the development of professional skills, abilities, and attitudes. Thus, it is paramount that teachers and students implement PBL in conformity to its core principles. However, such implementation becomes a challenge, considering that PBL is fundamentally different from traditional instruction methods. Without the appropriate pedagogical and technological support, PBL's quality, effectiveness, and authenticity are at risk. In an attempt to overcome these challenges, this paper presents the PBL‐Coach Virtual Learning Environment (VLE), built upon PBL learning theories. This paper describes in detail a scientific investigation on a practical PBL implementation, driven by the quest to achieve the following goals: (i) Scaffold teachers and students in implementing the PBL method; (ii) Investigate useful and usable technological resources in the PBL process; (iii) Maintain PBL core principles for an authentic and effective learning process. As a result, three case studies are described and one is detailed to validate our investigation, consisting of a real‐world setting with clients, teachers, tutors, and students. Each case study provided a good level of evidence that PBL‐Coach, used in a real‐world, authentic setting, can promote an effective learning process through a number of carefully designed technological resources. Encouraging results motivated to evolve PBL‐Coach through Augmented and Virtual Reality components as future works.
... The PBL methodology is based on principles [12]. In a research [16] performed from 4 key studies, it was defined 10 essential principles in PBL, that are a guide to an effective approach. The 10 principles are: ...
... To verify the result of the planning, a checklist was developed adapting an instrument called PBL-Test [16] that evaluates the maturity of the PBL process in the execution of its principles. The checklist consists of 10 questions of multiple choices, referring to the principles, which were filled from the perspective, perception and experience of the author of this research. ...
Article
The PBL (Problem-Based Learning) methodology provides many advantages to those adopting it in the teaching-learning process. To this end, a good planning that guides the usage of the methodology towards the goals established by the teacher is of fundamental importance, in order to prevent fundamental aspects to be neglected or forgotten. However, there is a lack of specific solutions to help teachers plan their teaching. This is even more pronounced in the specific case of PBL. As a solution to this problem, this paper proposes a tool composed of a PBL Canvas and a set of cards, used to guide the teaching planning in the PBL approach, called PBL Planner Toolkit. The initial results indicate a good acceptance of the tool as well as evidences of its effectiveness in the planning of the adoption of the PBL, elicited through the realization of real plans of 4 courses using the PBL Planner Toolkit. The data collection was done by means of the evaluation of the course plans and by observations recorded in the logbook made during the realization of the plans.
... To ensure that PBL principles that go beyond its educational objectives are met, the methodology xPBL is based on five elements: (1) Problem; (2) Environment; (3) Content; (4) The human capital and (5) Process. These elements reinforce ten principles that were established in [3]. ...
Poster
The PBL (Problem-Based Learning) methodology provides many benefits to those who use it in teaching. In this light, it is important to plan well when using this methodology, efficient to the purposes established by an educator, in a way to avoid those vital aspects to educational planning in the PBL approach that are neglected or forgotten. However, there is a lack of specific tools to help educators in the task of planning their teaching, specifically geared to the PBL approach. As an alternative to this problem, this paper proposes a tool consisting of a Canvas PBL and a set of cards intended to guide the planning of teaching in the PBL approach.
... Dos Santos, Figuerêdo and Wanderley [10], made a review of the literature to find the main PBL principles. The result of this mapping indicates that the PBL may be defined in 10 basic principles as shown in Table I. ...
Conference Paper
This a Research Full Paper. The requirements of Software Engineering need a learning environment that is not only practical but true to the reality of the market. As a student-centered approach, Problem-Based Learning (PBL) enables students to be collaborative and attitude-oriented during problem-solving. Despite this, PBL has its cultural challenges. Thus, this work intends to better understand how the students learn and behave through student meaningful learning profiles and the Myers Briggs Type Indicator (MBTI). Based on the principle that these students participate or participated in a discipline that used the PBL method and are students of a course in the Computer Science area. In this context, this article proposes a descriptive study to make initial analyses and kick start the research. Based on this study and its early results, we can conclude that learning dimensions require greater stimulation and it was identified an initial relationship between personality profiles and meaningful learning profiles.
Conference Paper
This Research Full Paper presents an overview of student assessment proposals for Problem-Based Learning (PBL) in Computing Education. Computing teaching has many challenges, as it requires different skills from students, often subjective and difficult to assess. In fact, technical knowledge alone is not enough to fully understand what is being taught, but the interpretive and logical skills to deal with practical problems and non-technical skills such as group work, creativity, critical vision, ability to cooperate and communicate. Active learning methodologies as Problem-Based Learning (PBL) have been used to dealing with such challenges, broadly developing technical and non-techniques skills in students. However, despite the benefits of PBL, the student assessment process is one of the points that present its own adversities and, therefore, an aspect that deserves greater attention. To better understand the nuances of this process and how it can contribute to the teaching and learning process based on PBL, this study aimed to investigate primary studies in the last two decades, seeking answers to the following research questions: RQ1) What assessment models are being used?; RQ2) Which aspects are evaluated?; RQ3) What criteria and media have been defined?; RQ4) Who gets involved in the assessment process?; RQ5) What is the ideal frequency to conduct the evaluations?; RQ6) What can these models reveal? As a research method, this study used the Systematic Literature Review method proposed by Kitchenham. As main conclusions, it was possible to identify that: generally, computing education based on PBL occurs at the undergraduate level, having as main educational objective the teaching of technical content; in practice, the need for a diverse teaching team is not reflected, the traditional student-teacher remains; to evaluate students, it is necessary to consider several aspects, technical and non-technical, defining specific criteria for each one of them; the main benefits for students are related to changes in behavior, development of soft skills and better absorption of technical knowledge; as main challenges for students, the difficulty to understand the nuances of the proposed problem and to be the main responsible for devising a solution for it without the figure of a teacher to give a clear definition of how to do it stands out.
Conference Paper
Full-text available
The continuous growth of the use of Information and Communication Technology in different sectors of the market calls out for software professionals with the qualifications needed to solve complex and diverse problems. Innovative teaching methodologies, such as the "Software Internship" model and PBL teaching approaches that are learner-centered and focus on bringing market reality to the learning environment, have been developed and implemented with a view to meeting this demand. However, the effectiveness of these methods cannot always be satisfactorily proved. Prompted by this, this paper proposes a model for assessing students based on real market practices while preserving the authenticity of the learning environment. To evaluate this model, a case study on skills training for software specialists for the Telecom market is discussed, and presents important results that show the applicability of the proposed model for teaching Software Engineering.
Conference Paper
Full-text available
In computing courses, the teaching and learning approach normally emphasizes theoretical knowledge at the expense of practical knowledge. The major disadvantages of this approach are learners' lack of motivation during class and their quickly forgetting the knowledge they have acquired. With a view to overcoming these difficulties, Problem Based Learning (PBL), an institutional method of teaching, has been applied to teaching computing disciplines. Despite the growth of the practice of PBL in various disciplines of Computing, there is little evidence of its specific characteristics in this area, the effectiveness of different PBL methodological approaches, or of benefits and challenges encountered. In this context, this paper presents a systematic mapping study in order to identify studies which involve best practices when using the PBL method in Computing between 1997 and 2011, answering five research questions: “What are the main characteristics of PBL that support teaching in Computing?”; “What are the criteria for applying PBL effectively in this area?”; “How is the PBL methodology applied?”, “What are the advantages and benefits of applying PBL in Computing?” and, finally, “What are the main challenges about learning in PBL in Computing?”.
Article
Full-text available
Many graduate and professional programs include Problem Based Learning (PBL) as a mainstay in their curricula. For many undergraduate students, this is a change from a teacher-centered to a student-centered learning method. This study was undertaken to learn about perceptions and test performances of college students (N=116) enrolled in liberal education classes when PBL is used vs. traditional teaching methods. Results indicated students perceived traditional teacher-centered learning more favorably than student-centered PBL. Nevertheless, test scores were similar. Negative student perceptions about learning in PBL classes did not support either teacher observations of learning activity in the classroom or compromised test performances.
Article
Full-text available
PBL (Problem-based Learning) has gained worldwide acceptance as an effective instructional approach that helps students to acquire knowledge as well as develop desired professional skills and attitudes. Contrary to other conventional methods that use problems after theory has been introduced, PBL uses a problem to initiate, focus and motivate the learning of new concepts. This paper presents and discusses the students' and the teacher's viewpoints on the implementation of PBL in an Administration Theory course of an engineering curriculum at a public university in Brazil. The data were collected by means of unstructured interviews, participant observation and an open-ended questionnaire responded by the students at the end of the course. The results show that despite increasing the workload for the teacher and the students, both evaluated PBL positively because it is more motivating and dynamic.
Conference Paper
Full-text available
The Information Communication and Technology - ICT industry is facing a market of constant changes and challenges. These characteristics demand the ICT professionals to have a wide vision of the problem, rather than just knowledge on the technology. In this context, the objective of this article is to propose an innovative pedagogical methodology based on PBL (problem based learning) to improve the learning effectiveness in software engineering, through the implementation of software factories, where students can work together to solve real problems. This methodology is running in a graduate course in software engineering managed by CESAR, a research institute with experience in the development of innovative software. Some important results about this course are presented and discussed in this paper.
Article
Full-text available
Incl. bibl., index.
Article
In a recent review article, Colliver concluded that there was no convincing evidence that problem-based learning was more effective than conventional methods. He then went on to lay part of the blame on cognitive psychology, claiming that 'the theory is weak, its theoretical concepts are imprecise. the basic research is contrived and ad hoc'. This paper challenges these claims and presents evidence that (a) cognitive research is not contrived and irrelevant, (b) curriculum level interventions are doomed to fail and (c) education needs more theory-based research.
Article
this paper is to provide a clear link between the theoretical principles of constructivism and the practice of instructional design and the practice of teaching. We will begin with a basic characterization of constructivism identifying what we believe to be the central principles in learning and understanding. We will then identify and elaborate on eight instructional principles for the design of a constructivist learning environment. Finally, we will exam what we consider to be one of the best exemplars of a constructivist learning environment -- Problem Based Learning as described by Barrows (1985, 1986, 1992) at the Southern Illinois University Medical School and at the Problem Based Learning Institute for high school teachers .