Content uploaded by Rohemi Zuluaga
Author content
All content in this area was uploaded by Rohemi Zuluaga on Dec 31, 2023
Content may be subject to copyright.
ERIES Journal
volume 16 issue 4
Printed ISSN
2336-2375
299
Electronic ISSN
1803-1617
ASSESSING THE RELATIVE IMPACT
OF COLOMBIAN HIGHER EDUCATION
INSTITUTIONS USING FUZZY DATA
ABSTRACT
KEYWORDS
Eciency, higher educaon, machine learning, predicve evaluaon
HOW TO CITE
Journal on Eciency and Responsibility in Educaon and
Science
Rohemi Zuluaga1
Alicia Camelo-Guarín2
Enrique De La Hoz3*
1
Escuela Militar de Cadetes General José
Colombia
*
Arcle history
Received
Received in revised form
Accepted
Available on-line
Highlights
• An empirical methodology is presented to evaluate, calculate, and predict the relave contribuon under a fuzzy approach.
• The evaluaon of homogeneous universies allows for correctly determining academic performance and associang
eciency with educaonal sustainability.
• The comparison of equivalent enes yields dierent average eciency values for the global analysis.
INTRODUCTION
Globalisation has catalysed what is now known as
the integration of economies, societies, and cultures.
Generally, these integrations manifest as global political
ideas such as Education for Sustainable Development
(Cars and West, 2015). Education for Sustainable
Development is an instrument created in December
2002 by the United Nations General Assembly in its
resolution 57/254. This instrument aims to provide
comprehensive education in values, knowledge, and
attitudes for discerning decisions and executing an action
plan, considering a country’s social, environmental, and
economic context.
According to the United Nations, Educational Institutions
are vital allies in this educational strategy due to their
role as transformers of society through education. Various
studies reveal a positive association between economic
growth and the number of professionals (Hoeg and
Bencze, 2017; Sharma et al., 2018). Meanwhile, Bianchi
and Giorcelli (2020) show how countries with higher
levels of education have higher levels of innovation, as
represented in patent registrations. Corlu and Aydin (2016)
demonstrated that science, technology, engineering, and
mathematics education increases enterprise creation.
Therefore, it is necessary to overcome the challenges
faced by educational institutions in Colombia to provide
their students with the best education. The reports
from the Organization for Economic Co-operation and
Development are alarming, as they indicate the poor
academic performance of Latin American countries.
Figure 1 shows that Latin American countries rank
at the bottom of the list of 79 evaluated countries in the
areas assessed by the PISA test. The nation’s average
score is below the estimated population’s average result.
Full research paper
Printed ISSN
2336-2375
300 ERIES Journal
volume 16 issue 4
Electronic ISSN
1803-1617
Academic Performance in Higher Education
in Colombia
The results of internal assessments conducted in Colombia
to evaluate the quality of secondary education conrm
the issue of low academic performance (see Figure 2). Since
2016, the average evaluation score administered to students
in professional training programs at Higher Education
Institutions (IES) in Colombia has been below the value of 150.
Figure 1: Ranking of PISA evaluaon results for the year 2018 (OECD, 2019)
Figure 2: Average of the overall score between 2016 and 2020 (ICFES, 2022)
Previous reports on student academic performance are an issue
that must be analysed, addressed, and resolved if the goal set by
the United Nations for countries worldwide concerning Education
for Sustainable Development is to be met. This is justied through
the Sustainable Development Goals (SDGs), a series of targets set
by the United Nations to address the world’s most pressing global
challenges to promote sustainable development worldwide. These
objectives cover many areas, from poverty eradication to climate
action and quality education (Chankseliani and McCowan, 2021).
Specically, one of the SDGs is Goal 4, which aims to “Ensure
inclusive and equitable quality education and promote lifelong
learning opportunities for all.” Quality education is essential
for achieving sustainable development, as it equips individuals
with the skills and knowledge required to understand current
and future challenges and nd innovative solutions (Ferrer-
Estévez and Chalmeta, 2021).
In engineering, quality education plays a crucial role
in promoting sustainability. Students and professionals
in engineering are fundamental in creating sustainable
solutions to social, economic, and environmental challenges.
Therefore, it is vital that quality education addresses
the principles of sustainability and equips students with
the necessary skills to design, develop, and manage projects
that are socially responsible, environmentally friendly, and
contextually appropriate (Kopnina, 2020) Education for
Sustainable Development (ESD).
In this vein, engineering programs incorporating sustainability
into their curriculum raise awareness of the environmental
and social impacts of engineering projects. Thus, it teaches
students to consider energy eciency, waste management,
responsible use of natural resources, and social equity when
designing technical solutions. At the same time, students must
ERIES Journal
volume 16 issue 4
Printed ISSN
2336-2375
301
Electronic ISSN
1803-1617
also evaluate and communicate the impacts of their projects
in terms of sustainability (Chankseliani and McCowan, 2021).
Additionally, engineering education can directly contribute
to the achievement of several SDGs, such as SDG 7
(Aordable and Clean Energy), SDG 9 (Industry, Innovation,
and Infrastructure), and SDG 11 (Sustainable Cities and
Communities). By equipping future engineers with knowledge
about renewable energies, clean technologies, and sustainable
urban design, the groundwork is being laid for more
sustainable and resilient development. Therefore, quality
engineering education that addresses sustainability principles
prepares students to tackle current and future challenges
from a responsible and sustainable perspective. Integrating
sustainability into engineering education can drive innovation
and promote more equitable, resilient, and environmentally
respectful development (Avelar et al., 2019) but evolving,
eld. To conceptualize the phenomenon, accumulated
ideas from a total of 193 articles were extracted through
a secondary data source, the Web of Science™. The analysis
proceeds in two sequential steps. First, the bibliometric
analysis identied the networks of co-authorship, periodicals,
higher education institutions (HEI).
However, all of this is overshadowed by the context of low
performance presented at the beginning of this section.
In response to this concern, various authors consider the possible
causes of low academic performance, which may include i) the
quality assessment approach for educational institutions (OECD,
2019), ii) how variables of interest are analysed (Rodríguez and
Huertas, 2016), and iii) diering information on variables that
determine academic performance (Pérez, 2019). These causes
may also be due to the lack of an educational management tool
to analyse student academic information and make accurate
decisions regarding academic performance.
The rst possible cause of low academic performance
contemplates that the quality assessment for educational
institutions is obtained by fullling three substantive activities
(teaching, research, and social outreach or extension) and
other specic requirements according to the accreditation
requested. Additionally, Duque Oliva and Chaparro Pinzón
(2012) consider that quality in education has dierent focuses:
quality as prestige-excellence, quality based on resources,
quality as a result, quality as change (added value), quality
as an adjustment to purposes, quality as perfection or merit,
quality as a program’s conformity with prior minimum quality
standards through accreditation processes, quality as a cost-
value ratio, and quality as suitability for meeting the needs
of the recipients or clients.
In Colombia, since 2016, the quality of educational institutions
is estimated using the concept of quality as a result, which
largely depends on the performance that students from
the institution achieve in various evaluations, and quality
as change (added value), which is granted based on the inuence
that the institution has on student performance (ICFES, 2022),
it is worth noting that education experts suggest using this
approach (Gamboa et al., 2003; Quintero Caro, 2018).
The second cause is that each approach mentioned considers
dierent sets of factors or variables that intervene in educational
processes based on an analysis, this makes quality in education
a complex concept to dene and possibly a multi-dimensional
concept with multiple methods for its estimation (Santos et al.,
2020) this process requires the development of a theoretical
framework in order to analyse the impact of universities’
social responsibility strategies on service quality and students’
satisfaction with higher education. The present study sought to
identify the factors dening students’ perceptions of university
social responsibility (USR). In the case of quality estimation
in Colombia, Pérez (2019) states that the controls carried out
on education measure a specic moment of education without
considering the evolution of students, evidencing a poor
understanding of the situation and, consequently, incorrect
solutions to this problem.
The quality of higher education institutions in Colombia is
estimated through information from the Saber PRO evaluations
(conducted by nal-year students in professional programs)
(ICFES, 2022). Table 1 presents the variables collected
for the evaluation model, and it is observed that they are
qualitative; moreover, only the socio-economic information of
the student is considered, and no past academic level is taken
into account. Therefore, the inferences about the results may
not be sucient to understand current academic performance.
Variable
Age Sex
Socio-economic status Scholarship
Region Student loan
Type of instuon Head of the household
Tuion fee Father’s educaon
Hours on the internet Mother’s educaon
Semester Public school
Socio-economic level Private school
Table 1: Survey Variables in the Saber PRO Assessment Used for the Quality Evaluaon Model
Lastly, the third cause relates to how variables are analysed,
as they are crucial for generating accurate conclusions.
According to Rodríguez and Huertas (2016), there are
degrees of correspondence between decient, acceptable,
and outstanding academic performance. These authors
argue that quality evaluation should consider, for instance,
to what extent performance is decient if an institution
exhibits poor academic results. Similarly, if an institution
has an acceptable academic performance, to what extent
is it considered acceptable? Moreover, if an institution
has an outstanding academic performance, to what extent
is it considered outstanding?
Printed ISSN
2336-2375
302 ERIES Journal
volume 16 issue 4
Electronic ISSN
1803-1617
Considering the challenges above, this research aims to
answer the question: What tool should Higher Education
Institutions utilise to identify the trajectory (in terms of
benchmarking) they should follow to improve their students’
academic performance?
LITERATURE REVIEW
Overview of the Colombian Higher Education System
The higher education system in Colombia is characterised by its
diverse range of institutions, which include public and private
universities, technological institutes, and technical professional
institutions (Altbach et al., 2009). The system is governed by
the Ministry of National Education, which denes policies and
regulations and evaluates and accredits institutions (Ntshoe and
Letseka, 2010) and quality assurance, movements have become
highly contested issues in the advent of new managerialism1 in
higher education. This is because while the notion of quality is
critical to institutional autonomy and academic freedom, there
are no universal criteria to determine quality in the current
conditions of global competitiveness and new managerialism.
In this chapter we analyze quality measures and the quality
assurance movement in the current global market economy.
We investigate ways in which the quality assurance movement
has shaped higher education policy and practice and impacted
national, regional, and international priorities. The chapter’s
emphasis is on the following areas: (a. There has been
signicant growth in higher education enrollment over the past
two decades, with a notable increase in private institutions
(Barr and Turner, 2013).
Despite the growth of the higher education sector, Colombian
higher education institutions (HEIs) face several challenges,
such as improving access, equity, and quality (Acosta and
Celis, 2014). Moreover, there is a need to enhance teaching
and research eectiveness and increase the internationalisation
of Colombian HEIs (Navas et al., 2020). On the other hand,
the higher education sector also presents opportunities
for growth and improvement, such as the potential for
collaboration between institutions, innovative teaching and
learning methods, and the integration of new technologies
(Castro, 2019) dynamics, and actors’ interactions, particularly
concerning technological innovations. This paper aims to
identify some of the most promising trends in blended learning
implementations in higher education, the capabilities provided
by the technology (e.g., datacation).
State evaluations of higher education institutions play
a crucial role in assessing the quality and performance of
these institutions, providing valuable information for decision-
making processes, and promoting accountability (Shriberg,
2002). State evaluations typically include assessments of
teaching, research, community engagement, governance, and
management (Abelson et al., 2003). Consequently, national
or regional agencies conduct these evaluations and can serve
various purposes, such as accreditation, funding allocation, or
performance benchmarking (Font, 2002).
In Colombia, state evaluations of higher education institutions
are conducted by the National Council for Higher Education
Accreditation (CNA) and the Colombian Institute for the
Evaluation of Higher Education (ICFES). The CNA is
responsible for accrediting institutions based on their
compliance with established quality criteria, while the ICFES
evaluates the performance of students and programs
through standardised tests. These evaluations are a basis
for developing national policies and strategies to improve
the higher education sector.
Application of Fuzzy Data Envelopment Analysis
in Higher Education Performance Evaluation
Fuzzy Data Envelopment Analysis (Fuzzy-DEA) has developed
as an essential method for evaluating the performance of higher
education institutions, especially when data are imprecise,
ambiguous, or subjective. Accounting for the inherent
imprecision of input and output characteristics, Fuzzy-DEA
has been utilised in several studies to assess the eciency of
higher education institutions in diverse scenarios.
Nojavan et al. (2021) utilised Fuzzy-DEA to evaluate eight
higher education institutes in Iran. The study resolved
the ambiguity of assessing research quality and its eect on
overall eciency scores by applying fuzzy logic. Their study
indicated considerable dierences in research eciency
across the examined institutions, shedding light on the aspects
contributing to successful research performance.
Similarly, Nazarko and Šaparauskas (2014) applied Fuzzy-
DEA to assess the eciency of university departments,
considering the uncertainty associated with the inputs and
outputs variables such as number of professors, number of
students, equipment and income. Their study found substantial
dierences in eciency scores among the university
departments, with most institutions operating below their
maximum eciency levels. Their research ndings highlighted
the need for resource equipment and space improvements to
enhance overall performance in the higher education sector.
Aparicio et al. (2019) used Fuzzy-DEA to evaluate
the performance of US students and schools participating
in PISA (Programme for International Student Assessment)
2015. Their study provided a more robust and comprehensive
assessment of educational performance by accounting
for the imprecision and subjectivity of input and output
factors. The results provide a framework to set the notion
of fuzziness in some variables, such as students’ socio-
economic status or test scores.
In addition to these studies, Fuzzy-DEA has also been used
to assess the efficiency of higher education institutions in
other countries, such as Phillipines (Mirasol-Cavero and
Ocampo, 2021), Taiwan (Liu and Chuang, 2009), and India
(Singh et al., 2022). These studies have demonstrated
the value of Fuzzy-DEA as a flexible and robust tool
for evaluating the performance of higher education
institutions, particularly in contexts where data are subject
to uncertainty, imprecision, or subjectivity.
In Colombia, the use of Fuzzy-DEA in evaluating
the performance of higher education institutions remains
restricted, giving a potential for more study and analysis. By
introducing fuzzy logic into the DEA framework, the present
study attempts to provide a more thorough and nuanced
evaluation of the relative contribution of Colombian higher
ERIES Journal
volume 16 issue 4
Printed ISSN
2336-2375
303
Electronic ISSN
1803-1617
education institutions based on state evaluations. Thus, this study
aims to create a tool for educational management to evaluate
students’ academic performance in the industrial engineering
program. Additionally, it is necessary to consider i) the quality
assessment approach for educational institutions, ii) how
the variables of interest are analysed, and iii) the variables that
determine academic performance.
Consequently, it is essential to recognise the research that has been
conducted to date in this eld. Table 2 presents a summary of
the literature review, in which only quantitative research studies were
considered due to the focus of this study. Additionally, it is important
to note that the identied research has an added-value approach for
quality assessment (used in Colombia) due to the implementation of
Data Envelopment Analysis models (De La Hoz et al., 2021).
Authors Variables Locaon Populaon
Johnes (2006)
Academic scores, number of undergraduate and
graduate students, library expenditure (Fuzzy
logic approach)
England 130 universies
Nazarko and Šaparauskas (2014) Financial expenses, faculty, student-to-
administrave sta rao Poland 19 universies
Do and Chen (2014)
Sta, expenses, university area, credit-hours,
publicaons, and scholarships (Fuzzy logic
approach)
Vietnam 18 universies
Galbraith and Merrill (2015) Academic performance and Burnout measures United States 350 graduate students in
economics and business
Alabdulmenem (2016) Faculty and administrave sta, number of
students, number of graduates Saudi Arabia 25 universies
Visbal-Cadavid et al. (2017) Financial resources, quality indicators,
accreditaons, and achievements Colombia 32 universies
Wolszczak-Derlacz (2017) Faculty, total income, number of students,
bibliographic producon, number of graduates
Europe and
United States 500 universies
Aparicio et al. (2019) PISA 2015 assessment outcomes (Fuzzy logic
approach) PISA Tests United States
Agasis et al. (2019) Faculty, government investment, and PISA
results Europe 24 countries
Kalapou et al. (2020) Faculty and administrave sta, spending on
research and development, and patents United States 182 regions
Nojavan et al. (2021)
Outcomes of academic performance
evaluaons for HEIs (Higher Educaon
Instuons) (Fuzzy logic approach)
Iran 30,000 Iranian students
Aparicio et al. (2021) the so-called
plausible values, which are frequently
interpreted as a representaon of
the ability range of students. In this
paper, we focus on how this informaon
should be incorporated into the
esmaon of eciency measures of
student or school performance using
data envelopment analysis (DEA)
PISA 2015 assessment outcomes (Fuzzy logic
approach) PISA Tests 72 countries
Table 2: A literature review of papers using the fuzzy data envelopment analysis model
MATERIALS AND METHODS
The current research focuses on three fundamental concepts:
Fuzzy Logic, Data Envelopment Analysis, Machine Learning
and Methodology.
Fuzzy Logic
The objective of fuzzy logic is to mathematically represent
the ambiguity of expressions or events that are observed
in everyday life. In other words, the fuzzy numbers
represent the uncertainty generated at the borders of
the qualifiers (high, medium, low) that describe an event,
for example, a student’s performance (Rodríguez and
Huertas, 2016).
On the other hand, mathematically, a fuzzy set is dened as
presented in equation (1).
( )
( )
{ }
,,
A
A x x xX
µ
= ∈
(1)
Thus, the expression
( )
Ax
µ
represents the membership level
of
x
in
A
and
A
µ
is the membership function associated with
A
. The equation denes the level at which each element of
X
belongs to the fuzzy set; it should be noted that
X
take values
in
[ ]
:,R−∞ +∞
.
Finally, there exists a series of fuzzy numbers whose usage depends
on the event or linguistic variable one wishes to represent.
Figure 3 shows the graphical representation of a triangular
fuzzy set
( )
a and another triangular fuzzy set
( )
b
. It should
be noted that these are the most commonly used sets.
The dierence lies in the results for the membership function
according to the same value of
X
.
Printed ISSN
2336-2375
304 ERIES Journal
volume 16 issue 4
Electronic ISSN
1803-1617
Data Envelopment Analysis
The Data Envelopment Analysis (DEA) methodology proposed
by Charnes, Cooper, and Rhodes (Charnes et al., 1978) is a non-
parametric approach for estimating the relative eciency of
Decision Making Units (DMUs). The outcome of the DEA
model is a frontier made up of the most ecient DMUs in the
study; it is essential to note that only the DMUs on this frontier
are considered ecient.
To construct the DEA model, it is necessary to establish its
conguration, which consists of scale return and orientation. First,
the scale return can be either constant or variable. It is constant
when estimating the system’s overall eciency, which involves
understanding all the parts contributing to eciency outcomes. On the
other hand, variable returns are used to observe resource utilisation for
each system unit. In other words, this scheme focuses on one aspect
of eciency; therefore, eciency with variable returns will always
be higher than with constant returns.
Additionally, orientation is important for the model’s conguration
and can be either input-oriented or output-oriented. Input orientation
implies that resources or inputs can be reduced to achieve a greater or
equal level of outputs. Conversely, an output-oriented model suggests
that products or outcomes can be increased using the same input level.
Lastly, equation (2) presents the linear programming model of
DEA (León et al., 2003). This model compares the ratio of outputs
to inputs. It is worth noting that one DMU will be more ecient
than another based on its ability to generate higher output levels
with a given input level.
0
min
θ
(2)
00
1
: , 1, ,
n
j ij i
j
Subject to x x i m
λθ
=
≤=…
∑
0
1
, 1, ,
n
j rj r
j
y yr s
λ
=
≥=…
∑
1
1,
n
j
j
λ
=
=
∑
0, 1, ,
jjn
λ
≥=…
where,
0
θ
is the value of the eciency of DMU
0
,
j
λ
is
the weighting of DMU
j
,
ij
x
is the fuzzy amount of resource
i consumed by DMU
j
,
0i
x
is the fuzzy amount of resource
i consumed by DMU
0
,
rj
y
is the fuzzy amount of output
r produced by DMU
j
,
0r
y
is the fuzzy amount of output r
produced by DMU
0
,
n
is the number of DMUs,
m
is the
number of resources, and
s
is the number of outputs.
Consequently, equation (3) presents the DEA model in its
version for fuzzy data analysis (León et al., 2003).
Figure 3: Graphical representaon of a triangular fuzzy set (a) and a trapezoidal fuzzy set (b)
0
min
h
T
P
θ
(3)
( ) ( )
00 0 0
11
: 1 1 , 1, ,
nn
j ij j ij i i
jj
subject to x h x h i m
λ λα θ θα
= =
−− ≤ −− =…
∑∑
( ) ( )
00 0 0
11
1 1 , 1, ,
nn
j ij j ij i i
jj
xh x hi m
λ λα θ θα
= =
+− ≤ +− =…
∑∑
( ) ( )
00
11
1 1 , 1, ,
nn
j rj j rj r r
jj
y h y hr s
λ λβ β
= =
−− ≥ −− =…
∑∑
( ) ( )
00
11
1 1 , 1, ,
nn
j rj j rj r r
jj
y h y hr s
λ λβ β
= =
+− ≥ +− =…
∑∑
1
1,
n
j
j
λ
=
=
∑
0, ,1h= …
0, 1, ,
jjn
λ
≥=…
ERIES Journal
volume 16 issue 4
Printed ISSN
2336-2375
305
Electronic ISSN
1803-1617
where,
ij
x
is the amount of resource i consumed by DMU
j
,
h
is the possibility level,
ij
α
is the alpha cut-o level
for resource i consumed by DMU
j
,
0i
x
is the amount of
resource i consumed by DMU
0
,
0i
α
alpha cut-o level for
resource i consumed by DMU 0,
rj
y
is the quantity of output r
produced by DMU
j
,
rj
β
is the betha cut-o level for output r
produced by DMU
j
,
0r
y
is the amount of output r produced
by DMU 0, and
0r
β
is the alpha cut-o level for output r
produced by DMU 0.
Machine Learning
Two machine learning algorithms are used to support this
research’s development: Random Forest and Logistic
Regression Boosted.
Random Forest
The Random Forest (RF) technique is a supervised machine-
learning model and is mainly used for classication (De La Hoz
et al., 2021). This model makes use of the democracy criterion,
which consists of the creation of multiple responses that will
be counted and the nal response is classied according to the
highest frequency (Louppe, 2014). On the other hand, the main
parameters of the RF technique are number of trees
( )
k and
number of variables needed to divide the nodes
( )
m
.
Logistic Regression
The Logistic Regression technique proposes the probability
ratio (odds). This is the ratio between success and failure in
a Bernoulli event. This algorithm predicts the probabilities of
success of the diverse levels of the response variable, using the
inverse of the logarithm of the probability ratio as a function of
the linear predictor.
Boosting Models
The algorithms belonging to the Boosting model family aim
to achieve robust and sophisticated predictions from a single
model. These algorithms train multiple weak models to generate
a robust nal model that feeds on information from the weak
models (Chen and Guestrin, 2016). This algorithm is also
known as a generic and non-specic algorithm, so it is crucial
to dene the base model (for example, DT, GLMNET, NB,
among others) and then it will be improved. This research will
apply Boosting to the Logistic Regression model (LogitBoost).
Methodology
The current research is divided into two stages (See Figure 4):
eciency analysis and predictive assessment. In the rst
stage, fuzzy data analysis is conducted using the technique
of Fuzzy Data Envelopment Analysis to estimate the relative
eciency of the Decision-Making Units. Then, in the second
stage, a predictive analysis of the eciency proles found
in the rst stage is designed. The results of these two stages
allow for generating useful information for decision-making in
educational environments.
Figure 4: Research methodology (own elaboraon)
Data
The data corresponds to the Mendeley’s repository
of the paper by Delahoz-Dominguez et al. (2020).
For the present research, 92 universities (DMUs) are
evaluated to summarise the results of the standardised
evaluations of high school (Saber 11 - inputs) and university
(Saber PRO – outputs) of 4,976 students of the Industrial
Engineering program in Colombia (See Table 3). It is
important to note that: rst, 57% of the institutions evaluated
in the database are private. Second, characteristics such as
size and age are not homogeneous. And nally, 13.27%
of the analysed universities are in socio-economic level 1
(low), 68.37% in level 2 (medium-low), 7.14% in level 3
(medium-high) and 11.22% in level 4 (high).
Variable Full name Test Average Deviaon
MAT_11 Math Saber 11 61.84 6.96
CR_11 Crical Reading Saber 11 58.83 5.10
CS_11 Cizenship skills Saber 11 58.93 5.11
BIO_11 Biology Saber 11 61.71 6.43
ENG_11 English Saber 11 58.67 7.50
QR_PRO Quantave Reasoning Saber PRO 73.45 12.33
CR_PRO Crical Reading Saber PRO 57.74 12.81
CS_PRO Cizenship skills Saber PRO 54.71 11.92
ENG_PRO English Saber PRO 62.46 14.72
WC_PRO Wring Communicaon Saber PRO 50.94 8.79
FEP_PRO Formulaon of Engineering Project Saber PRO 145.84 24.50
ACCP Academic Program - - -
Table 3: Data summary
Printed ISSN
2336-2375
306 ERIES Journal
volume 16 issue 4
Electronic ISSN
1803-1617
On the other hand, for the information analysis, the R software
is used (Coll-Serrano et al., 2018; R Core Team, 2013).
RESULTS
Stage 1: Eciency Analysis
As mentioned, the models used correspond to the two-scale
returns of the classic DEA model (CRS Constant, VRS
Variable) and scale performance (RTS = CRS/VRS). Table
4 presents the eciency results of the constant scale model;
Table 5 presents the eciency results of the variable scale
model and Table 6 presents the eciency results of scale
performance.
The tables mentioned (4, 5 and 6) contain the level of possibility
(h-level or alpha cut), the count of ecient DMUs (Count e)
and the percentage of ecient DMUs, the average (Mean),
standard deviation (SD), minimum value (min), quartile one,
two and three of the eciency levels of the DMUs.
Considering the above, Table 4 shows how level ℎ aects
eciency. As the
h
level increases, the number of ecient
DMUs, the average eciency level, the minimum eciency
value and the quartiles decrease.
On the other hand, although the eciency model with
variable scale return presents a similar behaviour as
the model with a constant scale, the eciency level is higher
(see Table 5).
Finally, the model scale performance results equal the constant
scale model. This indicates the diculty that some DMUs
could have in achieving the system’s overall eciency, so it
is necessary to generate strategies to increase the eciency of
these DMUs.
Consequently, Table 7 presents a non-random sample of the top
10 DMUs for the model with constant scale, variable scale,
and scale performance. Table 7 shows a similar eciency
behavior as in the summary tables (4, 5 and 6). For example,
for the model with constant scale, no DMU of the sample
has crisp eciency; that is, the DMU is always ecient for
the distinct levels of the possibility of
h
. On the other hand,
for the model with variable scale the DMUs U3, U4, U5, U6,
U9 and U10 have crisp eciency. Finally, the eciency of
the scale performance has results comparable to the model
with constant scaling; therefore, it does not have DMU with
crisp eciency. It should be noted that for the possibility level
0h=
, the eciency scores are always higher than those that
would be obtained in the conventional evaluation of the centers
of fuzzy triangular numbers (
1h=
).
h-level Count e Mean SD min Q1 Q2 Q3
0.000 68 (69%) 0.992 0.017 0.911 0.994 1.000 1.000
0.100 59 (60%) 0.990 0.019 0.903 0.990 1.000 1.000
0.200 57 (58%) 0.986 0.023 0.894 0.981 1.000 1.000
0.300 52 (53%) 0.982 0.027 0.883 0.972 1.000 1.000
0.400 43 (44%) 0.977 0.032 0.871 0.960 0.996 1.000
0.500 37 (38%) 0.970 0.038 0.859 0.950 0.990 1.000
0.600 36 (37%) 0.962 0.044 0.836 0.931 0.980 1.000
0.700 31 (32%) 0.953 0.051 0.803 0.913 0.971 1.000
0.800 29 (30%) 0.943 0.058 0.773 0.894 0.961 1.000
0.900 26 (27%) 0.932 0.065 0.745 0.877 0.949 1.000
1.000 20 (20%) 0.921 0.073 0.716 0.857 0.938 0.997
Table 4: Results of the eciency model with constant scale
h-level Count e Mean SD min Q1 Q2 Q3
0.000 85 (87%) 0.998 0.006 0.962 1.000 1.000 1.000
0.100 85 (87%) 0.998 0.007 0.957 1.000 1.000 1.000
0.200 82 (84%) 0.997 0.007 0.953 1.000 1.000 1.000
0.300 80 (82%) 0.997 0.008 0.948 1.000 1.000 1.000
0.400 77 (79%) 0.997 0.009 0.943 1.000 1.000 1.000
0.500 71 (72%) 0.996 0.009 0.939 0.999 1.000 1.000
0.600 68 (69%) 0.995 0.010 0.935 0.996 1.000 1.000
0.700 65 (66%) 0.994 0.011 0.931 0.993 1.000 1.000
0.800 63 (64%) 0.993 0.012 0.927 0.989 1.000 1.000
0.900 56 (57%) 0.992 0.013 0.923 0.988 1.000 1.000
1.000 52 (53%) 0.991 0.015 0.919 0.986 1.000 1.000
Table 5: Results of the eciency model with variable scale
ERIES Journal
volume 16 issue 4
Printed ISSN
2336-2375
307
Electronic ISSN
1803-1617
h-level Count e Mean SD min Q1 Q2 Q3
0.000 68 (69%) 0.994 0.014 0.927 0.999 1.000 1.000
0.100 59 (60%) 0.992 0.017 0.918 0.993 1.000 1.000
0.200 57 (58%) 0.989 0.021 0.910 0.989 1.000 1.000
0.300 52 (53%) 0.985 0.025 0.894 0.981 1.000 1.000
0.400 43 (44%) 0.980 0.030 0.879 0.969 0.998 1.000
0.500 37 (38%) 0.974 0.035 0.863 0.955 0.994 1.000
0.600 36 (37%) 0.966 0.042 0.836 0.937 0.984 1.000
0.700 31 (32%) 0.958 0.049 0.803 0.918 0.980 1.000
0.800 29 (30%) 0.949 0.056 0.773 0.900 0.975 1.000
0.900 26 (27%) 0.940 0.064 0.745 0.883 0.966 1.000
1.000 20 (20%) 0.929 0.072 0.716 0.866 0.953 0.998
Table 6: Model scale performance eciency results
CRS – Level of eciency
Level (h) U1 U2 U3 U4 U5 U6 U7 U8 U9 U10
0.000 1.000 1.000 0.956 0.961 1.000 0.999 0.974 0.911 0.962 0.981
0.100 0.999 1.000 0.949 0.948 1.000 0.993 0.954 0.903 0.955 0.974
0.200 0.989 1.000 0.941 0.934 1.000 0.982 0.935 0.894 0.950 0.968
0.300 0.977 1.000 0.930 0.921 1.000 0.968 0.922 0.883 0.939 0.960
0.400 0.957 0.997 0.912 0.901 0.992 0.955 0.912 0.871 0.928 0.952
0.500 0.938 0.990 0.892 0.870 0.982 0.941 0.900 0.859 0.917 0.942
0.600 0.922 0.983 0.871 0.836 0.964 0.928 0.886 0.842 0.902 0.929
0.700 0.908 0.976 0.849 0.803 0.940 0.913 0.872 0.823 0.884 0.914
0.800 0.892 0.968 0.826 0.773 0.918 0.898 0.855 0.803 0.861 0.898
0.900 0.873 0.957 0.804 0.745 0.896 0.880 0.836 0.782 0.838 0.882
1.000 0.854 0.943 0.781 0.716 0.874 0.859 0.817 0.761 0.816 0.864
VRS - Level of eciency
Level (h) U1 U2 U3 U4 U5 U6 U7 U8 U9 U10
0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 0.983 1.000 1.000
0.100 1.000 1.000 1.000 1.000 1.000 1.000 1.000 0.983 1.000 1.000
0.200 1.000 1.000 1.000 1.000 1.000 1.000 0.999 0.982 1.000 1.000
0.300 1.000 1.000 1.000 1.000 1.000 1.000 0.997 0.981 1.000 1.000
0.400 1.000 1.000 1.000 1.000 1.000 1.000 0.994 0.980 1.000 1.000
0.500 1.000 1.000 1.000 1.000 1.000 1.000 0.991 0.980 1.000 1.000
0.600 1.000 1.000 1.000 1.000 1.000 1.000 0.989 0.979 1.000 1.000
0.700 1.000 0.998 1.000 1.000 1.000 1.000 0.986 0.978 1.000 1.000
0.800 0.998 0.994 1.000 1.000 1.000 1.000 0.983 0.976 1.000 1.000
0.900 0.994 0.991 1.000 1.000 1.000 0.999 0.980 0.975 1.000 1.000
1.000 0.990 0.987 1.000 0.999 1.000 0.999 0.976 0.973 1.000 0.999
RTS - Level of eciency
Level (h) U1 U2 U3 U4 U5 U6 U7 U8 U9 U10
0.000 1.000 1.000 0.956 0.961 1.000 0.999 0.974 0.927 0.962 0.981
0.100 0.999 1.000 0.949 0.948 1.000 0.993 0.954 0.918 0.955 0.974
0.200 0.989 1.000 0.941 0.934 1.000 0.982 0.935 0.910 0.950 0.968
0.300 0.977 1.000 0.930 0.921 1.000 0.968 0.925 0.900 0.939 0.960
0.400 0.957 0.997 0.912 0.901 0.992 0.955 0.917 0.889 0.928 0.952
0.500 0.938 0.990 0.892 0.870 0.982 0.941 0.908 0.877 0.917 0.942
0.600 0.922 0.983 0.871 0.836 0.964 0.928 0.897 0.861 0.902 0.929
0.700 0.908 0.978 0.849 0.803 0.940 0.913 0.884 0.842 0.884 0.914
0.800 0.894 0.973 0.826 0.773 0.918 0.898 0.870 0.823 0.861 0.898
0.900 0.879 0.966 0.804 0.745 0.896 0.881 0.854 0.803 0.838 0.882
1.000 0.863 0.955 0.781 0.716 0.874 0.860 0.837 0.782 0.816 0.865
Table 7: Sample eciency result for 10 DMUs
Printed ISSN
2336-2375
308 ERIES Journal
volume 16 issue 4
Electronic ISSN
1803-1617
Table 8 also generates a concept called fuzzy set of eective
units. In this sense, a fuzzy set is represented as the name of
the DMU and the value of the maximum level
h
with which
the DMU is still ecient, for example, for the model with
constant scale the DMU U1 is ecient for the values
h
equal
to 0, 0.1 and 0.2, then the set is (U1, 0.2).
Model Eecve diuse assembly
CRS (U1, 0.2), (U2, 0.4), (U5, 0.3), (U6, 0)
VRS (U1, 0.8), (U2, 0.7), (U3, 1), (U4, 1), (U5,1), (U6, 1), (U7, 0.3), (U9, 1), (U10, 1)
RTS (U1, 0.1), (U2, 0.4), (U5, 0.3), (U6, 0.1)
Table 8: Fuzzy set of eecve units for 10 DMUs
On the other hand, the advantage of the model is the creation
of an ecient route (see Table 9), the path that a non-ecient
DMU must follow to become ecient and reach the maximum
level of eciency projected for its group. Two ecient
routes were created that correspond to the low-medium
eciency levels (range between the 0th percentile and the 66th
percentile of eciency) and high eciency (range between
the 67th percentile and the 100th percentile of eciency). For
the development of the two routes, all the non-ecient DMUs
of the model with constant scale were compared and grouped
by eciency level. Then, the score value of the references
between DMUs of the model (lambdas) was observed and
ordered from lowest to highest. Finally, DMU sequences were
selected more frequently.
Name group Eciency path Eciency level
Path 1 U61 - U48 - U45 [0 - 0.94]
Path 2 U39 - U48 - U69 (0.94 - 1]
Table 9: Ecient paths
The ecient routes are composed of the DMUs of Table 10,
each route has an expected increase from the competencies
of Saber 11 to the competencies of Saber PRO (Di). For
example, path 1 generates a 14.7% increase in learning
outcomes from Saber 11 to Saber PRO. It should be noted
that the increase must be gradual, that is, it must rst reach
the eciency of the rst DMU of the route, then the second
DMU and so, until reaching the last DMU of the route,
consequently, the DMU that passes through the path will
be ecient.
Finally, this section presents the analysis of two population
variables: type of institution and socio-economic level.
Table 11 presents a summary of the eciency of public and
private institutions.
Similarly, Table 12 shows the eciency analysis according
to the universities’ socio-economic level.
Path DMU Saber 11 Saber PRO Di
MAT CR CC ENG BIO Mean QR CR CC ENG WC Mean
1
U61 72.96 67.69 67.68 65.50 72.05 77.67 91.53 81.41 77.19 78.72 59.50 69.18 10.9%
U48 61.88 59.52 61.02 59.74 61.52 70.38 89.32 70.22 55.18 69.94 67.22 60.74 13.7%
U45 66.08 63.65 62.61 71.69 66.53 77.51 81.07 70.46 71.82 86.75 77.43 66.11 14.7%
2
U39 68.83 64.12 64.36 63.52 66.79 74.50 92.48 74.87 70.96 75.56 58.61 65.53 12.0%
U48 61.88 59.52 61.02 59.74 61.52 70.38 89.32 70.22 55.18 69.94 67.22 60.74 13.7%
U69 70.04 65.08 63.67 70.86 68.63 79.28 86.87 74.97 77.97 85.08 71.52 67.65 14.7%
Table 10: Characterisaon of ecient paths
University Count e Mean Standard deviant
CRS VRS RTS CRS VRS RTS CRS VRS RTS
Private 13 27 13 0.935 0.990 0.944 0.068 0.013 0.065
Public 7 25 7 0.902 0.991 0.910 0.076 0.017 0.076
Table 11: Descripon of the eciency of public and private universies
Socio-economic
level
Count e Mean Standard deviant
CRS VRS RTS CRS VRS RTS CRS VRS RTS
L1 2 6 2 0.916 0.985 0.930 0.071 0.019 0.073
L2 10 36 10 0.904 0.991 0.912 0.073 0.015 0.071
L3 2 3 2 0.977 0.993 0.984 0.020 0.007 0.018
L4 6 7 6 0.994 0.996 0.998 0.010 0.007 0.005
Table 12: Descripon of the eciency of the university’s socio-economic levels
ERIES Journal
volume 16 issue 4
Printed ISSN
2336-2375
309
Electronic ISSN
1803-1617
Stage 2: Prediction Analysis
Finally, this stage seeks to suggest a model for predictive
evaluation for non-ecient universities in the group analysed.
In this sense, the route universities must follow to achieve
maximum eciency is established as a response variable, on
the other hand, as predictor variables, the academic competencies
of the Saber 11 evaluation and the training program are selected.
The construction of the model consists of two stages: training and
evaluation. The data is divided into two groups, corresponding
to 70% for training and 30% for evaluation. In summary, two
models are used for the training phase: Random Forest and
LogitBoost. In addition, the cross-validation technique with
10 folds is used in this phase. The results show that the best-
performing model is Random Forest (see Table 13).
Model Metric AUC Accuracy F1 Sensivity Specicity
Random Forest Mean 0.641 0.650 0.725 0.892 0.600
SD 0.157 0.093 0.072 0.142 0.274
LogitBoost Mean 0.593 0.571 0.684 0.883 0.300
SD 0.146 0.145 0.114 0.153 0.222
Table 13: Results of model training
Then, the models are evaluated with 30% of the study
population, and their results are benchmarked. However, as in
the training phase, in the evaluation phase, it is observed that
the Random Forest model performs better (see Table 14).
Model AUC Accuracy F1 Sensivity Specicity
Random Forest 0.710 0.700 0.727 0.667 0.800
LogitBoost 0.570 0.577 0.649 0.545 0.800
Table 14: Results of model tesng
Finally, to generate additional information to understand
the model with the best performance, Table 15 is constructed.
Table 15 shows the importance of the variables of the Random
Forest model. It is possible to identify that the variable with
greater weight is the academic program, followed by English,
Mathematics, Biology, Citizenship Skills, and Critical Reading.
Variable Weight Variable Weight
ACCP 0.035 ENG_11 0.025
MAT_11 0.001 CR_11 0.000
BIO_11 0.000 CS_11 0.000
Table 15: Importance of the variables of the Random Forest model
DISCUSSION
Data Envelopment Analysis using fuzzy data oers an
interesting approach for creating decision-making tools in
the educational eld. First, a signicant advantage of this
tool is its ability to incorporate uncertainty when formulating
the evaluation model. Moreover, the results allow for analysing
eciency level changes concerning the decision variable
- results not provided by a classical DEA model. In other
words, if there is a substantial change from one level
h
of
measurement to another
1h+
, then it can be asserted that
the evaluated Decision-Making Unit (DMU) is sensitive to
the measurement variable. This could be a persuasive argument
for using the fuzzy approach to evaluate education quality
using DEA models. It should be noted that it is essential to
understand the context to adapt the model to the situation.
On the other hand, multiple eciency measures allow for
the creation of various alternatives within an action framework.
That is, decision-makers can establish an
h
level for a student’s
academic competencies and then observe the eciency level
and its ecient path (if it is not already ecient). In this vein, one
could know a student’s eciency level in advance to create an
action plan that improves their level of academic competencies
and, consequently, the eciency of the university.
According to the research results, variations in competency
levels cause signicant dierences in educational institutions’
eciency. Consequently, the eciency level of a student’s
basic competencies greatly impacts the university’s eciency
level. In other words, even if a university has an excellent
training program, the student’s competency level can be critical
and decisive in determining the university’s eciency.
The ndings on the economic aspect analysed complement this.
For example, in the present analysis, the socio-economic level of
the university is presented as a factor that has a small impact on
university academic eciency. Also, the diversity in eciency
within each socio-economic level suggests that institution-
specic strategies, beyond their economic context, are crucial
to achieving eciency in higher education. And nally,
the consistent eciency in specic academic programmes
indicates that the focus and quality of educational provision
may be more critical than socio-economic status. Considering
the above, it is necessary to generate crisply ecient DMUs,
meaning that a DMU can be ecient at any level of academic
competencies. This implies that higher educational institutions
should have a prior plan that contributes to raising the level of
academic competencies, not just for the university’s eciency
level but also because a student’s academic performance
signicantly determines their future professional performance.
Additionally, it is necessary to compare the present research
with similar works. For example, the research by Nazari-
Shirkouhi et al. (2020) develops a tool for evaluating academic
Printed ISSN
2336-2375
310 ERIES Journal
volume 16 issue 4
Electronic ISSN
1803-1617
performance based on an integrated fuzzy multicriteria
decision-making approach. Unlike our research, Nazari-
Shirkouhi et al. (2020) emphasise using the Fuzzy Decision-
Making Trial and Evaluation Laboratory and Fuzzy Analytic
Network Process tools to determine the indicators’ weight
for the model. This creates a robust framework for variable
selection and model construction. In contrast, the research
by Contreras et al. (2020) implemented classication models
(decision tree, KNN, and perceptron) to predict academic
performance. A dierentiating point in Contreras et al.’s
research is the use of data mining methodology for predicting
academic performance; however, failing to consider the fuzzy
aspect of information could be a weakness.
Similarly, Valdés Pasarón et al. (2018) research develops
an empirical model combining qualitative and quantitative
characteristics about the education system to estimate
education quality. A point in favor of Valdés Pasarón et al.’s
research is the addition of qualitative variables to provide more
information for training models using the fuzzy approach. On
the other hand, the research by Lee et al. (2019) constructs
a model for evaluating and analysing e-learning systems
through a matrix. In Lee et al.’s research, a dierentiating
point is avoiding the problem of potential sampling errors
and the complexity of collecting fuzzy linguistic data through
evaluative matrix systems.
Lastly, it should be noted that this model does not require
expensive or specialised software, but can be implemented
using standard DEA or linear programming packages. This
could greatly assist researchers who are just starting to develop
eciency models.
CONCLUSION
The present research aimed to design a tool for educational
management in a context of uncertainty. To accomplish this,
we utilised Data Envelopment Analysis methodology within
a framework of uncertainty represented by fuzzy inputs.
The research provided a new perspective on evaluating
quality in education using DEA models. The designed tool
successfully identies an “ecient path” consisting of
universities with standard or ideal eciency levels, serving
as a reference point for universities identied as inecient to
nd a path or goal towards increased eciency. A crucial point
in this development is that uncertainty is inherent in every
process within the service and production areas. Therefore,
the foundation of this research adapts classical DEA models
into equivalent “crisp” linear programming formulations.
In addition, the ndings show that there is a representation of
both public and private ecient universities, with a slightly
higher percentage of private universities; however, there is
no clear trend indicating that one type of institution (public or
private) is more ecient than the other in terms of the academic
programmes evaluated. Additionally, some academic
programmes, such as Electronic Engineering, Chemical
Engineering, Civil Engineering, Mechanical Engineering,
and Industrial Engineering, consistently stand out in terms of
eciency, regardless of socio-economic level.
Lastly, this research broadens the scope of knowledge to models
that analyse the quality level in education, providing a tool for
predictive evaluation under a fuzzy approach. Additionally,
future research will consider incorporating Machine Learning
models into eciency evaluation with fuzzy data.
REFERENCES
Abelson, J., Forest, P.-G., Eyles, J., Smith, P., Martin, E. and Gauvin,
F.-P. (2003) ‘Deliberations about deliberative methods: issues
in the design and evaluation of public participation processes’,
Social Science & Medicine, Vol. 57, No. 2, pp. 239–251. https://
doi.org/10.1016/S0277-9536(02)00343-X
Acosta, O. and Celis, J. (2014) ‘The emergence of doctoral
programmes in the Colombian higher education system: Trends
and challenges’, PROSPECTS, Vol. 44, No. 3, pp. 463–481.
https://doi.org/10.1007/s11125-014-9310-5
Agasisti, T., Munda, G. and Hippe, R. (2019) ‘Measuring the
eciency of European education systems by combining Data
Envelopment Analysis and Multiple-Criteria Evaluation’,
Journal of Productivity Analysis, Vol. 51, No. 2, pp. 105–124.
https://doi.org/10.1007/s11123-019-00549-6
Alabdulmenem, F. M. (2016) ‘Measuring the Eciency of Public
Universities: Using Data Envelopment Analysis (DEA) to
Examine Public Universities in Saudi Arabia’, International
Education Studies, Vol. 10, No. 1, pp. 137–143. https://doi.
org/10.5539/ies.v10n1p137
Altbach, P. G., Reisberg, L. and Rumbley, L. E. (2009) Trends in Global
Higher Education: Tracking an Academic Revolution, Trends
in global higher education: tracking an academic revolution; a
report prepared for the UNESCO 2009 World Conference on
Higher Education, Paris, [Online], Available: https://unesdoc.
unesco.org/ark:/48223/pf0000183219 [16 Oct 2023]
Aparicio, J., Cordero, J. M. and Ortiz, L. (2021) ‘Eciency Analysis
with Educational Data: How to Deal with Plausible Values from
International Large-Scale Assessments’, Mathematics, Vol. 9,
No. 13, 1579. https://doi.org/10.3390/math9131579
Aparicio, J., Cordero, J. M. and Ortiz, L. (2019) ‘Measuring
efficiency in education: The influence of imprecision
and variability in data on DEA estimates’, Socio-
Economic Planning Sciences, Vol. 68, 100698. https://doi.
org/10.1016/j.seps.2019.03.004
Avelar, A. B. A., da Silva-Oliveira, K. D. and da Silva Pereira, R.
(2019) ‘Education for advancing the implementation of the
Sustainable Development Goals: A systematic approach’,
The International Journal of Management Education,
Vol. 17, No. 3, 100322. https://doi.org/10.1016/j.
ijme.2019.100322
Barr, A. and Turner, S. E. (2013) ‘Expanding Enrollments and
Contracting State Budgets: The Effect of the Great Recession
on Higher Education’, The ANNALS of the American
Academy of Political and Social Science, Vol. 650, No. 1,
pp. 168–193. https://doi.org/10.1177/0002716213500035
Bianchi, N. and Giorcelli, M. (2020) ‘Scientific Education and
Innovation: From Technical Diplomas to University Stem
Degrees’, Journal of the European Economic Association,
Vol. 18, No. 5, pp. 2608–2646. https://doi.org/10.1093/
jeea/jvz049
ERIES Journal
volume 16 issue 4
Printed ISSN
2336-2375
311
Electronic ISSN
1803-1617
Cars, M. and West, E. E. (2015) ‘Education for sustainable society:
attainments and good practices in Sweden during the United
Nations Decade for Education for Sustainable Development
(UNDESD)’, Environment, Development and Sustainability,
Vol. 17, No. 1, pp. 1–21. https://doi.org/10.1007/s10668-014-
9537-6
Castro, R. (2019) ‘Blended learning in higher education: Trends and
capabilities’, Education and Information Technologies, Vol.
24, No. 4, pp. 2523–2546. https://doi.org/10.1007/s10639-019-
09886-3
Chankseliani, M. and McCowan, T. (2021) ‘Higher education
and the Sustainable Development Goals’, Higher Education,
Vol. 81, No. 1, pp. 1–8. https://doi.org/10.1007/s10734-020-
00652-w
Charnes, A., Cooper, W. W. and Rhodes, E. (1978) ‘Measuring
the eciency of decision making units’, European Journal of
Operational Research, Vol. 2, No. 6, pp. 429–444. https://doi.
org/10.1016/0377-2217(78)90138-8
Chen, T. and Guestrin, C. (2016) XGBoost: A Scalable Tree
Boosting System, In Proceedings of the 22nd ACM SIGKDD
International Conference on Knowledge Discovery and Data
Mining - KDD ‘16 (pp. 785–794), San Francisco, California,
USA: ACM Press. https://doi.org/10.1145/2939672.2939785
Coll-Serrano, V., Bolos, V. and Benitez Suarez, R. (2018) deaR:
Conventional and Fuzzy Data Envelopment Analysis, (Version
1.4.1), España: Universidad de Valencia, [Software], available:
https://CRAN.R-project.org/package=deaR [15 April 2023]
Contreras, L. E., Fuentes, H. J. and Rodríguez, J. I. (2020) ‘Predicción
del rendimiento académico como indicador de éxito/fracaso de
los estudiantes de ingeniería, mediante aprendizaje automático’,
Formación Universitaria, Vol. 13, No. 5, pp. 233–246. https://
doi.org/10.4067/S0718-50062020000500233
Corlu, M. A and Aydin, E. (2016) ‘Evaluation of Learning Gains
Through Integrated STEM Projects’, International Journal of
Education in Mathematics, Science and Technology, Vol. 4, No.
1, pp. 20–29. https://dx.doi.org/10.18404/ijemst.35021
De La Hoz, E., Zuluaga, R. and Mendoza, A. (2021) ‘Assessing
and Classication of Academic Eciency in Engineering
Teaching Programs’, Journal on Eciency and Responsibility
in Education and Science, Vol. 14, No. 1, pp. 41–52. https://doi.
org/10.7160/eriesj.2021.140104
Delahoz-Dominguez, E., Zuluaga, R. and Fontalvo-Herrera,
T. (2020) ‘Dataset of academic performance evolution for
engineering students’, Data in Brief, Vol. 30, 105537. https://
doi.org/10.1016/j.dib.2020.105537
Do, Q. H. and Chen, J.-F. (2014) ‘A hybrid fuzzy AHP-DEA approach
for assessing university performance’, WSEAS Transactions on
Business and Economics, Vol. 11, pp. 386–397.
Duque Oliva, E. J. and Chaparro Pinzón, C. R. (2012) ‘Medición
de la percepción de la calidad del servicio de educación por
parte de los estudiantes de la uptc duitama’, Criterio Libre, Vol.
10, No. 16, pp. 159–192. https://doi.org/10.18041/1900-0642/
criteriolibre.2012v10n16.1168
Ferrer-Estévez, M. and Chalmeta, R. (2021) ‘Integrating
Sustainable Development Goals in educational institutions’,
The International Journal of Management Education, Vol. 19,
No. 2, 100494. https://doi.org/10.1016/j.ijme.2021.100494
Font, X. (2002) ‘Environmental certication in tourism and
hospitality: progress, process and prospects’, Tourism
Management, Vol. 23, No. 3, pp. 197–205. https://doi.
org/10.1016/S0261-5177(01)00084-X
Galbraith, C. S. and Merrill, G. B. (2015) ‘Academic performance and
burnout: an ecient frontier analysis of resource use eciency
among employed university students’, Journal of Further and
Higher Education, Vol. 39, No. 2, pp. 255–277. https://doi.org/
10.1080/0309877X.2013.858673
Gamboa, L. F., Casas, A. F. and Piñeros, L. J. (2003) ‘La teoría del
valor agregado: una aproximación a la calidad de la educación
en Colombia’, Revista de Economía del Rosario, Vol. 6, No. 2,
pp. 95–116.
Hoeg, D. G. and Bencze, J. L. (2017) ‘Values Underpinning STEM
Education in the USA: An Analysis of the Next Generation
Science Standards: VALUES UNDERPINNING STEM
EDUCATION’, Science Education, Vol. 101, No. 2, pp. 278–
301. https://doi.org/10.1002/sce.21260
ICFES (2022) Resultados de la evaluación Saber PRO [Results of the
Saber PRO evaluation], Instituto Colombiano para la Evaluación
de la Educación, [Online], Available: https://www.icfes.gov.co/
web/guest/acerca-del-examen-saber-pro [26 Oct 2023]
Johnes, J. (2006) ‘Data envelopment analysis and its application to
the measurement of eciency in higher education’, Economics
of Education Review, Vol. 25, No. 3, pp. 273–288. https://doi.
org/10.1016/j.econedurev.2005.02.005
Kalapouti, K., Petridis, K., Malesios, C. and Dey, P. K. (2020) ‘Measuring
eciency of innovation using combined Data Envelopment
Analysis and Structural Equation Modeling: empirical study in EU
regions’, Annals of Operations Research, Vol. 294, pp. 297–320.
https://doi.org/10.1007/s10479-017-2728-4
Kopnina, H. (2020) ‘Education for the future? Critical evaluation of
education for sustainable development goals’, The Journal of
Environmental Education, Vol. 51, No. 4, pp. 280–291. https://
doi.org/10.1080/00958964.2019.1710444
Lee, T.-S., Wang, C.-H. and Yu, C.-M. (2019) ‘Fuzzy Evaluation
Model for Enhancing E-Learning Systems’, Mathematics, Vol. 7,
No. 10, 918. https://doi.org/10.3390/math7100918
León, T., Liern, V., Ruiz, J. L. and Sirvent, I. (2003) ‘A fuzzy
mathematical programming approach to the assessment of
eciency with DEA models’, Fuzzy Sets and Systems, Vol.
139, No. 2, pp. 407–419. https://doi.org/10.1016/S0165-
0114(02)00608-5
Liu, S.-T. and Chuang, M. (2009) ‘Fuzzy eciency measures in fuzzy
DEA/AR with application to university libraries’, Expert Systems
with Applications, Vol. 36, No. 2 (Part 1), pp. 1105–1113. https://
doi.org/10.1016/j.eswa.2007.10.013
Louppe, G. (2014) Understanding Random Forests: From Theory to
Practice, [PhD thesis], Ithaca, NY: Cornell University. https://
doi.org/10.48550/arXiv.1407.7502
Mirasol-Cavero, D. B. and Ocampo, L. (2021) ‘Fuzzy preference
programming formulation in data envelopment analysis for
university department evaluation’, Journal of Modelling
in Management, Vol. 18, No. 1, pp. 212–238. https://doi.
org/10.1108/JM2-08-2020-0205
Navas, L. P., Montes, F., Abolghasem, S., Salas, R. J., Toloo, M. and
Zarama, R. (2020) ‘Colombian higher education institutions
evaluation’, Socio-Economic Planning Sciences, Vol. 71,
100801. https://doi.org/10.1016/j.seps.2020.100801
Nazari-Shirkouhi, S., Mousakhani, S., Tavakoli, M., Dalvand, M.
R., Šaparauskas, J. and Antuchevičienė, J. (2020) ‘Importance-
performance analysis based balanced scorecard for performance
evaluation in higher education institutions: an integrated fuzzy
approach’, Journal of Business Economics and Management, Vol.
21, No. 3, pp. 647–678. https://doi.org/10.3846/jbem.2020.11940
Printed ISSN
2336-2375
312 ERIES Journal
volume 16 issue 4
Electronic ISSN
1803-1617
Nazarko, J. and Šaparauskas, J. (2014) ‘Application of DEA method
in eciency evaluation of public Higher Education Institutions’,
Technological and Economic Development of Economy, Vol. 20,
No. 1, pp. 25–44. https://doi.org/10.3846/20294913.2014.837116
Nojavan, M., Heidari, A. and Mohammaditabar, D. (2021) ‘A fuzzy
service quality based approach for performance evaluation of
educational units’, Socio-Economic Planning Sciences, Vol. 73,
100816. https://doi.org/10.1016/j.seps.2020.100816
Ntshoe, I. and Letseka, M. (2010) Quality Assurance and Global
Competitiveness in Higher Education, In L. M. Portnoi, V. D. Rust,
& S. S. Bagley (Eds.), Higher Education, Policy, and the Global
Competition Phenomenon (pp. 59–71), New York: Palgrave
Macmillan US. https://doi.org/10.1057/9780230106130_5
OECD (2019) Publications - PISA, Organisation for Economic Co-
operation and Development [Online], Available: https://www.
oecd.org/pisa/publications/pisa-2018-results.htm [4 Dec 2019].
Pérez, Á. (2019) ¿Por qué la calidad de la educación en Colombia
no es buena?, Semana, [Online], Available: https://www.
dinero.com/opinion/columnistas/articulo/por-que-la-calidad-
de-la-educacion-en-colombia-no-es-buena-por-angel-perez-
martinez/268998 [19 Nov 2019].
Quintero Caro, O. L. (2018) Efectos de la acreditación de alta calidad en
el valor agregado de la educación superior, [Master tesis], Bogotá:
Facultad de Ciencias Económicas y Administrativas, Ponticia
Universidad Javeriana. https://doi.org/10.11144/Javeriana.10554.38960
R Core Team (2013) R: A language and environment for statistical
computing, R Foundation for Statistical Computing, Vienna, Austria,
[Software], available: https://www.r-project.org/ [15 Jul 2023]
Rodríguez, M. and Huertas, Y. (2016) ‘Metodología para el Diseño
de Conjuntos Difusos Tipo-2 a partir de Opiniones de Expertos’,
Ingeniería, Vol. 21, No. 2, pp. 121–137. https://doi.org/10.14483/
udistrital.jour.reving.2016.2.a01
Santos, G., Marques, C. S., Justino, E. and Mendes, L. (2020)
‘Understanding social responsibility’s inuence on service
quality and student satisfaction in higher education’, Journal
of Cleaner Production, Vol. 256, 120597. https://doi.
org/10.1016/j.jclepro.2020.120597
Sharma, P. N., Shmueli, G., Sarstedt, M., Danks, N. and Ray, S.
(2018) ‘Prediction-Oriented Model Selection in Partial Least
Squares Path Modeling’, Decision Sciences, Vol. 52, No. 3,
pp. 567–607. https://doi.org/10.1111/deci.12329
Shriberg, M. (2002) ‘Institutional assessment tools for
sustainability in higher education: strengths, weaknesses,
and implications for practice and theory’, Higher Education
Policy, Vol. 15, No. 2, pp. 153–167. https://doi.org/10.1016/
S0952-8733(02)00006-5
Singh, A. P., Yadav, S. P. and Singh, S. K. (2022) ‘A multi-
objective optimisation approach for DEA models in a fuzzy
environment’, Soft Computing, Vol. 26, No. 6, pp. 2901–2912.
https://doi.org/10.1007/s00500-021-06627-y
Valdés Pasarón, S., Ocegueda Hernández, J. M. and Romero
Gómez, A. (2018) ‘La calidad de la educación y su relación
con los niveles de crecimiento económico en México’,
Economía y Desarrollo, Vol. 159, No. 1, pp. 61–79.
Visbal-Cadavid, D., Martínez-Gómez, M. and Guijarro, F. (2017)
‘Assessing the Eciency of Public Universities through DEA.
A Case Study’, Sustainability, Vol. 9, No. 8, 1416. https://doi.
org/10.3390/su9081416
Wolszczak-Derlacz, J. (2017) ‘An evaluation and explanation of
(in)eciency in higher education institutions in Europe and
the U.S. with the application of two-stage semi-parametric
DEA’, Research Policy, Vol. 46, No. 9, pp. 1595–1605. https://
doi.org/10.1016/j.respol.2017.07.010
