Content uploaded by Francisco José García-Peñalvo
Author content
All content in this area was uploaded by Francisco José García-Peñalvo on Dec 14, 2015
Content may be subject to copyright.
1
Monitoring Indicators for CTMTC: Comprehensive Training Model of the Teamwork Competence in
Engineering Domain
Á. FIDALGO-BLANCO
Laboratory of Innovation in Information Technologies. LITI. Polythecnical University of Madrid.
Calle de Ríos Rosas 21, 28003 Madrid, Spain.
E-mail: afidalgo@dmami.upm.es
D. LERÍS
Department of Applied Mathematics, University of Zaragoza, Campus Rio Ebro, University of Zaragoza.
María de Luna 3, 50018 Zaragoza, Spain.
E-mail: dleris@unizar.es
M. L. SEIN-ECHALUCE
Department of Applied Mathematics, University of Zaragoza, Campus Rio Ebro, University of Zaragoza.
María de Luna 3, 50018 Zaragoza, Spain.
E-mail: mlsein@unizar.es
F. J. GARCÍA-PEÑALVO
Research Institute for Educational Sciences. University of Salamanca
Paseo Canalejas 169, 37008 Salamanca, Spain.
E-mail: fgr.iuce@gmail.com
Abstract
This paper proofs three initial hypotheses in relation with the teamwork competence, Firstly, the new university
students access to the University without enough knowledge about teamwork competence because they have not been
trained or evaluated in this competence. This justifies the use of our Comprehensive Training Model of the Teamwork
Competence (CTMTC) method, which allows individualized training and assessment of teamwork competence in all
the stages, including the outcome of this teamwork too. Secondly, the realised fieldwork for obtaining indicators that
allow evaluate the required effort in order to use CTMTC, focusing on aspects of monitoring (formative evaluation) and
outcomes (summative evaluation) and also the enormous effort in time for the teachers to carry it out, are shown.
Thirdly, the final evaluation grades obtained by the teams and by their individual members, with the CTMTC method,
justify the need of applying that personalized evaluation method and the search of technological tools, such as learning
analytics, to help its implementation.
Keywords: Teamwork; Competences; Learning Analytics; White-box testing
1. INTRODUCTION
Teamwork is one of the most commonly required competencies that employees ask for (see [1-3]), as well as being an
educational dexterity that is widely used in teaching processes of any educational level. The teamwork competency
(hereafter the TWC) is one that is included in the majority of teaching syllabi for university subjects; being a learning
objective for all university studies in Europe.
Nevertheless, the implementation of teamwork in an academic environment is normally carried out by means of the
“black-box testing” model [4] (similar idea to the method of software testing). This means, that normally instead of
evaluating the TWC itself, only the result obtained from the work is evaluated, based on the quality of it once
completed. In this case, the students are left to develop the TWC simply by being exposed to or immersed in a team.
On the other hand, another reason why the “black-box testing” model is normally used in universities is to recognise
that students gain admission to university degree studies with the acquired TWC, given that they are used to carrying
out teamwork in a non-university learning context. In this article, we present the study undertaken regarding the
acquiring of the teamwork competency in a non-university learning context, with which we demonstrate that students
reach university without the TWC skills themselves and without having been evaluated by teaching staff.
There are currently research studies and models in which methodologies have been developed in order to create and
evaluate the TWC (in addition to the result itself). In this sense, our research team developed the model known as
Comprehensive Training Model of the Teamwork Competence (hereafter the CTMTC) ([5]) which allows a strategy to
be carried out that is based on the “white-box testing”. This means, it involves a model that consists of three layers: the
conceptual layer, which structures and describes the dimensions of the operative TWC in the context of education, in
addition to the methodological and technological layers, which cover the procedure and resources in order to carry out a
continual monitoring of the teamwork. Educational evaluations can therefore be produced in order to guarantee both the
teamwork competency being acquired and the quality of the result. [6].
PRE-PRINT
2
Throughout the work process [7], a table of factors is included that increase the efficiency of the group (equal
distribution of the work, organization of the projects, etc.) and therefore it can be demonstrated that our method creates
and evaluates aspects that have been demonstrated to be useful in order to increase the efficiency of the group.
The application of “white-box testing” methods is highly effective in the development of the TWC, however they
require a great amount of effort on behalf of the teaching staff. Subsequently, their application is practically relegated to
those subjects in which all of the assessment criteria are attributed to teamwork. Obviously, if the TWC represents a
small part of the overall grade given for the subject, it is to be expected that both the teachers and students dedicate time
and effort that correspond with the part attributable to the overall grade; and of course, if this is the case, to carry out an
assessment of the teamwork to determine if it is effective and influential or not for the teaching staff with common
methods
The widespread use of the “white-box testing” methods can become a reality thanks to ‘Learning Analytics’, the
objective of which is to analyse the data that is produced on behalf of the student in the learning processes. This data is
used to understand and optimize the learning process in the environment in which it takes place [8].
The problem is not having data available, as this can be gathered from the learning management system used, but rather
distinguishing which of the data is relevant and deciding how to use it so that it helps with the decision making. As is
supported by G. Siemens [9], the objectives of analysing the activity of trainees should be to address the problems that
are important for the teachers. In this sense, the work that we present provides an identification of relevant data which,
after being managed and processed by a Learning Analytics system, will allow the process of decision making, both for
the educational evaluation (monitoring of the TWC) to be carried out and the overall evaluation (result of the work and
the TWC). The benefit of using technology to carry out the monitoring of the TWC is demonstrated, and not only in
terms of collaborative and communicative aspects that are particularly fundamental in teamwork [10]..
2. CONTEXT
This research has been undertaken by the Polytechnic University of Madrid (hereafter the UPM) with the participation of
304 students from different university degrees and 82 students from the University of Zaragoza (hereafter the UNIZAR).
In total, 55 work groups have participated in the UPM and 22 groups in UNIZAR made up of six students and four
respectively, for whom data has been gathered and a monitoring process carried out.
The starting premise is to recognise that the TWC is important from a social and professional point of view and the
hypotheses or objectives to be demonstrated for this investigation are as follows:
• Hypothesis 1: “Students start university studies without having been taught or evaluated in the TWC”
• Hypothesis 2: “The “white-box testing” education of the TWC requires a large investment in terms of time”
• Hypothesis 3: “The development and evaluation of the TWC needs its educational process to be of the “white-box
testing” type”.
It is important to demonstrate hypothesis 1, given that a large number of university teaching staff expect students to have
already acquired the TWC, as a result of having carried out teamwork in their previous educational stages. Therefore, it
means that it is not specifically necessary to teach the TWC as part of the learning program. As is outlined in hypothesis
2, carrying out a daily supervision of each work group requires an educational and evaluation model, but this would
require the teaching staff to be very dedicated, in order to oversee both the monitoring process and the educational
evaluation and overall assessment. With regard to hypothesis 3, the CTMTC method allows the TWC to be taught by
means of the “white-box testing” format and the results of the evaluation for the experience justify the need to apply this
method.
In the following sections of this work, we will present the results that are demonstrated for each one of these hypothesis.
3. HYPOTHESIS 1: STUDENTS START UNIVERSITY STUDIES WITHOUT HAVING BEEN TAUGHT OR EVALUATED IN THE
TWC
It is therefore very important to identify relevant data for teamwork carried out by the students for their analysis and
decision making processes, in such a way that the effort required in the TWC learning process is minimised. In this
section we describe the methods used in order to carry out the study concerning the extent to which the TWC is acquired,
as well as the research undertaken in order to obtain the indicators.
In order to demonstrate that students start university studies without having been taught or evaluated in the TWC, a
survey was compiled with direct questions about the aspects of the TWC, as well as other indicators. Those indicators
should allow us to determine whether techniques have been used to carry out teamwork and if they have been evaluated
by their teachers. Below, the size of the sample and the structure of the survey are indicated, in order to finalise the
results that serve to support the hypothesis.
PRE-PRINT
3
3.1 SIZE OF THE SAMPLE
295 of the 304 students participating in the experience in the UPM and 65 of the 82 students participating in UNIZAR,
answered the survey questions. So the sample size is 360.
3.2 STRUCTURE OF THE SURVEY
The survey contains two kind of questions:
• Questions in order to obtain the basic profile of the students: demographic, university student and their
experience with academic teamworks.
• Questions about the phases and aspects of teamwork, that is, the phases of the CTMTC model: Mission and
Objectives, Planning, Execution and Presentation [5]. We highlight the questions focused on the second phase
(Planning) and the third phase (Execution).
The majority of the second kind of questions are formed according to a seven-point Likert scaling response. The scaling
used meant: 1- never, 2-rarely (less than 20%), 3-sometimes (between 20% and 40%), 4- half the time (between 40% and
60%), 5-quite often (between 60% and 80%), 6-very often (more than 80%) and 7-always.
The structure of this part of the survey focuses on three main types of aspects:
• Basic tasks associated with a teamwork competency (e.g. planning). Likert type questions.
• Indicators of the competency:
o Execution of the activity that represents the competency considered: Likert type question.
o Evaluation of the competency, evidence used and the time it is carried out. Likert type questions, open
and multiple choice, respectively.
3.3 RESULTS OF THE SURVEY
3.3.1 Survey question about academic teamwork previously undertaken.
From the answers to the questions concerning how many works they have undertaken throughout their studies prior to
university, the following percentages are deduced: 2% answer that they have not undertaken any, 40% have undertaken
between 1 and 5 works and 57% have undertaken more than 5 works.
3.3.2 Survey questions associated with the planning phase.
We w ill look at so me of the r e s ults f r o m th e survey questions concerning the planning skill and with the indicator for
carrying out and evaluating the activity that represents the said skill, the map of responsibilities.
• Basic tasks associated with planning phase:
The wording of Question 9 is: From the day that you were informed of the work to be carried out and the members
to participate in the teamwork, to the date on which the final result of the work was submitted, evaluate the extent
to which you carried out each one of the following tasks, on a scale from 1 to 7:
a) Part of the work was assigned to each member of the team and a date for its completion was established.
b) A method was established in order to be used to monitor the progress of the work carried out by each member of
the team.
c) The steps were identified that should be followed in order to carry out teamwork, before designating the tasks.
d) A calendar was created for the activities in which the result to be obtained in each activity should be included.
The students perceive that they have carried out tasks characteristic of the TWC. For example, in question 9 in which
they are asked about the basic tasks of planning, on average they answer that 'quite often' (5; maximum 7) have they
carried out the designation of tasks (9a-5.3, 9c- 4.9); whilst only a scale of 3.5 is reached, on average, for tasks
related with the monitoring of the planning (9b-3.6, 9d- 3.4). We have observed that these types of questions tend to
be answered as a result of the mean values of the scale used, in our case close to 4; so in order to receive answers in
more detail, more specific questions were asked, see [11].
• Indicators (execution and evaluation) of the planning skill. We call Indicator B: “Visible document that reflects
the tasks and responsibilities”.
About the Execution of the activity, if the students are used to make the type of document of indicator B.
The wording of Question 13 is: Have you ever made a map of responsibilities in any teamwork you have been
involved in (a map of responsibilities is a document that reflects the tasks and responsibilities of each member and
which is visible for the whole team)?
About the Evaluation of the TWC, namely, the evidence used by the teacher to evaluate the planning skill and the
time when it was carried out. The wordings of the three related questions are the following:
Question 14. Did the teachers grade the creation of the said map of responsibilities?
Question 15. What information did the teacher use in order to grade the said map of responsibilities?
PRE-PRINT
4
Question 16. If you have answered 1 (never) in question 14, do not answer this question. When did the teachers
carry out the evaluation of the map of responsibilities? Options to answer: Before completing the final project;
After completing the final project.
Nevertheless, when specific indicators are asked about, like the map of responsibilities (indicator B) for the team in
question 13, the distribution of the answers is clearly asymmetric in a way that the answers are mostly (51%) grouped
into the first value, 1 (never). See the first row in the table 1.
In terms of the evaluation of the competency indicator, the teaching staff on the whole did not evaluate the execution
of the indicator B (question 14, answer 1 - 82%, see the second row in the table 1) and which in the cases in which it
was evaluated, for 65% of the answers they reflected that it was done after the final project was completed (question
16, see the results in the first row of the table 2)
3.3.3 Survey questions associated with the execution phase of the teamwork.
In this case, we consider the following three indicators, in which both the execution of tasks that support it and their
evaluation on behalf of the teaching staff are considered. Indicator C. Resolve any doubts, ideas for improvements, etc.
Indicator D. Monitoring and progress of the work. Indicator E. Making decisions by means of debates.
Alike the indicator B (see subsection 3.3.2), the survey contains similar questions related to those three new indicators.
One question related to the execution and three questions related to the evaluation of every indicator. In table 1, five
ratings are presented instead of the seven of the Likert scale for the questions, grouping the scales 2 and 3 into a single
one (Rarely or Sometimes) and 5 and 6 (Quite often or Very often) and the following results are included:
• Execution. In a generic way, students carry out cooperative measures, the indicators C and E reach rather high
percentages for the execution. The situation is different when they are asked about indicators that reflect the
execution of specific tasks for teamwork, like B and D, for which we can confirm that they are not normally
used, as almost 50% state that they have never carried out this type of task.
• Evaluation of teamwork skill. Having said that, both in the case of the cooperation indicators, C and E, that
they say they normally use, as well as in the monitoring and planning indicators that they do not normally use,
the teaching staff did not evaluate the said indicators or tasks associated with the TWC. It is sufficient to look
at table 1 and read that the percentages for the option “Never”, in the Evaluation row of all indicators, are
greater than 80%
Not only was it made clear that the teaching staff had not evaluated the TWC, but rather that in addition, more than half
of the students that answered the survey, indicated that at least once they had been evaluated in the skill but that
evaluation was carried out after they had completed and submitted the work. See the second column of table 2.
Table 1. Distribution of the frequencies for the execution and evaluation of the indicators of the TWC
Table 2. Percentage of students evaluated in the TWC and time of their evaluation.
4. HYPOTHESIS 2 “THE “WHITE-BOX TESTING” METHOD OF THE TWC REQUIRES A LARGE INVESTMENT IN TERMS
OF TIME”
The teaching plan in the TWC was structured using the phases of the CTMTC model [5]: mission and objectives,
planning, execution and presentation. It included material with descriptions of each phase, examples and
recommendations, and the related activities. Next, the activities needed in order to carry out the individual monitoring
are shown for the CTMTC method, showing some of the differences in the application of the CTMTC model in each
university (which shows its flexibility), and which leads us to the same confirmation of the hypothesis.
At the UNIZAR the field work was carried out with 22 work groups, consisting of 3 or 4 students. Two groups did not
complete the designated work. The deadline for completing the teamwork was 6 weeks, the subject matter was chosen
by the work group and from four options offered by the teacher and the weight designated to the work with respect to
the final grade of the subject was 10%.
The grade obtained by the students for the teamwork was distributed in the following way: 15% for the participation in
the decision making processes (individual grade), 15% for the planning of the execution (joint grade), 40% for the
quality of the work result (joint grade), 15% for what they have learnt (individual grade) and 15% for the peer
evaluation (individual grade). As a result, the grade given specifically for the teamwork competency is 45%, the result
of adding the first two 15% to that of the peer evaluation.
PRE-PRINT
5
The participation in the decision making process was evaluated by means of their involvement in the forums that were
created for such a purpose. Some of the threads were designated expressly as part of the teaching process of the TWC
and others arose spontaneously.
In terms of the planning of the work to be carried out, it was assessed that the team would have created a teamwork
schedule in which the different work tasks were clearly defined, which were assigned to one or various team mates and
which were published in one of the private places (wiki or file depository) for the team of the Moodle course activated
for such a purpose. See [12].
For the evaluation in pairs, the students were asked to reflect on the contribution of the other team members, and that of
themselves as well, in terms of the work carried out. This reflection was guided by the questions of a survey created ad
hoc. The responses obtained were used in order to obtain the corresponding grade.
Global data of the process: forums, with 164 threads and 615 messages, 20 wikis and 64 files and a formative
evaluation during the execution phase of the work plan. Disaggregated data on the phases of teamwork are:
• Phase 1. Decision making: selecting the work to be undertaken and designating the team coordinator.Forum
phase 1 (160 messages) and Wiki team (21).
• Phase 2. Planning: construction and publication of the work calendar, indicating the tasks assigned and the
members of the team responsible for them. Forum phase 2 (310 messages) and Wiki team (20).
• Phase 3. Monitoring the work plan. Forum phase 3 (101 messages) and file depository of the team (20 folders).
• Phase 4. Conclusion of the work. Forum phase 4 (44 messages).
The magnitude of the effort made in the monitoring process (progress of the work by means of forums, of the wiki and of
the repository of the team) and evaluation can be deduced from the number of messages.
In the UPM the field work was carried out with 55 work groups, consisting of 6 students on average, and the deadline
to complete the teamwork was 3 months, the subject matter was chosen by the work group and the weight designated to
the work with respect to the final grade for the subject was 20%. The grade obtained by the students for the teamwork
was distributed in three parts of equal value: 1/3 for the planning, execution and participation (individual grade), 1/3 for
the analysis and synthesis skills (individual grade) and 1/3 for the quality of the work result (joint grade).
The tasks for the planning, execution and participation were undertaken using forums. On the whole, different threads
were used for the different phases of the teamwork. By means of these threads the TWC as individuals and a group was
evaluated: the effectiveness (cooperation, propositions, measures demonstrating knowledge, messages of motivation
and debates), responsibility (common knowledge, involvement in all of the threads, assuming responsibilities),
leadership, planning, organisation and monitoring.
The evaluation of the analysis and synthesis skills was carried out by means of the analysis of documents created and of
references used by each member of the work team. The tool used was Dropbox. The result of the work was captured in
a wiki created by the group and in a presentation (both the result of the work as well as the resources used during the
execution of the teamwork). Based on the wiki, the quality of the work, its use and brainstorming was evaluated.
With respect to the investment in time on behalf of the teaching staff, the following data can be deduced: 7 forums,
4,000 messages, 55 wikis, 600 documents, 500 threads. A “white-box testing” educational method of the TWC requires
that at the very least an educational evaluation of each team is carried out (in our case there were two) and an overall
evaluation. In addition, taking into account that the grade had to be individual, a monitoring of each student is
necessary.
In terms of the data from the forums used for the monitoring of the TWC, as well as for the educational evaluation, it
was as follows: number of threads, name of the threads, application of rules to the threads, type of threads, creating a
summary of each thread, person that starts the thread, person that closes the thread, dates the threads were started and
ended, involvement in the threads by each participant, period in which members were involved in the thread with
respect to its end date, type of involvement, total number of messages per thread and per forum.
The effort made by the teaching staff in this monitoring process, in the progress of the work by means of the forums and
Dropbox, meant an average amount of time of 10 minutes per student and revision, in addition to approximately 15
minutes per group and revision to write the report. In total, 3 hours and 45 minutes were needed per group, which
equates to an investment in terms of time of 206.25 hours for the 55 groups. The supervision time needs to be added to
that for the result of the work, which, on average, was 15 minutes for each wiki. As a result, the part that requires the
greatest amount of weight designated to it is in the monitoring, supervision and individual evaluation of each member of
the group. This data confirms the hypothesis with respect to the effort that is required in order to follow a “white-box
testing” method.
Once the educational process was finished and the final grade for the learning of each student was attained, two studies
were carried out concerning the said grades, which helped to somewhat demonstrate the validity of hypothesis 3. More
specifically, we chose to use two criteria: the comparison of the joint grades for the groups (average of its members), and
the analysis of the internal difference of the grades of the members of the group.
PRE-PRINT
6
Firstly, the data, of the figures which contain the team grades, are presented.
Data obtained for UNIZAR: in Figure 1 the average grades (range from 0 to 100) are presented for the teams, ordered
from lowest to highest.
Fig 1. Average grades (0 to 100) obtained by the teams in UNIZAR
Data obtained for UPM: the average grades (range from 0 to 100) for the 53 teams, that completed the work designated
to them, are shown on two consecutive graphs: Figure 2 and Figure 3. The first graph contains the average grades less
than or equal to 60 and the second one shows those between 60 and 100.
Fig 2. First part of average grades (0 to 60) obtained by the teams in UPM
Fig 3. Second part of average grades (60 to 100) obtained by the teams in UPM
Secondly, we can observe the variability between groups of both Universities. As a result, we judge the appropriateness
of carrying out a study concerning the variance that supports the idea that there are differences between the average
grades of the different teams. In both cases, the p-value of the analysis of the variance (ANOVA) is 2×10!!" , then it can
be rejected the null hypothesis: “the average calculated for the grades of the teams are the same”, with a level of
significance greater than 99%. This means, we can recognise that not all of the average grades are the same.
Lastly, the variability of the grades within each group can be viewed by means of the studentized residuals (differences
between the average grade of the group and that of each member).
Suppose that a grade within a team is defined as atypical grade if it has a studentized error lower than !2.01 or greater
than 2.01 (probability of 95%) in the UNIZAR case or which have a studentized error lower than !1.96 or greater than
1.96 (probability of 95%) in the UPM case. Then there are three teams (22, 25 and 32) or fourteen teams (G01, G06,
G11, G14, G22, G29, G31, G34, G36, G38, G43, G54, G55 and G60), respectively, in which the educational/evaluation
method has detected a significant difference in the performance within the team.
Somewhat less restrictive would be to consider as atypical grades those that produce an absolute error greater than
1.6775 (probability of 90%) for UNIZAR teams, or greater than 1.645 (probability of 90%) for UPM teams. In this case,
we add another four (4, 6, 9 and 26) or another three teams (G13, G16 and G21), respectively, with notable differences in
the performance. In summary, in the most moderate case, three UNIZAR teams out of twenty (15%) can be considered as
heterogeneous/varied, or even in a broader sense, seven teams out of twenty (35%) can be considered that way. See
Figure 4. And, in the most moderate case, fourteen UPM teams out of fifty three (26.4%) can be considered as
heterogeneous/varied, or even in a broader sense, seventeen teams (32.1%) can be considered that way. See Figure 5, the
group 38 has been excluded due to it containing a member that was extremely atypical (rstudent = -9.38), as it therefore
prevents us from seeing the rest of the variations with sufficient clarity.
Fig 4. Residual values of the grades obtained by the team members in UNIZAR
Fig 5. Residual values of the grades obtained by the team members in UPM
As a result, this method of evaluation and learning for the competency in teams leads to different individual grades. It is
observed that the majority of the teams are homogeneous (65% and 68% respectively), given that all of their members
obtain similar grades. However, the method that has been followed has been able to detect that there are teams (35% or
32%) in which the performance of some of its members has been notably greater or lower than that of the rest of the
members.
5. CONCLUSIONS
The students have carried out teamwork in their studies prior to going to university, however they do not use the
techniques that are characteristic of the TWC; in particular they have not developed skills that are required in order to
monitor their individual responsibilities or those of the progress of the work.
It has been demonstrated that the teaching staff did not evaluate any of their skills in terms of teamwork, not even the
skills that students normally carry out spontaneously, like how they help each other out. The lack of a form of evaluation
PRE-PRINT
7
reinforces that students do not practise these skills. Similarly, on the few times when teachers do carry out an evaluation,
they do so, on the whole, after the work has been completed and submitted. In conclusion, students start university
studies without having been taught or evaluated in the TWC. The methods based on “black-box testing” prevail, in which
only the result is evaluated and once the teamwork activity has been completed. Therefore, we can confirm hypothesis 1.
With regard to the effort made by the teaching staff, it is also clear, based on the volume of information created by the
students and the monitoring work and educational evaluation needed in order to channel the development of the
competency. Hypothesis 2 is thus demonstrated.
The results referring to the grades demonstrate the effectiveness of the CTMTC method for the learning of the teamwork
competency and for its subsequent individual evaluation based on the group activities. In fact, on the one hand, almost
84% of the groups of both experiences obtain an average grade that is greater than the central value of the spread of the
grades (see the data in figures 1, 2 and 3). Furthermore, it has been confirmed by means of the analysis of the variance
that there are significant differences between the grades obtained by the teams. On the other hand, 32% (UPM) and 35 %
(UNIZAR) of the groups (see figures 4 and 5) have had important variations in the grade of its members, meaning that it
can be concluded that giving the same grade to the whole team does not reflect the development of the competency of its
members, Therefore, it is necessary to use a “white-box testing” type method that allows the measures and achievements
of each member of a team to be distinguished. Hypothesis 3 is thus demonstrated.
Therefore, the “white-box testing” methods like the CTMTC are effective, but not productive, meaning that the
efficiency of the said methods would need to be improved. The TWC can be evaluated by means of the organisation of
the forums, the threads, the involvement in the threads and the content that is created temporarily by each member of the
team. The analysis of such data is therefore viable, by means of the Learning Analytics techniques and this research team
is working on it.
The said analysis would increase the effectiveness and the efficiency of the CTMTC. It would allow a reduction in the
time needed by teaching staff to carry out a monitoring process and an educational evaluation of the individual teamwork
competency. A Learning Analytics system would provide the analysis of the evidence data during the making process of
the teamwork. It would help the teaching staff to make decisions in relation with the training, evaluation and
accreditation of the TWC. Such as the supervision (to identify those individuals that are not developing the competency
and the evolution of the teamwork with respect to the anticipated planning), teaching evaluation (to carry out measures
depending on the evolution of the teamwork like the involvement of each member) and an overall evaluation (to assess to
what extent the individual competency is acquired and the planning of the teamwork).
ACKNOWLEDGEMENTS
We would like to thank the Government of Aragon, the European Social Fund and the Ministry of Education of the
Region of Castilla-León for their support. Finally, the authors would like to express their gratitude to the research groups
(LITI, http://www.liti.es; GIDTIC, http://gidtic.com and GRIAL, http://grial.usal.es).
REFERENCES
1. L.R. Lattuca, P.T. Terenzini, and J.F. Volkwein. Engineering Change. A Study of the Impact of EC2000. Executive
Summary, ABET, Inc. 2006 http://www.abet.org/engineering-change/, Accesed 10 March 2014
2. ANECA- Agencia Nacional de Evaluación de la Calidad y Acreditación Informe ejecutivo. El profesional flexible
en la Sociedad del Conocimiento ANECA, Madrid. 2007
http://www.aneca.es/var/media/151847/informeejecutivoaneca_jornadasreflexv20.pdf, Accesed 10 March 2014
3. AEIPRO – IPMA, Bases para la competencia en Dirección de Proyectos. 2009. [online]
http://www.lpzconsulting.com/images/CP-_Trabajo_en_Equipo.pdf, Accesed 10 March 2014
4. A. Fidalgo. Trabajo en equipo. Caja negra versus Caja blanca Innovación Educativa 25 April, 2007
http://innovacioneducativa.wordpress.com/2007/04/25/trabajo-en-equipo-caja-negra-versus-caja-blanca/, Accesed
10 March 2014
5. D. Leris, A. Fidalgo and M.L. Sein-Echaluce. A comprehensive training model of the teamwork competence.
International Journal of Learning and Intellectual Capital, 11(1), 2014, pp. 1-19.
6. M. Poblete, and A. García Olalla. Análisis y Evaluación del Trabajo en Equipo del alumnado universitario.
Propuesta de un modelo de Evaluación de Desarrollo del Equipo. III Symposium Iberoamericano de Docencia
Universitaria. 21-24 Enero 2004. Bilbao, Spain. [online]
http://paginaspersonales.deusto.es/mpoblete2/ANALISIS%20Y%20EVALUACION%20DEL%20TRABAJO%20
EN%20EQUIPO.htm, Accesed 10 March 2014
7. P. Hirsch and A. F. McKenna, Using Reflection to Promote Teamwork Understanding in Engineering
DesignEducation. Inernational Journal of Engineering Education (IJEE), 24 (2), 2008, pp. 377-385.
PRE-PRINT
8
8. SoLAR – Society for Learning Analytics Research. http://www.solaresearch.org/about/, Accesed 10 March 2014
9. G. Siemens. Learning Analytics: Envisioning a Research Discipline and a Domain of Practice. LAK Proceedings of
the Second International Conference on Learning Analytics and Knowledge, 2012, pp. 4-8.
10. A. Johri, C. Williams and J. Pembridge. Creative Collaboration: A Case Study of the Role of Computers in
Supporting Representational and Relational Interaction in Student Engineering Design Teams. International
Journal of Learning and Intellectual Capital, 29(1), 2013, pp. 33–44.
11. A. Fidalgo, D. Lerís, M. L. Sein-Echaluce,and F. J. García-Peñalvo. Indicadores para el seguimiento y evaluación
de la competencia de trabajo en equipo a través del método CTMTC. Libro de Actas de CINAIC 2013. Madrid,
Spain, 6-8 November 2013, 22 November 2013, pp. 280 - 285.
12. M. Alier, N. Galanis, M.J. Casañ, E. Mayol, J. Piguillem, F.J. García-Peñalvo and M.A. Conde. “Didactical
Patterns for the Usage of Wikis in Educational and Learning Activities”. International Journal of Engineering
Education (IJEE), 28(6), 2012, pp. 1347–1352.
SHORT BIOGRAPHIES OF THE AUTHORS
Á. Fidalgo-Blanco is responsible for educational innovation of the UNESCO Chair in University Management and
Policy and Director of the Laboratory for Innovation in Information Technology at the Polytechnic University of
Madrid. He has actively participated as principal investigator in R&D, both national and international. He has been the
organiser of many seminars and conferences for many years and he is currently the president of the organiser committee
of the International Conference of Learning, Innovation and Competitiveness (CINAIC, Spanish abbreviation). He is an
active researcher in educational innovation, knowledge management and educational technologies and educational
communities from the social networks, having generated as a result of both publications and information products.
D. Leris is Professor of Applied Mathematics in the School of Engineering and Architecture and member of the Quality
Evaluation Board of Electrical Engineering Degree at the University of Zaragoza, Spain. She is actively involved in the
Research and Innovation Group in Training supported by Information and Communication Technology (GIDTIC for its
Spanish abbreviation). Her current research interests focus on teamwork competency, team training, long-life learning
and mathematical competencies training. She coordinates or participates in research and innovation projects from
national, local or university funds. She has experience and expertise in the field of adaptive courses in LMS Moodle
with multi-conditionals.
M. L. Sein-Echaluce is Director of Virtual Campus and Professor of Applied Mathematics in the School of
Engineering and Architecture at University of Zaragoza, Spain. She is the main researcher of the Research and
Innovation Group in Training supported by Information and Communication Technology (GIDTIC, Spanish
abbreviation). She is the president of the scientific committee of the International Conference of Learning, Innovation
and Competitiveness (CINAIC, Spanish abbreviation) and takes part in evaluation committees of the local calls of
innovation projects and of the international conferences. Her research is currently focused on technologies applied to
cooperative methodologies and the usage of Open Source LMS and other tools for online adaptive learning. She takes
part in national projects with other research groups, in the organisation of events, in training activities for educational
institutions and also in conferences and papers cooperating with Spanish and foreign researchers.
F. García-Peñalvo is researcher in GRIAL (research GRoup in InterAction and eLearning) and he belongs to the
Research Institute for Educational Sciences at the University of Salamanca. His main research interests are related to
online education.
PRE-PRINT
9
Table 1. Distribution of the frequencies for the execution and evaluation of the indicators of the TWC
Never
Rarely or
Sometimes
Half of the
time
Quite often or
Very often
Always
Indicator B.
Visible document
that reflects the tasks
and responsibilities
Execution
51%
20%
10%
16%
3%
Evaluation
82%
7%
3%
7%
1%
Indicator C.
Resolve any doubts,
ideas for
improvements,…
Execution
1%
8%
15%
58%
18%
Evaluation
84%
5%
3%
7%
1%
Indicator D.
Monitoring and
progress of the work
Execution
47%
14%
14%
21%
4%
Evaluation
83%
8%
2%
5%
2%
Indicator E.
Making decisions by
means of debates
Execution
8%
15%
15%
46%
16%
Evaluation
83%
6%
4%
6%
1%
Table 2. Percentage of students evaluated in the TWC and time of their evaluation.
Indicator
At least evaluated
at some point
Evaluated
after completing the work
B. Visible document that reflects the tasks and
responsibilities
18%
65%
C. Resolve any doubts, ideas for
improvements,…
16%
60%
D. Monitoring and progress of the work
17%
49%
E. Making decisions by means of debates
17%
39%
Fig 1. Average grades (0 to 100) obtained by the teams in UNIZAR
Fig 2. First part of average grades (0 to 60) obtained by the teams in UPM
30.6
42.75
50.6
53.6
57.4
62.2
62.3
64.4
66.6
67.4
67.6
72.5
73
75.1
76.25
77.5
77.9
83.6
84.3
90.3
16.7
30.0
33.3
38.3
43.1
43.3
45.0
46.7
47.2
49.3
50.0
50.0
50.6
51.7
55.6
58.3
60.0
PRE-PRINT
10
Fig 3. Second part of average grades (60 to 100) obtained by the teams in UPM
Fig 4. Residual values of the grades obtained by the team members in UNIZAR
Fig 5. Residual values of the grades obtained by the team members in UPM
63.3
66.7
66.7
66.7
66.7
66.7
68.0
68.1
68.1
70.0
71.7
74.4
75.0
76.2
76.4
77.0
80.6
81.7
81.7
81.7
83.3
83.3
83.3
83.3
83.3
83.3
83.3
84.3
86.1
88.9
90.0
91.7
93.1
97.2
98.6
100.0
PRE-PRINT