Content uploaded by Silvija Hanžić Deda
Author content
All content in this area was uploaded by Silvija Hanžić Deda on Sep 25, 2017
Content may be subject to copyright.
CONTENTS
Foreword
Assessing skills, prospects and outcomes
17 Judit Dombi
Working towards a Model of Intercultural Communicative Competence for
Advanced-level EFL Learners
22 Katalin Doró
Source Distortion in Patchwritten EFL Academic Texts
31 VeraSavić
Learners?
43 Zoltán Lukácsi
The Language Instruction Scheme for Graduates with a Withheld Degree
52 GabriellaLőcsey
63 DarijaOmrčen
81 JasenkaČengić,SilvijaHanžićDeda&AnaGabrijelaBlažević
Individual differences, diversity and learners with disabilities
90 KatalinPiniel&ÁgnesAlbert
104 Robert Märcz
Exam
115 AlenkaMikulec&RenataŠamo
Preschool Teachers Not Majoring in English?
127 IvaGugo&RenataGeld
Frequency and Type of Gesture in the Blind and the Sighted in L1 and L2
139 AgnieszkaKałdonek-Crnjaković&ZrinkaFišer
4
1.1. Foreword
This volume stems from University of Zagreb Round Table 2016 (UZRT 2016)
conference, a yearly event which is organized alternately by the University of Pecs
and the University of Zagreb. The event started as a platform through which younger
peers and more experienced colleagues.
The volume comprises 12 papers which cover various empirically studied
Some studies propose new theoretical models, and other have a narrower focus and
have broadly categorized them into two areas of interest; studies pertaining to the
differences, diversity and learners with disabilities.
or pedagogical prospects and value of instructional material and assessment tools. Out
acquisition and use, and two papers deal with the characteristics and needs of learners
Even though most of the papers have the English language as their focus, we
believe that the range of educational and sociolinguistic settings as well as different
wider audience of researchers, teachers and students, as well as curriculum designers
and developers.
the initiators of this valuable collaboration between the two universities, for their
support to this yearly event.
The editors
81
FacultyofHumanitiesandSocialSciences,Zagreb,Croatia
jasenka.cengic@gmail.com
FacultyofHumanitiesandSocialSciences,Zagreb,Croatia
silvija.hanzic@gmail.com
FacultyofHumanitiesandSocialSciences,Zagreb,Croatia
anablazevic@gmail.com
1. Introduction
Evaluating academic programs has become an integral part of developing and improving
various programs at undergraduate as well as graduate and postgraduate university
levels. One of the problems in assessing the quality of university level programs is the
fact that most assessment programs rely on student satisfaction surveys only (Hurt,
2004). However, academic programs involve some other dimensions which should be
considered in the evaluation process with student satisfaction being only one of them.
Other dimensions include meeting the students’ desired outcomes, their relationship
as the thesis advisors’ experience (Cuseo, 2003).
Furthermore, a comprehensive assessment of academic programs should
include a variety of qualitative and quantitative assessment tools. The use of open-
ended questions and focus groups are the recommended procedures for general
(2005).
Ready-made instruments cannot always serve the purpose of evaluating
therefore, involve designing instruments that provide enough information for program
improvement.
the FHSS (Faculty of Humanities and Social Sciences), University of Zagreb (henceforth
the Program) was evaluated with the instruments designed by the Program directors.
of the multitude of dimensions encompassed by the Program, they were not validated
instruments.
Therefore, in 2014 the evaluation underwent considerable changes. The idea
was to recruit the students themselves in the process of collecting the necessary
information about what needs to be included in the new assessment protocol and
instrument design. After a period of brainstorming sessions and collaborative meetings
Design
82
between students and the faculty members involved in the Program, a decision was
made to design a new questionnaire for evaluating the Program. The aim of this paper
is to describe the process of designing the above-mentioned questionnaire.
2.
In this section, the Program will be described along with the evaluation procedures in
use prior to the Program change in 2014.
2.1. The description of the Program
The Program consists of two parts pertaining to the activities designed for doctoral
Table 1. The structure of the Program
The FHSS Doctoral Program in FLE
Coursework
Applied
linguiscs
courses
Courses
about
research
methods
FLE
themed
courses
Various
elecve
courses
Students’
current
research work
presentaon
Selecng the
area of doctoral
research
Dissertaon
work
Independent Research
2.1.1. Coursework
Module1
column shows the course titles, the second column shows their status (compulsory or
elective), and the third column shows the number of ECTS credits per course.
Table 2. Courses in Module 1
Course Status Credits
Second Language Acquisition compulsory 4
First Language Acquisition elective 2
Bilingualism and Multilingualism elective 2
The Age Factor elective 2
Cognitive Grammar in Foreign Language Learning and Teaching elective 5
elective 2
83
Module 2 includes courses pertaining to research methodology (see Table 3).
Table 3. Courses in Module 2
Course Status Credits
compulsory 3
Methods in Researching Foreign Language Education compulsory 3
compulsory 3
elective 2
Modern Technology and Foreign Language Teaching elective 5
Computational Linguistics and Language Technologies elective 2
Research Areas in Foreign Language Acquisition elective 2
Module3 consists of topics in foreign language education, and it offers fourteen courses
shown in Table 4.
Table 4. Courses in Module 3
Course Status Credits
Theories and Methods in Foreign Language Teaching elective 3
elective 3
Reading in a Second Language elective 3
Communicative and Other Competences in Foreign Language
Teaching elective 3
elective 3
Literature in Foreign Language Education elective 3
Cultural Elements in Modern Foreign Language Teaching elective 3
elective 3
Language Learning Strategies elective 3
Learner Autonomy elective 3
Reception Approach in Foreign Language Learning elective 3
Analysis and Evaluation of Teaching Materials elective 3
Assessing Communicative Competence in a Foreign Language elective 3
elective 3
Module4 contains eleven elective courses (see Table 5).
Table 5. Courses in Module 4
Course Status Credits
Cognitive Linguistics elective 2
Neurolinguistics elective 2
elective 2
Mental Lexicon – Theories and Models elective 2
84
Language Learning Impairments in Monolinguals and Bilinguals elective 2
Psychology of Children with Special Needs elective 2
Blindness and Foreign Language Learning elective 3
The Modern School in the Educational System elective 2
Curriculum Theory and Models elective 2
Education Strategies and Innovation elective 2
Interculturalism and Education elective 2
2.1.2. Independent Research Work
The doctoral students set their own goals and schedule their research activities, that
Module 5
Module 6 includes the necessary stages in the process of determining and
Module7 is dedicated to planning the research, reporting on the relevant stages
evaluation of the thesis and its public defense.
information, list of courses, and the program schedule. Students’ responsibility is to
attended, papers published, etc.).
2.2. Previously used questionnaires for evaluation of the Program
From 2006 to 2014 three questionnaires were designed for evaluation of the Program.
The evaluation was conducted after each semester and at the end of the Program.
In the questionnaires, the following categories were evaluated: the structure of the
courses, the relevance of the courses, course instructors’ performance and course
requirements. Semester evaluation was conducted using a questionnaire containing
the list of both compulsory and elective courses which were evaluated through the
the four listed categories using a 5-point scale. In addition to that, there was a text box
intended for comments and suggestions regarding individual courses as well as the
overall program.
general information about the students was required, such as previous education,
85
scale, and the subsequent 30 statements addressed the Program administration, the
Program director’s availability and the time provided for students’ needs, tuition fees,
assignment deadlines and reading materials, exam dates and requirements, teachers’
expectations, the balance between theory and research, collaboration with thesis
comprised three open-ended questions related to: a) three strongest aspects of the
Program, b) three possible improvements, and c) ideas pertaining to improving the
In the section that follows we are going to describe the concepts and instruments
that served as the basis for the new questionnaire design.
3.
The idea was to construct a questionnaire that would encompass the multidimensionality
of the Program structure and the complexity of the motives and goals of the students
prior to 2014, in terms of their applicability to the aims of evaluation of the Program.
In addition to the previously used questionnaires, we explored instruments used in
other universities. The instrument that we found most useful for constructing a new
questionnaire was The Students’ Evaluation of Educational Quality (SEEQ) instrument
3.1.
used by researchers at the university level (Richardson, 2005). It comprises a number
of the so-called teaching dimensions (Marsh, 2007).
The SEEQ is a type of the students’ evaluation of teaching effectiveness (SETs).
According to Marsh (2007), SETs is the most commonly used instrument for the
students’ assessment of their study programs. The SETs has an established construct
validity (e.g., Cashin, 1988; Howard, Conway, & Maxwell, 1985) and that was the main
motive for inserting it in the new questionnaire we were trying to design. SETs aims to
The standardized SEEQ is designed to evaluate one course at a time, and it
is most commonly administered upon completing the course. It consists of 34 items
questionnaire sections address the following teaching dimensions: the learning
process, enthusiasm, organization, group interaction, individual rapport, breadth of
Value, Instructor Enthusiasm, Organization/Clarity, Group Interaction, Individual
Rapport, Breadth of Coverage, Examinations/Grading, Assignments/Readings, and
have been proven constant in relation to time, courses and levels of teaching. The
86
3.2.
The questionnaire items were generated from 1) informal individual interviews with
the directors of the Program 2) focus group interview and 3) questionnaires previously
used for the evaluation of the Program.
Apart from the current Program director, one of the preceding directors of
the FHSS doctoral program in FLE was interviewed as well. The aim of the interviews
was learning about the Program structure and the changes after 2014 and, more
questionnaires affected and/or initiated the Program change.
2014 cohort. The goal of the focus group was to articulate the issues concerning the
structure and content of the Program. The focus group was held after the doctoral
students had already attended most of the courses planned for that cohort. The students
to say if they thought the items covered the issues related to the students’ satisfaction
with the Program. Having compared the issues that arose during the focus group
discussion to the items listed in the questionnaire previously used for the evaluation of
the Program, it was clear that the old questionnaires used for the evaluation purposes
of the Program failed to address most of the issued which emerged during the focus
group discussion.
There were several important issues that surfaced from the data collected
in the focus group. First, although the old questionnaires contained open-ended
questions for additional comments, it was concluded that the doctoral students had
to be provided with the opportunity to give much more detailed opinions about
individual courses and teachers. This would ensure that the comments collected would
Second, the doctoral students’ perception of course quality seemed to differ greatly
from one another due to their previous education, teaching experience and motivation
for enrolling in the Program. In other words, there were two basic ideas that emerged
during this focus group: 1) the questionnaire for the evaluation of the program had to
be more detailed in terms of addressing individual courses and teachers, and 2) the
program.
3.3.
The new questionnaire 3consisted of seven parts that had to be administered at
3 Contact the authors for the full version of the new questionnaire in Croatian and the English.
87
addressed the program itself – the courses available, exams and details about potential
1.
as age, gender and previous education. In addition to that, it collected data on
Finally, it addressed the question of motivation to enroll in this doctoral program.
2.
types of learners that can be found in a language classroom and examines with
3. The third set was intended for individual course evaluation and it consisted of
attending the course. The second subset related to the content and the schedule
of the course. The third part related to the assigning a grade to the overall
ended questions enquiring about the strong points of the course, as well as the
suggestions about what could be improved.
4. The fourth set examined the exam process. This part of the questionnaire was
administered after each semester, and included all the exams passed during that
subset related to the amount of time invested in completing the course
requirements and the number of ECTS credits earned in relation to the effort
invested in passing the exam.
5.
that described the types of teachers they preferred. For instance, the teachers
who presented topics systematically and clearly, the ones who used real life
situations to present research results, the teachers who were very objective and
demonstrated no personal opinions, the teachers who respected the variety of
theories and approaches even though they might not have agreed with them,
the ones who expressed their opinions clearly, the teachers whose teaching was
interactive, the ones who allowed interruptions, the teachers who encouraged
be guided by their students’ needs and interests.
6.
and the third subsection was intended for the overall program evaluation. The
last three subsets were open-ended questions examining the strong points of
the doctoral program’s organization, the suggestions for improvements, and the
study program. This part of the questionnaire would be administered at the end
of the doctoral studies.
88
7. The last, seventh set, of this questionnaire examined the quality of the doctoral
students’ collaboration with their thesis advisor(s). This section ended with an
open-ended question calling for further comments. This part of the questionnaire
would be administered after the doctoral thesis is defended.
3.4.
the questionnaire and comment on the following points: the wording of the items, the
organization of the questionnaire, and the time frame necessary for completion. The
comments obtained were used to improve the clarity and comprehensiveness of the
questionnaire items. The result was the version of the questionnaire that was piloted
further a few months later.
The participants were recruited from all the cohorts of the Program since its
founding in 2006, including the 2014 cohort. Altogether there were 38 participants,
out of which 12 were members of the 2014 cohort, and the remaining 26 were
students belonging to all the other cohorts of the Program since its founding. The
only those questions that relate to two out of four compulsory courses that were in
common to all the cohorts. The items included in the sixth and seventh set of the
questionnaire were not included in the piloting. As stated earlier those sets would
be administered at the end of the doctoral study, which is a stage the majority of the
doctoral students had not reached at the point the pilot study was conducted.
Our next step is to test the sixth and seventh module of the questionnaire on
the doctoral students of the 2014 cohort so that the questionnaire could be used in its
entirety with the 2016 cohort. One of the future steps will be designing an additional
well as thesis advisors to allow for even deeper insight into the needs of the Program.
4. Instead of a conclusion
Program in Foreign Language Education, faced the need for changing the Program
evaluation scheme. That resulted in constructing a set of questionnaires which, put
and as students, and had to cover the academic program in its entirety.
The novelties of the new questionnaire design are: 1) the opportunity for the
performance, and their exam experiences immediately upon completion, 2) items
designed to measure the students’ motivation to join the Program and their goals, i.e.,
what they are aspiring to upon completing their doctoral education, 3) items designed
Collecting relevant data at regular intervals, from every cohort, enables the
89
and remains up to date.
new questionnaire contains two distinctive subsets that collect information about each
doctoral student from two different perspectives: those of doctoral students as teachers
as teachers and as students is seen as a way of grasping the complexity of the nature of
the cohorts of students joining the Program. Both subsets of the questionnaire contain
the items examining various cognitive styles and personality traits, so they provide
valuable information that can clarify the reasons why individual students evaluated a
certain course, course teacher, and other components of the doctoral program the way
they did.
Having considered all the aspects of the Program requirements and the enrolled
Program and their goals, i.e., what they are aspiring to upon completing their doctoral
education. This new variable has the potential of explaining the variability in individual
student’s evaluation.
In sum, the data collected by means of the new questionnaire may carry great
value of such information can be twofold: it can be used for improvements in future
planning and curriculum design, and in understanding the needs of potential future
5. References
Usingsurveysinlanguageprograms.Cambridge, UK:
Cambridge University Press.
Retrieved May 5, 2017, from
Retrieved May 5, 2017, from http://www.psu.edu/dus/mentor/050211cd.htm
Questionnaires in Second Language Research: Construction,
Administration,andProcessing. London: LEA Publishers
Hurt, B. (2004). Using the balanced scorecard approach for program assessment of
(eds.). The Scholarship of Teaching and Learning in Higher Education: An
Evidence-BasedPerspective
literature. AssessmentandEvaluationinHigherEducation30(4): 387–415.
Evaluating-AcademicAdvising%2BAttachments.pdf
