Content uploaded by Margaret Thomas
Author content
All content in this area was uploaded by Margaret Thomas on May 27, 2020
Content may be subject to copyright.
Assessment of L2 Proficiency in Second Language Acquisition Research
Abstract
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
... It is advisable to describe the test format and content in sufficient detail and caution against following the partitioning employed by the local program, which may not necessarily be justifiable in the research context. Some researchers clump participants to a certain proficiency level based on institutional status, which however, is only a rough estimate of L2 proficiency and the standards by which learners hold a given status vary greatly among different institutions (Thomas, 1994). Participants with a particular status in one institution may not be at the same proficiency level as those with the same status in another institution. ...
... Avoiding impressionistic judgments of L2 reading proficiency. Finally, impressionist judgment, either from researchers or participants' instructors, gives no evidence leading to a particular judgment, and often neither the evaluator nor the evaluating process is specified (Thomas, 1994). Occasionally participants do self-assessments ranging from broadly framed descriptors (e.g., poor, good) to refined can-do statements, while not all learners have an accurate understanding of their own proficiency. ...
... Occasionally participants do self-assessments ranging from broadly framed descriptors (e.g., poor, good) to refined can-do statements, while not all learners have an accurate understanding of their own proficiency. Additional information of residence length in an L2-speaking environment or learning length may be unrevealing because it does not entail any certain amount of exposure to L2, even less attainment of any level of proficiency (Thomas, 1994). The context plays a mediating role: one year in foreign language (FL) vs second language (SL) context, or in classroom instructional vs naturalist setting may generate different proficiency levels. ...
This discussion piece explores issues and components that makes original research manuscripts submitted to Reading in a Foreign Language truly rigorous in terms of methodology. This paper will lay out issues of and provide suggestions on how to approach research questions (RQs), sampling, research design, reporting of results, discussion, and implications, in a way that can contribute meaningfully to research practices in the field. Finally, responses are invited. This paper aims at exploring components that makes original research manuscripts submitted to Reading in a Foreign Language truly rigorous in terms of methodology. This paper will lay out issues of and provide suggestions on how to approach research questions (RQs), sampling, research design, reporting of results, discussion, and implications, in a way that can contribute meaningfully to research practices in the field. Research Questions and Literature Review Proposing Research Questions (RQs) constitutes the first step of launching research. RQs should be clearly stated, and relevant concepts should be operationalized and explained. A core issue is to address the significance of the current study. Simply claiming that the topic has been overlooked does not seem sufficient to justify the necessity for probing it. Instead, researchers need to give a thorough and critical review of theoretical frameworks and previous literature, from which follows a logical segue to the RQs. A literature review should introduce relevant research with primary methodological information in an organized way. It should synthesize what has been achieved and what remains controversial or unclear, rather than only listing results. Missing key previous studies, failing to locate more recent sources, and including less relevant research would undermine the extensiveness of the coverage of the important works and the connection to the present study.
... Several systematic reviews of proficiency measurement and reporting practices (e.g. Thomas 1994Thomas , 2006Tremblay 2011) have argued that adequate proficiency measurement is essential to effective research design, since in the absence of such measures it is "unclear to what extent proficiency is intervening in the accurate interpretation of research findings" (Norris 2018 p. 9). A recent systematic review of this area (Park et al. 2022) painted a relatively pessimistic view of current proficiency measurement and reporting practices. ...
Recent studies of proficiency measurement and reporting practices in applied linguists have revealed widespread use of unsatisfactory practices such as the use of proxy measures of proficiency in place of explicit tests. Learner corpus research is one specific area affected by this problem: few learner corpora contain reliable, valid evaluations of text proficiency. This has led to calls for the development of new L2 writing proficiency measures for use in research contexts. Answering this call, a recent study by Paquot et al. (2022) generated assessments of learner corpus texts using a community-driven approach in which judges, recruited from the linguistic community, conducted assessments using comparative judgement. Although the approach generated reliable assessments, its practical use is limited because linguists are not always available to contribute to data collections. This paper, therefore, explores an alternative approach, in which judges are recruited through a crowdsourcing platform. We find that assessments generated in this way can reach near identical levels of reliability and concurrent validity to those produced by members of the linguistic community.
... For the written and spoken subcorpus, where the data came from the assessment of the preliminary year, we also consider the linguistic evaluation from a perspective of second language development. In second language research, learners' proficiency level is often determined by external criteria such as institution status or length of learning, and yet those external criteria are often not a reliable indicator of students' proficiency level (Callies, 2005;Thomas, 1994). An increasing number of studies, therefore, have turned to standardised language tests to determine L2 proficiency (e.g. ...
This paper describes the rationale for and design of a new multimodal corpus of L2 academic English from a Sino-British university in China: the Corpus of Chinese Academic Written and Spoken English (cawse). The unique context for this corpus provides language samples from Chinese students who use English as a second language (L2) in a preliminary-year programme, which prepares students for academic studies at university level, at a campus where English is used as the Medium of Instruction (emi). Data were collected from a variety of settings, including written (i.e., exam scripts and essays) and spoken assessments (i.e., interviews and presentations), covering the full range of grades awarded to those language samples, as well as from student group interactions during teaching and learning activities. The multimodal nature of the corpus is realised through the availability of selected audio/video recordings accompanied by the orthographically transcribed text. This open-access corpus is designed to help shed light on Chinese students' academic L2 English language use in a variety of written, spoken and multimodal discourses.
Against the backdrop of the critical importance of recognising the specificity of learning languages other than English (LOTEs) in Second Language Acquisition (SLA) research, this volume focuses on a state-of-the-art presentation of the research approaches and methods that characterise French as second language (L2) within contemporary SLA research. The presentation problematises those approaches and methods as a critique of what has been done, identifying a methodological roadmap of what needs to be done in order to advance the methodological agenda in L2 French and its contribution to wider SLA research. The discussion further aims to bridge the interface between methodological issues and the research investigation of a specific LOTE, French, such as in terms of its linguistic characterisation and developmental issues underpinning its acquisition. The analysis extends to approaches and methods across different theoretical paradigms in L2 French, in different areas of linguistic development, among learners in different learning contexts.
Second language (L2) viewing with captions (i.e., L2 on‐screen text) is now a proliferating as well as promising area of L2 acquisition research. The goal of the present meta‐analysis was to examine (a) the relationship between captioned viewing and incidental vocabulary learning and (b) what variables related to learners, treatment, methodology, and vocabulary tests moderate the captioning effect. Synthesizing 89 effect sizes from 49 primary studies (i.e., independent experiments), we fitted a multilevel meta‐analysis model with restricted maximum likelihood estimation to calculate the overall effect size based on a standardized mean difference of gain scores between captioned viewing and uncaptioned viewing groups. The results showed a medium effect of captioning on L2 vocabulary learning, g = 0.56, p <.001. Moderator analysis indicated moderating effects of instructional level, target audience of video materials, and administration of vocabulary pretest. These results are discussed with the aim of guiding future research and language learning through viewing.
In the late 20th and the beginning of the 21st centuries, language and culture teaching has seen a renewed interest. Debates and studies conducted in general and applied linguistics, anthropology, and ethnopragmatics have proved that culture and language are inextricably intertwined.1 Indeed, culture is negotiated by the members of the speech community through their discourses, and language is the tool used to codify cultural assumptions and values.
Even though the acquisition of Communicative Competence (CC) is still viewed as a primary goal, in modern times a new paradigm has begun to make its way in the understanding of Foreign Language (FL) instruction, which is now seen as the contact with and the discovery of alterity. Within this »cultural turn«, considered as the natural consequence of the »global village« effect,2 and the increased human mobility, Language Proficiency (LP) cannot but integrate the development of Intercultural Communicative Competence (ICC). This ability, which begins with a critical reflection and a growing awareness of both the own and the target culture, allows the learner to successfully understand and communicate with representatives of the target culture, therefore providing a method for overcoming prejudice, discrimination, and misunderstandings.
Acquiring ICC is undoubtedly a long-term and complex process, hard to finalize within the time and spatial constraints imposed by FL classroom education. This is particularly true for learners of Arabic as Foreign Language (AFL), who eventually (if not already in formal instruction, then later in real encounters) have to face an extremely eclectic and constantly evolving linguistic and cultural reality. Then ideally, becoming an interculturally competent interlocutor of Arabic implies not only to acquire dexterity of diglossia, but also to develop awareness of sociolinguistic values embedded in the choice of varieties. The question that arises is: How can ICC be easily and gradually practiced in AFL classes? Taking a cue from the intercultural oriented approach adopted in German as Foreign Language (Deutsch als Fremdsprache, DaF) at Goethe-Institut, the present paper aims to provide a theoretical foundation and a practical example for the integration of ICC in Teaching Arabic as Foreign Language (TAFL).
Aims and objectives
Proficiency assessment is a key methodological consideration in the field of bilingualism, and previous reviews have highlighted significant variability in both the use and type of assessment methods. Yet, previous reviews of proficiency assessment methods in bilingualism have failed to consider key study characteristics (e.g., methodology and subfield) that may impact the choice of proficiency assessment method. This paper provides an updated systematic review of proficiency assessment methods in the field of bilingualism, analyzing trends within different methodological approaches and linguistic subfields.
Design/methodology/approach
A systematic review was conducted, examining recent research articles in the field of bilingualism, broadly defined. A total of 17 journals (of 100) and 140 empirical research articles (of 478) with bilingual participants fit the relevant inclusionary criteria.
Data and analysis
Studies were coded for several characteristics, including methodology (e.g., quantitative vs qualitative), linguistic subfield (e.g., psycholinguistics), and the method of proficiency assessment (e.g., standardized testing, self-reporting).
Findings/conclusions
Analyses revealed a number of different methods of proficiency assessment currently used in bilingualism research. However, different trends were found by methodology type and linguistic subfield. Broadly, the results revealed greater use of proficiency assessments in quantitative research than qualitative research. Moreover, while there was significant variability in all of the subfields examined, several within-subfield trends were identified.
Originality
This study provides an update to previous findings, establishing current proficiency assessment practices in bilingualism research. In addition, acknowledging the unique needs of different types of research, this study is the first to examine trends within different methodological approaches (i.e., quantitative vs qualitative) and subfields of bilingualism (e.g., psycholinguistics, sociolinguistics).
Significance/implications
The notable variability in proficiency assessment methods within different subfields suggests a greater need for subfield-specific norms to facilitate comparative analysis. Several key considerations are given for the selection of proficiency assessment methods in bilingualism research.
The articles in this special issue contribute to second language (L2) pragmatics by investigating different speech acts and interactions in Chinese as the target language. The studies all examined learners’ pragmatic performance in different contexts and compared their performance with that of native Chinese speakers. The findings of the articles showed what the learners could achieve and what they still had problems in. In this commentary, I first analyze and discuss the aspects in these articles that impress me the most, including research design and focus on L2 pragmatics, theoretical contributions, and methodological considerations. Then, I will address some important implications that the articles provide for future research in L2 pragmatics.
This article reconsiders one of the classic articles in the second language literature, Schachter's 'An error in error analysis', in the light of subsequent work on avoidance. Such work is essentially of two kinds, either building on and refining the concept of avoidance, or offering alternative explanations for Schachter's findings. While avoidance as a genuine phenomenon is not proved or disproved by Schachter's data, hypotheses based on her figures suffer from the lack of methodological detail in her original study. In order to be able to establish whether avoidance is a feasible explanation for relative underproduction by a group of learners, it is necessary to look at the first language form, distribution and function of the entity supposedly being avoided in the L2 as well as the means being used to establish whether and to what extent the entity is already part of the L2 knowledge of members of that group.
This paper focuses on a parametric difference between French and English, namely the issue of whether or not the language allows verb movement. The lack of verb-raising in English causes a potential learnability problem for francophones, as far as English adverb placement is concerned. In particular, an adverb in English is not allowed to interrupt a verb and its direct object, in contrast to French. It is argued in this paper that form-focused classroom instruction, including negative evidence, is more effective in helping L2 learners to arrive at the appropriate properties of English than positive input alone. An experimental study on the effectiveness of teaching adverb placement was conducted with I 1 and 12 year-old francophone learners of English. One group (n = 82) was explicitly instructed on adverb placement, and another on ques tion formation (n = 56). Subjects were tested on a variety of tasks relating to adverb placement; they were pretested, and post-tested twice, immediately after the instructional period, and again five weeks later. Some of the subjects were followed up a year after the original testing. Results show significant differences between the two groups: only the group that received positive and negative evidence that was specifically oriented towards adverb placement came to know that adverbs may not interrupt the verb and object. The results from the follow up, however, suggest that this knowledge is not retained in the long-term.
The issue of completeness in adult second language acquisition is critical in the development of a theory of second language acquisition. Assuming the Chomskyan definition of core grammar as being those aspects of the language determined by the interaction of the innately specified Universal Grammar and the input to which the learner is exposed, we need to ask if it is possible for an adult learner of a second language to attain native-speaker competence in the core aspects of the grammar of the second language. This paper examines evidence for presence or absence of one principle of UG, Subjacency, in the grammars of groups of proficient nonnative speakers of English. There are three groups whose native languages - Korean, Chinese, Indonesian - differ from English with regard to Subjacency, Korean showing no evidence of it, Chinese and Indonesian showing partial evidence of it. There is one group whose native language, Dutch, shows the full range of Subjacency effects that English does. If all groups show the same Subjacency effects in English that native speakers do, then it must be the case UG is still available for adult second language learning and completeness in second language grammars is possible; if not, then completeness cannot be included as a possible characteristic of adult second language acquisition. Proficient nonnative university students with the above native languages were given grammaticality judgement tests on a set of sentences containing a variety of structures (islands) and Subjacency violations involving those structures. Analysis showed that though all groups were able to correctly judge grammatical sentences (containing islands) as grammatical, only the Dutch group was able to correctly judge ungrammatical sentences (containing Subjacency violations) as ungrammatical; the Korean subjects performed randomly on this task. This native language effect was shown not to be due to attribute variables, such as age of first exposure to English, number of months in an English-speaking country, number of years of English study, etc. The results support the conclusion that completeness is not a possible property of adult-acquired grammars since adults no longer have access to UG for the second language learning process.
Much of the work on the second language acquisition of restrictive relative clauses has made reference to the similarities between learners' order of diffi culty and Keenan and Comrie's (1977) typologically determined noun phrase accessibility hierarchy for relativisation (AH). There has been little considera tion, however, of whether this 'theory of markedness' (for that is the implica tion of citing the AH in the context of second language learning) actually determines the way that second language learners develop rules for restrictive relative clauses. The present study examines the way that learners of L2 French construct rules for French relativiser morphology from this perspective. It is found that there is no evidence to support the view that learners make use of a theory of markedness like the AH in constructing such rules. Rather, learners appear to construct rules on the basis of the linear ordering of the constituents of restrictive relative clauses in surface configurations. From the evidence it is suggested that 'markedness' in the development of L2 restrictive relative clauses is not a feature of the grammatical component of learners' linguistic knowledge, but is a feature of their L2 processing capacity.
In this chapter, I first examine Tarone’s sociolinguistic approach to the study of interlanguage (IL) variability. I focus on Tarone’s conception of the “vernacular style” as the speech style in which the speaker (L2 learner) pays least attention to language form. Tarone’s approach is then contrasted with a cognitive, information-processing view on IL variability. I try to demonstrate how the two approaches can be combined so as to provide a more fruitful basis for research on second-language acquisition (SLA) processes. I then describe a study of my own, to illustrate two methodological points in the empirical study of IL variability concerning (1) the usefulness of pretest procedures to increase the chances that L2 learners will in fact exhibit variability in their IL performance, and (2) the importance of making a conceptual and empirical distinction between “task” and “task requirements.”