Although some piecemeal efforts have been made to investigate the validity and use of the Iranian PhD exam, no systematic project has been specifically carried out in this regard. The current study, hence, tried to attend to this void. As such, to ensure a balanced focus on test interpretation and test consequence, and to track evidence derived from a mixed-method study on the validity of Iranian
... [Show full abstract] PhD entrance exam of TEFL (IPEET), this study drew on a hybrid of two argument-based structures: Kane's (1992) argument model and Bennett's (2010) theory of action. Resting on the network of inferences and assumptions borrowed from the hybridized framework, the study investigated the extent to which the proposed assumptions would be supported by empirical evidence. It also examined the unintended consequences that may possibly be revealed through this validity investigation. Three sources of data informed the present study: (a) Test score data from about 1000 PhD applicants' taking IPEET test administered in 2014, (b) questionnaires completed by university professors and PhD students of TEFL, and finally, (c) telephone and focus-group interviews with university professors and PhD students of TEFL, respectively. The results from the analysis of mixed-method data indicated that all the inferences proposed for this study were rebutted, suggesting that some unintended consequences have happened to the technical as well as the decision quality of this test, hence its invalidity. Findings also The Journal of Teaching Language Skills / 7(2), Summer 2015, Ser. 79/4 2 provided valuable insights and suggestions for the betterment of the present content and current policy of IPEET in Iran. Keywords: mixed method study, argument-based validity, theory of action, unintended consequences 1. Introduction A decentralized assessment system was previously practiced to screen PhD applicants in Iran. In the decentralized PhD exam no central bodies from top-tier decision makers such as Ministry of Science, Research and Technology (hereafter, MSRT) and the National Organization for Educational Testing (NOET) were in control of this admission system. Different universities administered their own examination differently in different formats and at different times. The screening was based on a written performance-based assessment (or sometimes an MC test) in which applicants were required to respond to essay-type knowledge questions (or MC questions) based on which those who passed the cut-score (determined and decided by each specific university) were allowed to attend an interview. The overall evaluation was based on the local written performance assessment, the MC test, and the oral interview. However, this traditional system was claimed in the oral literature not to be scientific and fair enough; that is, most of the PhD students were selected from the MA students of the same university. Furthermore, most of the PhD applicants were relatively dissatisfied with the entrance criteria of the higher education in Iran (Kiany, Shayestefar, Ghafar Samar, & Akbari, 2013). Therefore, these problems casted some doubts on the reliability and validity issues of this type of evaluation. Currently, following the criticism leveled against the decentralized admission system, a semi-centralized assessment system is practiced for screening PhD applicants. Every year, a resounding number of MA graduates (NOET news, 2013) from different majors, in 30 capital cities in Iran take part in IPEET. As released on the official sites of NOET and MSRT, the apparent intentions behind introducing this test were both to solve some of the deleterious effects of the decentralized local examinations and to take more control and power on the acceptance and non-acceptance of candidates for doctoral programs. Annually, this test is administered in March and the primary results are released on NOET site at the end of May. The IPEET test subsists of a test of academic talent, a general English proficiency test and a specialized knowledge test, all appearing in MC format. The knowledge test which is aimed at measuring the candidates' expertise in the field of Teaching English as a Foreign Language (TEFL) is supposedly related to the courses students have passed in the MA or even BA program. In fact, it assesses the students' specialized knowledge in areas which are assumed to be the prerequisite for entering the PhD program since the PhD program is built on such areas of knowledge. As such, the knowledge test of IPEET includes questions on