No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
A systematic review of evidence on psychometric properties of information literacy tests
Abstract
Purpose
This paper aims to present the results of a systematic review of the evidence on psychometric properties of information literacy (IL) tests.
Design/methodology/approach
A two-stage search strategy was used to find relevant studies in two subject and three general databases. A descriptive review of test characteristics and psychometric properties was presented. The review included 29 studies describing psychometric properties of 18 IL tests.
Findings
It was found that the classical test theory was applied for all tests. However, the item response theory was also applied in three cases. Most of the psychometric tests were developed in the USA using ACRL IL competency standards. The most commonly used psychometric analyses include content validity, discriminant validity and internal consistency reliability.
Research limitations/implications
Only studies in English language are included in this review.
Practical implications
The study recommends that standards should be developed for the use and reporting of psychometric measures in designing IL tests. Librarians need to be trained in psychometric analysis of tests.
Originality/value
It is the first study that systematically reviewed psychometric properties of IL tests. The findings are useful for librarians who are teaching IL courses.
... Measures of knowledge, usually multiple-choice tests, are another common way to measure IL (see Mahmood, 2017;Walsh, 2009 for reviews). These are less reliant on students' more or less accurate self-assessments, and they tend to be relatively easy to administer and score. ...
... These are less reliant on students' more or less accurate self-assessments, and they tend to be relatively easy to administer and score. Despite their popularity, relatively few of these measures have been psychometrically evaluated (Mahmood, 2017), and some of the more thoroughly evaluated measures are only commercially available and, therefore, less accessible to most IL practitioners. ...
... Another peculiarity of the literature on the development and evaluation of IL tests is the reliance on internal consistency measures of reliability, in particular Cronbach's alpha. Of the 16 studies reviewed by Mahmood (2017), 14 report alphas, while only four report temporal consistency (i.e., test-retest reliability). Given the aforementioned lack of factor analytic evidence regarding the dimensionality of IL tests, and the seemingly pervasive construal of IL as a multifaceted construct in extant definitions and frameworks, this is potentially problematic. ...
This study touches upon three major themes in the field of information literacy (IL): the assessment of IL, the association between IL knowledge and skills, and the dimensionality of the IL construct. Three quantitative measures were developed and tested with several samples of university students to assess knowledge and skills for core facets of IL. These measures are freely available, applicable across disciplines, and easy to administer. Results indicate they are likely to be reliable and support valid interpretations. By measuring both knowledge and practice, the tools indicated low to moderate correlations between what students know about IL, and what they actually do when evaluating and using sources in authentic, graded assignments. The study is unique in using actual coursework to compare knowing and doing regarding students’ evaluation and use of sources. It provides one of the most thorough documentations of the development and testing of IL assessment measures to date. Results also urge us to ask whether the source-focused components of IL – information seeking, source evaluation and source use – can be considered unidimensional constructs or sets of disparate and more loosely related components, and findings support their heterogeneity.
... Reliability and validity are the fundamental concepts for the classical response theory. Whereas the item response theory models the association between latent traits and responses to test items (Mahmood, 2017a). The reporting of psychometric characteristics of data collection instruments utilized in a particular research study ensures readers about appropriate utilization and interpretation of the scale. ...
... The common definitions of different types of reliability and validity measures were used by the authors for data extraction and its interpretation. These definitions given in Table 2 were adopted from similar studies in the area of information literacy (Mahmood, 2017a(Mahmood, , 2017b. Table 2 Definitions of reliability and validity measures adopted in the review ...
... The extent that measures of constructs that are theoretically unrelated and are independent of one another. Sources: Adopted from Mahmood (2017aMahmood ( , 2017b ...
This systematic review examined the evidence of psychometric properties of scales available in studies reporting surveys measuring information related anxieties such as library anxiety, information seeking anxiety, and information anxiety. A systematic search in four databases such as Web of Science, Scopus, LISA, and LISTA was carried out using the keywords 'library anxiety', 'information anxiety', 'information seeking anxiety', and 'information seeking' AND 'anxiety'. This review included those studies reporting the use of any scale measuring information related anxiety published in the English language and included all type of documents (e.g. journal articles, conference papers, book chapters, theses/dissertations, research reports). The screening process resulted in 45 studies meeting the eligibility criterion. The extracted data included author names, year of publication, type of scale used, scale title, background, type of construct assessed, number of items in the scale, scale origin, studies reporting use, studies contributing psychometric information, scale availability, and psychometric properties reported. The results indicated nine instruments assessing information-related anxieties. The classical test theory was applied for eight instruments. No psychometric properties were reported for a single instrument. Most psychometric instruments were developed in the United States. Face/Content validity through experts, construct validity through exploratory and confirmatory factor analysis, and internal consistency reliability through Cronbach alpha was the most commonly used psychometric analysis. None of these studies applied the Rasch model of modern item response theory for psychometric examination. This review has serious implications on the inferences drawn by the practitioners and researchers based on the earlier assessment of information related anxieties. It suggests the development of standards for not only designing new psychometric tests but also for the use and reporting of such tests. This study contributes to the existing research on information-related anxieties by systematic reviewing the evidence of psychometric properties as no such study available so far.
... The results further corroborate with the work of Mahmood (2017). The purpose of Mahmood's (2017) was to present the results of a review of the evidence on psychometric properties of information literacy tests. ...
... The results further corroborate with the work of Mahmood (2017). The purpose of Mahmood's (2017) was to present the results of a review of the evidence on psychometric properties of information literacy tests. The study found that the most commonly used psychometric analysis included content validity, discrimination validity and internal consistent reliability. ...
Background: The validity and reliability of research outputs are important elements of the research trail. They drive accuracy, transparency, and minimize researcher biases, contributing to rigor and dependability. This paper reviews the frequency of published articles reporting the psychometric properties of the scales/subscales employed in educational research.
Methods: We conducted a systematic review of psychometric properties in educational research papers published between 2010 and 2020 from 15 education-related journals. In our search, we included quantitative studies with primary data. The methodological quality assessment was performed using trained reviewers. The search was conducted using PRISMA 2020 to identify, screen eligible papers for inclusion. The extracted was analyzed using SPSS v25 while reported and interpreted in descriptive statistics.
Findings: We extracted 763 papers published between 2010 and 2020 from 15 education-related journals. More than half of the articles reviewed did not report either validity (n=456 out of 763, 59.8%) or reliability (n=400, out of 763, 52.4%) statistic. For those reporting either validity or reliability, the alpha coefficient was the most widely used statistic to establish reliability (n=185, 50.9%) and correlation coefficient was frequently reported (n=219, 71.3%) for validity.
Conclusions: The paper concluded that to produce dependable conclusions and recommendations in educational research, it is imperative for researchers to pursue psychometric properties to ground their findings and take-home learning.
Keywords: validity, reliability, educational research, psychometric properties
... With regard to the validation of the quality of the assessment instruments, the classical test theory was applied in most of the tests identified, and the most commonly performed analyses were content and discriminant validity and internal consistency reliability [59]. Therefore, experts argued the need to have free available assessment instruments for measuring IDL, performing a more effective assessment, validated and independent of the domain and the context [57]. ...
Until recently, most of the digital literacy frameworks have been based on assessment frameworks used by commercial entities. The release of the DigComp framework has allowed the development of tailored implementations for the evaluation of digital competence. However, the majority of these digital literacy frameworks are based on self-assessments, measuring only low-order cognitive skills. This paper reports on a study to develop and validate an assessment instrument, including interactive simulations to assess citizens’ digital competence. These formats are particularly important for the evaluation of complex cognitive constructs such as digital competence. Additionally, we selected two different approaches for designing the tests based on their scope, at the competence or competence area level. Their overall and dimensional validity and reliability were analysed. We summarise the issues addressed in each phase and key points to consider in new implementations. For both approaches, items present satisfactory difficulty and discrimination indicators. Validity was ensured through expert validation, and the Rasch analysis revealed good EAP/PV reliabilities. Therefore, the tests have sound psychometric properties that make them reliable and valid instruments for measuring digital competence. This paper contributes to an increasing number of tools designed to evaluate digital competence and highlights the necessity of measuring higher-order cognitive skills.
How information literate are students in higher education, and how accurate is their metacognition related to that ability? Are students’ perceived needs to learn more and their level of interest in becoming information literate related to their pursuit of information literacy (IL) skill development? First-year undergraduates, master’s, and PhD students ( N = 760) took an objective IL test and estimated their scores both before and after the test. IL ability, as well as students’ estimation of their IL ability, increased with higher education experience and IL test experience, though also varied notably within groups. Low-performers tended to overestimate their abilities, while high-performers tended to underestimate them—both evidence of the Dunning-Kruger effect. Furthermore, gender comparisons revealed that men tended to estimate higher, and more accurate, scores than women. Finally, PhD students reported greater interest in becoming information literate than undergraduates. Although undergraduates felt a greater need to learn more, PhD students were more inclined to pursue IL growth. For both groups, interest in becoming information literate correlated far more with their likelihood to invest effort into developing IL competencies than their perceived need to know more. What implications might these findings have for how we conceptualize the teaching of IL?
Purpose
Advances in information technology now permit the recording of massive and diverse process data, thereby making data-driven evaluations possible. This study discusses whether teachers’ information literacy can be evaluated based on their online information behaviors on online learning and teaching platforms (OLTPs).
Design/methodology/approach
First, to evaluate teachers’ information literacy, the process data were combined from teachers on OLTP to describe nine third-level indicators from the richness, diversity, usefulness and timeliness analysis dimensions. Second, propensity score matching (PSM) and difference tests were used to analyze the differences between the performance groups with reduced selection bias. Third, to effectively predict the information literacy score of each teacher, four sets of input variables were used for prediction using supervised learning models.
Findings
The results show that the high-performance group performs better than the low-performance group in 6 indicators. In addition, information-based teaching and behavioral research data can best reflect the level of information literacy. In the future, greater in-depth explorations are needed with richer online information behavioral data and a more effective evaluation model to increase evaluation accuracy.
Originality/value
The evaluation based on online information behaviors has concrete application scenarios, positively correlated results and prediction interpretability. Therefore, information literacy evaluations based on behaviors have great potential and favorable prospects.
The construction and validity of an assessment tool mapped to objectives in a high enrollment credit-bearing information literacy course delivered primarily online is the focus of this article. An open book and non-proctored objective test can be a reliable measure for assessing student competencies in basic information literacy skills, both at the course level and for reporting to national accrediting bodies and state agencies. An analysis of overall student performance on test items that are mapped to information literacy outcomes helps to identify competencies that need improvement in a course, as well as provide a baseline for informing the process of assessing student learning outcomes in an undergraduate curriculum.
: Protocols of systematic reviews and meta-analyses allow for planning and documentation of review methods, act as a guard against arbitrary decision making during review conduct, enable readers to assess for the presence of selective reporting against completed reviews, and, when made publicly available, reduce duplication of efforts and potentially prompt collaboration. Evidence documenting the existence of selective reporting and excessive duplication of reviews on the same or similar topics is accumulating and many calls have been made in support of the documentation and public availability of review protocols. Several efforts have emerged in recent years to rectify these problems, including development of an international register for prospective reviews (PROSPERO) and launch of the first open access journal dedicated to the exclusive publication of systematic review products, including protocols (BioMed Central's Systematic Reviews). Furthering these efforts and building on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines, an international group of experts has created a guideline to improve the transparency, accuracy, completeness, and frequency of documented systematic review and meta-analysis protocols--PRISMA-P (for protocols) 2015. The PRISMA-P checklist contains 17 items considered to be essential and minimum components of a systematic review or meta-analysis protocol.This PRISMA-P 2015 Explanation and Elaboration paper provides readers with a full understanding of and evidence about the necessity of each item as well as a model example from an existing published protocol. This paper should be read together with the PRISMA-P 2015 statement. Systematic review authors and assessors are strongly encouraged to make use of PRISMA-P when drafting and appraising review protocols.
Impairments in social functioning are associated with an array of adverse outcomes. Social skills measures are commonly used by health professionals to assess and plan the treatment of social skills difficulties. There is a need to comprehensively evaluate the quality of psychometric properties reported across these measures to guide assessment and treatment planning.
To conduct a systematic review of the literature on the psychometric properties of social skills and behaviours measures for both children and adults.
A systematic search was performed using four electronic databases: CINAHL, PsycINFO, Embase and Pubmed; the Health and Psychosocial Instruments database; and grey literature using PsycExtra and Google Scholar. The psychometric properties of the social skills measures were evaluated against the COSMIN taxonomy of measurement properties using pre-set psychometric criteria.
Thirty-Six studies and nine manuals were included to assess the psychometric properties of thirteen social skills measures that met the inclusion criteria. Most measures obtained excellent overall methodological quality scores for internal consistency and reliability. However, eight measures did not report measurement error, nine measures did not report cross-cultural validity and eleven measures did not report criterion validity.
The overall quality of the psychometric properties of most measures was satisfactory. The SSBS-2, HCSBS and PKBS-2 were the three measures with the most robust evidence of sound psychometric quality in at least seven of the eight psychometric properties that were appraised. A universal working definition of social functioning as an overarching construct is recommended. There is a need for ongoing research in the area of the psychometric properties of social skills and behaviours instruments.
To explore inter-rater agreement between reviewers comparing reliability and validity of checklist forms that claim to assess the communication skills of undergraduate medical students in Objective Structured Clinical Examinations (OSCEs).
Papers explaining rubrics of OSCE checklist forms were identified from Pubmed, Embase, PsycINFO, and the ProQuest Education Databases up to 2013. Included were those studies that report empirical validity or reliability values for the communication skills assessment checklists used. Excluded were those papers that did not report reliability or validity.
Papers focusing on generic communication skills, history taking, physician-patient communication, interviewing, negotiating treatment, information giving, empathy and 18 other domains (ICC -0.12-1) were identified. Regarding the validity and reliability of the communication skills checklists, agreement between reviewers was 0.45.
Heterogeneity in the rubrics used in the assessment of communication skills and a lack of agreement between reviewers makes comparison of student competences within and across institutions difficult.
Consideration should be afforded to the adoption of a standardized measurement instrument to assess communication skills in undergraduate medical education. Future research will focus upon evaluating the potential impact of adoption of a standardized measurement instrument.
Copyright © 2015 Elsevier Ireland Ltd. All rights reserved.
Three studies were conducted to develop a test for academic information-seeking skills in psychology students that measures both procedural and declarative aspects of the concept. A skill decomposition breaking down information-seeking into 10 sub skills was used to create a situational judgment test with 22 items. A scoring key was developed based on expert ratings (N = 14). Subsequently, the test was administered to two samples of N = 78 and N = 81 psychology students. Within the first sample, the scale reached an internal consistency (Cronbach’s Alpha) of α = .75. Scale validity was investigated with data from the second sample. High correlations between the scale and two different information search tasks (r = .42 to .64; p < .001) as well as a declarative information literacy test (r = .51; p < .001) were found. The findings are discussed with regard to their implications for research and practice.
At the end of 2012 a short course, the Certificate of Information Literacy, was registered by the Cape Peninsula University of Technology Libraries and was first introduced in 2013. The course consisted of five modules and a summative assessment. The assessment, accessed via the Learning Management System, comprised 100 multiple-choice questions of which students received a randomly selected fifty to answer in ninety minutes. This article reports on the effectiveness and validity of the online assessment for the Certificate of Information Literacy: whether the assessment instrument accurately measures the information literacy skills of the students and whether the test can be considered valid and reliable. There are two parts to the study, but this paper focuses on the first part which is an item analysis on the assessment data of the students who completed the test in 2013. The results highlight weaknesses and problematic areas in the test. The second part of the study will focus on comparing the results students received in the test with the results of a subject-specific essay assignment to measure the level of application of their information literacy skills.
This pilot study was developed to determine if the University’s students were proficient in information literacy (IL) based on the requisite skills defined by ALA (2000), to define faculty and student perceptions and behaviours related to IL and to test an evaluation rubric using empirical inquiry and triangulated methods. Findings suggested that not all students ( n =164) had satisfactory IL skills even at the senior student level. While 4th year college students (seniors n =91) fared better on an IL survey when compared to 1st year college students (freshmen n= 53), analysis of the senior students’ theses led researchers to believe that students were most likely not skilled in this area, and had an inflated opinion of their own IL abilities. Overall, students felt they were less IL challenged compared to the faculty’s ( n= 55) observation of the IL challenges experienced by the students. Students’ self-assessment of their literacy skills may have been coloured by the propensity of the faculty to over-edit students’ papers rather than simply providing constructive feedback, thus altering the natural end result. These authors used a triangulated approach including thesis review, comparisons between student and faculty survey responses and comparison of findings from the theses and the student and faculty surveys. Findings and discussion of methodology will hopefully provide valuable lessons for those interested in assessing students’ IL.
This article synthesized evidence for the validity and reliability of the Strengths and Difficulties Questionnaire in children aged 3-5 years. A systematic review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement guidelines was carried out. Study quality was rated using the Consensus-based Standards for the Selection of Health Measurement Instruments. In total, 41 studies were included (56 manuscripts). Two studies examined content and cultural validity, revealing issues with some questions. Six studies discussed language validations with changes to some wording recommended. There was good evidence for discriminative validity (Area Under the Curve ≥ 0.80), convergent validity (weighted average correlation coefficients ≥ 0.50, except for the Prosocial scale), and the 5-factor structural validity. There was limited support for discriminant validity. Sensitivity was below 70% and specificity above 70% in most studies that examined this. Internal consistency of the total difficulty scale was good (weighted average Chronbach’s alpha parents’ and teachers’ version 0.79 and 0.82) but weaker for other subscales (weighted average parents’ and teachers’ range 0.49-0.69 and 0.69-0.83). Inter-rater reliability between parents was moderate (correlation coefficients range 0.42-0.64) and between teachers strong (range 0.59-0.81). Cross-informant consistency was weak to moderate (weighted average correlation coefficients range 0.25-0.45). Test-retest reliability was mostly inadequate. In conclusion, the lack of evidence for cultural validity, criterion validity and test-retest reliability should be addressed given wide-spread implementation of the tool in routine clinical practice. The moderate level of consistency between different informants indicate that an assessment of a pre-schooler should not rely on a single informant.
In this research, the relations between the information literacy self-efficacy and computer self-efficacy as well as the achievement of information literacy are examined. The scale of information literacy self-efficacy, computer self-efficacy and information literacy achievement test are applied to 783 teacher candidates receiving education at Faculty of Education in Fırat University. It is determined with the regression analysis how the achievement of information literacy self-efficacy, computer self-efficacy and information literacy affect each other; when the regression (beta) coefficients standardized as a result of the analysis are examined, it is observed that the computer self-efficacy has a positive effect on information literacy self-efficacy. Nonetheless, it is observed that the information literacy self-efficacy has no significant effect on the achievement of information literacy, and also the computer self-efficacy has no significant effect on the achievement of information literacy.
Protocols of systematic reviews and meta-analyses allow for planning and documentation of review methods, act as a guard against arbitrary decision making during review conduct, enable readers to assess for the presence of selective reporting against completed reviews, and, when made publicly available, reduce duplication of efforts and potentially prompt collaboration. Evidence documenting the existence of selective reporting and excessive duplication of reviews on the same or similar topics is accumulating and many calls have been made in support of the documentation and public availability of review protocols. Several efforts have emerged in recent years to rectify these problems, including development of an international register for prospective reviews (PROSPERO) and launch of the first open access journal dedicated to the exclusive publication of systematic review products, including protocols (BioMed Central's Systematic Reviews). Furthering these efforts and building on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines, an international group of experts has created a guideline to improve the transparency, accuracy, completeness, and frequency of documented systematic review and meta-analysis protocols-PRISMA-P (for protocols) 2015. The PRISMA-P checklist contains 17 items considered to be essential and minimum components of a systematic review or meta-analysis protocol.This PRISMA-P 2015 Explanation and Elaboration paper provides readers with a full understanding of and evidence about the necessity of each item as well as a model example from an existing published protocol. This paper should be read together with the PRISMA-P 2015 statement. Systematic review authors and assessors are strongly encouraged to make use of PRISMA-P when drafting and appraising review protocols.
© BMJ Publishing Group Ltd 2014.
Objective:
To develop and psychometrically evaluate an information literacy (IL) self-efficacy survey and an IL knowledge test.
Methods:
In this test-retest reliability study, a 25-item IL self-efficacy survey and a 50-item IL knowledge test were developed and administered to a convenience sample of 53 chiropractic students. Item analyses were performed on all questions.
Results:
The IL self-efficacy survey demonstrated good reliability (test-retest correlation = 0.81) and good/very good internal consistency (mean κ = .56 and Cronbach's α = .92). A total of 25 questions with the best item analysis characteristics were chosen from the 50-item IL knowledge test, resulting in a 25-item IL knowledge test that demonstrated good reliability (test-retest correlation = 0.87), very good internal consistency (mean κ = .69, KR20 = 0.85), and good item discrimination (mean point-biserial = 0.48).
Conclusions:
This study resulted in the development of three instruments: a 25-item IL self-efficacy survey, a 50-item IL knowledge test, and a 25-item IL knowledge test. The information literacy self-efficacy survey and the 25-item version of the information literacy knowledge test have shown preliminary evidence of adequate reliability and validity to justify continuing study with these instruments.
Abstract Despite coming of age with the Internet and other technology, many college students lack the information and communication,technology (ICT) literacy skills necessary to navigate, evaluate, and use the overabundance ofinformation available today. This paper describes the development and early administrations of the ETS iSkills™ assessment, an Internet-based assessment of information literacy skills that arise in the context of technology. From the earliest stages to the present, the library community has been directly involved in the design, development, review, field trials, and administrations to assure the assessment and scores are valid, reliable, authentic, and useful. , Assessment of ICT literacy 3 Katz, I.R. (2007). Testing information literacy in digital environments: The ETS iSkills™ assessment. Information
The Information Literacy Test (ILT) was developed to meet the need for a standardized instrument that measures student proficiency regarding the ACRL Information Literacy Competency Standards for Higher Edu- cation. The Web-based, multiple-choice ILT measures both lower- and higher-order skills. Evidence is presented that the ILT scores provide reliable and valid measures of information literacy. In addition, a rigorous standard setting method was used to identify score values correspond- ing to various absolute levels of proficiency. The ILT can be used to help institutions measure student information literacy outcomes and determine the effectiveness of instruction programs.
Purpose
This article seeks to provide an overview of the development, construction and evaluation of an information literacy testing instrument connected with the VOILA! online information literacy tutorials, developed by librarians at Hunter College libraries for students enrolled in the college's first‐year orientation seminar (ORSEMs).
Design/methodology/approach
The VOILA! online information literacy tutorial, developed by librarians at Hunter College, has a test as one of its components. All incoming students registered in the college's orientation seminars (ORSEMs) are required to go through the VOILA! tutorials and take the test. The longitudinal development and evaluation of the test are reviewed, including initial statement of competencies, test creation, test construction (assessment type, question formulation and question type) and test evolution. Validity and reliability are also discussed in detail. Finally, a review of the test results including general statistics and the connection of student performance to test construction is provided.
Findings
Test construction was given a great deal of attention during the development of VOILA!, as librarians wanted an instrument that was valid and reliable. The importance of iterative test design and attention to the details of test construction is stressed in the article, and the methodology used to determine validity and reliability is presented. Those interested in testing information literacy skills must be able to examine the instrument, the results and the design process in a way that contributes meaningfully to the preparation of students to undertake research at the college level.
Research limitations/implications
Librarians will continue to modify the VOILA! test in keeping with evaluation results, and the assessment will continue. Thus the results presented may not fully represent the final evaluative outcome.
Practical implications
This research is useful for information professionals who are interested in test design, construction and evaluation. The longitudinal study presents clear, practical information that can be used as a case study for other libraries interested in this area. The VOILA! test itself is included as an Appendix, and can be used as a model for other libraries to consult.
Originality/value
A literature review revealed a number of articles about testing and information literacy, but very few, if any, articles approached the discussion from a longitudinal test construction standpoint. In this way, the study presents information that is original. The study would be valuable for information professionals who are interested in test design, construction and evaluation.
Background
Despite a plethora of studies examining the effect of increased urbanisation on health, no single study has systematically examined the measurement properties of scales used to measure urbanicity. It is critical to distinguish findings from studies that use surrogate measures of urbanicity (e.g. population density) from those that use measures rigorously tested for reliability and validity. The purpose of this study was to assess the measurement reliability and validity of the available urbanicity scales and identify areas where more research is needed to facilitate the development of a standardised measure of urbanicity.
Methods
Databases searched were MEDLINE with Full Text, CINAHL with Full Text, and PsycINFO (EBSCOhost) as well as Embase (Ovid) covering the period from January 1970 to April 2012. Studies included in this systematic review were those that focused on the development of an urbanicity scale with clearly defined items or the adoption of an existing scale, included at least one outcome measure related to health, published in peer-reviewed journals, the full text was available in English and tested for validity and reliability.
Results
Eleven studies met our inclusion criteria which were conducted in Sri Lanka, Austria, China, Nigeria, India and Philippines. They ranged in size from 3327 to 33,404 participants. The number of scale items ranged from 7 to 12 items in 5 studies. One study measured urban area socioeconomic disadvantage instead of urbanicity. The emerging evidence is that increased urbanisation is associated with deleterious health outcomes. It is possible that increased urbanisation is also associated with access and utilisation of health services. However, urbanicity measures differed across studies, and the reliability and validity properties of the used scales were not well established.
Conclusion
There is an urgent need for studies to standardise measures of urbanicity. Longitudinal cohort studies to confirm the relationship between increased urbanisation and health outcomes are urgently needed.
No population exists where it is more important to produce information literate individuals than teacher candidates, yet few would suggest that practitioners newly entering the field are adequately prepared to model and teach information literacy to their students. Consequently, information literacy has recently been established as a key outcome by a number of teacher education accrediting bodies and professional associations. Only in the last few years has there been an attempt to develop a standardized scale to assess general information literacy skills, and at the time of this writing no standardized tool exists that measures the information literacy levels of teacher candidates. This study documents the development and validation of a standardized instrument to measure teacher candidates' information literacy skills levels based on the International Society for Technology in Education's 2000 National Educational Technology Standards for Teachers and the Association of College and Research Libraries' 2000 Information Literacy Competency Standards for Higher Education. Undergraduate students enrolled in the teacher education program at the University of Central Florida were identified and asked to complete a test consisting of 22 multiple-choice test items and 13 demographic and self-percept items. A number of procedures designed to enhance validity and reliability of the scale were integrated throughout its development. Results of the test were also submitted to analysis. This project is part of a national initiative to develop standardized information literacy assessment tools specific to a discipline, and is spear-headed by the Project for the Standardized Assessment of Information Literacy Skills and the Institute for Library and Information Literacy Education. Use of the instrument described herein will allow librarians and teaching faculty a means to inform curricular and instructional decisions, and results can be used for internal and external benchmarking of education students' information literacy skills levels. (Contains 16 tables and 1 figure.)
Interest in developing ways to assess information literacy has been growing for several years. Many librarians have developed their own tools to assess aspects of information literacy and have written articles to share their experiences. This article reviews the literature and offers readers a flavour of the methods being used for assessment: those which are popular within the field and also illustrative examples from some of the case studies found, particularly where they show how reliability and validity of the methods, have been considered. It does not aim to be an exhaustive list of case studies or methods, but a representative sample to act as a “jumping off point” for librarians considering introducing assessment of information literacy into their own institutions.
Few constituencies exist where it is more important to produce information literate individuals than teacher candidates, yet rarely is it suggested that practitioners entering the field are adequately prepared to teach and model information literacy to their students. As a result, information literacy has been established as a key outcome by a number of teacher education accrediting bodies and professional associations. Corollary to this initiative is the effort to develop valid instruments that assess information literacy skills. Yet, at the time of this dissertation, no rigorously reviewed instruments were uncovered that measure the information literacy skills levels of teacher candidates. The study describes the development and validation of the Beile Test of Information Literacy for Education (B-TILED). Funded in part by the Institute for Library and Information Literacy Education and the Institute of Museum and Library Services, the study is part of a national initiative spear-headed by the Project for the Standardized Assessment of Information Literacy Skills (SAILS). Test content is based on nationally recognized standards from the International Society for Technology in Education and the Association of College and Research Libraries. Procedures designed to enhance the scale's validity were woven throughout its development. 172 teacher education students at a large, metropolitan university completed a protocol consisting of 22 test items and 13 demographic and self-percept items. This instrument can be used to inform curricular and instructional decisions and to provide evidence of institutional effectiveness for program reviews.
This paper describes the South Dakota state universities' development of a dual-measure instrument to test Information Literacy (IL) following the Association of College & Research Libraries (ACRL) Information Literacy Competency Standards for Higher Education and using classical and modern psychometric methods. This tool can be used as an exit assessment for undergraduates and as an entrance assessment for advanced learners to indicate deficiencies in IL that must be addressed early in graduate education.
This contribution summarizes the results of the evaluation of information literacy (IL) of 677 higher education students enrolled in study programs of life sciences, health, technologies, and social sciences at six Slovenian faculties. The information literacy test (ILT) that was developed, verified and validated by the authors in a previous work served as the IL measuring instrument. Statistical analyses of ILT responses were performed in SPSS. The results suggest that, on average, Slovenian students’ IL is satisfactory and improves with years of education. On average, students know information sources and adequately evaluate the collected information. They possess skills to use the information in academic work and to synthesize data into knowledge. However, students are less proficient in advanced search strategies available in scientific and patent databases. The main deficit in students’ knowledge is evident in topics related to intellectual property rights and in ethical issues related to acquisition and use of information. Students that participated in an IL-specific study course significantly improved their ILT achievement, most significantly in topics where their pre-knowledge was lower.
Psychometrics and measurement are important for all aspects of psychological research and especially so in social/personality psychology.
This volume provides conceptual and practical foundations in scale construction and psychometrics for producers and consumers of social and personality research. It covers basic principles, practices, and processes in scale construction, scale evaluation, scale use and interpretation of research results in the context of psychological measurement. It explains fundamental concepts and methods related to dimensionality, reliability, and validity. In addition, it provides relatively non-technical introductions to special topics and advanced psychometric perspectives such as Confirmatory Factor Analysis, Generalizability Theory, and Item Response Theory.
Social/personality research is often grounded in effective measurement, but poor measurement can and does compromise the meaningfulness of psychological research. This volume is intended to raise awareness and understanding of issues that will enhance even further the generally good conduct and interpretation of research in social and personality psychology.
This text will be perfect for all advanced students and researchers in social and personality psychology using psychometrics or measurement as part of their studies or research.
The construction and validity of an assessment tool mapped to objectives in a high enrollment creditbearing information literacy course delivered primarily online is the focus of this article. An open book and non-proctored objective test can be a reliable measure for assessing student competencies in basic information literacy skills, both at the course level and for reporting to national accrediting bodies and state agencies. An analysis of overall student performance on test items that are mapped to information literacy outcomes helps to identify competencies that need improvement in a course, as well as provide a baseline for informing the process of assessing student learning outcomes in an undergraduate curriculum.
Information literacy is defined as the capacity to identify, locate, evaluate, use, create, store and index information. This article deals with the assessment of information literacy. Therefore a list of skills an information literate person should possess in the 21st century was developed. Based on these indicators, multiple-choice tests were developed which focus on the target groups of pupils, university students, teachers and researchers. With these tests it is possible to assess the knowledge in terms of information literacy.
BACKGROUND
Bullying is a significant problem in schools and measuring this concept remains problematic. The purposes of this study were to (1) identify the published self-report measures developed to assess youth bullying; (2) evaluate their psychometric properties and instrument characteristics; and (3) evaluate the quality of identified psychometric papers evaluating youth bullying measures. METHODSA systematic review of the literature was conducted using 4 electronic databases. Data extraction and appraisal of identified instruments were completed using a standardized method and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. RESULTSThirty-one articles describing 27 self-report instruments were evaluated in our analysis. Quality assessments ranged from 18% to 91%, with 6 papers reaching or exceeding a quality score of 75%. Limited evidence supporting the reliability, validity, and responsiveness of existing youth bullying measures was identified. CONCLUSIONS
Evidence supporting the psychometric soundness of the instruments identified was limited. Many measures were in early development and additional evaluation is necessary to validate their psychometric properties. A pool of instruments possesses acceptable initial psychometric dependability for selected assessment purposes. These findings have significant implications for assessing youth bullying and designing and evaluating school-based interventions.
Purpose
– This paper aims to report on the development of an information literacy test for German psychology students.
Design/methodology/approach
– The test consists of 22 items covering two facets of information literacy (searching for information and evaluating information) which are mapped on Standards Two and Three of the information literacy framework provided by the Association of College and Research Libraries (ACRL, 2000). A sample of N =64 German psychology students including n =22 freshmen, n =21 advanced students and n =21 PhD students completed the test. The freshmen and advanced students also performed an academic literature search task.
Findings
– The test has acceptable internal consistency (Cronbach's Alpha between 0.73 and 0.82). An analysis of variance showed that PhD students scored higher than advanced students, who, in turn, scored higher than freshmen. Additionally, the score on the first scale (Evaluating) showed a significant relationship with the performance on the academic literature search task.
Practical implications
– The test can be used to determine training needs among German psychology students, or to evaluate instruction programs.
Originality/value
– The new measure is the only standardized information literacy test for German-speaking populations for which psychometric properties have been reported.
Purpose
Since the education of Master of Science students in the discipline of agriculture is catalectic without libraries, the present paper seeks to assess the information literacy competency (ILC) of post graduate (PG) students at Haryana Agricultural University, Hisar, India, along with the impact of instruction initiatives in this respect.
Design/methodology/approach
Based on the Association of College and Research Libraries (ACRL) Information Literacy Standards for Science and Engineering/Technology, 2006, an instrument was developed with 79 scoring items in the first part of the questionnaire along with non‐scoring items in the second part. Broad headings of the questions asked are shown in detail. Validity and reliability of the instrument are also reported.
Findings
Information literacy competency of PG students has been found satisfactory along with significant difference between the first and second year students. Results of the paper distinguish between the first and second year students in the light of five information literacy (IL) standards along with other considerations. Various instruction initiatives were found to have a positive impact upon the ILC of PG students.
Practical implications
Since the study has been able to establish a positive relationship between instruction initiatives and ILC, such initiatives may be introduced in other academic institutions.
Originality/value
None of the earlier reported research instruments had used science and engineering/technology IL standards, especially upon PG students in agriculture. This study further paved the way for the development of such an nstrument along with assessing the ILC of PG students and impact of instruction initiatives in an agricultural university.
The purpose of this study was to investigate the relationship between self-directed learning and information literacy. Participants completed the Personal Orientation in Self-Directed Learning Scale ([PRO-SDLS], Stockdale, 2003) and the Information Literacy Test ([ILT], James Madison University, 2003). The PRO-SDLS is a self-report scale consisting of 25 statements about self-directed learning preferences in college classrooms. The ILT is a 60-item multiple-choice test that assesses the information literacy skills of college students. Correlation, ANOVA, and multiple regressions were used to test relationships and differences between self-directed learning and information literacy. Despite claims that teaching information literacy creates self-directed learners, composite scores on the PRO-SDLS and the ILT indicated no statistically significant relationship exists. Likewise, no statistically significant differences were found between the bachelors, masters, or doctoral level participant scores. While composite scores on the PRO-SDLS did not predict scores on the ILT, there was a negative, statistically significant relationship between the Initiative factor on the PRO-SDLS and ACRL (2000) Information Literacy Competency Standard 5 – Ethics & Understanding sub-scale of the ILT. Implications for practice and suggestions for further research are proposed along with discussions and conclusions.
Purpose
With a call for increased accountability for student learning across higher education, it is becoming more important for academic libraries to show their value to the greater university community with the use of quantitative data. This paper seeks to describe the development of an information literacy test at the University of Arizona to measure student learning in an online credit course. In order to measure the impact of an online course, a test that was statistically valid, and reliable was created by local librarians.
Design/methodology/approach
The methodology involved administering test items to undergraduate students enrolled in an online information literacy course and applying both classical test theory and item response theory models to evaluate the validity and reliability of test items. This study included the longitudinal and cross‐sectional development of test items for pre and post‐testing across different student groups. Over the course of two semesters, 125 items were developed and administered to over 1,400 students.
Findings
The creation of test of items and the process of making test items reliable and valid is discussed in detail. Items were checked for construct validity with the use of a national standardized test of information literacy (SAILS). Locally developed items were found to have a higher than average reliability rating.
Practical implications
The process described here offers a method for librarians without a background in assessment to develop their own statistically valid and reliable instrument.
Originality/value
One of the unique features of this research design was the correlation of SAILS items with local items to test for validity. Although SAILS items have been used by many libraries in the past, they have not been used to create new test items. The use of the original SAILS test items is a valuable resource for instruction librarians developing items locally.
Librarians and learning researchers at Central Michigan University collaboratively developed an online tool that assesses how student research attitudes and perceptions correlate to their actual research skills in order to educate them about state-of-the-art library resources and prepare them to write high-quality research papers. This article describes the reasons for developing the assessment as well as the design process and technical characteristics.
on a project to develop an instrument for program-matic-level assessment of information literacy skills that is valid—and thus credible—to university administrators and other academic person-nel. Using a systems approach for test development and an item re-sponse theory for data analysis, researchers have undertaken a rigor-ous and replicable process. Once validated, this instrument will be ad-ministered to students to assess entry skills upon admission to the uni-versity and longitudinally to ascertain whether there is significant change in skill levels from admission to graduation. biblical parable on the virtue of tenacity tells of a widow who repeatedly beseeches a judge to grant her request. Fi-nally, the judge, although not sympathetic to her cause, grants her request lest she eventually exhaust [him] with her com-ing. In the golden age of higher educa-tion, when expansion was rapid and funding abundant, the widow's tech-nique may have been effective. In the cur-rent era of finite resources and increased fiscal accountability, however, when li-braries plead their cases for resources to support their information literacy pro-grams, persistence alone does not suffice.
In an era of easy access to information, university students who will soon enter health professions need to develop their information competencies. The Research Readiness Self-Assessment (RRSA) is based on the Information Literacy Competency Standards for Higher Education, and it measures proficiency in obtaining health information, evaluating the quality of health information, and understanding plagiarism.
This study aimed to measure the proficiency of college-age health information consumers in finding and evaluating electronic health information; to assess their ability to discriminate between peer-reviewed scholarly resources and opinion pieces or sales pitches; and to examine the extent to which they are aware of their level of health information competency.
An interactive 56-item online assessment, the Research Readiness Self-Assessment (RRSA), was used to measure the health information competencies of university students. We invited 400 students to take part in the study, and 308 participated, giving a response rate of 77%. The RRSA included multiple-choice questions and problem-based exercises. Declarative and procedural knowledge were assessed in three domains: finding health information, evaluating health information, and understanding plagiarism. Actual performance was contrasted with self-reported skill level. Upon answering all questions, students received a results page that summarized their numerical results and displayed individually tailored feedback composed by an experienced librarian.
Even though most students (89%) understood that a one-keyword search is likely to return too many documents, few students were able to narrow a search by using multiple search categories simultaneously or by employing Boolean operators. In addition, nearly half of the respondents had trouble discriminating between primary and secondary sources of information as well as between references to journal articles and other published documents. When presented with questionable websites on nonexistent nutritional supplements, only 50% of respondents were able to correctly identify the website with the most trustworthy features. Less than a quarter of study participants reached the correct conclusion that none of the websites made a good case for taking the nutritional supplements. Up to 45% of students were unsure if they needed to provide references for ideas expressed in paraphrased sentences or sentences whose structure they modified. Most respondents (84%) believed that their research skills were good, very good, or excellent. Students' self-perceptions of skill tended to increase with increasing level of education. Self-reported skills were weakly correlated with actual skill level, operationalized as the overall RRSA score (Cronbach alpha = .78 for 56 RRSA items).
While the majority of students think that their research skills are good or excellent, many of them are unable to conduct advanced information searches, judge the trustworthiness of health-related websites and articles, and differentiate between various information sources. Students' self-reports may not be an accurate predictor of their actual health information competencies.
To appraise the reported validity and reliability of evaluation methods used in high-quality trials of continuing medical education (CME).
The authors conducted a systematic review (1981 to February 2006) by hand-searching key journals and searching electronic databases. Eligible articles studied CME effectiveness using randomized controlled trials or historic/concurrent comparison designs, were conducted in the United States or Canada, were written in English, and involved at least 15 physicians. Sequential double review was conducted for data abstraction, using a traditional approach to validity and reliability.
Of 136 eligible articles, 47 (34.6%) reported the validity or reliability of at least one evaluation method, for a total of 62 methods; 31 methods were drawn from previous sources. The most common targeted outcome was practice behavior (21 methods). Validity was reported for 31 evaluation methods, including content (16), concurrent criterion (8), predictive criterion (1), and construct (5) validity. Reliability was reported for 44 evaluation methods, including internal consistency (20), interrater (16), intrarater (2), equivalence (4), and test-retest (5) reliability. When reported, statistical tests yielded modest evidence of validity and reliability. Translated to the contemporary classification approach, our data indicate that reporting about internal structure validity exceeded reporting about other categories of validity evidence.
The evidence for CME effectiveness is limited by weaknesses in the reported validity and reliability of evaluation methods. Educators should devote more attention to the development and reporting of high-quality CME evaluation methods and to emerging guidelines for establishing the validity of CME evaluation methods.
The development and validation of all Four TRAILS (Tool for Real-Time Assessment of Information Literacy Skills) tests for K-12 students”, PhD dissertation
- J A Salem
Salem, J.A. Jr. (2014), "The development and validation of all Four TRAILS (Tool for Real-Time
Assessment of Information Literacy Skills) tests for K-12 students", PhD dissertation, Kent State
University.
Comparing information literacy needs of graduate students in selected graduate programs through the technology acceptance model and affordance theory
- J Magliaro
Magliaro, J. (2011), "Comparing information literacy needs of graduate students in selected graduate
programs through the technology acceptance model and affordance theory", PhD dissertation,
University of Windsor.
Initial development of a medical information literacy questionnaire
- S K Morley
Morley, S.K. (2014), "Initial development of a medical information literacy questionnaire", PhD
dissertation, University of New Mexico, Albuquerque.
Information
literacy tests
Can college students evaluate information sources? Validating a Web-based assessment of evaluation skills
- E Snow
Snow, E. (2008), "Can college students evaluate information sources? Validating a Web-based
assessment of evaluation skills", PhD dissertation, University of Colorado, Boulder, CO.
Thinking beyond technology: using the iSkills assessment as evidence to support institutional ICT literacy initiatives
- T Egan
- I R Katz
Egan, T. and Katz, I.R. (2007), "Thinking beyond technology: using the iSkills assessment as evidence
to support institutional ICT literacy initiatives", Knowledge Quest, Vol. 35 No. 5, pp. 36-42.
Assessing information literacy skills: developing a standardized instrument for institutional and longitudinal measurement
- L G O'connor
- C J Radcliff
- J A Gedeon
O'Connor, L.G., Radcliff, C.J. and Gedeon, J.A. (2001), "Assessing information literacy skills: developing
a standardized instrument for institutional and longitudinal measurement", Crossing the Divide:
ACRL Tenth National Conference, Denver, CO, 15-18 March, pp. 163-174.
Using the SAILS test to assess information literacy
- J A Salem
- C J Radcliff
Salem, J.A. and Radcliff, C.J. (2006), "Using the SAILS test to assess information literacy", Proceedings
of the Second Library Assessment Conference on Building Effective, Sustainable, Practical
Assessment, Charlottesville, VI, pp. 131-137.
Preferred reporting items for systematic review and meta-analysis protocols (PRISMAP) 2015: elaboration and explanationInformation literacy competency of post graduate students at Haryana agricultural university and impact of instruction initiatives: a pilot survey
- L Shamseer
- D Moher
- M Clarke
- D Ghersi
- A Liberati
- M Petticrew
- P Shekelle
- L A Stewart
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P. and Stewart, L.A.
(2015), "Preferred reporting items for systematic review and meta-analysis protocols (PRISMAP) 2015: elaboration and explanation", BMJ, Vol. 349 p. g7647
Singh, D. and Joshi, M.K. (2013), "Information literacy competency of post graduate students at
Haryana agricultural university and impact of instruction initiatives: a pilot survey", Reference
Services Review, Vol. 41 No. 3, pp. 453-473.
The Standards for Educational and Psychological Testing
American Educational Research Association, American Psychological Association and National
Council on Measurement in Education (2014), The Standards for Educational and Psychological
Testing, AERA Publications, Washington, DC.
Assessing information literacy: creating generic indicators and target groupspecific questionnaires
- L Beutelspacher
- S In Kurbanoglu
- S Špiranec
- E Grassian
- D Mizrachi
- R Catts
Beutelspacher, L. (2014), "Assessing information literacy: creating generic indicators and target groupspecific questionnaires", in Kurbanoglu, S., Špiranec, S., Grassian, E., Mizrachi, D. and Catts, R.
(Eds), Information Literacy: Lifelong Learning and Digital Citizenship in the 21st Century,
Springer International, pp. 521-530.