The capability of scales to detect change, i.e. sensitivity to change, is important in longitudinal studies. We present a quantitative study comparing the effects of alignment to course content and item granularity (generic vs. concrete wording) on the capability of different scales to capture changes in ICT self-efficacy. In an evaluation of a course on teaching with ICT, we used ICT self-report questionnaires for pre-service teachers at the beginning and end of a one-semester in a treatment and a control group (Nintervention = 278 vs. Ncontrol = 395). After propensity score matching on the imputed data, we applied descriptive, frequentist, and Bayesian methods to examine differential scale behavior. Results show more pronounced longitudinal changes for scales on instructional ICT self-efficacy, especially for scales formulated for concrete tasks. Our results suggest that better alignment and specific formulation enhance sensitivity to change in self-reports, and relate to scale construction and selection in educational evaluations.

Responses to open-ended items (OEI) in surveys in evaluation research are valuable but often short or even missing. We investigate how response behavior can be altered by means of survey design. In a survey on tutoring courses, 3,183 students responded to OEI. Participants were randomly assigned to conditions with or without a motivational cue and with OEI positioned at the beginning or the end of the survey, respectively. We simultaneously inspected response rates and answer lengths using Bayesian multilevel hurdle models. Positioning at the beginning enhanced response rates but reduced answer lengths. The motivational cue raised response rates only at the end of the survey. We also applied two different topic modeling approaches. Resulting topics cover different aspects of teacher behavior and positioning slightly altered the content of the responses. Based on motivational and cognitive theories, we address both the design of questionnaires and analysis approaches to OEI responses.