Technical ReportPDF Available
TextNow Transition Programme
Evaluation Report and Executive Summary
October 2014
Independent evaluators:
Dr Bronwen Maxwell
Professor Paul Connolly
Sean Demack, Dr Liam O'Hare, Anna Stevens, Lucy Clague
The Education Endowment Foundation (EEF)
The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to
breaking the link between family income and educational achievement, ensuring that children from all
backgrounds can fulfil their potential and make the most of their talents.
The EEF aims to raise the attainment of children facing disadvantage by:
Identifying promising educational innovations that address the needs of disadvantaged
children in primary and secondary schools in England;
Evaluating these innovations to extend and secure the evidence on what works and can be
made to work at scale;
Encouraging schools, government, charities, and others to apply evidence and adopt
innovations found to be effective.
The EEF was established in 2011 by the Sutton Trust, as lead charity in partnership with Impetus
Trust (now part of Impetus-The Private Equity Foundation) and received a founding £125m grant from
the Department for Education.
Together, the EEF and Sutton Trust are the government-designated What Works Centre for improving
education outcomes for school-aged children.
For more information about the EEF or this report please contact:
Robbie Coleman
Research and Communications Manager
Education Endowment Foundation
9th Floor, Millbank Tower
2124 Millbank
SW1P 4QP
p: 020 7802 1679
e: robbie.coleman@eefoundation.org.uk
w: www.educationendowmentfoundation.org.uk
About the evaluators
The project was independently evaluated by a team from the Centre for Education and Inclusion
Research at Sheffield Hallam University (SHU) and the Centre for Effective Education at Queen's
University Belfast (QUB).
The evaluation was co-directed by Dr Bronwen Maxwell (SHU) and Professor Paul Connolly (QUB).
The project directors were supported by Dr Liam O’Hare (QUB) and Sean Demack, Anna Stevens and
Lucy Clague (SHU). Professor Guy Merchant (SHU) acted as expert literacy advisor to the project.
Contact details:
Dr Bronwen Maxwell
Centre for Education and Inclusion Research
Sheffield Hallam University
Unit 7 Science Park
Howard Street
Sheffield
S1 1WB
p: 0114 225 5166
e: b.maxwell@shu.ac.uk
Professor Paul Connolly
Centre for Effective Education
School of Education
6971 University Street
Queen's University Belfast
Belfast
BT7 1HL
p: +44 (0)28 9097 5929
e: paul.connolly@qub.ac.uk
Education Endowment Foundation 1
Contents
Executive summary .................................................................................................. 2
1. Introduction ........................................................................................................... 5
2. Methodology ....................................................................................................... 10
3. Impact evaluation ............................................................................................... 17
4. Process evaluation ............................................................................................. 35
5. Conclusion .......................................................................................................... 43
References .............................................................................................................. 47
Appendix I: Outcomes ............................................................................................ 49
Appendix II: Approach to modelling and terms used in tables .......................... 53
Appendix III: Effect size .......................................................................................... 55
Appendix lV: Measuring MyChoice! Engagement ............................................... 59
Appendix V : Parent/ carer consent form ............................................................. 60
Appendix VI: Memoranda of Understanding for Schools ................................... 62
Appendix VII: Security of finding .......................................................................... 89
Appendix VIII: Cost rating ...................................................................................... 90
Executive summary
Education Endowment Foundation 2
Executive summary
The project
The TextNow Transition Programme aimed to improve the reading comprehension skills of pupils at
the transition from primary to secondary school by encouraging engagement in, and enjoyment of,
reading. The programme was delivered by Unitas, a national charity that helps young people access,
participate, and progress in mainstream education and training.
Participating students received 20-minute one to one sessions with a volunteer coach each weekday
for five weeks at the end of primary school and for a further 10 weeks at the start of secondary school.
Children were expected to read independently for a further 20 minutes per day, and were rewarded for
attendance with credits that could be used to buy books online.
The trial examined the impact of the programme on 501 pupils in 96 schools across England who had
been identified as unlikely to achieve Level 4a or above by the end of Key Stage 2. Pupils who were
not likely to gain at least Level 2 were not included in the trial.
The study was funded by the Education Endowment Foundation as one of 24 projects in a themed
round on literacy catch-up at the primary-secondary transition. Projects funded within this round aimed
to identify effective ways to support pupils not achieving Level 4 in English at the end of Key Stage 2.
The project was one of four funded with a particular focus on reading for pleasure.
Key conclusions
1. The trial has not provided any evidence that the TextNow Transition Programme improved reading
comprehension or attitudes towards reading for pleasure over the transition from primary to secondary
school.
2. On average, pupils who participated in the programme made slightly less progress than similar pupils who
did not. However, this finding was not statistically significant, meaning that it could have occurred by
chance.
3. The programme was found to have a differential effect for pupils eligible for free school meals compared
to their peers. A small positive (but not significant) effect was found for pupils eligible for free school
meals, while a negative (and statistically significant) effect was detected for pupils not eligible for free
school meals. It is unclear why this differential effect was found.
4. Higher attendance at the 20-minute daily coaching sessions was found to have a positive impact on
reading comprehension. However, this was only found to be statistically significant for attendance at
sessions in secondary schools. Attendance of the coaching sessions was not found to have an impact on
the secondary outcomes (liking reading and motivation to read).
5. The programme appeared to be more effective when coaches were highly trained, enthusiastic and
committed, and when secondary schools worked closely with feeder primaries to coordinate all elements
of the programme.
Executive summary
Education Endowment Foundation 3
What impact did it have?
On average, pupils participating in the programme made less progress than those who did not. The
size of the difference was small, and can be envisaged as saying that pupils who participated in the
programme made approximately one months less progress than those who did not. However, the
finding was not statistically significant, meaning that it is not possible to conclude with confidence that
the observed effect was due to the programme rather than chance.
The programme appeared to have a different impact on pupils eligible for and claiming free school
meals compared to their peers. To look at this difference more closely, additional, separate analyses
were conducted on pupils eligible for free school meals and their peers. This revealed a positive effect
for pupils eligible for free school meals. However, this effect was not statistically significant, meaning
that it could have occurred by chance. A negative effect equivalent to three months’ less progress was
found for pupils not eligible for free school meals. The effect was found to be statistically significant,
which means that is it unlikely to have occurred by chance.
The programme did not appear to have a differential effect according to gender or prior attainment.
Group
No. of
pupils
Effect
size
(95% confidence
intervals)
Is this
finding
statistically
significant?
Evidence
strength*
Cost of
approach**
Intervention vs. Control
391
-0.06
(-0.22, +0.09)
No
££
Intervention vs. Control
(FSM only)
116
+ 0.18
(-0.13, +0.48)
No
Intervention vs. Control
(Non-FSM only)
275
-0.19
(-0.01, -0.36)
Yes
*For more information about evidence ratings, see Appendix VII in the main evaluation report.
Evidence ratings are not provided for sub-group analyses which will always be less secure than overall findings
**For more information about cost ratings, see Appendix VIII in the main evaluation report.
In addition to attainment, pupils’ enjoyment of reading and motivation to read was assessed. No
statistically significant difference was detected between participating pupils and those in the
comparison group on either measure.
Overall attendance at sessions was good, but higher attendance rates at the 20-minute daily coaching
sessions were found to have a positive impact on reading comprehension. However, this was only
found to be statistically significant for attendance at sessions in secondary schools. The programme
appeared to be more effective when secondary schools worked closely with feeder primaries to co-
ordinate all elements of the programme. In addition, more effective coaches were highly trained,
enthusiastic and committed.
How secure is this finding?
Overall, the evaluation findings are judged to be of moderate security. This assessment takes into
account a number of factors including the study’s design, size and the level of drop-out.
Executive summary
Education Endowment Foundation 4
This evaluation was set up as an effectiveness trial. Effectiveness trials aim to test the intervention
under realistic conditions in a large number of schools. A randomised controlled trial design was
employed to compare outcomes of pupils receiving the intervention to similar pupils who did not.
This study had some limitations with regard to recruitment and attrition (e.g. where pupils dropped out
of the intervention, or where test data for participating pupils was not available). 501 pupils were
initially recruited compared to the desired sample size of 600 and of these, 391 provided full data for
analysis of the primary outcome. However, despite this attrition, pre-test comparisons showed the
comparison and intervention groups were well balanced. The study was large enough to detect an
effect size of 0.2 (equivalent to approximately three months’ additional progress). This means that any
impact at or above this level could have been detected with a high level of confidence.
The sample of schools in the study was broadly comparable with other English schools with above
average levels of disadvantage and thus the results are somewhat generalisable beyond the
immediate context.
Generally, the literature suggests there is potential in tutoring programmes for improving both
academic and social and emotional outcomes. However, existing evidence also suggests that highly
structured tutoring programmes provide the best effects on outcomes (i.e. where tutors are well
trained, lessons focus on specific reading skills and sessions are timetabled systematically). This
recommendation may be counter to the TextNow underpinning model of choice enjoyment
comprehension. Furthermore, there is lack of systematic evidence in the wider academic literature for
a pathway of causality between choice and comprehension.
The study provides important insights into the feasibility of the programme in terms of: its potential
effects on outcomes; an estimate of these potential effect sizes (providing valuable information on
sample sizes for future study); the impacts on different groups (particularly with regard to free school
meal eligibility); and key facilitators and barriers to implementation (from the process evaluation and
the exploratory analysis of dosage).
How much does it cost?
Based on estimates from Unitas, the direct cost that schools would be required to pay for TextNow
programme is £112 per pupil. This estimate includes resources (£50 per pupil), salary costs (£46),
administration and other (£16). The estimate assumes a minimum cohort size of 10 and that sufficient
schools are in the programme to make it viable. In addition, to run the TextNow programme each
school must appoint a coordinator to recruit and manage volunteer coaches, manage programme
delivery and undertake quality assurance and reporting. The programme also requires volunteer
coach time of 20 minutes per pupil per day for 75 days, plus time for initial training, preparation for
coaching sessions and keeping records.
This estimate does not include estimates of the cost of co-ordinators, coaches and other resources in
kind.
Introduction
Education Endowment Foundation 5
1. Introduction
1.1 Intervention
The TextNow Transition Programme, developed by national charity Unitas, is a reading for pleasure
literacy programme with one to one sessions with a volunteer coach occurring 20 minutes a day each
weekday. This occurs at two time points: in Year Six, the final year of primary school for five weeks in
the second half of the summer term; and at the beginning of Year 7, the first year of secondary school,
for ten weeks. The role of the coach is to support the three TextNow core principals of choose, enjoy
and understand. Coaches are usually: teachers; teaching assistants; non-teaching staff such as
technicians or administrative staff; parents; university students; community volunteers; or older year-
group pupils (usually Years 10-13). A TextNow programme coordinator in each school recruits and
supports the coaches and manages the administration of the programme. Coordinators are usually
teachers, teaching assistants or school librarians. Schools are free to timetable the sessions whenever
suits them best, for example through withdrawal from lessons, during registration or tutor periods, or in
breaks, at lunchtime or after school.
The programme also includes access to My Choice!, an online resource that has news, quizzes and
competitions to engage young people in activities to do with reading, as well as giving them access to
a book store. Participants are given credits to choose books from MyChoice! for summer holiday
reading at the beginning of the project. They are able to earn further credits for their attendance and
engagement with the coaching sessions during the five-week programme in Year 6 and the ten-week
programme in Year 7: these can be used to buy books for reading at home and to build their own
personal library.
This trial is the first time that the TextNow Transition Programme has been delivered. It was adapted
from the Unitas TextNow programme that targets young people in their middle teenage years.
TextNow was originally designed to meet the needs of young people in the youth justice system that
were not in education, training or employment (NEET) and struggling with poor reading skills.
Gradually TextNow was made available to the education sector, initially focusing on alternative
education providers, but increasingly being used in mainstream secondary schools from 2010, when it
was provided as a ten-week programme. At any one time there are normally between 25 and 35
schools active in the secondary TextNow programme involving 400 to 600 pupils. During its
development phase, the TextNow programme had working titles of ‘Intensive Reading Programme’
and ‘Reading Matters’, but these were insufficiently distinctive from other programmes, so the
programme was named ‘TextNow’ in 2009.
1.2 Background evidence
There is a wealth of professional literature on literacy coaching (e.g. Blamey et al., 2008) and also
quite widespread use of ‘literacy coaches’. The research on literacy coaches is almost entirely US-
based, and in this context refers to specialised teachers equivalent in status to literacy advisers,
tutors or consultants in the UK context. In the US these coaches are often attached to a university
reading laboratory or literacy research centre. The Unitas TextNow programme uses the word ‘coach’
in a more general sense as teachers, parents and peers who are involved in supporting pupils with
their reading thus suggesting a more general ‘tutoring’ approach.
As the Unitas TextNow programme involves a range of tutoring practices it is necessary to look at the
evidence of effectiveness in this literature. According to a systematic review of RCT’s evaluating adult
volunteer tutoring interventions (Ritter, Barnett, Denny, & Albin, 2009) it was found these programmes
Introduction
Education Endowment Foundation 6
have a small but significant effect on pupils reading outcomes (hedges g = 0.23) including
standandardised reading achievement scores e.g. SATs (g = 0.26), letters and words (g = 0.41), oral
fluency (g = 0.30), and writing (g = 0.45). However, these programmes were not found to have
significant effects on reading comprehension and mathematics outcomes. This systematic review also
considered three implementation issues: (1) types of tutors; (2) age of tutees; and (3) highly structured
versus unstructured programme delivery. Of these factors, only level of programme structure was
found to have a significant influence on effectiveness. In this respect highly structured refers to the
following programme characteristics: tutors are well trained; lessons focus on specific reading skills;
and sessions are timetabled systematically. In another review of effective programmes for struggling
readers (Slavin et al., 2009), one to one volunteer tutoring was found to be an effective method for
improving reading. The three main conclusions in this research were: (1) that tutoring focused on
phonics was most effective; (2) teachers were more effective than teaching assistants; and (3) that
one to one tutoring was more effective than group-based sessions.
Randomised trials of peer tutoring (where pupils tutor other pupils) have also shown promising results.
For example a large cluster randomised trial of 129 schools showed that peer tutoring could produce
small effects (d = 0.20) on reading and mathematics outcomes (Tymms et al., 2011). However, these
effects were only seen for cross-age tutoring (where older more experienced readers tutored younger
readers).
TextNow is an established programme from which the transition version in this study is developed. A
previous evaluation of the established TextNow programme (Brooks, Tarling and Adams, 2011)
showed moderate effect sizes on secondary school children’s reading outcomes (ES = 0.49) which are
in fact relatively large compared to other tutoring programmes like those discussed above. However,
this study was not a RCT and used standardised scores as a comparison group to the TextNow
intervention group. As a result, there is likely to be bias in comparisons between the intervention group
and the standardised sample. Furthermore, there may be within-group bias in that those who
completed the full intervention as well both pre- and post-tests were initially significantly lower in
reading age than those who did not complete all elements of the study. Regardless of these design
limitations, which may have led to inflated or spurious effects, the study showed that the programme
was more effective for older children (mean sample age 14 years 11 months) and for those children
who received an increased input (more sessions) which is useful information in relation to programme
implementation guidance.
The TextNow programme utilises a choose-enjoy-understand rubric. Like similar models in literacy
education (e.g. read-and-respond for comprehension, or look-cover-write-check for spelling) it
conveys the programme’s intent to foster independent reading, enjoyment and comprehension. This
type of model is well supported in the general literature (e.g. Hall et al., 2013) and in policy, and has a
strong common sense appeal. With the exception of ‘understand’ (assuming that is taken as a cipher
for comprehension) these attributes are hard to define and measure. What might improved choosing
look like? How might reading enjoyment be measured in a reliable way? Despite these measurement
issues, there is a literature that suggests a link between choice, enjoyment and motivation to read
(Flowerday & Schraw, 2000; Gambrell, 1996; Hunt, 1970; Sanacore, 1999). There is also literature
that suggests a link between enjoyment and comprehension (Moss and Hendershot, 2002). However,
this literature is often non-experimental and formed from the views of teachers and pupils. In addition,
at least one experimental study shows no link between pupil choice and their engagement with
reading (Flowerday, Schraw, & Stevens, 2004). So, given the inconclusive evidence around pupil
choice and reading outcomes along with the systematic evidence in support of highly structured
tutoring programmes (potentially diminishing choice), there remains a need to further investigate
pathways of causality between choice enjoyment comprehension particularly in the context of
tutoring programmes.
The young people in the study also have access to online activities on MyChoice! The evaluation team
are not aware of any existing research on the benefits of supporting reading in this way, although it is
increasingly common in online book circles and web-based resources such as those provided by
Booktrust in the UK. Furthermore, the TextNow programme provides credits to buy books and this
dimension of the work clearly relates to the notion that book ownership is beneficial (Clark & Poulton,
2011). However, the direction of causality is ambiguous reading enjoyment might well result in a
Introduction
Education Endowment Foundation 7
higher levels of book ownership, but book ownership may not be sufficient to promote reading
enjoyment.
Finally, the Unitas TextNow Transition Programme has been adapted to overlap with transfer from
primary to secondary school. Considerable attention has been paid to the potentially negative effects
of transition, particularly with respect to the apparent performance dip in transfer. The evaluation team
is not aware of any quantitative research to support effects from an alleged ‘transition dip’, even
though the notion has informed the discourse of policy-makers (e.g. DfE, 2013). However, there is
some more general quantitative evidence of the impact of summer vacations. Cooper et al's (1996)
meta-analyses of 13 American studies indicated that overall achievement test scores declined by
about one month over the summer vacation. However, the decline in reading was less than for
mathematics and while scores for reading comprehension decreased, scores for vocabulary and
reading recognition increased. Importantly, this study indicates that social class is an important
indicator of the extent to which a child experiences a reading dip over the summer. As Alexander and
Entwhisle's (2007) analysis of Baltimore children's reading scores over the summer vacations in
primary school also shows, children from low-income families experience a larger overall decline in
reading scores than children from highe -income families. The difference is particularly striking for
reading recogniition where children for higher-income families were found to make gains but children
from low-income families experienced a dip (Cooper et al., 1996). The authors of both of these studies
point to the greater availability of books in higher-income families as a potential explanation for their
findings.There is also some qualitative work which suggests that regression or stagnation of learning
may occur during transition and may be due to secondary teachers underestimating the capabilities of
Year Seven students (Galton, Gray and Ruddock 1999 & 2000; Kirkpatrick, 1992). Further research
suggests 16% of transition students feel unprepared for secondary school but only 3% of students
remain worried about secondary school in the term after their transition (Evangalou et al., 2008).
As indicated above, the TextNow Transition Programme is an adapted version of the Unitas TextNow
programme. Therefore, the evaluation is set within three early phases of development as outlined by
Campbell et al. (2007):
Phase 0: Theoretical development (why should this intervention work?).
Phase 1: Logic Modelling (how does it work?).
Phase 2: Exploratory or pilot trial (optimising trial measures and looking at potential effects
overall and on sub-groups).
This would map onto the Education Endowment Foundation system as an effectiveness trial.
1
The rationale for the present evaluation is to test rigorously whether the adapted TextNow Transition
Programme has a measurable effect on students’ reading skills and comprehension, as well as on a
small number of other secondary outcomes.
The project was funded as part of a £10 million grant awarded by the Department for Education to the
EEF for projects dedicated to literacy catch-up for pupils at the primary-secondary transition who do
not achieve Level 4 in English by the end of Key Stage 2.
1
Further information regarding EEF trial types may be found here:
http://educationendowmentfoundation.org.uk/uploads/pdf/EEF_approach_to_process_evaluation.pdf
Introduction
Education Endowment Foundation 8
1.3 Evaluation objectives
The aim of this evaluation was to measure, through a randomised controlled trial, the impact of the
Unitas TextNow Transition Programme on children who were unlikely to achieve Level 4a in reading
by the end of Key Stage 2 (KS2). The process evaluation aimed to examine the implementation of the
programme and capture the perceptions and experiences of key stakeholders engaged with the
programme.
As set out in the trial protocol and registration, the evaluation was designed to address the following
questions:
What is the impact of the programme, at post-test (i.e. following the delivery of the
programme), on a number of specific reading outcomes for participating children?
Is the programme having a differential impact on children depending on:
o their gender?
o their socio-economic status?
Does the impact of the programme vary significantly with any variations found in
implementation?
1.4 Project team
Intervention team
A dedicated Unitas programme manager led the implementation of the TextNow Transition
Programme. This included recruiting schools, securing school and parental consent to the trial,
supporting schools in implementing the programme, and accessing TextNow training and resources,
monitoring implementation and agreeing test dates with the schools. A dedicated full-time
administrator supported the separate management arrangements and there was a significant amount
of line management support, for example from the Director of Operations.
There were 53 coordinators recruited in primary schools and 31 recruited in secondary schools to take
responsibility for implementing the TextNow Transition Programme in their own school or across two
schools. The coordinators were generally school leaders, teachers, teaching assistants or support
staff responsible for the school library/learning centre. There were 379 coaches recruited to support
young people in primary school and 381 in secondary schools. The majority of coaches were teaching
assistants, but they also included teachers, library/learning centre support staff, parents, student
coaches from the secondary schools involved and higher education students.
Evaluation team
The evaluation was co-directed by Dr Bronwen Maxwell (SHU) (responsible for overall direction and
process evaluation) and Professor Paul Connolly (QUB) (impact evaluation). The project directors
were supported by Sean Demack (SHU), Dr Liam O’Hare (QUB) and Anna Stevens (SHU) for the
impact evaluation, and by Lucy Clague (SHU) for the process evaluation and project management.
Professor Guy Merchant (SHU) acted as expert literacy advisor to the project.
Introduction
Education Endowment Foundation 9
1.5 Ethical review
The evaluation was approved by the ethics committees at Sheffield Hallam University and Queens
University Belfast. Participating primary and secondary schools entered into a licence agreement with
Unitas for use of the TextNow resources and signed a memorandum of understanding that set out
their roles and responsibilities for the trial as well as the role and responsibilities of Unitas and the
evaluators (Appendix VI). Opt-in consent was sought from parents/carers prior to trial. The
parent/carer consent form (Appendix V), which covered both participation in the programme and the
evaluation was distributed to parents through their child's primary school. Only children who had
written parental/carer consent where included in the randomisation process.
Methodology
Education Endowment Foundation 10
2. Methodology
2.1 Trial Design
The intervention was delivered at the individual level. Randomisation was at the individual pupil level
but blocked within schools (see Randomisation section for more detail). There were two trial groups
only (i.e. control and intervention). Children in the control were given credits for MyChoice! after the
trial. The design was chosen to provide a robust evaluation of the Unitas TextNow Transition
Programme with regard to its effects on literacy outcomes. Analysis was conducted using multi-level
models as the pupils were naturally clustered into primary and secondary schools. No post-hoc
changes were made to the trial design due to recruitment issues.
2.2 Eligibility
Unitas recruited 34 secondary schools across England to take part in the trial, selecting schools where
the proportion of pupils eligible for free school meals was above the national average and pupil
performance below average. Full details of how schools were selected are given in Section 3.2.
Across the secondary schools a total of 62 feeder primary schools took part in the trial. Eligible
participants were drawn from pupils in Year 6 in primary school at the start of the intervention in June
2013 who were making the transition to the 34 secondary schools involved in the study. Only pupils
identified through teacher assessments as not likely to achieve Level 4 in reading, or who were likely
to achieve a Level 4, 4b or 4c by the end of KS2 were eligible to participate. Pupils who were not likely
to achieve a minimum of Level 2 were not included in the study due to their need for more extensive
additional support. No minimum or maximum numbers of participants were placed on schools.
Opt-in parental/carer consent was sought prior to randomisation. Primary schools organised the
distribution and the collection of consent forms.
2.3 Intervention
The Unitas TextNow Transition Programme comprises one to one sessions with a volunteer coach
and access to the MyChoice! online resource and book selection system. The programme is
organised around secondary schools and their feeder primary schools. Implementation of the
programme in each school (primary and secondary) is managed by a school coordinator. Most school
coordinators were teachers, teaching assistants or school librarians/learning centre managers. The
school coordinator liaises with, and is directly supported by, the Unitas programme manager. The
school coordinator is responsible for:
identifying and preparing young people to participate;
recruiting and supporting coaches (for primary schools this may be done by the linked
secondary school);
monitoring, reviewing and assessing young people's progress this includes a short reading
interview and an end of programme final review with the coordinator, coach and young
person, as well as ongoing oversight of progress and addressing any issues such as
attendance;
Methodology
Education Endowment Foundation 11
maintaining quality assurance systems and reporting to Unitas;
setting up a starter library of books from MyChoice!
School coordinators are expected to undertake four online training modules in preparation for their role
comprising information, videos, supporting resources and training assessments. While there is no
face-to-face training or monitoring of completion of training, school coordinators are also supported
directly by Unitas by email and telephone throughout the set-up and delivery of the programme.
Coaching is undertaken on a one to one basis, however one coach may support more than one young
person. Coaches may be from a variety of backgrounds. Most frequently they are teachers; teaching
assistants; non-teaching staff such as technicians or administrative staff; parents; university students;
community volunteers; or older year-group pupils (usually Years 10-13). Coaches meet with their
young person for 20 minutes each weekday for five weeks in the second half of the summer term
when the pupils are in Year 6 in their primary schools; and 10 weeks at the beginning of Year 7 when
pupils are in secondary school. For most young people a different coach works with them in their
primary and secondary schools.
Sessions are timetabled at times that suits the school: this may be in lesson time, before or after
school or at lunchtime. The coaching role involves planning sessions, initially using the reading profile
created by the school coordinator after the reading interview with the young person and as they
establish a relationship with the young person, increasingly focusing in on the young person's interests
and needs. Coaches also review and record the young person's progress. Coaching is focused around
three strands choosing texts, enjoying texts and understanding texts. 'Choosing texts' focuses on
improving young people's skills in choosing reading materials that they are interested in and which are
suitable for their level of ability, giving them a sense of ownership about their choice. It is also
designed to help young people to become familiar with environments where they can find the texts
and to be relaxed in those environments. Related coaching activities include talking about book
choice, developing strategies to assess a book choice, visiting a library and using MyChoice!
To support young people in 'enjoying texts' coaches are recommended to deploy a range of strategies
and resources focused on shared reading: the use of ICT (such as e-readers and the internet),
discussing related books and films, and using audio books. The 'Understanding texts' coaching strand
aims to help young people understand what they read at a literal and inferential level and to talk, with
understanding, about what they have read. Techniques included in the coaching framework to support
understanding of texts include: skimming and scanning; kinaesthetic activities such as acting out a
scene; KWL (what do I know, what do I want to find out, and what did I learn); visual representations
such as a storyboard or book cover design; and reciprocal teaching involving summarising, asking
questions, clarifying and predicting and embellishing. Coaches are expected to undertake four online
training modules prior to working with their young person.
A key feature of the TextNow Transition Programme is to provide rewards to young people for their
participation. This is achieved through credits (worth 25p) being awarded for each day of attendance
and participation with reading on TextNow. The credits are added to each young person’s account on
MyChoice!, a web resource with an ‘online bookstore’ and book club which aims to support and
encourage independent reading. MyChoice! includes questions to guide young people's choice of
books, as well as news, quizzes and competitions to engage young people in fun activities to do with
reading. Young people use the credits they 'earn' to 'buy' their own books and build their own personal
library. In addition young people were given credits at the beginning of the trial to 'buy' books from
MyChoice! over the summer holiday to sustain their engagement during transition. Coaches are
expected to use MyChoice! at least once a week with their young person during the intervention at
primary and secondary school.
The control group was 'business as usual' during the trial and received no TextNow coaching.
Following the post-test the control group was provided with credits to 'buy' books from the MyChoice!
website.
Methodology
Education Endowment Foundation 12
The process evaluation indicates that there were significant fidelity issues in the implementation of the
TextNow Transition Programme. Some pupils did not receive the full dosage of coaching and some of
the young people who received peer coaching reported that that the coaching was poor quality. Some
pupils had no or limited access to MyChoice! Fidelity was most problematic where schools lacked
commitment to the programme. Due to the timing of the primary intervention (post SATs) some
primary schools did not perceive any benefit to the school from their participation. Some secondary
schools also lacked commitment to the programme, particularly when they had been recruited as part
of a chain of schools. These issues are discussed in more detail in the process evaluation in Section 4
below.
2.4 Outcomes
The primary outcome in the study was reading comprehension. The New Group Reading Test was
used as a post-test reading ability measure and employs sentence completion and passage
comprehension items
2
. Appendix I contains a statistical and graphical summary for this primary
outcome. The New Group Reading Test was administered online through GL online testing software.
Independent invigilators were sent to the schools to facilitate the testing process and a teacher in each
school oversaw this process. The invigilators were blinded to the group status of the individual pupils.
The KS2 reading score (standardised national school literacy assessment, taken from the spring
census 2013 and accessed through the National Pupil Database) acted as a pre-test measure in the
analysis models.
The secondary outcome of interest was ‘reading for pleasure’ or children enjoying and fostering a love
of books and reading. It was chosen because it was an intermediate attitudinal change that would lead
to end-point outcomes in assessed literacy ability and a core aim of the TextNow programme. Two
sub-scales from the Progress in International Reading Literacy Study (PIRLS) were used as a post-
test reading attitudes measure related to reading for pleasure. The two scales were: 1. 'Students Like
Reading' Scale and 2. 'Students Motivated to Read' Scale. Appendix I provides a statistical summary
of these two scales and the questionnaire items they were derived from. As with the primary outcome,
standardised national school literacy assessments were used as a pre-test measure in the secondary
outcome models. The resulting observed correlations between this pre-test and the two secondary
outcome measures were fairly weak
3
and so the value of the pre-test measure was more limited for
the secondary outcomes compared with the primary outcome. The secondary outcome tests were
administered during the same session and under the same conditions as the primary outcome test.
2.5 Sample size
At the design stage, the sample size calculation for this individually randomised, multi-site block
randomised, RCT used the following indicators:
4
Probability: to detect effects with a probability <0.05.
Power: with a power of 0.8.
2
NGRT Information available at: http://www.gl-assessment.co.uk/products/new-group-reading-test.
3
r = 0.19 (KS2 v 'Students like Reading’); r = 0.08 (KS2 v 'Students Motivated to Read').
4
Calculation made using Optimal Design software: http://hlmsoft.net/od/
Methodology
Education Endowment Foundation 13
Effect size: with an effect size in the range of g = 0.2-0.3 (from previous literature).
Site size: with an average site size of 20 pupils.
Intra-Cluster Correlation (ICC) = 0.05 i.e. the proportion of variation that is 'explained' by
clustering at the school level. At this design stage, clustering at the secondary school was
used in the power calculations.
Covariate with the proportion of explained variance by level 1 covariate being estimated to
be 0.5 (Pearson's Correlation Coefficient).
This calculation produced an estimate of number of sites (i.e. secondary schools) required to be 16-30
depending on effect size in the range outlined above (i.e. 0.2-0.3). Therefore, the original design for
this trial was based upon a large sample of pupils (N=600) from approximately 30 secondary schools.
Although more than the target number of schools were recruited (34) there was a shortfall in the
number of children entered into the randomisation process (N=501). This was due to there being fewer
eligible pupils per secondary school than had been predicted and the non-return of consent forms by
parents/carers.
Following the collection and processing of data, the Minimum Detectable Effect Size (MDES) for the
primary outcome (New Group Reading Test) was calculated more precisely. This was done by using
the actual values of the ICC and the correlation between the pre- and post-test measures. Please note
that two sets of calculations are provided. The first assumes clustering by secondary school (this was
the approach used to estimate the MDES at the design stage see above) whilst the second assumes
clustering by primary school. The statistical analyses (multilevel models) presented in this report
assume clustering at the primary school level because of the higher ICC.
501 pupils at randomisation
34 secondary schools, 62 primary schools
Average site size (pupils per school) at randomisation = 14.7 (secondary school); 8.1 (primary
school)
Inter-class correlation = 0.048 (for secondary), 0.053(for primary)
Pre-post correlation = 0.54
MDES estimates:
Assuming clustering at secondary school level = 0.20
Assuming clustering at primary school level = 0.19
Despite the smaller than expected sample size, the randomisation worked well providing two balanced
groups for analysis.
2.6 Randomisation
The randomisation was conducted at an individual level. The individuals were block randomised
initially by school and then by reading level as indicated by teacher-predicted KS2 results to create
pairs.
A random number generator (provided by Stata v13) assigned a number between 0 and 1 to each
member of a pair and the individual with the highest random number in each pair was assigned to the
intervention group. In cases where uneven numbers of pupils were present in a school, the remaining
individual would be assigned to the intervention group if they had a random number of above .5 and if
they had a random number below 0.5 they were assigned to the control group. The randomisation was
Methodology
Education Endowment Foundation 14
conducted by a member of the independent evaluation team who had been blinded to any identifying
school or pupil data.
2.7 Analysis
Multilevel analyses were conducted using the Stata v13 software that took account of how participants
were clustered into primary schools at the beginning of the trial. Because this was a project around the
transition of young people from Y6 to Y7, they were also clustered differently at the end of the trial in
relation to the different secondary schools they entered. The main analysis was run twice to assess
whether taking into account the clustering of young people by primary or secondary school made a
difference. Both approaches resulted in similar findings. Moreover, the degree of clustering effects
was more evident in relation to the young people’s attendance at primary school, as would be
expected, and thus the main multilevel analysis was based on taking their clustering by primary school
into account. The main analyses were conducted on an "intent to treat" basis following which an
examination of 'dose' or fidelity was undertaken. This means that the assessment attempted to
measure the impact of a school running the programme in real-world conditions, recognising that
attendance or implementation may not be perfect.
Binary dummy variables were used to identify the intervention (coded “1”) and control (coded “0”)
group participants. Similarly, binary dummy variables were used for the FSM (“1”=FSM, “0”=non-
FSM)
5
and gender (“1”=female, “0”=male) variables. A pre-test measure of Key Stage 2 reading score
was also used within the analyses.
The modelling was conducted in stages; first, a main effects stage followed by an interaction stage.
The main effects models included the intervention / control dummy variable along with the KS2 pre-
test measure, FSM and gender dummies. The main effects model assumes that any impact of the
intervention is consistent across different participant subgroups (for example males and females; FSM
and non-FSM participants): to explore whether this was a reasonable assumption, the three interaction
terms were introduced one at a time. The interaction terms included were: FSM*intervention (isolating
FSM participants who received the intervention); a gender*intervention (isolating females who
received the intervention); and KS2 pre-test*intervention (isolating the pre-test scores of intervention
group participants). If an interaction term was found to be statistically significant, a subgroup analysis
was used to explore this in more depth. For example, if a gender*intervention interaction term was
found to be statistically significant, separate models would be run for the male and female subsamples
to explore this.
Following the impact analyses, further analyses focused on intervention group participants to explore
'dosage'. In this context, 'dosage' is defined as the number of days a participant attended a TextNow
session. Multilevel models explored whether attendance had an impact on the three outcome
measures. For each outcome, two models were constructed: the first used an overall (primary and
secondary school) attendance variable whilst the second used separate attendance details for the
primary and secondary school sessions.
5
The FSM measure identifies young people who are eligible and claiming free school meals from
young people who are not eligible or eligible and not claiming them. Although this measure is widely
used as a proxy for the socio-economic status of young people, it should be noted that there is a
known undercount of FSM claimants that is estimated at approximately 200,000 (or 2%) of all 4-15
years olds in England (Iniesta-Martinez and Evans, 2012). Whilst FSM is a rather simple measure of
socio-economic status and has this problem of inaccuracy, it is readily available and until better socio-
economic detail is collected, remains likely to be the main tool for taking socio-economics into account
in educational research in England.
Methodology
Education Endowment Foundation 15
2.8 Process evaluation methodology
The process evaluation was designed to illuminate the implementation of the TextNow Transition
Programme and assess fidelity. Key areas explored were pupil and school engagement, the quality of
implementation of the intervention, perceived programme outcomes and adherence to the planned
programme. Data was collected through a variety of activities at different time points as shown in
Table 2.1.
Table 2.1 Process evaluation data collection
Date
Activity
June to July 2013
5 weeks of TextNow intervention in primary schools
September to November 2013
Surveys of all primary coordinators and coaches
September to mid-December 2013
10 weeks of TextNow intervention in primary schools
Mid-Dec 2013 to February 2014
Surveys of all secondary coordinators and coaches
Jan 2014
Four pupil focus groups
The survey of primary and secondary coaches covered: their background and training, the
arrangements for coaching sessions, approaches to coaching, pupil engagement, behaviour and
outcomes, as well as coaches' suggestions for improving the programme. The survey of primary and
secondary coordinators covered: their background and training, the numbers of coaches they worked
with, activities and tasks carried out in the coordination role and programme outcomes. Coaches and
coordinators were also asked for suggestions to improve the programme. All surveys were conducted
anonymously online and analysed by the evaluators using descriptive statistics for numerical data and
thematic analysis of open questions. Links to the coordinator survey were provided directly to all
coordinators, initially by the evaluators and then to boost the response by Unitas. Coordinators were
asked to pass the links to the coach survey to coaches in their school. Response rates from both
primary and secondary coordinators were high (primary: 64%; secondary: 68%). Although a lower
response rate was achieved for coaches (primary: 11%; secondary 21%), a response from at least
one coach was received from over half of the schools recruited to the trial (Table 2.2).
Methodology
Education Endowment Foundation 16
Table 2.2 Process evaluation survey response rate
Role
No. of
responses
Response rate
(%)
No. of schools
Percentage of
schools (%)
Primary coordinator
366
64%
30
55.6
Secondary coordinator
217
68%
19
61.2
Primary coach
42
11%
27
50.0
Secondary coach
72
21%
18
58.1
Four pupil focus groups were conducted by one researcher from the evaluation team after the end of
the programme. A purposive sample of participating schools was selected for the focus groups to
maximise variation in terms of geographical distribution, rural/urban school location, socio-economic
profile, and type of governance. Of the six schools originally approached, two declined, one due to a
forthcoming Ofsted inspection, and another did not have staff capacity to make the necessary
arrangements for the visit. The four schools which agreed to participate in the focus groups varied in
their degree of engagement with the TextNow programme. The TextNow coordinator in each school
was asked to select 6 pupils from the intervention group that were representative of the range of pupils
who had received the intervention in terms of gender, reading ability, attitudes towards reading,
programme outcomes and English as a second language status (EAL). Due to pupil absence, there
were fewer focus group pupils at one school. In total, 22 pupils (10 girls and 12 boys) from the
intervention group participated in focus groups conducted in four secondary schools. The participant
characteristics broadly matched those requested by the evaluators. Pupils were asked about their
experiences of the programme at both their primary and secondary schools and how they thought
participation in the programme had impacted their enjoyment of reading and reading skills. Each focus
group was recorded and a thematic focus group report with illustrative quotes was prepared.
6
This equated to 34 individuals, with two working across two different primary schools.
7
Two schools had responses from two coordinators.
Impact evaluation
Education Endowment Foundation 17
3. Impact evaluation
3.1 Timeline
Date
Activity
Sept 2012-Jan 2013
Recruitment of schools
May 2013
Pre-test data (Key Stage 2 finely graded reading point score) was
collected from schools to identify eligible pupils (Note: KS2 data used for
analysis made available to the research team from DfE in January 2014)
April-May 2013
Collection of parental consent
20th May 2013
Randomisation of pupils to intervention and control groups
3-17 June 2013
Commencement of five weeks of TextNow in primary schools
School summer
holidays 2013
Access to MyChoice! continued throughout the school holidays
4-25 September
Commencement of ten weeks of TextNow in secondary schools
December 2013
Intervention finishes in secondary schools
December 2013
Post-test (standardised age score from the NGRTB test) collected
January 2014
Pupils (N=35) at two schools who was unable to arrange an earlier test
were tested
3.2 Participants
Recruitment of schools was undertaken by Unitas who undertook a scoping exercise drawing on pupil
performance and FSM data to identify secondary schools likely to have eligible pupils. Due to the short
time period over which schools were to be recruited, the strategy was to approach secondary schools
which were part of networks or chains which would provide routes into multiple schools or schools with
which Unitas had a pre-existing relationship. Between October 2012 and December 2012 Unitas made
telephone and/or email approaches to head teachers in selected schools (the number of schools
contacted was not recorded) to gain senior leader support for participation. Where some interest was
expressed, Unitas then explained the trial through meetings with head teachers/relevant staff and
phone calls. This resulted in an 'interested list' of 50 schools. Unitas communicated further information
on the trial methodology in January 2013 via email, telephone and/or meetings. This was summarised
in a Memoranda of Understanding for primary and secondary schools (Appendix V). At this point a
number of schools withdrew. Reasons for withdrawal cited were that the evaluation was too onerous
or that they were unwilling to take part in the RCT methodology of selection of young people. In
addition, some schools revised downwards the number of pupils they felt they could work with. In
some instances, although the secondary school had committed to the project, primary feeder schools
could not be convinced to participate, so the secondary school was forced to withdraw. Unitas chose
to recruit secondary schools rather than primary schools as they would undertake the bulk of delivery
Impact evaluation
Education Endowment Foundation 18
of the programme, and would be in the best position to identify a cohort of eligible pupils from multiple
feeder primary schools. Secondary schools, supported by the Unitas programme manager, brought
their feeder primary schools into the trial. None of the secondary schools which elected to participate
and were able to bring on board their feeder primary schools had previously worked with Unitas or had
run the secondary TextNow programme. At the point of randomisation 34 secondary schools (four
above the original target number) across England and 62 primary schools were participating in the
trial.
All pupils in Year 6 in the feeder primary schools of the 34 participating secondary schools were
assessed for eligibility by the secondary schools. Potential eligible pupils were identified by the
secondary schools based on primary teachers' assessments of predicted reading point score at KS2.
Parental/carer consent was requested for all pupils who were not predicted to achieve Level 4, or who
were likely to achieve a fragile Level 4 (4b or 4c) and were likely to gain at least a Level 2.
Only data on the 501 pupils that schools identified as eligible, and for whom schools had collected
parental/carer consent, were passed to the evaluators. It is therefore not possible to compare those
pupils whose parents did not give consent with the pupils included in the trial in terms of key
characteristics. The number of pupils who met the pilot selection criteria and accepted a place at the
participating secondary school was far below the initial prediction given by the secondary schools,
resulting in an under-recruitment of pupils for the pilot (target number 600). The under-recruitment of
pupils was further exacerbated by the process of gaining parental/carer consent. Issues included: the
timing of the consent request clashing with the Easter holiday; some schools not adhering to the
processes for distribution of consent forms requested by Unitas and/or not chasing missing forms; and
the complexity of the form (Appendix V).
As the flowchart (Figure 3.1) illustrates, 501 eligible pupils were entered into the trial. Of the 501 pupils
who were randomised, 252 were allocated to the intervention group and 249 to the control group.
Three of the control group pupils were given the intervention in primary school but were not given the
intervention in their secondary school. Nineteen pupils in four primary schools in the intervention group
did not receive the five-week intervention in their primary school and did not have access to
MyChoice! over the summer, but did receive the intervention in the secondary school.
8
Of the 252 pupils allocated to the intervention group, 53 were lost to follow up: 32 of these 53 were in
the five secondary schools that discontinued the programme and dropped out of the trial. Four of the
five secondary schools did not start the 10-week secondary programme. Reasons given included: that:
their feeder primaries had dropped out and not run the 5-week programme; logistical reasons making
it impossible to implement the programme in the timescale required; and in one school, an impending
Ofsted inspection. These four schools were unwilling, or did not have the capacity, to arrange the end-
tests. The drop-out of these schools does not compromise the internal validity of the study but does
have minor implications for the external validity or representativeness of the results for schools in
general. Schools which were less engaged with the programme also tended to drop-out of the testing,
meaning that the treatment effect is estimated on more engaged schools. The fifth secondary school
dropped out in week eight of the ten-week secondary programme citing a lack of capacity to either
complete the programme or arrange the end-test. Twenty-one of the 53 pupils lost to follow-up did not
attend the end-test in their school. In total 199 intervention pupils were entered into the analysis of the
primary outcome. Of the 249 pupils allocated to the control group, 56 were lost to follow-up, 28 in the
secondary schools who withdrew from the trial and 28 who did not attend the end-test in their school.
We were unable to match one of the control pupils to the NPD, so a total of 192 pupils from the control
group were entered into the analysis of the primary outcome. In total pupils from 53 primary schools
and 29 secondary schools were entered into the analysis.
8
Of these 19 pupils, seven were lost to follow- up due to their secondary school withdrawing from the
trial near the end of the ten- week programme in the secondary school.
Impact evaluation
Education Endowment Foundation 19
The number of pupils entered into the analysis for the secondary outcomes was lower than for the
primary outcome of reading comprehension (primary outcomes n=391; secondary outcome 'Students
Like Reading n=331; secondary outcome 'Students Motivated to Read' n=352). This was due to
missing responses relating to the questionnaire components used to construct the scales for the
secondary outcomes.
Randomised (n=501)
Secondary Schools = 34
Primary Schools = 62
Allocated to intervention (n=252)
Received allocated intervention (n=252)
Did not received allocated intervention (n=0)
Allocated to control (n=249)
Received allocated intervention (n=249)
Did not received allocated intervention (n=0)
Allocation
Lost to follow-up (n=53)
Discontinued intervention (n=0)
Lost to follow-up (n=56)
Discontinued intervention (n=0)
Unable to match to NPD (n=1)
Analysed (n=199)
Excluded from analysis (n=0)
Analysed (n=192)
Excluded from analysis (n=0)
Follow-Up
Analysis
Figure 3.1: Trial participants
Schools lost to
follow up
Secondary = 5
Primary = 9
Schools included:
Secondary = 29
Primary = 53
Impact evaluation
Education Endowment Foundation 20
3.3 School characteristics
In terms of percentage of pupils with SEN, the average percentage for the sample was slightly higher
than the national average, this was also the case for the percentage of pupils with English not as a
first language. Compared to the national average, the schools sampled for the study had a higher
average percentage of pupils eligible for free school meals and a lower average percentage of pupils
achieving 5+ A*-C or equivalent GCSEs including A*-C in both English and Mathematics. Comparing
the original sample to the achieved sample used in analysis reveals little difference between these
sample characteristics (Table 3.1).
Table 3.1 Secondary school characteristics of sample compared with national data
England - national
(secondary state-
funded)
Original
sample
(n=34)
Sample for
analysis
(n=29)
Percentage of pupils with SEN
1.9%
3.1%
3.2%
Percentage of pupils with English not as a first
language
13.6%
16.8%
16.1%
Percentage of pupils eligible for free school
meals
16.3%
25.4%
25.4%
Percentage of pupils achieving 5+ A*-C or
equivalents including A*-C in both English and
Mathematics GCSEs
60.6%
51.4%
51.7%
Around half the schools in the original sample had an Ofsted rating of Good whilst 9 had a rating of
Requires improvement. Three schools had an Outstanding rating whilst 4 were deemed
Inadequate. The distribution of Ofsted ratings was similar across the sample used for analysis (Table
3.2).
Table 3.2 Secondary schools Ofsted ratings
Ofsted rating of
overall effectiveness
of the school
Original
sample (n=34)*
Sample for
analysis
(n=29)
Outstanding
3
3
Good
15
13
Requires
improvement
9
7
Inadequate
4
3
*Ofsted reports unavailable for 3 schools
Impact evaluation
Education Endowment Foundation 21
The majority of schools in the original sample were Academy sponsor-led schools (n=18). In total 9
schools were community schools, 4 were foundation schools, 2 were academy convertors and 1 was a
Free School. Four of the 5 schools who dropped out of the trial were Academy sponsor-led schools
(Table 3.3).
Table 3.3 Secondary school type
School type
Number of schools
Original
sample (n=34)
Sample for
analysis
(n=29)
Foundation School
4
3
Community school
9
9
Academy Converter Mainstream
2
2
Academy Sponsor-Led
18
14
Free School Mainstream
1
1
As noted above, the schools in the sample were mostly in the upper range in terms of percentage of
pupils eligible for free school meals; 21 schools were in the upper quintile (Table 3.4).
Table 3.4 Pupils eligible for FSM
Number of schools
FSM quintiles
Original sample (n=34)*
Sample for analysis (n=29)*
Lowest quintile
0
0
Lower quintile
3
2
Middle quintile
7
6
Upper quintile
21
19
Highest quintile
2
1
* One school is not included in the 2013 KS4 Performance Tables
The majority of schools (19) were in the lower quintile for attainment (Table 3.5). Although all drop-out
schools were in this category, all of these were at the top end of the lower quintile. Hence with regards
to the average percentage of pupils achieving 5+ A*-C or equivalents including A*-C in both English
Impact evaluation
Education Endowment Foundation 22
and mathematics GCSEs little difference can be seen between the original sample and the achieved
sample (Table 3.1).
Table 3.5 Secondary school attainment
Number of schools
Percentage of pupils
achieving 5+ A*C or
equivalents including A*C
in both English and
Mathematics GCSEs
quintiles
Original sample
(n=34)*
Sample for analysis
(n=29)*
Lowest quintile
2
2
Lower quintile
19**
14
Middle quintile
7
7
Upper quintile
3
3
Highest quintile
2
2
* One school is not included in the 2013 KS4 Performance Tables
** Although all drop-out schools appear in the lower quintile, these schools were close to the top end of this quintile, and the
mean % attainment for the original sample is close to that of the analysis sample.
3.4 Pupil characteristics
Table 3.6 Number of participants involved in the TextNow Transition Programme RCT
evaluation
Description
Pre-test (Baseline)
Post-test (Outcome)
Intervention
Control
Intervention
Control
Participants
252
2499
199
192
Primary Schools
62
62
53
53
9
It was not possible to match two of the control group pupils to the NPD; therefore the analysis of pre-
test pupil characteristics is based on 247 control group pupils.
Impact evaluation
Education Endowment Foundation 23
Description
Pre-test (Baseline)
Post-test (Outcome)
Secondary Schools
34
34
29
29
In terms of FSM, gender and the pre-test measure, randomisation resulted in intervention and control
group samples that were comparable at baseline. The samples remained statistically comparable at
the post-test stage suggesting that the small attrition did not introduce bias in terms of FSM, gender
and the KS2 pre-test measure (Tables 3.6-3.9).
Table 3.7 FSM statistical snapshots at baseline and outcome
FSM
Pre-test (Baseline)
Post-test (Outcome)
Intervention
Control
Intervention
Control
Eligible and registered /
claiming
79 (31%)
76 (31%)
57 (29%)
59 (31%)
Not eligible or eligible
and not registered /
claiming
173 (69%)
171 (69%)
142 (71%)
133 (69%)
Total
252 (100%)
247 (100%)
199 (100%)
192 (100%)
Value (d.f.)
p=
Value (d.f.)
p=
Pearson chi-square test
0.02 (1)
0.889
0.20 (1)
0.65
Table 3.8 Gender statistical snapshots at baseline and outcome
Gender
Pre-test (Baseline)
Post-test (Outcome)
Intervention
Control
Intervention
Control
Female
116 (46%)
114 (46%)
96 (48%)
86 (45%)
Male
136 (54%)
133 (54%)
103 (52%)
106 (55%)
Total
252 (100%)
247 (100%)
199 (100%)
192 (100%)
Value (d.f.)
p=
Value (d.f.)
p=
Pearson chi-square
test
0.001 (1)
0.978
0.47 (1)
0.49
Impact evaluation
Education Endowment Foundation 24
Table 3.9 KS2 Pre-test (Reading) statistical snapshots at baseline and outcome
KS2 Pre-test
(Reading)
Pre-test (Baseline)
Post-test (Outcome)
Intervention
Control
Intervention
Control
Mean (sd)
24.7 (4.16)
24.4 (4.16)
24.7 (4.29)
24.3 (4.27)
Median (IQR)
25.4 (5.04)
25.1 (6.18)
25.4 (5.04)
25.1 (6.18)
N
252
247
199
192
mean difference
(d.f.)
p=
mean difference
(d.f.)
p
t-test (df)
+0.28 (497)
0.46
+0.34 (389)
0.43
Effect Size (Hedges
g)
+0.07
+0.08
Impact evaluation
Education Endowment Foundation 25
3.5 Outcomes and analysis
10
Table 3.10 Primary outcomereading comprehension
11
(n=391 participants in 53 primary schools)
Description
Main Effects Model
Interaction Models
Pre-test
interaction
Gender
Interaction
FSM
Interaction
coef.
s.e.
coef.
s.e.
coef.
s.e.
coef.
s.e.
Group (Intervention)
-0.72
0.874
-0.72
0.868
-1.58
1.221
-1.96
1.043
Pre-test (KS2 Reading)
1.54
0.111
1.33
0.152
1.54
0.111
1.54
0.110
Interaction (Pre-
test*Intervention)
-
-
0.40
0.206
-
-
-
-
Gender (female)
1.58
0.926
1.59
0.921
0.65
1.309
1.75
0.924
Interaction (gender*Intervention)
-
-
-
-
1.84
1.834
-
-
FSM (eligible & claiming)
-1.35
1.018
-1.31
1.013
-1.23
1.023
-3.42
1.408
Interaction (FSM*Intervention)
-
-
-
-
-
-
4.41
1.948
Constant
88.81
1.000
88.76
1.000
89.20
1.073
89.3
7
1.033
-2 Log Likelihood
-1497.891
-1418.657
-1420.047
-1418.310
School level variance (s.e.)
17.68 (6.239)
18.07 (6.309)
17.84 (6.259)
17.92 (6.233)
Pupil level variance (s.e.)
73.39 (5.721)
72.48 (5.652)
73.13 (5.700)
72.39 (5.640)
Intra-Class Correlation (s.e.)*
0.19 (0.059)
0.20 (0.060)
0.20 (0.059)
0.20 (0.059)
10
See Appendix II for the approach to modelling and the key terms used in the tables in this section.
11
Reading comprehension was assessed using the GL New Group Reading Test.
Impact evaluation
Education Endowment Foundation 26
Hedges g effect size
12
for main effects model (95% Confidence Intervals): = -0.06 (-0.22, +0.09)
A small negative effect size was found but this was not statistically significant and so we must
conclude that there is no evidence of impact on the primary outcome.
The FSM*intervention interaction term, however, was found to be positive and statistically significant,
and when this is included into the model the (negative) FSM main effects term becomes statistically
significant. This suggests that the TextNow Transition Programme had a different impact for FSM and
non-FSM participants and to look more closely at this, models were re-run on the FSM and non-FSM
subsamples.
The pre-test KS2 measure was statistically significant in all models, reflecting the correlation between
this test score and the primary outcome (Pearson's r=0.5).
The following tables (3.11 and 3.12) summarise the primary outcome models for the FSM and non-
FSM participant subsamples.
Table 3.11 Primary outcome reading comprehension
(FSM Participant only, n=116 participants in 44 primary schools)
Description
Main Effects
Model
Interaction Models
Pre-test
interaction
Gender
Interaction
coef.
s.e.
coef.
s.e.
coef.
s.e.
Group (Intervention)
2.01
1.767
2.17
1.780
3.03
2.601
Pre-test (KS2 Reading)
1.33
0.238
1.15
0.334
1.35
0.240
Interaction (Pre-test*Intervention)
-
-
0.35
0.451
-
-
Gender (female)
1.08
1.866
0.90
1.873
2.03
2.586
Interaction (gender*Intervention)
-
-
-
-
-1.99
3.718
Constant
86.11
1.775
86.12
1.766
85.55
2.055
-2 Log Likelihood
-430.804
-430.509
-430.663
12
For detail on how the Hedges g effect size was calculated see Appendix III.
Impact evaluation
Education Endowment Foundation 27
Description
Main Effects
Model
Interaction Models
School level variance (s.e.)
19.38 (12.344)
17.99 (12.043)
18.67 (12.276)
Pupil level variance (s.e.)
82.82 (13.477)
83.26 (13.530)
83.06 (13.540)
Intra-Class Correlation (s.e.)
0.19 (0.111)
0.18 (0.110)
0.18 (0.111)
Hedges g effect size for main effects model (95% Confidence Intervals): +0.18 (-0.13, +0.48)
Amongst the FSM subsample, a positive effect size was found. However, this does not reach
statistical significance and so we conclude no impact.
Table 3.12 Primary outcome reading comprehension
(Non-FSM participants only, n=275 participants in 52 primary schools)
Description
Main Effects
Model
Interaction Models
Pre-test
interaction
Gender
Interaction
coef.
s.e.
coef.
s.e.
coef.
s.e.
Group (Intervention)
-2.08
0.999
-2.16
0.990
-3.35
1.350
Pre-test (KS2 Reading)
1.56
0.121
1.34
0.166
1.57
0.120
Interaction (Pre-test*Intervention)
-
-
0.45
0.231
-
-
Gender (female)
2.09
1.077
2.23
1.071
0.59
1.519
Interaction (gender*Intervention)
-
-
-
-
2.91
2.085
Constant
89.27
1.030
89.19
1.032
89.85
1.107
-2 Log Likelihood
-986.895
-984.983
-985.927
School level variance (s.e.)
17.79 (6.652)
18.61 (6.816)
17.78 (6.637)
Pupil level variance (s.e.)
65.42 (6.218)
64.07 (6.100)
64.91 (6.171)
Impact evaluation
Education Endowment Foundation 28
Description
Main Effects
Model
Interaction Models
Intra-Class Correlation (s.e.)
0.21 (0.068)
0.23 (0.070)
0.22 (0.068)
Hedges g effect size for main effects model (95% Confidence Intervals): - 0.19 (-0.36, -0.01)
Amongst the non-FSM subsample, a statistically significant negative effect size was found.
Overall, the findings for FSM participants are encouraging but the evidence is insufficient to claim a
positive impact whilst for non-FSM participants, it seems that the TextNow programme had a negative
impact that was small but statistically significant.
The analyses then focused on just the intervention group participants to look at whether attendance at
the TextNow Transition Programme coaching sessions had an impact on the primary outcome
measure. The attendance 'dosage' was approximated using attendance details provided by Unitas for
all participating pupils. The time in days spent involved in the (20 minute) coaching one to one
sessions was selected as the measure. The time spent in primary school and secondary school was
kept separate in the analyses. Table 3.13 summarises these dosage measures for the 199
intervention group participants.
Table 3.13 Attendance dosage (weeks and days) intervention group only (n=199)
Primary School Dosage
number of primary schools = 53
Secondary School Dosage
no. of secondary schools = 29
Weeks
Days
Weeks
Days
Expected dose
5
5 x 5 = 25
10
10 x 5 = 50
Mean (sd)
4.4 (1.61)
17.8 (7.11)
9.9 (1.24)
42. (6.901)
Median (IQR)
5.0 (0.00)
20.0 (6.00)
10.0 (0.00)
44.0 (10.00)
Min
0
0
0
0
Max
6
25
13
52
Table 3.14 Primary outcome reading comprehension dosage analysis
(Intervention sample only, n=199, across 53 primary schools)
Description
Total Time
(days)
Separating
Primary &
Secondary
coef.
s.e.
coef.
s.e.
Pre-test (KS2 Reading)
1.69
0.152
1.67
0.150
Impact evaluation
Education Endowment Foundation 29
Description
Total Time
(days)
Separating
Primary &
Secondary
Dosage Variables
Days attended (overall)
0.09
0.061
-
-
Days attended (primary school)
-
-
-0.05
0.106
Days attended (secondary school)
-
-
0.15
0.072
Constant
88.42
0.815
88.42
0.803
-2 Log Likelihood
-722.59
-721.26
School level variance (s.e.)
14.07 (8.380)
13.29 (7.931)
Pupil level variance (s.e.)
72.49 (8.867)
71.89 (8.711)
Intra-Class Correlation (s.e.)
0.16 (0.089)
0.16 (0.086)
A statistically significant coefficient does emerge around attendance at the secondary school stage of
the intervention. Increased time (in days) spent engaged with the TextNow intervention during Y7
(secondary school) is associated with an increased score on the primary outcome. Time spent
engaged with TextNow at the end of Y6 does seem to have an impact on the primary outcome.
Table 3.15 Secondary outcome Students Like Reading'
13
(n=331 participants in 53 primary schools)
Description
Main Effects
Model
Interaction Models
Pre-test
interaction
Gender
Interaction
FSM
Interaction
coef.
s.e.
coef.
s.e.
coef.
s.e.
coef.
s.e.
Group (Intervention)
-0.10
0.584
-0.10
0.584
-0.51
0.804
-0.16
0.700
13
This was measured using the Progress in International Reading Study (PIRLS) 2011 'Students Like
Reading' Scale.
Impact evaluation
Education Endowment Foundation 30
Description
Main Effects
Model
Interaction Models
Pre-test (KS2 Reading)
0.26
0.070
0.23
0.100
0.26
0.071
0.26
0.070
Interaction (Pre-test*Intervention)
-
-
0.06
0.138
-
-
-
-
Gender (female)
3.17
0.596
3.19
0.597
2.70
0.866
3.18
0.599
Interaction (gender*Intervention)
-
-
-
-
0.90
1.199
-
-
FSM (eligible & claiming)
0.74
0.654
0.75
0.654
0.80
0.657
0.64
0.926
Interaction (FSM*Intervention)
-
-
-
-
-
-
0.21
1.288
Constant
12.93
0.561
12.92
0.562
13.13
0.617
12.96
0.590
-2 Log Likelihood
-1026.459
-1026.369
-1026.175
-1026.445
School level variance (s.e.)
1.50 (1.016)
1.51 (1.017)
1.53 (1.024)
1.50 (1.016)
Pupil level variance (s.e.)
27.63 (2.274)
27.61 (2.272)
27.56 (2.269)
27.63 (2.274)
Intra-Class Correlation (s.e.)*
0.05 (0.034)
0.05 (0.034)
0.05 (0.035)
0.05 (0.034)
Hedges g effect size for main effects model (95% Confidence Intervals): = -0.02 (-0.22, +0.18)
A very small negative effect size was found but this is not statistically significant and so we must
conclude that there is no evidence of impact on the secondary outcome 'Students Like Reading'. No
evidence was found for differential effects across participant subsamples (indicated by a lack of
statistically significant interaction terms). The only statistically significant terms were (a) the KS2 pre-
test measure, reflecting its correlation with the PIRLS 2011 'Students Like Reading' outcome
(Pearson's r=0.19) and (b) participants' gender (females had a significantly higher outcome score,
although this was found to be independent of participation in the TextNow programme).
Impact evaluation
Education Endowment Foundation 31
Table 3.16 Secondary outcome Students Like Reading scale dosage analysis
(Intervention sample only, n=173, across 53 primary schools)
Description
Total Time
(days)
Separating
Primary &
Secondary
coef.
s.e.
coef.
s.e.
Pre-test (KS2 Reading)
0.26
0.105
0.26
0.105
Dosage Variables
Days attended (overall)
0.00
0.037
-
-
Days attended (primary school)
-
-
0.03
0.059
Days attended (secondary school)
-
-
-0.03
0.045
Constant
14.53
0.445
14.53
0.444
-2 Log Likelihood
-550.47
-550.14
School level variance (s.e.)
0.00 (0.000)
0.00 (0.000)
Pupil level variance (s.e.)
33.99 (3.654)
33.86 (3.640)
Intra-Class Correlation (s.e.)
0.00 (0.000)
0.00 (0.000)
There is no evidence of attendance having an impact on the 'Students Like Reading' secondary
outcome measure.
Impact evaluation
Education Endowment Foundation 32
Table 3.17 Secondary outcome'Students Motivated to Read' scale
14
(n=352 participants in 53 primary schools)
Description
Main Effects
Models
Interaction Models
Pre-test
interaction
Gender
Interaction
FSM
Interaction
coef.
s.e.
coef.
s.e.
coef.
s.e.
coef.
s.e.
Group (Intervention)
0.24
0.388
0.23
0.388
0.39
0.540
0.14
0.461
Pre-test (KS2 Reading)
0.01
0.047
0.04
0.067
0.06
0.047
0.06
0.047
Interaction (Pre-test*Intervention)
-
-
0.04
0.092
-
-
-
-
Gender (female)
0.69
0.398
0.69
0.398
0.85
0.575
0.70
0.399
Interaction (gender*Intervention)
-
-
-
-
-0.31
0.797
-
-
FSM (eligible & claiming)
-0.33
0.439
-0.32
0.439
-0.34
0.441
-0.50
0.628
Interaction (FSM*Intervention)
-
-
-
-
-
-
0.34
0.869
Constant
13.05
1.061
13.04
1.060
13.06
1.063
13.05
1.061
-2 Log Likelihood
-959.635
-959.522
-959.560
-959.558
School level variance (s.e.)
0.72 (0.550)
0.72 (0.550)
0.70 (0.548)
0.71 (0.548)
Pupil level variance (s.e.)
13.05 (1.061)
13.04 (1.060)
13.06 (1.063)
13.05 (1.061)
Intra-Class Correlation (s.e.)*
0.05 (0.039)
0.05 (0.039)
0.05 (0.039)
0.05 (0.039)
Hedges g effect size for main effects model (95% Confidence Intervals): = +0.06 (-0.14, +0.27).
14
This was measured using the Progress in International Reading Study (PIRLS) 2011 'Students
Motivated to Read' Scale
Impact evaluation
Education Endowment Foundation 33
A small positive effect size was found but this was not statistically significant and so we must conclude
that there is no evidence of impact on the secondary outcome 'Students Motivated to Read'. No
evidence was found for differential effects across participant subsamples (indicated by a lack of
statistically significant interaction terms). The KS2 pre-test measure was also not statistically
significant, reflecting its weak correlation with the PIRLS 2011 'Students Motivated to Read' outcome
(Pearson's r=0.08).
3.18 Secondary outcome PIRLS 'Students Motivated to Read' Scale dosage analysis
(Intervention sample only, n=183 across 53 primary schools)
Description
Total Time
(days)
Separating
Primary &
Secondary
coef.
s.e.
coef.
s.e.
Pre-test (KS2 Reading)
0.09
0.064
0.09
0.064
Dosage Variables
Days attended (overall)
0.00
0.024
-
-
Days attended (primary school)
-
-
-0.01
0.037
Days attended (secondary school)
-
-
0.01
0.030
Constant
13.60
0.273
13.59
0.275
-2 Log Likelihood
-497.29
-497.22
School level variance (s.e.)
0.07 (0.911)
0.13 (0.935)
Pupil level variance (s.e.)
13.36 (1.660)
13.29 (1.659)
Intra-Class Correlation (s.e.)
0.00 (0.068)
0.01 (0.070)
The coefficient for attending sessions overall was zero and when separating these out into primary
and secondary school stages, was close to zero. Therefore we conclude that there was no evidence
that attendance had an impact on the 'Students Motivated to Read' secondary outcome measure.
Impact evaluation
Education Endowment Foundation 34
Cost
Based on estimates from Unitas, the direct cost that schools would be required to pay for the
resources and support provided by Unitas for the TextNow Transition Programme is £112 per pupil.
This assumes a minimum cohort size of 10 and that sufficient schools are in the programme to make it
viable. In addition, to run the TextNow programme each school must appoint a coordinator to recruit
and manage volunteer coaches, manage programme delivery and undertake quality assurance and
reporting. The programme also requires volunteer coach time of twenty minutes per pupil per day for
75 days, plus time for initial training, preparation for coaching sessions and keeping records.
The direct costs quoted per pupil are an underestimate of the actual full economic cost of the
programme as they do not include estimates of the cost of coordinators, coaches and other resources
in kind. These are important to include as they represent significant ‘opportunity costs’ in that they are
resources that could be used on other programmes. However, an estimate of the total cost of this
programme would require a full economic cost-effectiveness analysis and was beyond the scope of
this present study.
Schools did not pay for the TextNow Transition Programme and received additional incentives through
the provision of MyChoice! credits, which young people exchanged for books. Schools were also
assisted on an individual basis to meet other costs incurred through their participation in this trial such
as payments for additional teaching assistant time and, where secondary school pupils undertook the
role of coaches, the cost of transport from the secondary to the primary school.
Process evaluation
Education Endowment Foundation 35
4. Process evaluation
4.1 Implementation
Exposure and dosage
Full dosage for pupils engaged in the TextNow Transition Programme comprised 15 weeks of daily
one to one 20-minute sessions (five weeks in primary school and ten weeks in secondary school) with
a volunteer coach. This was supplemented by access to the MyChoice! website so that pupils could
engage in online activities related to reading and 'buy' books with the credits earned from their
attendance and participation in the coaching sessions. Pupils were encouraged by coaches to read for
20 minutes per day in their own time. This aspect of the programme was not formally monitored. As
detailed in the fidelity section below, there was variation across the participating schools in their
adherence to guidelines regarding the quantity and content of coaching sessions and in providing
access to MyChoice! This, in turn, impacted on the extent to which pupils were enabled to access the
full dosage for the programme.
Attendance data supplied by the participating schools to Unitas is tabulated in Table 3.13 in the impact
evaluation section. This shows that the mean number of days on which pupils received coaching in
primary schools was 17.82 (varying from 0 to 25, against the programme specification of 25). In
secondary school the mean number of days on which pupils received coaching was 41.99 (varying
from 0 to 50, against the programme specification of 50). Data on usage of MyChoice! during the
secondary period of TextNow indicates that some pupils did not receive access to this aspect of the
intervention. Just under one quarter of pupils in the intervention group (61 of the 252) placed no
MyChoice! book orders. Most of these (n=41) came from five secondary schools where little or no
usage of the MyChoice! website was evident and the rest from a further seven schools with low usage.
Data was not available for usage of MyChoice! in the primary schools. Reasons for variation in
exposure and dosage are discussed in the fidelity section (4.2).
Pupils' engagement, motivation and experiences
The majority of coaches (84% of primary coaches and 79% of secondary coaches) that responded to
the coach survey felt that the young people were engaged during the coaching sessions and had not
encountered any behavioural issues. It is important to note here and elsewhere throughout this section
where the coach survey is used as evidence, that the response rate was low (primary 11%; secondary
21%). Although this represents coaches in over 50% of all schools it is likely that responses were
received from more committed coaches.
Pupils reported varying levels of engagement and enjoyment of the TextNow programme. Pupils’
engagement and enjoyment appeared to be related to their attitude towards reading, as well as being
greatly influenced by the quality and consistency of the relationship with, and input of, their coach.
Focus group pupils who reported high levels of engagement and enjoyment also talked highly of the
relationship they built up with their coaches who remained the same over the term and their
enjoyment of the regular one to one attention in a separate, quiet room. Where the coach relationship
was good this could overcome a reluctance to read, as one focus group pupil explained: 'Reading’s
really boring, but I was happy to get out of my lesson to do it because [the coach] was kind and very
nice'. Pupil engagement was also enhanced where the coaching sessions were perceived to be fun,
this appeared particularly important for disengaged boys.
Across the focus groups, pupils who were supported by staff coaches more consistently reported
greater levels of engagement and enjoyment than those supported by other student coaches. The
Process evaluation
Education Endowment Foundation 36
engagement and enjoyment of pupils supported by other students in their school was dependent on
the attitude and engagement of their student coach. For example, one pupil reported they read to
themselves as their coach wasn't listening, and another explained: 'He wouldn't let me read the book I
wanted to read, he said it was boring! I got fed up with it after that'. Some pupils' motivation and
enjoyment were negatively affected by a concern with what they were missing when they attended
TextNow coaching sessions. In one focus group, school pupils attended TextNow during tutor
sessions and felt that they were missing out on the fun activities with friends and information-giving
that was important to them settling in during their first term. Similarly, in another school, where the
coaching sessions were scheduled during assembly, pupils resented missing their friends and
explained that there were 'lots of things we didn’t know or find out about because we weren’t at
assembly'. While some pupils were happy to come out of lessons, there was also some reluctance
due to the 'hassle' of having to catch up and finish class work. It is, however, important to note that
some pupils did see the benefits of extra reading, even when balanced against missing other school
activities or a disinterested coach 'I would rather do the reading because it gets you better at doing
reading". However, a few remained resentful even though they thought their reading had improved.
While the pupil focus groups provided indicative evidence of the range of activity that pupils missed,
data on this was not collected across all schools.
There appeared to be a gender difference in terms of pupils’ engagement with reading for pleasure at
home or outside of the coaching sessions. Girls across most of the focus groups were generally more
likely to report that they enjoyed reading in their own time either on their own or to a member of their
family. While some of the boys stated that they did enjoy the TextNow sessions, this less frequently
translated to reading outside sessions 'The (coaching sessions) were well fun but I still don’t like
reading on my own it's boring':
Access and use of the MyChoice! website varied across schools. In one of the focus group schools,
where coaches and pupils regularly recorded points and played online games and activities, this
seemed to engage and motivate the boys in particular: 'It was awesome, I loved going on it nearly
every day.' Whilst most pupils across the other three focus group schools generally enjoyed spending
their points and ordering books, the recording of these, and use of the website at school, only
happened irregularly. For student-coached pupils, this required additional input from the TextNow
school coordinators who they only saw occasionally. Some pupils had problems remembering their
usernames and passwords which limited their use of MyChoice! at school and at home. Some pupils
expressed frustration at using the MyChoice! website, as the books they thought were available could
not be found or ordered when they searched for them, an issue also raised in the coach and
coordinator surveys.
School coordinators and coaches engagement and experiences
There was variation in the enthusiasm and commitment of schools involved in the TextNow Transition
Programme. Where there was less commitment from schools, there was less adherence to the
intended intervention which created fewer or poorer quality opportunities for pupils to engage with the
programme.
A number of factors led to lower levels of commitment and, in some cases, school dropout. The most
frequently mentioned factors identified from a range of evidence sources, spanning feedback from
Unitas staff; coordinator and coach surveys; informal feedback from secondary school coordinators to
invigilators and the focus group researcher; and telephone calls to schools to ascertain reasons for
drop out, were:
Lack of senior leader commitment. This tended to occur where recruitment had been via
academy chains so the 'buy-in' of individual heads was not necessarily secured at the start of
the project.
Delays in appointing coordinators and coordinators changing (or, in a very few cases, no-one
being in place to take on the role). The role of the coordinator was crucial in recruiting and
Process evaluation
Education Endowment Foundation 37
supporting coaches and facilitating access to MyChoice! In addition, secondary coordinators
played an important role in building relationships with primary schools.
Perceived lack of benefit to primary schools. Many primary schools saw no benefit in
committing to the TextNow programme. SATs had been completed prior to the start of the
programme, so any improvements in pupils' reading would not improve their published results.
In addition, maintaining daily reading sessions became difficult because of school trips and
other planned end of year school activities. Some primary schools also found it hard to release
staff to attend meetings related to the project.
Lack of well-established pre-existing relationships between secondary schools and their
feeder primary schools. Some secondary schools explicitly used the trial as an opportunity to
engage their feeder primary schools. However, the quality of the relationship and
communication between secondary and primary schools varied and, even though Unitas
helped broker discussions between primary and secondary schools, the intervention was
easier to implement where secondary schools had stronger relationships with their primary
counterparts.
Duplication of reading initiatives. Some secondary schools already had established reading
intervention schemes to support the target pupils, so some coordinators were unclear about
why they were participating in the TextNow programme, or felt forced to participate.
Amount of administration/paperwork. Some school coordinators and coaches felt
overburdened with the administrative requirements of the programme. However, for
coordinators it was sometimes unclear to what extent the burden was accounted for by trial
itself rather than programme-related administration.
Although a ‘Memorandum of Understanding (Appendix VI) was supplied to primary and secondary
schools to help formalise arrangements, Unitas considered that this could have been developed
further to provide simple but clearer statements of responsibilities.
4.2 Fidelity
Training for coordinators and coaches
Coordinator and coach training programmes (comprising modules, videos and downloadable
resources) are made available on the Unitas website, and the expectation is that all coordinators and
coaches will undertake the training and draw on the materials to support them in their role. The vast
majority of coordinators and coaches in both primary and secondary schools who responded to the
surveys had engaged at least to some extent with the training resources. Overall, primary coaches
had higher levels of engagement in the training than secondary coaches. In contrast, there was little
variation between primary and secondary coordinators' engagement in the training. There was
stronger engagement by both coaches and coordinators with the online modules than either the
training videos or downloadable resources (for example, 92% of coaches responding to the coach
survey reported engaging with the online modules, whereas 75% engaged with the online videos and
77% accessed the downloadable resources). Reasons given by coordinators who had engaged less
with the online training materials were: a lack of time, other people having taken on some aspects of
the role so they did not have to carry out some tasks themselves, or direct contact and emails from
Unitas had provided sufficient information. Reasons given by coaches for not engaging with, or not
making more use of, the online training materials were related to their prior knowledge and experience
and difficulties in accessing online videos. Some more experienced coordinators and coaches
reported that they did not engage with the guidance as they had already received training in
supporting reading and found there was too much information for their needs. However, some less
experienced coordinators and coaches, particularly student coaches, felt that the materials did not
Process evaluation
Education Endowment Foundation 38
provide sufficient guidance. This indicates a need for a more differentiated approach to training to
maximise adherence to the TextNow programme (as indicated by Coordinator and coach surveys).
Undertaking the coordinating role
Approximately 80-90% of coordinators undertook, at least sometimes, most of the key activities and
tasks required of them as set out in Unitas guidance, with stronger compliance in secondary schools
than primary schools. Coordinators reported high levels of compliance in both primary and secondary
schools in meeting with coaches, monitoring pupil attendance, discussing pupil progress and future
plans, and (in secondary schools only) providing resources, monitoring pupil progress and conducting
the final review. However, compliance was very low in primary and secondary schools in organising
trips (for example to a public library), carrying out celebration events or (in secondary schools only)
sharing progress with parents/carers. In addition, in primary schools at least 25% of coordinators did
not: (a) motivate coaches or advise coaches on suitable texts; (b) suggest approaches to planning to
coaches; or (c) advise coaches on the use of MyChoice! with their young person. Coordinators, in the
relevant survey, cited a lack of time, issues regarding the sharing of tasks with coaches, and (for
secondary coaches), not knowing the young people when they arrived at the school as reasons for not
being able to carry out specific tasks. (Coordinator surveys).
Recruitment and allocation of coaches
As anticipated in the Unitas guidance, coaches were drawn from a variety of backgrounds. We do not
have data on the whole coach population. As noted earlier, the low response rate for the coach
surveys and the likelihood of over-representation of more committed coaches amongst respondents
means that findings drawn from the coach surveys should be regarded as tentative indications.
Responses to the coach survey indicate that the majority of coaches in both primary schools and
secondary schools were either teaching assistants or learning support assistants (primary, 59.5%;
secondary, 41.7%). Teachers or head teachers were more likely to take on the coach role in primary
schools (primary, 19.1%; secondary, 5.6%), whereas just over a third of coaches in secondary school
(36.1%) were peers from the school. Peers were usually drawn from older year groups. Other coaches
were higher education students, librarians, administrators, and parent link workers.
Table 4.1 Background of coaches
Role
Primary (n=42)
Secondary (n=72)
n
%
n
%
Teaching assistant (TA)/ learning support
assistant
25
59.5
30
41.7
Teacher or head teacher
8
19.1
4
5.6
Peer coach from a participating secondary
school
3
7.1
26
36.1
Higher education student
0
0
3
4.2
Other categories with 2 or fewer responses:
includes librarians; administrators, parent link
workers and not specified
6
14.3
9
12.5
The majority of TextNow young people in primaries and half of those in secondary schools worked
with one coach only. However, around a tenth of all coordinators said that most of their young people
Process evaluation
Education Endowment Foundation 39
in the intervention group in their school had more than one coach each whilst in their school. Coaches
who shared their coaching with someone else were supposed to ensure effective handover and
shared planning. However, amongst those coaches who shared their coaching with someone else,
40% of primary and 21% of secondary coaches did not undertake any shared planning, which may
have undermined consistency and impinged on young peoples' experiences and
outcomes.(Coordinator and coach surveys).
One to one coaching sessions
As explained above, the exposure, dosage and consistency of the intervention was highly variable in
both primary and secondary schools, thus undermining the fidelity of the intervention both within and
across schools. Evidence from the focus groups suggests that many pupils experienced sub-optimal
levels of the intervention at some stage whether at primary school, secondary school or both. This
negatively affected pupil engagement and enjoyment. Nineteen pupils in the intervention group only
received coaching sessions in their secondary school.
Coordinators are expected to ensure that coaches know how to make sense of the initial reading
interview information and the pupils' reading profile, however just under a fifth of primary coaches and
just under half of secondary coaches were unaware of this requirement. The vast majority of coaches
who were given the information did take it into account when working with their young people. (Coach
surveys.)
As intended, coaches reported the use of a range of strategies for choosing texts and engaging young
people, but often did not, as set out in the guidance, plan their session. Coaches felt that planning
would lead to more formality in the sessions, making them less enjoyable for the young people, or said
that they had not had time to plan. Coaches tended to use 'summarising texts', 'asking questions',
'paired reading' and 'KWL' techniques to 'a great extent' rather than the other techniques. Only 50% or
fewer coaches used 'visual activities', kinaesthetic activities' and 'ICT', with use of visual and
kinaesthetic activities being lowest in secondary schools. The 20-minute session time constraint also
led to the coaches using some techniques less often. Coaches also noted that some
techniques/approaches were not suitable for certain young people. However, a few coaches stated
that they were not aware that they should be carrying out any of the techniques and approaches. This
may have been due to not engaging with the training materials. Primary coaches were more likely to
use specific strategies than their secondary counterparts. The vast majority of sessions took place in
an area where the coach and young person were rarely disturbed, although in primary schools there
were some instances where the coaching took place in the classroom while teaching continued, and in
a few secondary schools coaching took place during a form period lesson. (Coach surveys and pupil
focus groups.)
While some pupils experienced 20 minutes a day with a coach, the surveys and pupil focus groups
indicate that this was not consistently applied. Sessions were reported to have varied from 10 to 30
minutes in duration and from daily to a few times a week in frequency. For example, in one primary
school pupils reported that: 'we just did it whenever the teacher had time and came and got us… some
days we didn't do it because the teacher was busy'; and in another 'I read every day but sometimes for
only 10 minutes'. The inconsistencies in delivery were even greater at secondary school, particularly
for pupils with student coaches where the commitment and competence of their student coach was an
issue. Some pupils coached by other students received little or no consistent structured support, whilst
other student coaches took the role seriously. In one school with student coaches a pupil noted that:
'About seven times no one was there at all so we would just sit there and we just talked to each other'.
The variation in the approach of student coaches was summed up by a pupil in another school: But
when you had a good buddy, it was different, much better if they did it properly or we went to the
library'. Pupils with staff coaches more often reported that they had received more consistent and
higher quality support than pupils with peer coaches. (Pupil focus groups.)
Process evaluation
Education Endowment Foundation 40
Feedback from school coordinators during pupil focus group visits confirmed that the consistency and
quality of coaching was impacted by school-related factors including: timetabling issues; some subject
teachers not allowing pupils out of their lessons or allowing them out on time; staff/student coach
availability; difficulties in providing cover for coaches when absent and the low commitment of some of
the student coaches. (Coordinators surveys.)
Use of MyChoice!
As noted in the exposure and dosage section, Unitas usage data evidenced considerable variation in
the extent to which pupils engaged with MyChoice! Almost a third of secondary coaches and a tenth of
primary coaches who had completed the surveys reported that they had not encouraged the use of the
MyChoice! website with their pupils. Some coaches commented that they did not know enough, or, in
a very few cases, anything at all about MyChoice! A few pupils in the focus group who had been given
their login details at primary school continued to access the website from home even though their
secondary coach did not use the website with them. Some schools found the MyChoice! site difficult to
navigate. (Coach surveys and pupil focus groups.) There is insufficient data to ascertain whether the
lack of use of the MyChoice! website was attributable to pupil choice or access issues.
4.3 Perceived outcomes
Coaches and coordinators in both primary and secondary schools reported positive impacts on
participants, with coaches reporting stronger outcomes than coordinators. Between a third and half of
coaches and coordinators felt that the TextNow programme had increased the enjoyment and interest
in reading of all of the young people that they had worked with. Interestingly, they identified a stronger
impact on the development of participants' reading skills and comprehension around 60% of
coaches and 40% of coordinators felt that the programme had increased young peoples' reading skills
and comprehension. Positive unexpected outcomes noted by coaches and coordinators were the
development of relationships between coaches and young people, improved links between primary
and secondary schools, and the increased confidence and eagerness of the young people involved.
Some secondary school coordinators that had deployed student coaches noted an increased sense of
responsibility in the student coaches (Coach and coordinator surveys and feedback from secondary
coordinators.)
Negative outcomes were noted by coaches and coordinators in terms of resentment and loss of
interest from young people due to the time consuming nature of the programme. While focus group
pupils pointed to the negative impact of missing assemblies, form periods and lessons, there is
insufficient data to substantiate whether or not this led to negative social or learning outcomes.
Regardless of the quality and consistency of the intervention, overall most pupils in the focus groups
felt that their confidence in reading, and their reading level, had improved as a direct result of their
additional reading, albeit to differing extents. Pupils in three of the four focus group schools reported
making greater use of their school or public library: 'I borrowed a book and when I was home and
bored, I started to read it, and I finished it last week… It’s made a difference cos when I was at primary
school I didn’t like reading, now I wish I could read even more'. Some pupils, mainly girls, reported
reading more regularly and for longer than before sometimes reading different authors, or longer and
more challenging texts: 'I used to read two times a week and now I read every day for about an hour'.
However, for some pupils, especially boys, their increased confidence and regular in-school practice
during the TextNow programme did not result in reading for pleasure more frequently. Even at schools
where the intervention was the most consistent and of the highest quality, a few pupils who did not
read for pleasure before the intervention still did not at the end of the trial, despite their stated
enjoyment of the one to one sessions with their coach.
Process evaluation
Education Endowment Foundation 41
Notwithstanding the sometimes negative aspects of their experience, nearly all pupils thought that, on
balance, the additional reading had benefitted them and they would do it again if they could. The
factors they felt made the most positive difference were:
the one to one attention, particular with a engaged and encouraging coach;
having someone to listen to them read and help them concentrate and guide them when they
got stuck;
the regularity of the sessions and the additional practice;
the MyChoice! website.
4.4 Formative findings
Summary
There was considerable variation across and within schools regarding adherence to the TextNow
programme. This was particularly problematic where schools did not feel a strong sense of
commitment to the programme. Coaching sessions varied in duration and frequency and in the extent
to which they adhered to the Unitas guidance. Consequently pupils experienced varying levels of
consistency and quality of coaching. Where the relationship with their coach was positive, respectful,
and consistent the pupils were most likely to find reading enjoyable. Some of the student coaches who
were deployed in some schools were uninterested in their coaching role. Use of the MyChoice!
website also varied across schools. Where pupils had access, many were positive about using the
site, although some had been unable to 'purchase' the books they wanted as they were not available.
Coordinator and coach engagement with training was also variable, with the training considered too
detailed for coaches with a literacy teaching or support background and insufficiently detailed for less
experienced coaches.
Despite inconsistency in the implementation of the programme and some pupils' reluctance to miss
lessons to attend the coaching session, the pupil focus groups indicated that most intervention pupils,
including the most reluctant readers, thought that their reading had improved as a result of the
TextNow Transition Programme. Many young people also reported increased confidence, reading
more, reading for longer periods of time, and reading more challenging books; borrowing books from
school and public library had also increased. Girls appeared to be keener to read for pleasure than
boys. Coaches and coordinators considered that the intervention had had a stronger impact on the
development of young people's reading skills than their enjoyment of reading or their reading levels,
with coaches reporting stronger outcomes overall than coordinators.
The process evaluation has highlighted inherent problems in adopting the TextNow programme for
use at transition. There is little incentive for primary schools to participate and the coaching sessions
in primary are disrupted due to clashes with end of year activities. In the first term of secondary school
young people miss out on essential information and friendship-building when they attend coaching
sessions.
Necessary conditions for the success of the Unitas TextNow Transition
Programme
Our evaluation indicates that successful implementation of the programme requires:
Process evaluation
Education Endowment Foundation 42
Commitment from primary and secondary schools involved that is underpinned by a sense of
ownership by the school and a clear understanding of how the intervention will benefit the
school and pupils. This is crucial to ensuring that the TextNow programme is implemented as
intended.
Full engagement in training by coordinators and coaches so that they are knowledgeable
about their role and responsibilities, and for those new to supporting literacy development,
training regarding the use of appropriate strategies. Engagement in training could be improved
by Unitas developing a differentiated training approach that takes account of coaches' prior
knowledge and experience.
The selection of coaches that are enthusiastic and committed to supporting young people and
have the skills to build positive relationships.
The programme could be improved by resolving the technical difficulties and the clarity of the My
Choice! website, and ensuring that all books advertised on the site are available. Coaches and
coordinators recommended that the programme should be less time-consuming and intensive, with
the duration of the sessions extended but the frequency reduced. They also recommended that only
one coach should work with each young person, that more preparation time is required, and that face-
to-face, rather than online, training would have been more effective.
4.5 Control group activity
Feedback to Unitas from the schools indicated that three pupils in the control group were given the
intervention at their primary school. This was faded out at secondary school. No other pupils in the
control group received coaching sessions or access to MyChoice! for the duration of the project.
Access to MyChoice! was given to control pupils after the post-test. The small scale of the process
evaluation did not allow for data gathering from control group pupils so we are unable to make any
claims about their perceptions of the trial. However, some data does indicate that the intervention
group pupils resented missing out on activities, such as assemblies and lessons that the control group
attended.
Conclusion
5. Conclusion
5.1 Key conclusions
Initially, it should be understood that the trial results and therefore conclusions in this report relate to
the Unitas TextNow Transition Programme which was designed and implemented (for the first time)
specifically for this trial. As such, it is important to note that the following conclusions do not refer to
the established 10-week TextNow programme which targets mid-teens.
Key conclusions
1. The trial has not provided any evidence that the TextNow Transition Programme improved reading
comprehension or attitudes towards reading for pleasure over the transition from primary to secondary
school.
2. On average, pupils who participated in the programme made slightly less progress than similar pupils who
did not. However, this finding was not statistically significant, meaning that it could have occurred by
chance.
3. The programme was found to have a differential effect for pupils eligible for free school meals and their
peers. A small positive (but not significant) effect was found for pupils eligible for free school meals, while
a negative (and statistically significant) effect was detected for pupils not eligible for free school meals. It
is unclear why this differential effect was found.
4. Higher attendance at the 20-minute daily coaching sessions was found to have a positive impact on
reading comprehension. However, this was only found to be statistically significant for attendance at
sessions in secondary schools. Attendance of the coaching sessions was not found to have an impact on
the secondary outcomes (liking reading and motivation to read).
5. The programme appeared to be more effective when coaches were highly trained, enthusiastic and
committed, and when secondary schools worked closely with feeder primaries to coordinate all elements
of the programme.
5.2 Limitations
Despite some issues with recruitment of schools, the final number of secondary schools recruited
before randomisation (N=34) was above the required number set out in the sample size calculation
(N=30). Regarding pupil sample size, there were some issues with recruitment and attrition. The initial
trial size was anticipated to be N=600, however. However, there were some issues with gaining
consent that led to a lower than anticipated participant sample (N=501 recruited). Furthermore, some
children were lost to the post-test in both the intervention (N= 53) and control groups (N=56 plus one
pupil not matched to the NPD ) providing a final analytical sample of N=391 and an attrition rate of
22%.% Despite the smaller than expected pupil sample size, pre-test comparisons of the intervention
and control groups indicated that the randomisation had worked well providing two balanced groups
for evaluation. Therefore, it is suggested that the trial had good internal validity, with no major
limitations with regard to the study sample size and potential bias between the study groups.
Conclusion
Education Endowment Foundation 44
With regard to external validity, the trial schools had a slightly lower percentage of pupils with SEN
statements or on School Action Plus (3.1%) compared to English national secondary state-funded
school norms. (7.7%); a slightly higher percentage of pupils with English not as a first language
(16.8%) compared to national norms (13.6%); and a higher number of pupils eligible for freeFree
school mealsmeal pupils (25.4%) compared to national norms (16.3%). Overall, it is suggested that
that the schools represented in the study were slightly higher than average on characteristics of
disadvantage. Furthermore, the drop-out of five secondary schools does not compromise the internal
validity of the study but does have implications for the external validity. A total of 60 pupils were lost to
the trial due to the drop- out of the five secondary schools (intervention = 32, control = 28). Schools
whichwho were less engaged with the programme also tended to drop -out of the testing, meaning
that the treatment effect is estimated on more engaged schools.
5.3 Interpretation
The evaluation results indicated that the Unitas TextNow Transition Programmeprogramme was
having no significant effects on pupil outcomes i.e., either their reading ability or attitudes towards
reading for pleasure. However, there was evidence that the programme had a differential impact for
FSM and non-FSM participants with respect to the reading comprehension primary outcome measure.
Specifically, the evidence suggests that those eligible for FSM were more likely to experience a small
positive effect (although this was not statistically significant) and non-FSM participants were more
likely to experience a small negative effect. In other words, once the research design and sample size
are taken into account, there is insufficient evidence to conclude that the small positive effect seen
amongst FSM participants is due to the TextNow Transition Programme. However, amongst non-FSM
participants the evidence is stronger and it is reasonable to conclude that the negative effect found is
due to the TextNow Transition Programme experience.
The process evaluation found that implementing the TextNow programme at transition created a
number of issues that are not evident in the established 10-week programme for older secondary
pupils. In primary schools issues included a lack of commitment to a post-SATs initiative that did not
directly benefit the school and, clashes with trips and end of term activities. In secondary schools,
pupils resented being removed from assemblies, classes or other activities to attend coaching
sessions as they missed important information and opportunities to build friendships. Furthermore, the
most intensive aspects of the intervention took place during school time, with only limited support, via
MyChoice! during the summer holiday period. These issues indicate that delivering the TextNow would
require a more- coordinated approach between primary and secondary schools possibly with added
support during the summer break for schools and pupils.
Fidelity problems were most marked in schools that did not feel a strong sense of commitment to the
programme. As a consequence, some pupils did not receive the intended amount or frequency of
coaching and/or received no, or limited, access to the MyChoice! website. In some instances where
student coaches were deployed, they lacked commitment to supporting their pupil and the quality of
coaching was perceived to be poor by the recipient pupil. The process evaluation findings indicate that
for transition programmes to be successful, leaders and all staff involved in secondary schools and
their feeder primary schools need to committed and engaged and feel a sense of ownership of the
programme. In addition, effective communication between primary and secondary schools is essential,
as is the high quality coordination of the programme and ensuring that everyone engaged in managing
the programme and supporting young people undertake high quality training that takes account of their
prior knowledge and experience.
There are some salient points in the literature that can help with the interpretation of the results of this
evaluation. A systematic review of RCT evaluations of programmes that deploy volunteer tutors
highlights the importance of the programmes being well structured in order to produce positive results
Conclusion
Education Endowment Foundation 45
(Ritter et al., 2011). In this review a high degree of programme structure was typified by three
elements, namely: a focus on learning specific literacy skills; regularly timetabled sessions; and the
deployment of well-trained tutors. Reflecting on these three key factors of tutoring programme
structure, and in light of the current results, it can be said firstly that the TextNow Transition
Programme was timetabled but there were problems in fidelity to this timetabling (particularly in the
primary schools). Secondly, the focus of the sessions was fairly flexible both in choice of text and
literacy instruction provided. Furthermore, from the process evaluation, building and maintaining
relationships was a large part of the focus in the sessions rather than learning specific literacy skills.
Thirdly, there were problems with fidelity relating to the participation in training by the coordinators and
coaches, and coaches with no prior experience of supporting literacy initiatives were given insufficient
training in how to develop specific literacy skills.
A number of these implementation problems stem from contextual issues. Firstly, the fact that the
programme was running in the transition period, and secondly that it was provided for primary school
and young secondary students. A high level of coordination between two schools (i.e. the secondary
school and a feeder primary school) would be required to provide consistent highly-structured
sessions that focus on literacy skill development. However, McGee et al. (2004) suggest that good
links between secondary schools and their feeder primary schools are rare, and coordination
regarding individual pupil needs and past achievements are often neglected. In addition, previous
evaluations of the related TextNow programme for pupils predominately in their mid-teens showed that
age was a significant predictor of effectiveness with the greatest benefits of this programme being
provided to older teens (Brooks, Tarling & Adams, 2011).
15
The reading age test used as a measure of
effectiveness in Brook, Tarling & Adam's (2011) evaluations was able to capture and these contextual
issues may contribute to explaining the lack of positive results on the primary outcome measure.
However, as the earlier discussion has indicated, it is possible that this may be due to the lack of focus
on key literacy skill development and the need for a strong supporting structure to underpin skill
development.
At this point, it is worth exploring the notion of book choice as a central tenet of the TextNow
programme. Indeed there is growing practice literature to support the link between choice and
motivation to read (Flowerday & Schraw, 2000; Gambrell, 1996; Hunt, 1970; Sanacore, 1999; Worthy,
Moorman, & Turner, 1999). However, there is little systematic evidence of causality or theory of
change to explain the