Measuring impact: The results and implications of UCL’s evaluation of the AllChild Impact Programme
While we have powerful data demonstrating the impact of, and providing insightful learnings into, our work with children and young people, we want to go further. In particular, we are keen to try to understand the difference in social, emotional, and academic outcomes for children who participate in our Impact Programme relative to comparable children who do not participate.
For this reason, in 2020, we commissioned an external evaluation with the UCL Centre for Education Policy and Equalising Opportunities (CEPEO) and the Helen Hamlyn Centre for Pedagogy (0 – 11 years), collectively referred to hereafter as ‘UCL’. We detailed the interim findings of this evaluation in February and, previously, the methodological challenges this approach has posed. Now we have the full findings and recommendations of this work. While the results are somewhat promising, they continue to reveal the significant challenges faced in trying to develop a robust evaluation design for our model of support using comparison groups.
The methods
In addressing the challenge of defining meaningful comparison groups for children and young people participating in our Impact Programmes, UCL employed two different quantitative methods and one qualitative.
1. Propensity score matching
This approach compared the outcomes of the AllChild cohort to those of two matched non-AllChild comparison groups in the same schools where we work: wide and narrow.
- In the wide comparison group, UCL matched each AllChild young person with two young people in their school not participating in the programme who are most similar at baseline, based on a range of observable characteristics. However, they found that the effectiveness of AllChild’s approach to identifying the children and young people most suited for our Impact Programme means that the young people available for selection into this comparison group are still substantially different to AllChild participants.
- For the narrow comparison group, the approach was fundamentally the same but UCL only attempted to match each child on an AllChild programme with one other (based on that same range of observable characteristics) and only where the fit between the participant and non-participant was within certain parameters. This leads to a better match — albeit still with some differences — but smaller sample size.
Given the challenges of identifying comparable pupils among non-AllChild participants in schools where AllChild works, UCL has acknowledged that, for both groups, there will be, on average, a lower level of risk factors than for those on our Impact Programmes. It said: “Even where we have taken quite an aggressive approach to identifying well-matched comparison groups at the expense of sample size – ultimately young people selected into AllChild are extremely different to their peer group and, as a result, it is extremely difficult to identify a suitable number of truly comparable individuals as a comparison group”.
Over the course of this four-year project, the UCL team have consistently found that most children who are suitable for AllChild’s Impact Programmes within a school have already been selected to be part of the programme. This is of course reassuring in terms of our approach to identifying the children and young people that could most benefit from our work but, for the purposes of evaluation, makes benchmarking our impact a real challenge.
2. Discontinuity analysis
UCL also conducted a discontinuity design analysis. This takes advantage of AllChild’s (necessary) thresholds on school attendance when selecting children for our programme; a child with school attendance above 96% is less likely to be considered ‘at risk’ and therefore be selected. The discontinuity analysis compares the outcomes of children who fall just either side of the 96% school attendance cut-off, so with quite different selection probabilities but otherwise likely to be quite similar (on average) given that the difference between just under and just over might mean just a single extra day off school. As such, this methodology is likely to lead to more comparable groups and so a greater chance of avoiding bias in finding the causal impact of our programmes.
3. Process evaluation
Alongside the quantitative studies, UCL also undertook qualitative research to examine how AllChild Impact Programmes are delivered and how they might enable change in children and young people’s social, emotional, and academic outcomes. This included observations of key components of the programme and regular interviews with young people over the course of their time with AllChild.
The findings
1. Propensity score matching
The midpoint results (measured after year one of the Impact Programme) for primary schools painted a positive picture of impact on school attendance and on social and emotional wellbeing, although the effect sizes are small and in some cases not statistically significant. They painted a more mixed picture on academic attainment: the estimate of impact is positive in the case of the wide matched sample but negative in the narrow matched sample (contrary to expectations). However, again, the results are small and not statistically significant.
The endpoint results (measured after year two and for which there was a smaller sample size) showed no evidence of difference across primary school participants, i.e. the positive midpoint picture on attendance and social and emotional wellbeing at primary school age was not seen again in endpoint outcomes. The endpoint results for secondary schools include a mixture of both positive and negative findings across all outcome areas, the majority of which were not statistically significant.
In summary, while some positive impact was identified at points, there were no clear patterns seen under this method across the full evaluation when comparing participants in the AllChild programme to comparison group children. It is hard to say exactly why this is, although the methodological challenges in finding a suitable comparison group are likely a significant factor.
Nevertheless, as a result of this work. as well as our learning from ongoing internal analysis, we are exploring possible changes to how we measure outcomes for young people on our Impact Programmes (e.g. how we might better understand our social and emotional impact). We have also made some changes to the phasing of our programme delivery to allow for delivery to start sooner in year one and to strengthen delivery in year two, which we believe will support with maintaining impact across the course of the programme.
2. Discontinuity analysis
UCL reported that, at mid-point: “The estimates are in the direction that we would hope they would be and are of a substantial magnitude. Nevertheless, likely due to the limitations of the discontinuity method with the sample size available, are not statistically significant.”
And, again, at end-point: “Extending then to end-point outcomes (which further reduces sample size, especially as we do not observe end-point outcomes for the most recent cohort)…the estimates are in the direction that we would hope they would be but, likely due to the limitations of the discontinuity method with the sample size available, are not statistically significant”.
In short, at both mid-point and end-point there was evidence of impact in the right direction — that children just under the 96% cut-off who were selected onto the programme achieved better attainment and socio-emotional outcomes than those above the threshold who were not on the programme — however, this was not statistically significant.
UCL notes further that: “Despite the fact that these findings are not statistically significant, it is encouraging to note that, with what we would hope to be the estimates closest to being causal from this, all are consistent with the direction of impact that we would hope for from the intervention”.
Our learning for the future is that we would need our sample sizes to be larger before repeating this type of analysis.
3. Process evaluation
In its process evaluation analysis, UCL noted a number of strengths related to the programme model and to AllChild’s impact with children and young people, including:
- our focus on socio-emotional outcomes for children and the knock-on benefits of this work
- supporting and enabling children’s agency, particularly through activities allowing creative expression and choice
- Link Workers as ‘trusted adults’
UCL said: “Overall, AllChild’s approach of prioritising socio-emotional support laid a foundation for improvements in other areas, including academic performance and personal development. This holistic impact suggests the programme’s effectiveness in addressing the diverse needs of the participants.
“The majority of the [children] reported feeling comfortable discussing personal issues with their Link Workers and expressed a desire for more one-on-one time. In-school Leads attributed a significant portion of the programme’s benefits to the presence and engagement of Link Workers, noting that [children] eagerly sought opportunities to spend time with them. The continuous presence and availability of Link Workers in schools were highlighted as crucial factors contributing to the programme’s success.”
UCL also provided some useful observations and recommendations in a number of areas related to the design and delivery of our programmes. These include:
- Allowing young people more agency within the setting of programme plans
- Further developing our approach to family engagement
- Reducing cohort size
We have been acting on these recommendations as part of our ongoing improvements to programme design and delivery and as they’ve emerged throughout the course of the UCL evaluation.
· Family engagement and young person agency - we have reworked our engagement process during the first term of each child’s programme to enable more time for Link Workers to help young people understand the programme, build a trusted relationship, and inform child-centred goal setting. This deepens our ability to engage with both child and family and better tailor programmes according to each child’s context.
· Cohort size - while we haven’t changed the numbers of young people on the programme in each school, we have made significant changes to our resourcing model to expand capacity. Our delivery managers are now based in schools to reduce travel times, improve team collaboration, and offer more immediate programme oversight and co-delivery. We have also added Associate Link Workers to act as further support for existing link workers and delivery managers. These changes have all worked to add capacity and increase support and been well received by schools and the AllChild team.
Our learnings
In the context of discovering more about the impact of our work, we have commissioned a specialist external learning partner, Coram Impact and Evaluation, to help us evaluate the process and impact of our new programme in Wigan Borough. This extensive project will help us:
- learn rapidly from delivery on the ground and more quickly adapt programme design and delivery to meet local needs and contexts
- document our learning from launching a new, co-designed programme in a new place, to help us plan for working in new places in future
- document learning from AllChild, Wigan Council, schools, and other stakeholders on how our collaborative and cross-sector approach is improving the local early help system
- better understand the impact of our Impact Programmes with children and young people, families, schools, and communities in the first two years of delivery
This is the next step in our evaluation journey. We have intentionally embedded this research from the beginning of our work in Wigan to learn about and improve our design and delivery in real-time, as well as to provide a summative review of our impact. Our next evaluation blog will explore our partnership with Coram.
In conclusion
It remains challenging to compare outcomes for children and young people on the AllChild programme against any progress they might have made in their social, emotional, and academic outcomes without our support. We are committed to exploring ways to better understand, learn from and report on our relative impact, not just for our children and young people, but also their families, schools, communities, and wider local systems of support. We’re looking forward to seeing the outputs of our new learning partnership in Wigan both to inform ongoing improvements in programme design and delivery and learn more about potential methodologies for evaluating our impact.
