One of the exciting things about the SCREEN project in Zambia is the opportunity to evaluate the work from start to finish. It’s natural to assume that we would think of evaluating a project after it has finished. With the SCREEN project, we plan the evaluation right from the start. Here are the five stages of our evaluation plan.
We needed to know exactly what we wanted to measure before we produced the films – what changes in knowledge, attitudes, care-seeking, and lifestyle we hoped to see. This meant each film had clear learning objectives that we could measure progress towards.
Measuring the effectiveness of communications projects is trickier than measuring a time-bound, geographically-based project. In our connected lives there is so much – sometimes too much – information that crosses our path every day. We might also converse with lots of people with different opinions every week. So how do you work out if your films have communicated effectively?
The first step is to work out the current level of knowledge and dominant health practices amongst the rural young people in the districts where we will first screen the films. In July trained data collectors will ask young people a range of questions as part of a baseline survey. Having baseline data means that we can measure any reported changes after the films have been screened.
It’s also important to monitor how the project is progressing during implementation and to collect results and feedback. Listening to young people as they watch the films will give us some idea of their responses to the films’ messages and content.
Once the films have been screened in all the schools and health centres in every project district, we will conduct an end-line survey at the end of 2022 to gather data that we can then compare to the baseline data. Although this approach is not a scientific experiment that can confidently identify causal links between watching the films and subsequent actions, analysing end-line data means that we can draw associations between any reported changes and the film screening.
But how do you know whether it was the films that influenced any reported changes? As well as conducting an end-line survey, gathering qualitative data through interviews and focus group discussion helps to provide an in-depth understanding of young people’s responses to the film screenings. Experienced facilitators and interviewers will explore whether and how watching the films might have led to different decision-making or actions around reproductive, sexual, and mental health issues.
This work of evaluation is not completed until findings and learnings are shared with local communities and other organisations working with young people or involved in health communications. Of course at Medical Aid Films we’ll be making a short film about what we’ve learned through the SCREEN project and how this will influence our film-making in the future – we hope it will be fascinating viewing!