Impact Evaluation of the Pakistan Reading Project: Cohorts 1 and 2

USAID and the Government of Pakistan (GOP) created the Pakistan Reading Project (PRP) to address the reading deficit in Pakistani schools. During the seven years of program implementation, approximately 1.3 million students have benefited from the intervention in seven provinces of Pakistan. PRP aimed to have a positive effect on students’ reading outcomes by improving classroom learning environment for reading, policies and systems, and community-based support for reading. The program employed an array of interventions that are geared towards changing the way reading is taught in the classrooms. Treatment ingredients, which were provided to students over the course of two school years from the beginning of first grade to the end of second grade, include teaching and learning materials and the provision of professional development opportunities for teachers, including face-to-face training, teacher inquiry groups, and school support visits. Research evidence from a quasi-experimental study indicate that PRP is had positive and statistically significant effects on students’ reading outcomes and positive, medium to large and statistically significant effects on teacher’s instructional practices. 

To evaluate the effects of PRP on students’ reading skills, the IRC used a quasi-experimental design to compare the reading outcomes of two cohorts of students in Urdu-medium schools who received the PRP intervention (Cohort 1 and 2) with the outcomes of a comparison group that had not participated in the program at the time of data collection (Cohort 3). A cross-sectional sample was used to assess students’ reading performance and teachers’ instructional practices at baseline and endline to identify the effect of the program on students’ reading skills and teachers’ instructional practices. The project collected data from 192 schools (132 treatment, 60 in control), 344 teachers (233 treatment, 111 in control) and 5523 students (3767 treatment, 1756 in control) using Early Grade Reading Assessment (EGRA) to measure students’ reading skills and a classroom observation tool to capture teachers’ instructional practices. Researchers used a difference-in-differences approach to identify the learning gains observed in students and teachers in the treatment (Cohorts 1 and 2) and control (Cohort 3) groups, from baseline to endline. 

Findings suggest that PRP had positive and statistically significant effects on students’ reading outcomes and teachers’ instructional skills. Students in first grade who received one year of the intervention showed small non-significant gains on their reading skills, but second graders who received two years of the intervention showed significant moderate-to-large learning gains, reflecting the accumulated effect of the program on students’ reading skills. Girls exhibited higher baseline performance in all reading outcomes than boys. While first grade girls reaped greater benefits from PRP than boys, in second grade, the results are mixed with boys obtaining larger gains in key reading outcomes such as oral reading fluency and reading comprehension. 

With regard to teachers, findings suggest that PRP may have had a positive, moderate-to-large and statistically significant effect on teachers’ instructional practices, which increase with higher dosages. The effect of PRP on teachers’ instructional practices is moderate for Cohort 2 teachers who received one year of continuous professional development (CPD) and large for Cohort 1 teachers who received two years of CPD. Teachers’ ability to promote students’ participation and well-being in the classroom exhibited a high correlation with students’ oral reading fluency. When teachers focused on teaching reading at the expense of students’ participation and well-being, oral reading fluency scores decreased. 

The report also discusses the limitations of the study and implications for researchers, practitioners, and policy makers. A main limitation to the study is that the cohorts were not randomly assigned and therefore we cannot be certain that the effects resulted directly from PRP domains, as well as measures of program implementation quality.

The Airbel Impact Lab at IRC is a team of researchers, strategists and innovators committed to the accelerated design, rigorous evaluation and cost-effective scaling of the most impactful solutions supporting people affected by crisis.