Leave No Girl Behind: Evaluation of EAGER in Sierra Leone

The Every Adolescent Girl Empowered and Resilient (EAGER) project is a DFID-funded Girls’ Education Challenge (GEC) Leave No Girl Behind (LNGB) project in Sierra Leone. EAGER is implemented by the IRC in partnership with Concern Worldwide (CWW), Restless Development, and BBC Action Media in ten districts across Sierra Leone. The project duration is approximately four years running from February 2019 to January 2023. EAGER targets out-of-school (OOS) adolescent girls aged 13-17 – those who have never attended school or who have dropped out – and lack basic literacy and numeracy skills.

EAGER aims to significantly improve learning outcomes for functional literacy and numeracy as well as business and life skills through an eleven-month learning programme. Additionally, girls will receive one-on-one mentorships from selected female mentors within the community to develop individual transition plans focused on empowerment (economic, learning, household, community and personal). EAGER aims to reach a total of 32,500 adolescent girls across three cohorts in 500 communities with activities for cohort one (7,500 girls) starting in January 2020. Cohort one is the focus of the external evaluation.

Research questions

The IMC evaluation team was required to develop an evaluation approach that answers the following overarching questions, with support from the EAGER Consortium Coordination Unit (CCU):

  • Process – Was the project successfully designed and implemented?
  • Impact – What impact did the project have on the learning and transition of marginalized girls, including girls with disabilities?  How and why was this impact achieved?
  • Effectiveness – What worked (and did not work) to increase the learning and transition of marginalized girls as defined by the project?  What adaptations and improvements can be made to the project for cohorts 2 and 3?
  • Sustainability – How sustainable were the activities funded by the GEC-LNGB and in what way? Was the project successful in leveraging additional interest and investment?

Specific questions that IMC will be required to answer using a mixed-methods evaluation approach are:

  1. How do the effects of the EAGER learning project on girls’ learning and transition outcomes vary for different subgroups of girls (e.g. single vs. married, pregnant vs. not pregnant, disabled vs. non-disabled, rural vs urban, older vs younger girls, etc.)?
  2. What individual (including psychometric measures), home, and community level characteristics are associated with girls’ learning and transition outcomes?
  3. What implementation characteristics (e.g. attendance to interventions, community members’ engagement with radio show and community group discussions) moderate the effect of the EAGER learning project and Business training on girls’ learning and transition outcomes?
  4. What are the girls’ perceptions and experiences with the interventions?
  5. What are the profiles and experiences of girls that were successfully able to transition to formal education, training or (self) employment and how do they differ from girls who were unable to transition into new paths?
  6. What are some of the facilitators/barriers to successful transition? Are the girls able to remain in their chosen transition pathway after the project and mentoring concludes? What obstacles remain for girls who were unable to transition?
  7. What are the community attitudes and facilitators/barriers to girls’ education and employment? How do these change over the course of the project?
  8. How can the project improve for future cohorts? What elements of the intervention work/do not work, and what adaptations can be made?

Methodology

Baseline

A convergent mixed methods design informed this baseline evaluation; quantitative and qualitative data collection occurred simultaneously. The evaluation design includes a pre- and post-test assessment of a group of treatment girls and a cluster-based sampling method accounting for marginalization status and geographic diversity. Surveys and learning assessments were conducted with 2,084 beneficiary girls from 215 communities, and their caregivers or head of households for quantitative data collection. The qualitative component provides context and depth to the findings of the quantitative impact evaluation and increases validity by triangulating findings. Qualitative data collection relied on a purposeful approach across 10 communities (1 per each of the 10 implementation districts) and reached 441 individuals (247 females and 194 males, including 144 beneficiaries) via key informant interview and focus group discussions.

Tools used during the baseline evaluation were:

  • Adapted Early Grade Mathematics Assessment (EGMA)
  • Adapted Out of school Learning Assessment (OLA)
  • Girls’ Combined Survey
  • Life Skills Survey
  • Head of households Survey
  • Caregiver Surveys
  • Key informant interviews (KIIs)
  • Focus group discussions (FGDs)
  • Program Data Sheet

Main Findings and Conclusions

Marginalization analysis

Thorough subgroup analyses identify that beneficiaries face several important barriers to education. Along with district-level differences that arose during analysis, these barriers require nuanced consideration from EAGER. They are not mutually exclusive and may intersect with other barriers.

  • A higher than anticipated percentage of beneficiaries are married (44.1 percent) and many have children (57.5 percent).
  • An unspecified subgroup of 9 percent of beneficiaries are their own heads of household.
  • 14.6 percent of the sample have one or more disabilities;  60 percent of these qualify due to reported daily experiences of anxiety or depression.
  • Nearly half (45.3 percent) of girls in the sample never went to school and nearly the same proportion (45.2 percent) went to school but dropped out with less than six years of education completed (equivalent of primary education level).
  • In the past, girls have experienced the following barriers to education: families not having enough money to pay fees (72.2 percent), followed by girls needing to help around the house (23.4) and a girl having a child or being pregnant (18.9 percent).

Analysis of project’s gender approach

  • EAGER’s approach is largely gender-accommodating with opportunities to shift towards a more deliberate gender transformative approach.
  • While men and boys interviewed are interested in women and girls gaining further education and contributing more substantially to household expenses, empowerment will likely be limited due to existing gender norms.
  • Although interviews with partners and caregivers revealed little overt resistance, only a small number of males pledged to actively support girls in their EAGER pursuits.
  • Transition options, which though not fully articulated at baseline, may simply reinforce traditionally female pathways like catering, hairstyling and soap making, limiting beneficiaries’ opportunities.

Transformative programming will likely be needed to increase the proportion of males actively supporting girls for EAGER participants to succeed in the program, especially given the high percentage of girls who are married and/or who have children. EAGER will also need to closely scrutinize its transition plan as it develops for gender transformative opportunities.

Outcome 1: Learning

  • Literacy: Three distinct groups of beneficiaries emerge in terms of literacy skills: 1) 40 percent could not read any letters, 2) 35 percent achieved letter recognition but could not read or comprehend proficiently, and 3) 25 percent who could not read proficiently but succeeded with the majority of the easy comprehension questions.
  • Numeracy: On average, all test-takers performed equally or slightly better on the real-world settings items. Girls who had never attended school performed significantly better (9 percentage points) on real-world problems while girls who had attended school performed only slightly better (3 percentage points).
  • Life skills: Girls demonstrated a weak to moderate awareness on the wide range of knowledge, attitudes and skills assessed by this tool. Sub-task analysis shows that while over half of girls (58.8 percent) assumed hostile intent in response to story prompts, a much smaller percentage of girls (21.1 to 39.1 percent) say they would act in an emotionally dysregulated way. Results also show that girls with disabilities and younger girls may benefit more significantly from exposure to these particular life skills.

Takeaways: EAGER’s learning program aligns with beneficiaries’ needs in focusing on real-world numeracy skills. Because of beneficiaries’ significantly different reading abilities, however, EAGER will need to emphasize a tailored approach for literacy instruction based on each individual group of girls. In life skills, EAGER presents an opportunity for girls to strengthen their awareness of important topics like good health practices as well as strengthening girls’ reactions to unpleasant situations and building upon problem-solving tendencies.

Outcome 2: Transition

EAGER’s transition component will focus on five types of empowerment, although details of the approach are still being finalized: economic, learning, household, personal and community.

  • Challenges for economic empowerment: The majority of girls are not employed (57.5 percent) at baseline while 4.5 percent are employed by others and 29.7 percent are self-employed. Almost none (five individuals) are engaged in formal paid employment.
  • Interviews with men, especially male partners emphasized education as a means to generate household income but did not approve of changes that would alter traditional gender norms. Social and community norms will likely be obstacles for girls.
  • Challenges for learning empowerment: 75 percent of the sample of girls demonstrated the inability to read or comprehend proficiently. The same barriers (lack of financial means, household constraints and child rearing/pregnancy) that complicated schooling previously likely will also complicate other schooling opportunities.
  • Challenges for household, personal and community empowerment: Girls demonstrate limited ability to make decisions in their lives. While 75 percent of girls expressed the capacity to make important decisions and voice opinions in their home, 66 percent also state that they cannot choose their educational path and are subject to the decisions of others.
  • Challenges of the mentor model: Mentors, who are to guide girls in their transition, exhibit more similarity in age and background to girls entering the project than initially expected.

Takeaways: Adequately trained mentors and connections to community resources, networks, and knowledge necessary for identified transition pathways will require proactive planning and resource building in the first year of the project. Attention to various identities, such as girls’ marital status, parental status and disability status should all be addressed, and accommodations made. Entrenched norms, specifically around gender, are likely to continue to be major barriers to girls’ transitions. EAGER’s community programming will need to be substantial and effectively target community members, especially boys and male partners to foster a transformative approach.

Outcome 3: Sustainability

  • Nearly all boys’ focus groups, caregiver focus groups and community leaders (91.5 percent) demonstrate a “latent” rating according to the sustainability scorecard, indicating some changes in attitude towards girls’ education and empowerment already at baseline, but behavior may not yet have followed.
  • A notable 8.5 percent of stakeholders demonstrated a “negligible” status showing no support for girls’ education. These stakeholders included 2 boys focus groups (out of 10).

Takeaways: EAGER should build upon participants’ changing views to move towards changed practices.

The Airbel Impact Lab at IRC is a team of researchers, strategists and innovators committed to the accelerated design, rigorous evaluation and cost-effective scaling of the most impactful solutions supporting people affected by crisis.