Description


This thought-piece was authored by Ashley Barry, Vice President for Strategy and Impact at Education for Employment, and Linda Fogarty, Director of MERL at International Youth Foundation, with contributions from Lucia Hernandez, Senior MERL Manager at IYF, and Jeetwan Tripathy, Monitoring & Evaluation Specialist at Generation Unlimited. This thought-piece was produced as part of collaborative efforts facilitated by the Technical Advisory Group (TAG) on Monitoring, Evaluation, Research, and Learning (MERL), convened by UNICEF's Generation Unlimited.

The social sector has lagged behind the private sector in the effective collection and use of data, but that is rapidly changing. An increasing number of social sector organizations are lending greater intentionality to the long-term development of data collection, learning, and adaptive management systems given their significant potential to support improved results through streamlined Key Performance Indicator (KPI) selection, data sharing, and learning utilization. Collectively, these approaches form the basis for high frequency monitoring (HFM), which enables organizations to understand real-time results, long-term outcomes, and organization-wide impact. Through the process of implementing HFM approaches, Education For Employment (EFE) and the International Youth Foundation (IYF) have enhanced organizational capacity to serve youth by bringing the right data to the right staff at the right time.


Implementing High Frequency Monitoring Systems

Implementing HFM and enabling technologies begins with building a collective understanding of how the organization and those it serves will benefit from the shift away from the status quo toward the use of more sophisticated data management systems. Although results measurement and adaptive management are critical for supporting impact, many organizations have failed to meet their potential by relying on ad hoc data collection systems, such as spreadsheets, that limit the depth and breadth of their learning activities relating to youth employment programs. An overreliance on fragmented, project-specific data collection approaches at the expense of organization- and multi-stakeholder-level KPIs further complicates these efforts and limits coordination across development actors. Change leaders need to effectively communicate to key stakeholders at all levels of the organization how the development and adoption of HFM approaches will provide a solution to these challenges and why the effort needed to implement the change over time will yield dividends for the organization, as well as for the youth they serve.


Staff leading the change effort should engage key stakeholders holding leadership, project implementation, and monitoring, evaluation, research, and learning (MERL) positions to ensure that efforts will be mutually reinforcing. At EFE, all Affiliates within the EFE Network were engaged in this process to ensure that the Network-wide indicators and the proposed technology solution would meet the diverse needs of staff working across nine countries in the MENA region.


Once change leaders secure sufficient buy-in to begin the effort, project scope should be clearly defined, including technology and indicator selection, by documenting business processes and goals across operating units. Similarities and - in particular - differences that the monitoring system will need to accommodate should be noted. Technology systems should then be reviewed against the requirements, with a focus on identifying a system that will support growth and adaptation over time as data collection needs and organizational size changes within the evolving international development landscape and youth employment contexts. Key considerations include flexibility of the technical features, the ease with which the organization could engage in future technical configurations, as well as the potential for the subscription model to accommodate organizational growth in a financially sustainable manner.


Indicator selection, accompanying data collection tools, and a comprehensive data collection plan should be developed in partnership with key stakeholders during this phase to prepare for their incorporation into the technology solution. Indicators should support the measurement of organization-wide objectives as well as outcomes of interest for the broader youth development field. For example, EFE measures quarterly job placement and retention results across all of its job-track graduates, regardless of the project under which they are trained. IYF measures changes in life skills and technical skills, disaggregated by sex and age group, to track reach across target audiences.


As the technical selection and development process advances, change leaders should prioritize ongoing, proactive communication with staff in order to maintain and expand their investment in the effort. Staff engagement in technical testing should be explored to ensure that system features meet staff needs and are functioning properly prior to launch. A comprehensive staff training plan that anticipates evolving training needs throughout the first year of system usage should be developed. Identifying system champions and clearly communicating roles and responsibilities for staff are critical for supporting adoption efforts following launch. Change leaders should respond proactively to any emerging staff concerns regarding potential risks associated with the shift in operational approaches that will occur with the implementation of the new system and should be mindful of any unstated concerns about the effects that the shift could have on existing power dynamics within the organization.


Efforts to engage staff in discussions about the results data generated through the HFM system are important for encouraging data utilization and continued system usage. For example, IYF engages in semi-annual all-staff meetings to reflect on results dashboards and EFE engages in semi-annual learning workshops with EFE Affiliate teams. These efforts support staff in drawing conclusions from the data and identifying action items for youth program improvement, while celebrating and acknowledging everyone it takes to make the work happen.



Key Learnings and Recommendations to Support High Frequency Monitoring

  1. Grow with the platform: Organizations should undergo a thorough selection process that accounts for current and future needs, choosing a platform that can grow with the organization and adapt as it evolves.
  2. Build organization-wide ownership and support accessibility: Organizations should consider building ownership by moving beyond a MERL function to a decentralized approach that actively engages program management, business development departments, executive management, and others. As the system is socialized, training should be integrated into onboarding activities and non-MERL staff should take on increasing roles. Adding HFM-specific milestones to annual performance objectives helps clarify roles and endorses HFM as an organizational priority.
  3. Evolve indicators: Organization-wide indicators will evolve along the journey. This evolution should be embraced and socialized, becoming part of the organizational culture.
  4. Manage the change: Developing a structured change management plan will support organizations in gathering the needed buy-in to begin the process of rolling out HFM and to move it forward during challenging moments along the way. Change leaders should engage champions across departments to build a scaffolding of support and should manage any emergent concerns about potential risks associated with new approaches and the effects that the new system could have on existing power dynamics.
  5. Actively engage all staff in results discussions: Organizations will benefit from involving a broad range of staff in results discussions, which are data quality boosters.
  6. Contribute to efforts beyond individual organizations: Lessons learned from HFM may also be applied to the macro-level, where organizations stand to collectively benefit from coordinated and frequent tracking of their progress.

‍