The world is suffering from multiple challenges that are setting back our collective push to achieve the sustainable development goals by 2030 as planned. Most significant today is the ongoing atrocities happening in Gaza. There’s no shortage of recommendations to jump-start progress.
But one key ingredient is missing: robust measurement of impact.
One of us can recall an encounter with an international NGO claiming they did research, which turned out to be a simple evaluation of trainees pre- and post-training. For a scientist, that is disappointing.
Large organizations may well have the resources for sophisticated monitoring and evaluation, or M&E, activities. But in most cases, measurement of program effectiveness is a simple M&E report produced by the same entity that runs the project, whether a social entrepreneur or large INGO.
Such an approach to measurement is problematic on two levels. One, it is not scientific and rigorous enough: simply not independent, let alone peer-reviewed or published in academic journals. Two — and here both academia and INGOs are culprits — it tends to adopt survey and self-reporting methodologies that fail to capture the complexity of human behavior, and are not rooted in the local Indigenous culture.
One example comes to mind, from a study of a local program on empowerment. The actual quantitative tool did not show impact. But when the research team tried to understand what the women were describing in their perception of what empowerment was about, we were able to discover the impact — that’s because it was rooted in the women’s narrative, and not in the survey, which was designed based on Western notions of how empowerment is expressed.
Measuring impact to achieve the SDGs
The tools many INGOs use to assess their impact are usually drawn from academia but not necessarily adapted to the local context, and not even challenged for authenticity of the local voice. And publishing does not — and should not — in itself lend authentic credibility of localization. This is a Western framework of science, and it is problematic in both academia and in development.
The bottom line of not having culturally rooted and robust-enough measurement of impact is that we don’t really know what works and what doesn’t when it comes to efforts to meet the Sustainable Development Goals. Nor is it inconsequential for national budgets, considering estimates that put the cost of achieving the SDGs at $5 trillion-$7 trillion per year globally between 2015 and 2030.
It’s a blind spot that was also spotted by the independent group of science advisers on the SDGs, who released the 2023 Global Sustainable Development Report, titled “Times of Crisis, Times of Change: Science for Accelerating Transformations to Sustainable Development,” during the SDG Summit last September. The text is peppered with examples of how “transformations” have been achieved in the past, an effort to help decision makers visualize the necessary changes. But you’d be hard pressed to find specific policies that have been evaluated and shown to work across the board. In part, that’s because this information is missing from the literature. There’s a need for more systematic evidence on what works best, and where.
Innovative tools of measurement
In INGO programming, rigorous M&E can go a long way to help with providing systematic evidence of what works. But there is also a need for a different way in assessing impact — and this begs for INGOs to create partnerships with academia on one hand and to come up with innovative, culturally rooted tools of measurement that capture the complexity of individual and community mindsets.
This is important to understand how impact is achieved and how to make it sustainable. And it can be achieved through collaboration between academics and local social entrepreneurs — because without the input from the local social entrepreneurs on the ground, the academic is working in isolation and not in touch with reality, which is one of the perennial challenges of academic work.
This, too, is stressed in the United Nations’ 2023 GSDR report, which argues that the actions needed to steer the world toward a sustainable pathway must be rooted in “socially robust science”— in short, science that’s relevant to society and effective for policymaking.
In practice, one way that the group of scientists say the science sector can evolve in that direction is to pursue implementation research in the form of “real world labs” at the local level — an idea not far from the joint evaluation work we propose.
Research funding for impact measurement
One key challenge in this new approach is how to fund the research for impact measurement — because funders of academic research are usually different from those who fund social entrepreneurs and INGOs.
Now, we are proposing a new model, where the social enterprise/NGO applies for grants from academic research funders in collaboration with local researchers. This will open up a new source of funding for local social enterprises/NGOs — which tend to struggle for funding when competing with INGOs — and foster collaboration with academia. On their side, funders will have higher, more sustainable local impact and more evidence-based programming for the projects they fund. It’s a win-win strategy.
We have seen that this approach works. Taghyeer, an NGO that Rana founded in Jordan in 2010, runs the flagship program We Love Reading, or WLR, designed in 2006. Taghyeer has been a partner in multiple research grants — in collaboration with universities locally, such as University of Jordan, regionally, such as New York University Abu Dhabi , and globally, with Yale University and Queen Mary University of London — to conduct research on the impact of the program on refugee children and adults, showing how WLR impacts social networks. This helped to understand how to develop better programs that have sustainable impact. Recently Taghyeer was a partner in a grant from the British academy, to study the impact of the WLR program under Taghyeer on Syrian refugees.
We envision that similar initiatives will encourage academic funders to include NGOs in their calls. But this can work the other way around too: if INGO funders do more to include academic partners in their calls for better measurement of impact. The humanitarian organization Elrha has recently begun to implement this approach by insisting that academia and NGOs apply together, as well as including local partners.
This strategy of INGOs partnering with academia to improve impact will result in collaboration across disciplines and across sectors, which is important in moving forward with efforts to achieve the SDGs in time.
Finally in order to have maximum impact we must disseminate research results in local language in simplified terms to raise awareness, influence policy and share the results with those who participated in the study.
Rana Dajani is currently a Richard Von Weizsacker Fellow, Robert Bosch Academy, Germany A former visiting Professor at MIT, Harvard and Cambridge, International Science Council Fellow and a Morse Yale Fellow.
Anita Makri is a freelance writer/journalist, producer, and editorial adviser specializing in global development and science in society, drawing from 25 years of experience across sectors. She is based in London and has produced stories from Africa, Latin America, and South Asia.
