Building on the success of our last assignment for UNFPA, Scalway consulting will be helping UNFPA develop an Africa-wide strategy for adolescents and young people. It is a challenging assignment in many ways. It involves working with radically different kinds of countries, and trying to develop a framework for consensus and shared vision across many cultural and political divides. A key element of the work with involve moving beyond the kinds of general, visionary language that we often find in the United Nations, to come up with a measurable, focused and effective strategy.

My ambition in this  job is to give UNFPA great strategy tools and materials so that the staff from different offices can shape something that really responds to their local circumstances. As with most of my work these days, the emphasis will be on creating a results-based strategy.

The evaluation lends itself to a qualitative research design. A rapid appraisal (RA) methodology will be employed to ensure quality systematic research at a low cost. We will undertake country visits to three of the five Southern African countries in which SAT is fully operational.

The evaluation will concentrate on looking at the performance of SAT against its own progress indicators. A special emphasis will be placed on looking at the added value of SAT’s contribution as a regional entity.

The collaborating partners of this initiative are the World Bank, UNAIDS, the World Health Organization, the Bill and Melinda Gates Foundation, the Global Fund to Fight AIDS, Tuberculosis and Malaria, and PEPFAR.

The Champions for an HIV Free Generation is undertaking an evaluation and strategy development process. The first stage of this undertaking involved a brief assessment of the CHFG’s institutional mechanisms, its strategy, governance, and operations. I have just completed this phase and created a document that makes some observations of how the organisation operates, the challenges it will face in the future, and the key considerations for future strategy.

This process was based on the following methods:

• Semi-structured interviews with a range of stakeholders
• A three hour SWOT Analysis with secretariat staff
• A two hour Stakeholder Analysis Session (involving an adapted Porter’s framework approach)
• A series of informal meetings with staff to discuss the different processes and documents that shape the work of the organisation
• Direct observation of the work of the organisation, through four weeks of sharing office space
• A desk review of literature and documentation relating to the work of the organisation
• Inputs taken from another recent evaluation

As I will be based mainly in Botswana during this period, please use my local cell: +267 72231839

1. The official UN site for the MDG indicators have a breakdown for family planning here.
2. Measure Evaluation have a Handbook of Indicators for Family Planning Program Evaluation here.
3. Programme Manager’s Planning Monitoring & Evaluation Toolkit from UNFPA  here.
4. Nice generic example: A results based model for primary health care (in Canada) here.
5. The obvious reference in this area: UNFPA, State of the World Population, 2010 here.
6. Looking Back, Moving Forward Results and recommendations from the ICPD-at-15 process, from UNFPA, here.
7. USAID: Health and Family Planning, Measuring Sustainability, here.
8. Results Accountability Matrix from UNAIDS, contains info on reproductive health here.
9. 1996 Framework of Selected Indicators from UNFPA, here.
10. A few ideas on family planning indicators in emergency situations from UNFPA here.

by Thomas Scalway, Strategy and Evaluation Consultant, July 2011

I am currently developing a regional strategy for adolescent reproductive health for one of my UN clients. This strategy, like many of the others I have been doing recently, will have a results-based format. In this blog I thought I’d share a few notes on this move towards results-based approaches.

Getting uptodate on the literature, its clear the debates between results-based vs process-based development have been going on for about as long as international development itself. An online essay, “Aid on the edge of reason” rehearses some of the arguments and offers many relevant insights.  On a lighter note, the New York Times published a piece entitled, “Are metrics blinding our perception.” Another blog “Worth the Risk” by Leni Wild talks about the dangers and benefits of results-based aid, and from there a number of other links can be found. More insights in this vein can be found in this rich source of linkages on RBM by Greg Armstrong.

Boiling some of the debates down with a relevance to public health strategy, it seems that results-based approaches have a number of risks, including:

• cultivating a reductionist model of public health, which by definition has to be holistic
• supporting indelicate and potentially damaging outcome-related actions within sensitive complex institutional and cultural processes
• focusing on the short term, rather than the bigger picture.

However, many seem to agree that results based approaches are useful if they encourage efficiency, focus and an emphasis on impact. Most people in public health that I know worry about the waste of resources associated with seemingly endless meetings, report writing, conferences and travel that forms the background to much of our work. My own sense, for what its worth, is that being more orientated towards results will help us feel better about what we do. More importantly, when done well, results-based strategies for public health will maybe make a measurable dent in some of the health problems we work with.

The pervasiveness of results-based approaches within public health.

Within the UN, the focus on results-based approaches stretches back at least to the 90’s.

According to the “Programme for Reform”, presented by the UN Secretary General to the UN General Assembly in 1997, it was proposed that the UN “place greater emphasis on results in its planning, budgeting and reporting and that the General Assembly moves the budget of the United Nations from input accounting to accountability for results… shifting the focus of planning, budgeting, reporting and oversight from how things are done to what is accomplished”. Click here for more.

Since then, the notion of results and “Results-Based Management” (RBM) has apparently become a central aspect within the UN system and is shaping a global trend among international organisations. Putting financial incentives behind results-based approaches, “Cash on Delivery” is gaining purchase among the world’s big vertical health funds, such as Global Alliance on Vaccinations and Immunisations and the Global Fund (GFATM).  This article from the UK Guardian looks at the trend in more detail.

A new book presented at the World Bank recently by the Center for Global Development proposes progress-based aid for education. It is a trend that has been growing in the World Bank for a while, as part of their quest for aid effectiveness, as outlined here. And not only there – according to the Centre for Global Development, the UK Department for international development is moving increasingly towards adopting results-based approaches to aid.  It will allocate future resources across country and regional programmes on the basis of results-based strategies. DFID spends annually almost 3 billion pounds, about $4.5 billion, on these programs – exclusive of its allocations for humanitarian assistance and for support of multilateral programs. Read more here.

Results-based approaches underlie most of the US government funded public health initiatives.  It is likely the results-based trend within US public health funding is set to continue. USAID’s new procurement processes, for example, are aligning the agency towards results, as discussed here.

Results based approaches are increasingly at the core of African development discourse. During the September General Assembly’s summit, October 2010, Ibrahim Assane Mayaki, Chief Executive Officer of The New Partnership for Africa’s Development (NEPAD), called for a results-based approach if African countries are to achieve the Millennium Development Goals (outlined in more detail, here).

Countries such as Rwanda and Burundi are adopting payment for performance models (an extension of RBM) within their domestic health systems. In developing the Lesotho strategy, we drew on other countries results-based examples, and were encouraged by the relevant national and regional UN agency offices to do so. In HIV, results-based approaches have become enscribed within multi-year national, regional and international strategies and frameworks for governments and major international organisations. That is already having a major knock-on effect for the way HIV programmes function, but the full impact is yet to be felt.

The writing is on the wall. Results placed approaches are becoming hard-wired within the public health machine. They are here to stay. This is going to challenge governments, major international organisations and the non-profit sector to make a number of difficult changes. If this brings greater emphasis on efficacy, and real results for public health, these changes will be well worth it.

The South Africa National AIDS Council (SANAC) has a number of Technical Task Teams. I was contracted by JHHESA (John Hopkins Health and Education South Africa) to evaluate the Communications Technical Task Team (CTTT).

This review was requested by the CTTT in response to a number of sensitive fundamental challenges that were perceived differently by various stakeholders. JHHESA consequently commissioned the review, with funding by USAID/PEPFAR.

The review was based on a large amount of information -much of it in grey literature that has only been seen by a few people. Other desk research information sources were the  minutes and other documentation developed by, or about, the task team.

Interviews were undertaken with a number of key stakeholders from the CTTT and other institutions. This involved talking to inspiring and interesting individuals who were at the head of the health communication sector.

HIV Communication in South Africa benefits from both NGOS and private sector organisations. Together they produce a number of excellent outputs, strong research, and a fair amount of information sharing. Their programming includes a blend of mass-media, community and inter-personal elements, following the best practice models recommended by UNAIDS and others. Some South African HIV Communication organisations have become internationally recognised, with their evaluation, creativity, research, design and implementation methods being emulated worldwide. Despite such examples, the overall HIV communication response in South Africa has many areas of considerable weakness. Central and foremost is the fact that there is no government-led communication

Furthermore, despite the high quality of South African HIV communication providers, they are receiving decreasing amounts of funding. The efficacy of social and behavioural change communication is now also questioned in the light of evidence produced by clinical trials, for example on PMTCT (Preventing Mother To Child Transmission), male circumcision and prevention among positives. Evaluation of communication programmes rarely produces impact level results that can be interpreted simply through randomised controlled trials, and this adds to its legitimacy being questioned within some scientific circles (a number of other documents explain how communication does have a measurable and cost-effective impact). The communication sector is being forced to intensify its efforts to collect and use strategic information for programme design, monitoring and evaluation, and so far it is only receiving minimal support for this.

The fact that influential people are questioning the value of SBCC combines with other factors, relating to the global economy, changing donor priorities and increasing HIV treatment costs, to contribute to an overall decline in funding and support for communication. Programmes are reducing in size, and have to compete increasingly aggressively for diminishing funding resources.

It is against this challenging background that the work of the CTTT needs to be understood. My evaluation report is confidential, but do contact me if you would like to discuss this work further.

The major communication programmers operating in public health in South Africa generate a large quantity of strategic information through research and surveys. Johns Hopkins Health and Education, South Africa, together with partners Soul City, loveLife and CMT commissioned me to assess this data in light of its potential for impact evaluation. While it was clear that the programmes could clearly show successes on a number of outcomes within a logic framework linking inputs to final impacts, it was also clear that more could be done to gather and analyse strategic information. I developed a paper that was shared internationally and that highlighted the major lessons that came out of the review.

Some insight into this work can be gained from looking at the Second National Communication Survey of 2009 which is the most recent and comprehensive study that can currently be drawn upon in investigating the impact of HIV communication in South Africa.[i] It used a sophisticated multi-stage sampling approach where 9728 participants between the ages of 16 and 55 years were interviewed using a structured questionnaire. Both uni-variate and multi-variate analytical methods were applied. Regression analysis was used to ensure that any reported changes could be attributed to the communication intervention and not to other interventions or associated variables that could have caused these.

It shows that communication programmes have had a positive and measurable effect on the knowledge, attitudes and beliefs that are contributory to HIV prevention. The survey shows that the exposure to communication programmes is also responsible for a number of positive behavioural changes, such as HIV testing and condom use. The survey has found some evidence that partner reduction messages, which are relatively new in the SA communications landscape, have an impact with increased levels of knowledge of multiple and concurrent partners and some reported behavioural changes amongst the intended target market.

More on this paper can be found on Comminit.com

http://comminit.com/africa/content/presenting-evidence-hiv-communication

Quality public health projects depend on the availability of individuals and appropriate M&E experience and skills. As a skilled evaluator, I can support your programmes M&E efforts.

Developing M&E frameworks is an integral step in the development of M&E plans. It is also the core of most strategy work and is imperative for fund-raising. I am available to support your work in developing conceptual, results and logic frameworks to help you and your partners:

• make decisions about project management and service delivery
• ensure effective and efficient use of resources and provide accountability to donors
• assess whether the project has achieved its objectives or has the desired effects
• learn from program activities and provide information to design future project

You can learn more about my evaluation work, here