One of the things that has always bothered me about Monitoring and Evaluation (M&E) of international Rule of Law programs has been the innumerable risks involving incorrect data collection and input, and the ever present danger of conflicts of interest, not to mention lack of experience and expertise. Add to all this the challenges of violence, a hostile host government, war, and other calamities, and the writing on the wall is quite clear.
Oversight of a project does not mean that M&E is actually being carried through. A while ago, I gathered some of my thoughts. I wonder how current they are. My review of reports and audits makes me think that not much has changed.
Historically, some U.S. Government agencies were late in understanding the importance of M&E to determine the impact that foreign assistance programs were having. In the last few years, I always kept hearing that we needed to answer then Secretary of State Hillary Clinton’s “so what?” question regarding how effective our international aid projects were.
Many multi-million dollar programs had no internal nor external M&E experts to provide guidance. In Afghanistan, for example, the U.S. Embassy’s 2013 rule of law strategy failed to incorporate any performance measures. (For an interesting report that reveals what the problems relating to M&E were at the time, I suggest you read the Special Inspector General for Afghanistan Reconstruction (SIGAR) audit).
Through evaluation tools, M&E programs aim to demonstrate program impact. This, in turn, provides feedback to guide program implementation staff to enhance future programming by identifying planned and unplanned results to allow donors, implementers and host country beneficiaries to understand what works and does not work, how to maximize efficiencies, and address any issues that might arise before they become a problem or a cataclysmic risk.
In government contracts, the Statement of Work (SOW) may provide the indicators to be used. Sometimes, the implementer may develop a series of iterative evaluations as well, which might include a training evaluation and an audit, a trainee-satisfaction survey, a mentoring plan, and -depending on the program- a public outreach component.
Performance indicators may combine the Foreign Assistance Framework Indicators (F-Indicators), as well as customized indicators, with the goal to develop and utilize indicators that measure outputs and impact in the short, medium and long-term of the project.
Of course, the most perfect and all-encompassing M&E plan will not work unless both donors, implementers and beneficiaries take into account the critical risks inherent in, or coming from, the place of performance, and agree on some critical assumptions that, at the very least, encompass three contexts: political, security, and operational.
What I have learnt is that decision-makers and bureaucrats from both the government side and the corporate side make choices and issue “diktats” without having had the benefit of operating in the environment where the program is being carried out. I never gave it much thought until I witnessed it first-hand. Therefore, it is imperative that the “experts” who are hired to handle M&E issues understand that they may be dealing with people who have little or no knowledge of the hurdles the technical staff face day in and day out.
Sometimes, the mere fact that electricity is not available or the internet connection does not work, may mean that M&E data cannot be incorporated into a database.
While I applaud the importance of M&E in program management, I see some problem areas:
Who monitors and evaluates the authenticity and the accuracy of the M&E plan and its implementation in-house? In other words, if I am the donor, would I fully trust the contractor or grantee to monitor and evaluate itself?
Through my own observations, I came to the conclusion that the Chief of Team of a project would have to have not only “Rule of Law” and international development experience, but a keen understanding of the host country – for example Afghanistan -, business development, human resources, management, accounting, psychology, history, security, culture, philosophy, and -most importantly- an understanding of how the State Department works. Who can meet all those requirements?
If the donor hires a third-party to do an indepent M&E of a program, how comfortable can the donor and implementer be that the third-party will do an unbiased and truly objective M&E assessment? What are the chances that the M&E firm will have a former implementer employee evaluating the very same program that person put in place?
Rule of Law programs are not immune from a myriad of conflicts of interest. Who pays attention to these things?
Dum Spiro Spero 