Project Everest

3 Principles for Measuring Impact (Pt. 2)

by
Riley Harris
Riley Harris | Jun 11, 2017 | in Ideas Box

 

If you’ve ever conducted quantitative research this might seem obvious. I’m not being condescending, this stuff is really really important and often gets overlooked. In any case, these are the most crucial things to keep in mind when evaluating impact:

A sceptical Prior

A natural way to think about gathering evidence for a program is to directly look for evidence that it works. However, assuming the program doesn’t have a positive outcome, then attempts to find enough evidence to disprove this hypothesis (equivalent to a null hypothesis that there will be no effect) is more effective. It ensures that the programs which have strong evidence showing they have a positive impact will be supported.

Measures the Right Thing(s)

Making sure that the right thing is being measured is extremely important, and an impact evaluation is really only useful if it’s measuring the right things. Evidence of the outcome of the project (e.g. increased hours of employment of disabled students) rather than information about the implementation of the project (e.g. number of students given access to the project). This is important as many organisations are measuring activities rather than the important results, which is fine, but the goal of the impact evaluation require that you measure the outcomes directly, rather than measuring activity levels in a program.

A Counterfactual

In order to really know the effect that a project has on the community, we need to know what would have happened if the project didn’t exist. For example, imagine if there was a project that sold textbooks to students, and then saw that test scores rose. Did the text books raise test scores? Possibly, but maybe the test scores would have risen anyway. The way to find out is measure some people who didn’t receive text books and see what happened to their test scores (Vivalt, 2012). This can be seen in the figure below (AidGrade, 2012): the first row shows the test scores improving, in the second row where there is a control group that doesn’t receive textbooks. The group that didn’t receive textbooks score higher; thus, the orange rectangle is the negative outcome of the project. Obviously, there is no way to both run a project and observe the community without the project at the same time, however there are ways to get some idea of what would have happened if the project didn’t intervene. Dividing a population into those that receive the treatment and a control group who does not receive the treatment is one way of achieving this. The counterfactual is essentially the best estimate of what would happen if people didn’t have access to the program, and in order to know the effect of a project we need a counterfactual to measure against.

 

 

Jessica Stephanie Arvela Jun 11, 2017

Measure progress of those who do not receive service

Users tagged:

Reply 1

Riley Harris Jul 3, 2017

Yes, exactly!

Reply 0