Tricky indicators and how to deal with them

Melanie Uhlen

Indicator development can be quite a challenge and far too often, no M&E expert is involved in this process. Consequently, M&E staff often has to work with difficult indicators. Based on my experience as an M&E consultant, in this article I give some examples of tricky indicators and make suggestions on how to report and present their progress.

Ideally, an indicator should cover only one aspect. In that case, it is easy to display its progress in a chart, because there is a clear and unidimensional progress trajectory. For example, if the indicator says "Three national bills for safe working conditions in the manufacturing sector are passed", then as soon as one bill is passed, the indicator is at 33% progress, two bills mean 66% progress and three bills will be 100% progress. This is both easy to report and to communicate visually in a chart:

 

 

 

 

 

In practice, however, we often have to deal with indicators that describe more than one aspect, meaning there is an "and" explicit or hidden in there. If context allows, it is usually best to simply report all the details. But sometimes it may be required to summarize the progress in one single figure. How to do this? Read some examples and suggestions below.

Cumulative aspects

Example: A business-process map is developed for 4 services using a structured analysis mechanism.

So, the product we are counting is a finished business-process map. We aim for four of these. But there is a second condition, namely that they were developed using a structured analysis mechanism. First, we would need to define more in detail what that analysis mechanism entails. But then it is not really an additional aspect, but a necessary implicit pre-condition, meaning that we can only count a business-process map, IF it has been developed using that specified mechanism. Any business-process map developed in some other way, does not count. So the progress display is still unidimensional:

"One finished business-process map" AND "It being developed using the structured mechanism" = 25% progress

 

 

 

 

 

Additive aspects

Example: 80% of the beneficiaries report that the improved cookstove uses less firewood and emits less smoke.

In this case, we face the challenge that it is very well possible that in the survey we conduct a beneficiary reports that the stove does emit less smoke, but still uses the same amount of firewood as the regular stove. Now we have two options:

  1. Only consider the conditions fulfilled, if a beneficiary reported both improvements. So any beneficiary reporting only one of the two improvements will not be counted for this indicator.
  2. Report the average. Assume 50% of beneficiaries report that the cookstove uses less firewood, but 90% report that it emits less smoke. Then the overall progress is (50% + 90%) / 2 = 70%

 

 

 

 

Multiple target values

Example: Of the 1000 young people who have participated in the job trainings, 50% are in a qualification-related, dependent or independent employment within 8 months after the end of the trainings.

Here, the problem is that we have two target values: On the one hand we have the absolute number of planned beneficiaries (1000) and on the other hand, the share of them that will find employment after completing the job trainings (50%). So it is very well possible that we may achieve a 50% of employment for the beneficiaries, but overall only reach 600 instead of 1000 youngsters. How, then, do we report and visualize progress for this indicator? Either we report both numbers, or:

  1. If we are obliged to decide on one main progress number, we could ask the donor what the more important aspect is, the absolute number of young people reached with the job trainings or the share of them finding employment. If the share of people is more relevant, we can report for example that 55% found employment and then add a note that we failed to reach the anticipated total number of people.
  2. We can merge the progress values of both targets into one by taking the average. E.g.

    1. We reached only 600 instead of 1000 (600 / 1000 = 60% progress)
    2. Their employment rate was 55% (55% / 50% *100 = 110% progress)

--> this gives us an average progress of (110%+60%)/2 = 85% progress

 

 

 

 

Sub-groups

Example: The satisfaction of users (of whom 20% are women and 10% are from vulnerable groups) of public services has increased by 30%.

Assuming that we have clearly defined how exactly user satisfaction is measured and how 30% improvement is calculated, we will still have the challenge of having one main figure to report (of all users), as well as the two numbers of the two sub-groups. Now we can either:

  1. Report the main figure only.
  2. Take the average of the three figures. The tricky question is: how exactly? Just dividing the three progress numbers by three? This would mean the main figure and the sub-groups are equally important. However, the main figure is probably more important than the two sub-groups. So we could introduce a weighing factor: The main figure would weigh twice as much as each of the subgroup. So the progress figure would be calculated like this: ((progress all users * 2) + progress women + progress vulnerable groups)/4

 

 

 

 

What do you think?

Have you encountered other tricky indicators? How did you deal with them? Please share your examples and experience, I would be curious to learn about them!