The Observer Effect

"Efficiency is doing things right; effectiveness is doing the right things" - Peter Drucker

Drucker’s famous saying is usually intended to remind us that it’s not enough to do things right, you also have to do the right things. However, in social services, we often think that we are doing the right things, but many times we don’t measure ourselves in a way that enables us to know whether we have indeed achieved our objectives.

Let’s take, for example, interventions designed to help families lift themselves out of poverty. Before the program starts and during the program, we measure its short-term effects.  But in most cases the goal is for the outcomes of the program to be retained for several years after the intervention. The effectiveness of most intervention programs is only measured during or immediately afterwards, while the long-term impact is rarely examined.  Short term measurement is important to let us know if we are going in the right direction, but it is not enough.  Without long term measurement, we cannot know if the intervention really succeeded.  In the case of relieving poverty, the family’s income 3 years after the end of the program is a far more significant measure of success than immediately after the intervention, but as stated, we do not usually measure this.

Sometimes, measurement is done using data and metrics that are readily accessible to the organization.  We tend to choose existing metrics and available data when we define how the organization’s success is to be measured.  This process is efficient but not "the right thing" (now we can read Drucker’s quote in his original intention).  It results in measuring products or outcomes that are not necessarily the main goals of the program.  A correct definition of metrics is the result of a process in which we first define how we perceive the success of the program (even though it may not be measurable). Once we have defined the ‘winning picture’ of the intervention, we must think which metrics (including those that are not available) will enable us to decide whether it has been achieved.  Sometimes we may discover that we have to develop these metrics, collect the data, and invest resources in the whole process.  But there is no other choice, if we really want to learn about the effectiveness of our activity – we have to invest the necessary resources.

Measuring impact is essential but may also lead to negative results.  I am not talking about failure to achieve a program’s objectives, but of the damage that may be caused by the measurement process itself.  One of the processes that measured quality in the health system produced an almost perfect result.  The program succeeded beyond all expectations.  However, after a qualitative examination of the process, it emerged that in order to comply with the defined metric, not only were other metrics neglected, but there was even some incorrect clinical activity that led to a decline in other aspects of performance that were not measured, and adversely affected the health of part of the population.

As the Jewish New Year (Rosh Hashana) approaches, we must stop for a moment and assess our measurement processes (in our professional lives and in our personal lives) and ask ourselves the following questions:

1. Do our measurements cover the long-term, sustainable success of the program or just the outcomes during and immediately after the intervention?
2. Have we defined what we want to achieve? Do our metrics measure these desired results?
3. Are we aware of the negative outcomes of measurement? Which areas are we neglecting or sacrificing in order to achieve the desired results?

I invite myself and you to take the few months between Rosh Hashana and the end of 2023, as we prepare to set our targets in the work plan for 2024, to assess our measurement processes and think about how we can achieve better measurement and assessment of our impact.

Get to know the author

Nir Kaidar (WSL 2022) is father to Beeri, Tamar, and Maayan, and married to Zohar. He lives in Jerusalem as part of a community that works mostly in the public sector. Between the years 2006-2019, he worked in the Ministry of Health in a variety of positions. In his last position, he was Head of the Administration for Strategic and Economic Planning. Since 2019, he has served as Head of the Administration for Strategy, Economic Planning, Research and Regulation. He is responsible for promoting the ministry's strategy, leading the ministry's policy planning, leading the ministry's activities with local authorities, digitizing the welfare system, and being responsible for all regulatory processes. He has a bachelor's degree in economics, philosophy and political science, and a master's degree in the Outstanding Public Policy Program, both from the Hebrew University. In addition, he has a master's degree in humanitarian affairs from the University of La Spianca in Rome.