Measuring What Matters – The Risks of Overreliance on a Single Story

Date

15 Nov 2023, 17:30 – 15 Nov 2023, 19:00

Location

Central European University, Auditorium, Vienna Campus, 1110 Vienna, Quellenstrasse 51

Presentation by Valentin Seidler (LBIDH, CEU) and Michael Woolcock (World Bank, Harvard University) and open discussion at the Policy Talks der Central European University (CEU)

The phrase “What gets measured gets managed; and what gets measured gets done” is a ubiquitous phrase in management. Responding effectively to the many challenges of public administration requires recognizing that access to more and better quantitative data is necessary, but an overreliance on quantitative data comes with risks, especially when reforms, policies or programs are highly complex. They typically entail numerous discretionary interactions between multiple agents over long periods of time, while the stakes are high and the specific solutions to solve problems are largely unknown ex ante.
Valentin Seidler dug deep into the most widely trusted demographic and health data available for sub-Saharan Africa. In the absence of reliable census and population data, many African countries and international agencies rely on household surveys for policymaking and development interventions. He found that we cannot blindly assume uniform data quality where it matters most – at the district and village level. Remote villages in particular suffer from increasingly unreliable data, with negative consequences for planning reforms and aid efforts for these populations. (Link to study)
Michael Woolcock’s study focuses on four data related risks and offers four cross-cutting principles for building an approach to the use of quantitative data—a “balanced data suite”—that strengthens problem-solving and learning in public administration. The four principles are: 1) identify and manage the organizational capacity and power relations that shape data management, 2) focus quantitative measures of success on those aspects which are close to the problem, 3) embrace a role for qualitative data, especially for those aspects that require in-depth, context-specific knowledge, and 4) protect space for judgment, discretion, and deliberation in those decision-making domains that inherently cannot be quantified. Practical examples are provided of complex initiatives, and the unwarranted conclusions that can be drawn from assessments of them, when only a single source of data is considered.

Datum: November 15 2023 5:30 pm – 7:00 pm
Location: