We’re back! We will have new articles for you starting next week.
In the meantime, we have reader responses to our previous pieces. We’d love to publish your reactions as well — please email letters@indevelopmentmag.com. Letters may be edited for clarity and to conform to house style.
In response to What South Sudanese Data Can Teach Us All, and Why We Should Care
Dear editors,
My name is Augustino (August) Ting Mayai. I am currently the Director General of the National Bureau of Statistics.
I am grateful for your thoughts, shared in your August 6 article. Hon. Margaret is certainly one of a kind, having experienced her outstanding dedication in 2007, when I paid the bureau a visit as a doctoral student at Wisconsin looking for population data for my dissertation. Margaret and fellow demographer Eliaba Damundu gave their all to ensure I had the data I needed to shine light on South Sudan’s health problems. Indeed, it is people like Margaret and Eliaba who make countries and organizations stand out.
I would also like to note that your article is particularly heartwarming. For about 10 years now, the world hasn’t presented much positivity regarding South Sudan. It is encouraging to see that not everyone has given up on us. It has personally energized me as I now prepare the country for its first census since independence.
Augustino (August) Ting Mayai
Director General of the National Bureau of Statistics, South Sudan
Dear editors,
I’m a bit sad and frustrated to be writing this response. I love the idea of this success story in South Sudan, but I was suspicious at the bold claims that when the author was in South Sudan in 2012, “the data” “was extremely high quality” and that “at first pass, at least, it seemed accurate.” These claims unfortunately seem straightforwardly false. When I looked at the data, it seemed to be of low quality.
Although the author doesn’t make clear what “the data” he discusses refers to, an earlier article which he quotes heavily makes it clear that Margaret was referring to her work on the 2008 census. For instance, he says in that article “the first census in 2008 involved 11,000 data collectors … how could Labanya know if her staff had collected accurate data, if there was nothing to compare it to?” If Dan was referring to other data besides the 2008 census, this should have been made clear.
Regarding the 2008 census:
- The article states that “People trusted the data they got; donors, government, NGOs seemed to report government data to me without the subtext of an eye roll.” This ignores a pretty significant incident. The government of South Sudan publicly rejected the census’ population count and claimed the real population figure was 50% higher than in the census.
- When I looked at the census data, there were three red flags that the data quality was poor:
- The Whipple index looks at how many people end their age with a 0 or a 5. Over 175 is the worst category here, and the 2008 census number was 178.4.
- The male to female ratio was 108 men to every 100 women. Given poor access to healthcare and an active conflict, this seems very unlikely.
- The 2008 census is incompatible with a 2021 population estimate. In 2021, the population was estimated at 12,444,018. For this to be consistent with the 2008 estimate, the population would have to grow 5% per year, given plausible assumptions about migration and mortality. This also seems unlikely.
Either one or both of these studies are inaccurate.
If this is the primary example of high-quality South Sudanese data, it seems to undermine the article’s point.
Furthermore, the specific impressive statistical methods praised happened well after the author was there in 2012. The High Frequency Survey studies took place from 2015 onwards, and were largely reliant on the 2008 census data (already flagged as inaccurate). The World Bank researchers who say the data was particularly accurate are also the ones who conducted the research.
Lastly, I do not think the positive framing reflects the failed institutions which typify South Sudan. Margaret Labanya stated herself that the South Sudanese government undermined the statistics department and effectively ran them into the ground. She says that “those with political power began to marginalize statistics.”
South Sudan is a failed state, with millions of refugees still outside the country. It is a completely savage dictatorship with some of the worst institutions in the world. Though the article is trying to frame things positively, it still feels a bit disingenuous to go a whole article about South Sudan without even mentioning this situation.
In a huge, extremely poor, low-trust country with failed institutions, sending thousands of poorly supervised, largely green enumerators around the country will not generate very good data, no matter how fantastic Margaret or other leaders might be. Indeed, given the situation, I only have more respect for Margaret Labanya.
She herself doesn’t make the author’s bold claims that the statistics are very good. She talks about doing everything possible to get the best stats possible. She’s also willing to criticize the South Sudanese government, which could well have cost her dearly.
The data she helped generate can be at the same time useful, inform to some extent programs and make a difference to people’s lives while still being pretty inaccurate. I think it is better to be honest about these realities rather than sugarcoat them.
Nick Laing
CEO of OneDayHealth
Response from Dan Honig:
I’m grateful to the critique’s author for taking my piece seriously enough to dig into the evidence behind it. On some factual and presentational matters we do disagree, which I’ll come to below. But I want to begin with what strikes me as the most important point in the response: one on which I think we agree almost completely. Margaret is remarkable.
Indeed, that claim is much closer to the central thrust of my article than the accuracy of South Sudan’s 2008 census.
My interest in this story began with something I observed while working in South Sudan in 2012. In a difficult operating environment, I had the experience (rather than systematic research/evidence, to be clear) of receiving government data that was useful, coherent and treated by the people around me — government officials, donors and NGOs — as sufficiently reliable to inform their work. That surprised me.
Years later, while researching “Mission Driven Bureaucrats,” I encountered Margaret Labanya and began to understand one important part of how a statistical agency in such an environment had been able to perform as well as it did. She is an exemplar of what I call a mission-driven bureaucrat: someone deeply motivated by the purpose of her work, who both exercises judgment herself and helps cultivate commitment among those she manages.
I’m uninterested in defending (and did not mean to assert) the claim that every statistic produced by the National Bureau of Statistics, including the 2008 census, must have been accurate. The claim I intended to make (and if this was ambiguous in the piece, that’s on me, and I appreciate the opportunity to clarify) is: Even in a state facing institutional challenges, there can be pockets of genuinely impressive bureaucratic performance. Understanding the people, management and organizational environments that make those pockets possible is an important — and underexplored — part of understanding state capacity. On this, I suspect the author of the critique and I are much closer to agreement.
The Census, High Frequency Surveys and South Sudanese Data
The critique argues that the performance I discuss could not have happened while I was in South Sudan in 2012, because the High Frequency Survey took place from 2015 onward. There were multiple generations of the High Frequency Survey, the first of which occurred before I was in South Sudan (they did not in fact begin in 2015). And, of course, there were other statistics agency products collecting data between the 2008 census and these surveys. It is simply not true that I observed the bureau in 2012 and then retroactively attributed techniques developed years later to what I had seen.
That said, I think the critique has identified a place where my writing could have been more precise. I cite, after discussing Margaret’s approach to building the organization, the High Frequency Survey as evidence. I intended the claim to describe the surprising capacity of the bureau and the quality of a data product that had undergone rigorous external investigation (which is of course relatively rare). I did not intend to certify every statistic the NBS had ever produced, or to seem to use the High Frequency evidence “for” the 2008 census or any other specific output of the South Sudanese National Bureau of Statistics. I could have been clearer that my general claim was that this was an agency producing generally strong data products.
Let’s turn to that 2008 census, to which the critique devotes substantial attention. Let me be explicit: The 2008 census is not what my description of South Sudanese data in the article principally rests upon. That is why I do not mention the 2008 census at any point in the piece. It is possible some of what I was exposed to in 2012 came from that census; but I don’t know that to be the case, nor do I think that my central claims in the article rest upon the answer to that question.
I agree that some of the statistical claims in the critique do seem to suggest flaws in the 2008 census, and I am happy to concede this. I don’t believe myself to have ever made a claim that contradicts this view, in my In Development piece or anywhere else.
And, as the critique rightly observes, Margaret herself speaks less in terms of statistical perfection than of the extraordinary effort required to generate the best information possible under extremely difficult circumstances.
That, to me, is the important takeaway when we look at South Sudan’s statistical products — including potentially the 2008 census. “Data quality” is not a binary in which a dataset is either perfectly accurate or worthless. That the data was useful for decision-making is the main point for informing how we should think about South Sudanese statistics and for what the lessons of Margaret’s time at the agency are for the world, rather than the technical details of a nearly 20-year-old census. The surprising accomplishment was not creating a perfect statistical system; it was building an institution capable of generating information useful enough that actors relied upon it under circumstances in which doing so was extraordinarily difficult.
The critique says that the positive framing of my article feels “disingenuous” given South Sudan’s wider record of state failure and suggests that in such a poor, low-trust country with weak institutions it is simply very difficult to imagine thousands of enumerators producing very good data. And it is true that South Sudan has faced — and continues to face — political, humanitarian and institutional difficulties.
But as I write in the piece, states are not unitary actors. “Juba,” “Washington” or “London” do not reflect just one thing. Within governments that perform badly overall, some organizations perform well. Within high-performing governments, some organizations perform badly. Within individual agencies, some teams flourish while others fail.
Much of my In Development piece is about evidence that the same basic phenomenon appears all over the world. Government performance varies enormously within countries and even within organizations. Where we find pockets of exceptional performance, we very often find committed public servants, effective managers, meaningful autonomy, strong peer relationships and a sense of connection to purpose.
That matters because much of the state capacity conversation starts somewhere else. We see weak systems and reach for more rules, more monitoring, more compliance systems to make sure mistakes cannot happen. Sometimes those things help. But people are not simply inputs into organizational machinery. Their motivation, judgment and relationships affect what states can and do do.
If we thought more seriously about empowered, mission-driven bureaucrats — and about the organizational conditions that attract them, sustain them and help others become more like them — I believe we could achieve much more in our efforts to strengthen state capacity in developing and developed countries alike.
In response to Why Economists Don’t Listen
Dear editors,
The post conflates two points. One is uncontroversial: Economists should use more kinds of data. Transcripts and open-ended interviews are evidence, and nothing in the credibility revolution excludes them. The Nick Bloom-John Van Reenen management agenda is built on open-ended interviews. Many of us write text-as-data papers.
The second point is ideological: that distance from subjects invalidates research, that respondents should become co-analysts, that separation is a mere “performance of objectivity.” This idea leads researchers astray. I want to be able to check your findings, not replace verifiability with sympathy. If you write a paper, I will not read it if I know you are in advance determined to tell a given story. I don’t care whom you sympathize with; your sympathy reduces verifiability, and the interest of your research. Yes, separation is an aspiration more than it is always possible. But the moment we give up, and we know what the abstract and title will say before we look at the data, then it is research itself that has become performative.
Luis Garicano
Professor of Public Policy at LSE
Dear editors,
I read Biju Rao’s piece on why economists don’t listen with a lot of interest. It captures well the kinds of things my advisors told me when I started my PhD. I’d spent two years in Vanuatu thinking about small island states, how many people a country really needs to run itself and how to get policymakers to engage with economic analyses. When I took some of these ideas to my supervisor, he said “these are interesting ideas … for a newspaper article. Not an economics journal. Your job is to publish in economics journals.” Another mentor reacted to my suggestions with “this is a good idea for sociology. But remember, we’re economists. Our comparative advantage is doing large-n, we have to leave small-n questions to other disciplines.”
So I have a lot of sympathy for Biju’s perspective on this subject. But his piece left me with more questions than answers:
- Should we have disciplinary boundaries at all? Is a field defined by its questions or by its methods? One could interpret Biju’s prescriptions as yet another form of economics imperialism. You could hear sociologists and anthropologists scream, “First you came for our questions, you wrote in your papers that yet-little-is-known-about-x when there is a LOT known about the subject in journals you don’t read, and now you’ll take our methods too?” I generally think comparative advantage is good, epistemic humility is good and we should find a way to reward collaborations between economists and practitioners of other social sciences rather than teach economists to do everything.
- Is this nature or nurture? I don’t think we can make mixed-methods masters of people who join economics programs. We know that qualitative work is not just about “listening to people.” It is about knowing enough to make sense of what you’re listening to, to form theories based on what you hear and don’t hear. This is a different skill from quantitative analysis and coding. There are other disciplines that are known for those skills, and the people who are interested in those skills often join those programs.
- How is cognitive empathy built? Jeremy Weinstein tells his students: You have to be proximate to the politics you wish to study. Somewhat less charitably, where you live has bearing on whose validation you seek and that then affects what you work on and what methods you use.
If I was passing judgement here, I could say that I hope more social scientists like Biju would move back to India over time, but I know life is not so simple. The Global North has far more capital, amenities and freedoms. To the extent that we need capital to do research, it is important to be legible to that capital, and so really the distance that matters is not the distance between economists and the people they study, but between owners of philanthropic capital and the people they want to benefit.
Karan Nagpal
Senior Director at IDinsight