measures for democratic outcomes
Answers
One of the great challenges for policymakers is taking abstract concepts like “power” or “democracy” and using them to measure concrete policies. Each year, for example, the United States spends several billion dollars on democracy promotion. It would be great to know – not just for government officials, but for all of us – whether this money actually helps to nudge countries toward democracy.
The problem is figuring out what we mean by democracy. Somewhere in the vast space between Norway and North Korea is a gray zone composed of countries that are neither full democracies nor full dictatorships, and sometimes it can be extremely hard to measure the quality of their government.
A story of two post-Soviet states can illustrate the difficulty. Look at the graph below – the two counties started at about the same level of democracy after the Soviet collapse. But one saw the rise of crony capitalism, growing restrictions on media freedom, and the increasing centralization of presidential power. The other cut down on corruption, established economic ties with the West, and had a parliament that successfully stood up to the president’s unpopular choice of a prime minister. By the turn of the century, as the graph shows, their levels of democracy had completely diverged:
Two post-Soviet nations’ differing levels of democracy. (Data: Polity IV, Freedom House. Figure: Seva Gunitsky)
Any guesses for what these countries might be? Perhaps one of the Baltics for the “good” case, or Belarus for the “bad” case?
I’ll save you the suspense. What the graph actually shows are two measures of democracy for the same country: Russia. The solid line is a measure of Russian democracy by a commonly used index called Polity IV. The second, decidedly less optimistic scenario is the same measure done by Freedom House, another popular and well-known index of democracy.
Both Polity and Freedom House are commonly used indices that claim to measure the same phenomenon. (In Google Scholar, they are mentioned about 15,000 and 47,000 times, respectively). So how did they come away with such vastly different portrayals of Russian democracy? In both cases, plausible narratives could be constructed to explain the outcome – as I did just two paragraphs ago – but which one is closer to reality?
Anyone who has ever used a measure of democracy in their research should be very frightened by this graph. And anyone who has tried to figure out what makes countries more or less democratic should be worried. How can we begin to answer basic questions about the causes of democracy if we cannot even agree on what democracy is?
The problem is that democracy means different things to different people. Polity really cares about constraints on the elites – how much the president is checked by parliament, for example. In 1998, the Russian Duma rejected Yeltsin’s nomination of Viktor Chernomyrdin as prime minister (the unpopular Chernomyrdin received only 94 votes out of a possible 450). The rejection signaled legislative independence, and since executive constraint is the most important sub-component of Polity, the supposed increase in the power of the Duma led to a significant increase in the country’s score.
Freedom House, on the other hand, cares much more about individual rights and personal freedoms. By that measure, it sees Russia as a pretty bad place. In fact, it ranks the country on a par with places that allow the death penalty for women who commit adultery. As a result, it has been accused of having an anti-Russian bias – a charge the organization has consistently denied.
Disagreements like these are less about proper measurement and more about a philosophical debate on what counts as a democracy – an inherently complex and contested idea. This is a problem, because it suggests that there are no easy solutions. The Economist, for example, produces a measure of democracy that sees mandatory voting as bad for democracy because it infringes on individual rights. If I want to stay home and watch football on Election Day, that should be my right, goes the argument.