Download Criteria for Good Theories: Comparative Analysis of Formal Theory, Case Studies, and Stats and more Study notes Performance Evaluation in PDF only on Docsity!
Chapter 3 Criteria for Evaluating Causal Theories
Much of our knowledge about politics is factual or descriptive. It requires useful concepts and valid, reliable measurement, as discussed in chapter 2. Often, it goes beyond describing static, unchanging situations and constructs narrative accounts of specific events. These accounts can be richly detailed, amounting to histories of events and processes that enable us to feel that we understand what happened and why. Much of our understanding of the birth and death of democracies consists of this kind of knowledge. If one wants to understand how democracy evolved in Great Britain, there are hundreds of books that recount the process in exhaustive detail. Most other countries are not quite as well studied, but there are still dozens of books on democratization in Brazil and Japan and even a healthy number of books or articles on relatively neglected cases such as Paraguay, Botswana, Sri Lanka, and Mongolia. As I argue below, the knowledge contained in these works is essential and extremely valuable; it is a great scholarly achievement that has been tragically undervalued. This literature is so massive that it would be foolish to attempt to summarize it in this book. Instead, I focus here on attempts to develop a theoretical understanding of democratization, which is a quite different kind of knowledge. (Readers can judge for themselves whether it is a superior or inferior kind of knowedge.) Theoretical understanding is knowledge of the general laws that give rise to the specific events we observe. If we had perfect theoretical understanding, we could claim that the causes singled out in our explanations will have the same effects in cases that we have not yet observed, and would have the same effects even in hypothetical situations that we can never observe. We do not have this quality of understanding yet, and we may never have it. But the goal of political science, just like physics or biology or meteorology, is constantly to improve our theoretical
understanding and to learn useful things along the way. Practitioners of different approaches to comparative politics have jealously disputed one another’s claims to theoretical understanding. Area studies specialists have accused quantitative comparativists of either comparing the incomparable or quantifying the obvious. Statistical analysts have condescendingly thanked country experts for digging up the anecdotal evidence that only multi-country comparisons can can transmute into theory. Both complain about the lack of realism in rational-choice theory, yet formal theorists have tried to brand propositions in both large-sample and case-study approaches "atheoretical" because they are not integrated into a larger, systematic body of propositions. All these charges reflect a narrow view of theory. In reality, all three approaches make indispensable contributions to good theorizing. In this chapter I define three fundamental criteria for good theories, and I use these criteria to evaluate three approaches in comparative politics–formal theory, case studies and small-sample comparisons, and large-sample statistical analysis. My purpose in doing so is to advocate a broader view of theory, in which each approach has one unique strength and two weaknesses. From this broad perspective, our three main approaches can be seen as complementary. I illustrate the tradeoffs with examples from research on democratization, which has been studied so long and so variously that it affords examples of the strengths and weaknesses of every method. Three Central Criteria for Good Theory An overview of criteria for good theorizing provides a good foundation for a comparison of the advantages and disadvantages of different approaches. In a literature too vast to summarize here, scholars have defined more than a dozen criteria for good theory.^1 However, I contend that three criteria are especially central: generality, integration, and thickness.^2 They are central in the sense that each of the three major approaches in comparative politics achieves one at the expense of the
generalization is, at best, optional, and at worst, impossible and pernicious. This is false. Generality is an indispensable characteristic of theory. In the standard (nomological) view of what theories are, an explanation interprets an event or a tendency as a specific instance of universal laws. If the laws are not universal, then there is no solid foundation for the explanation; the explanation itself then requires explanation, and that explanation requires explanation, and so on. The phenomenon is not explained until it is understood as a necessary consequence of laws recognized as universally true. Because true universals are unattainable in social science today, practicality forces us to confine our theories to bounded times and places. We must take care not to bound our theories arbitrarily, that is, for no good theoretical reason. But as long as we can presume that there is potentially a good theoretical reason for limiting a theory to, say, postwar Europe, even if the reason is vague or implicit, then we can treat the theory as provisionally valid, pending completion (and empirical confirmation). All actual theories in comparative politics are therefore incomplete and provisional. The admission that they are works in progress is not damning, because this is all that one can claim about any scientific theory. Even physicists are still searching for a Theory of Everything, which could overturn the insights of Einstein and quantum mechanics. It is probably much harder to generalize in comparative politics than it is in physics, but we still have an obligation to generalize, for generalization is a defining characteristic of theory.^5 The notion of a truly general theory of democratization in units as diverse as countries is difficult to imagine. Even those who dare to generalize about a region typically take care to disclaim any application beyond that region. After all, it is very likely that the nature and causes of democratization were fundamentally different in 19th-Century Western Europe than they were in Latin America in the 1980s. In Europe, democratization required extending the suffrage to all adults
in elections that were already competitive; in Latin America, it required restoring electoral competition to countries that had usually already practiced universal adult suffrage. These were fundamentally different processes involving different sets of actors, interests, and goals. How could one general theory fit both processes? The answer is that it is fine to explain these different processes in different ways as long as our general theory includes a reason for knowing where to apply which part of the theory. A general theory might state, for example, that the European variant of the theory applies only to the first countries to develop mass liberal democracy, while a different version of the theory applies to the latecomers, because pro-democracy elites in the latecomers benefited from the example of the first democracies. Without such a reason, a theory would be arbitrarily bounded. It would state, in effect, that the theory applies everywhere except where it doesn’t, or that it applies only in Western Europe because we say so. The justification for bounding a theory does not have to be fully developed, because general theory is a work in progress; but scholars have an obligation at least to recognize this issue and to suggest possible reasons for circumscribing the relevance of their theories to certain times and places. Integration Generalization to the entirety of observed reality is not enough. In order to explain, we must also generalize to what is unobserved and hypothetical.^6 As Donald Moon wrote, The nomological pattern of explanation, as its name implies, requires the presence of general laws in any explanatory account.... But not just any kind of general statement can perform this explanatory function. Not only must laws be unrestricted universals (i.e., they must be in universal form and must apply to an unrestricted class of objects), but they must also support "counter-to-fact" and subjunctive conditional statements.... But to make such an assertion requires a great deal more information than that conveyed by a particular law, and
John Gerring notes, "The proposition that sits by itself in a corner is likely to be dismissed as 'ad hoc,' or 'idiosyncratic.' It does not fit with our present understanding of the world. It refuses to cumulate."^8 The more ways in which a proposition meshes with other propositions, the richer our understanding becomes. Second, consistent and complete theories are more “fertile”: they generate a larger number of observable implications. This results in part from the completeness of the theory, but it is also a by-product of the number of propositions that are integrated together. The greater the number of propositions that are linked together, the more hypotheses they can generate. One can derive more theorems from fifty axioms than from three. As a result, complete, well-integrated theories have many observable implications and are therefore potentially more testable.^9 The basis for the systematic structure of a theory is often logic, but it can be other branches of mathematics as well, such as calculus, game theory, or probability theory. I believe that it can also be, and typically and unavoidably is, common sense: our own informal understandings of how the world works. No elaborate proof is needed to show that money talks, that united organizations are stronger than divided ones, or that you can fool some of the people some of the time. These understandings of how the world works are less foolproof than mathematical or logical tools, but consciously or not, we rely on them all the time. For example, if a general calls for the overthrow of an elected president, we may not know exactly what will happen, but the range of possible consequences is actually quite small. The general may be forced to retire or sent overseas, other officers may rally around him, the U.S. ambassador will weigh in or one side or another, and so on; but we know the consequences will not include the discovery of a platinum mine, a major earthquake, or the outbreak of world peace and harmony. Our common sense guides the translation of theoretical symbols into meaningful referents (interpretive theory) and informs and constrains the
range of possible causal connections (causal theory). In fact, few hypotheses in comparative politics have been derived purely from the formal assumptions of a theory. In almost all cases, at some point, researchers have had to draw on their common-sense knowledge of the political world to translate the logical implications of a theory into observable implications. Thickness Finally, theory should be thick. A “thick” theory is a thorough one, a theory that provides a complete explanation for the phenomenon in question. It is useful to think of theoretical thickness as having two dimensions: depth and breadth. A theory is “deep” if it traces the chain of causation far back from the eventual effect. Depth is desirable to avoid overly proximate “explanations,” which tend to be superficial or trivial. For example, Higley and Burton argued that “a disunified national elite... produces a series of unstable rgimes that tend to oscillate between authoritarian and democratic forms.. .” while “a consensually unified national elite... produces a stable regime that may evolve into a modern democracy... .”^10 Although their argument fit their cases well, the authors never explained why a country’s elite is divided or consensually unified. The cause is suspiciously close to the effect, so the explanation is unsatisfying. It avoids the more interesting, and more difficult, question of what causes elite unity or disunity. A deeper explanation that took us farther back along the causal chain would be more useful and satisfying. The breadth or complexity of a theory concerns the number of parameters it includes and the degree of interconnection among them. Every theoretical model in social science has five parameters. First, every model pertains to a certain level of analysis--individual, group, national, world-systemic, or some intermediate gradation. Second, it has one or more dependent variables. Third, it has one or more explanatory variables. Fourth, it applies to a certain relevant universe of cases. And fifth, it applies to events or processes that take place during a certain period of time. We
number of recent studies have examined the democratic diffusion hypothesis that conditions in other countries influence democratization.^13 First-order complexity is common. Second-order complexity involves causal relationships between two different parameters. All hypotheses about an independent variable causing democracy (or democracy causing something else) are of this order; but so are various complications that could be introduced into a model. If the meaning of democracy varies over time or the best way to operationalize an independent variable depends on the world region, then one is dealing with this degree of complexity. Third-order complexity comes into play when there are plausible hypotheses relating three parameters. Most common among these are hypotheses that the relationship between the dependent variables and an independent variable is partly a function of time or place. A good example is the hypothesis that the impact of economic development on democratization depends on a country's world-system position.^14 With fourth-order complexity, a causal relationship could be a function of both time and place (or level of analysis). This may sound far-fetched, but in small-sample comparisons such relationships are fairly commonly asserted--for example, the notion that increasing wealth has not favored democracy in the Arab oil-producing states since the Second World War;^15 or the claim that the U.S. has become more sincerely interested in promoting democracy in the Caribbean Basin since the end of the Cold War.^16 Increasing complexity does not render a theory more esoteric; on the contrary, it is only by increasing complexity that a theory begins to approximate common sense. Orders of complexity can increase only so far. Eventually, one arrives at the extremely inelegant "saturated" model that explains each outcome perfectly by providing different and unique explanations for each case. Laypersons who have not been socialized into social science know that the saturated model is the truth: every country is unique, history never repeats itself exactly, and every event is the product of a long and densely tangled chain of causation stretching back to the
beginning of time. We political scientists know on some level that a true and complete explanation for the things that fascinate us would be impossibly complex. But we willfully ignore this disturbing fact and persist in our research. We are a community of eccentrics who share the delusion that politics is simpler than it appears. Although I would be as delighted as any other political scientist to discover simple, elegant, and powerful explanations, I think the common sense of the layperson is correct: we must presume that politics is extremely complex, and the burden of proof rests on those who claim that it is not. Guided by our own experience in the world, we should presume that most of these complex possibilities could be true and that only a complex theory can explain politics well. Unfortunately, this is a controversial position. Most influential works on the methodology of comparative politics emphasize the fact that all models necessarily simplify reality, and these texts usually exalt parsimony as a methological virtue. However, parsimony is frequently misunderstood. It is not a rule that we should always prefer the simpler of two theories; properly understood, it is a rule that if two theories explain a phenomenon equally well , we should prefer the simpler one.^17 In my experience in comparative politics, simplifications almost always sacrifice some accuracy. I see no reason to prefer a simple but less accurate theory over a complex but more accurate one. Of course, theoretical thickness does not guarantee a grasp of the truth; any creative person could dream up ten complex theories that are wrong for every one that is right. But very few of the right theories are likely to be simple. We should not let a misguided preference for parsimony blind us to the truth. We have to consider complex theories; the trick is to find the right ones. This is the role of testing, which will be discussed in chapter 7. Multiple Paths to Theory All approaches in comparative politics are deficient in satisfying some requirements for
are integrated into this intuitive interpretive theory. The bulk of democratization research consists of case studies and small-sample (usually within-region) comparisons. Every transition to democracy in the past two decades has been thoroughly analyzed in several books and numerous articles. Some of the most influential books in the field have been compendia of case studies.^19 Scholars seeking a thorough knowledge of a particular transition, breakdown, or regime survival are practically cursed with a superabundance of information. Often such studies prefer specific concepts to general ones: Duma to parliament, Clinton to president, Chiapas to province; but such precision reflects not a simple interpretive theory, but a more elaborate one that captures some of the idiosyncracies of each case. What is striking at this level is that we collectively know so much and disagree so little. Research of this type has created what is probably the most thorough understanding of specific democratic transitions, breakdown, and survival, and has done so for practically every country one could mention. These works, whether they are academic research in a British or anthropological tradition, current history, or journalistic analyses, do an excellent, often unsurpasssed, job of recounting events, identifying key actors and their motives, asssessing the strength of organizations, and tracing connections among causal forces. The authority of this work is such that we rarely argue about who the key players were, what the basic chronology was, or who won and who lost. Ironically, the lack of controversy about these inferences diminishes the prestige of the scholars who make them. But the high degree of consensus around their work makes their accomplishment more impressive, not less so. All theories should be as convincing as these. But these studies are just one pole of a continuum in small-sample research. At the opposite pole, some small-sample studies interpret every specific actor, institution, trend, and situation as a specific instance of a general type. They take literally Przeworski and Teune's call to "replace proper
names of social systems” with “the relevant variables."^20 The kind of theory generated by this type of research tends to have two characteristics. First, most of it is qualitative and categorical. The causal relationships it identifies link types to types and kinds to kinds rather than matching quantities or degrees. Relationships are hypothesized to be true or false, necessary or sufficient, rather than partially true, stronger or weaker, likely or iffy. This qualitative bent does not make this style of theorizing inferior; rather, it is merely different from a mathematical style. Second, the theoretical propositions that emerge from these studies, if examined with care, turn out to possess a high order of complexity. The more faithfully a theory represents our complex world, the more complex it must be. ( How faithfully, of course, is a question to be resolved by testing.) In the Latin American democratization literature, the conventional wisdom presumes that each wave of democratization is different, that each country has derived different lessons from its distinct political and economic history; that corporate actors vary greatly in power and tactics from country to country, and that both individual politicians and international actors can have a decisive impact on the outcome. This is the stuff of thick theory, and comparative politics as a whole benefits when a regional specialization generates such rich possibilities. For these two reasons, case and area studies have made many of the best-known and most original contributions to comparative political theory. Dependency theory germinated in a study of Argentina's terms of trade.^21 Consociational democracy was inspired by Lijphart’s Dutch origins.^22 The debate about the impact of parliamentary and presidential constitutions began as an effort to understand the fall of the Weimar Republic and its renewal was inspired by the success of the Spanish transition.^23 The hypotheses generated by this literature have reflected high-order, complex theorizing.^24 Daniel Lerner’s seminal work on modernization was a case study of Turkey that identified parallel
Nevertheless, the small-sample approach has two weaknesses. First, although its propositions are integrated with theory, they are integrated more loosely. By "loosely," I mean that such propositions are not derived from other propositions according to any strict logic. Rather, they are borrowed from other theories and taken out of their original theoretical context or generated by observation, induction, and intuition. Loose integration has two consequences. One is that the facts can be used to support an embarrassing variety of theories. This happens because the question,"What is this a case of?" has many possible answers. The leap from specific to general can go in many different directions. What, for example, was Venezuela in 1989 a case of? Every theoretical framework suggests a different direction. To a progressive political economist, it was an oil-dependent economy;^31 to an institutionalist, it was a presidential partyarchy;^32 to a liberal political economist, a case of delayed structural adjustment;^33 to a student of labor, it was a corporatist system;^34 to a cultural theorist, a nation with unrealistic trust in a "magical state.”^35 In reality, all of these labels may have been accurate. The point is that moving from the specific to the general forces us to describe our cases more selectively, and we make our selections so as to integrate the case into a larger body of theory. The second consequence of loose theoretical integration is that it is less clear which tests would confirm or disconfirm the theory. Without rigorous logic or mathematical tools to generate hypotheses, there is no straightforward way to derive necessary implications: what must be true if the theory is true. In contrast to formal theory, the theories of small-sample analysis are less clear about their assumptions; they rely more on the tacit assumptions of common sense, which leads to conditional and vaguely probabilistic predictions, which are hard to falsify. The second weakness of small-sample theories is that they are, by definition, not general. These propositions (when they are explicitly integrated into a theory) merely assert generality;
whether such assertions are empirically valid or not is a matter for large-sample testing to decide. Until the testing takes place, these are only general hypotheses, not generally confirmed theory. Replacing proper names with variables is indeed our goal, but generalizing is far harder than affixing general labels to particulars. It is one thing to call the United States a presidential democracy, but quite another to assert that what one observes in the United States is true of presidential democracies in general. The former is a description of one case; the latter is an inference about a population (all presidential democracies) from one case (the United States), which is not justified. To summarize, case studies and small-sample comparisons yield a type of theory that is qualitatively thick and empirically well-grounded, and therefore plausible in bounded times and places; but also provisional, pending extension to more general samples; and often ambiguous in its theoretical implications, and therefore difficult to test decisively, especially beyond its original boundaries. It is, to caricature a bit, a soft theory built on a hard foundation. Brian Downing’s The Military Revolution and Political Change is a a small-sample comparison that illustrates this tradeoff well. As we have come to expect of comparative historical analysis, it is admirably thick and well-grounded. Downing delves deeply into the resurrection of Roman Law in medieval consitutionalism, the rights of towns in the Holy Roman Empire, the terms of feudal levies, the advantages of Swiss pikemen over mounted knights, the powers of the estates of Brandenburg and Pomerania, 17th-Century French tax farming, the consensus voting rule in the Polish Seym, the financing of English wars, and the logistics of Swedish armies. The specificity of his arguments, however, forces him to refrain from generalizing beyond Europe: To say that European social, political, and economic history is markedly different from that of the rest of the world is to say nothing new.... Three principal conditions in medieval Europe provided a predisposition to democracy: a rough balance between crown and
tools used in hypothesis testing encourage, and sometimes require, conversion of theories from a qualitative logic to a quantitative logic. Theories become less about kinds and types and true/false or necessary/sufficient relations and more about the magnitudes of impacts, partial impacts, probabilities, and curvilinear relationships. These relationships are difficult to handle in a qualitative idiom. The reverse is not true, fortunately. Statistical analysis can also handle the kinds of relationships found in qualitative theories, such as conditional relations, necessary or sufficient conditions, and the direction of causation. Examples of distinctly quantitative theory abound in democratization research. The qualitative hypothesis that wealthy countries tend to be democracies has been converted into a rococo variety of quantitative hypotheses:
- the wealthier the country is, the more democratic it is;
- the wealthier the country is, the more democratic it is, but to a logarithmically diminishing degree;
- the wealthier the country is, the more democratic it is, but with logarithmically diminishing increases and a dip at an intermediate level of wealth (the “N-curve” hypothesis);
- the wealthier the country is, the more democratic it is, except when economic growth temporarily worsens inequality, which undermines democracy;
- the wealthier the country is, the more democratic it is, although the impact of wealth is mediated by state size, which has an “inverted U” relationship with democracy;
- increasing wealth does not make countries become more democratic but improves the life expectancy of any regime; and so on. Another line of research has begun to explore the notion of democratic diffusion. Although
Rustow and Huntington wrote about various possible types of transnational influences on democratization, quantitative scholars have found that “democratic diffusion” can refer to a tremendous variety of causal paths.^37 In the course of testing for them, they have had to refine the theory in order to distinguish among neighbor effects, regional effects, and superpower effects; impacts on the probability of change, change vs. stasis, the direction of change, and the magnitude of change; and change influenced by ideas, trade, investment, population movement, military pressure, and national reputations, many of which were not contemplated in smaller-sample or qualitative research. The principal advantage of the kind of theory that emerges from large-sample work is that it is relatively general, both in its aspirations and in its empirical grounding. The degree to which it is general varies depending on the coverage of the universe by the sample, of course, but it is by definition more general than small-sample research. Formal theory makes universalistic assumptions, which are even more general, but large-sample research has the advantage of making at least some assumptions that are guaranteed to have empirical support. (The assumptions of statistical analysis are rarely fully supported, such as the assumption of normally distributed random errors. I will discuss the consequences of this problem in later chapters on testing.) For example, the most consistent finding in the large-sample statistical literature is that democracy is associated with high levels of economic development. The association is a rough one, not strong enough to predict small differences in demoracy or differences between any two cases with a high degree of certainty; but it remains a very general statement. The two weaknesses of large-sample comparisons are thinness and loose theoretical integration. A "thin" proposition is a simple statement that assumes very little about the world and identifies an association between just two narrowly conceived phenomena, such as democracy and
