Download Correlational Design, Ex Post and more Schemes and Mind Maps Design of Wood Structures in PDF only on Docsity!
Chapter 15. Nonexperimental Research Designs: Correlational Design, Ex Post
Facto Design, Naturalistic Observation, and Qualitative Research
Introduction to Nonexperimental Designs
Correlational Design
Importance of Correlational Research Direction of Control and Third Variable Problems Addressing Directionality and Third Variable Problems Correlational Ruling Out Factors Interpretation of Correlational Data
Ex Post Facto Design
Naturalistic Observation
Qualitative Research
Case Study Phenomenology Ethnography
Case Analysis
General Summary
Detailed Summary
Key Terms
Review Questions/Exercises
Introduction to Nonexperimental Designs
We have said much about true experiments and we have described their strength in drawing strong, confident conclusions. A word of caution is advisable. An experiment may use random assignment and involve manipulation of the treatment variable and still be essentially worthless as a basis for drawing conclusions. It is essential that rigorous controls, careful execution, planning, thoughtfulness, etc., accompany a valid design. We have also noted the qualities of designs termed quasi-experimental. Recall that these were characterized as designs in which the independent variable was manipulated but the study lacked random assignment of participants to conditions. As we have seen thus far in the book, experimental research is a very powerful tool for generating a scientific database for drawing cause-effect conclusions, for testing hypotheses and evaluating theory, for answering questions and satisfying our intellectual curiosity, for systematic manipulation of variables, and, at times, for discovering principles that may be relevant to everyday life. After considerable discussion of the virtues of experimental designs, you might wonder why researchers would use other types of nonexperimental designs. Actually, there are several good reasons to use nonexperimental designs. Many very interesting questions in psychology do not lend themselves to experimental designs. Some of these questions involve independent variables that simply cannot be manipulated by a researcher. If we wish to study the effects on a dependent measure of such naturally occurring variables as gender, ethnic background, intelligence, temperament, or body size, we cannot say to the participants, "For the purposes of this experiment, I am going to declare you a female, or a black, or a person with an IQ of 130!” In addition, some questions involve independent variables that could theoretically be manipulated by a researcher but are not because the opportunity does not present itself, the financial cost would be too high, or the ethical concerns too great. For example, we may ask, "Do individuals who have left hemispheric brain damage show greater verbal impairment than those who have comparable damage to the right hemisphere?" Obviously, it is not possible to randomly assign people to an experimental and control group and then conduct brain surgery to answer this question. However, if we are to shed any light on the question, we are forced to look into the histories of people who have suffered brain damage as a result of adverse circumstances. Similarly, as we have repeatedly explored the issue of TV violence and aggressive behavior in children, we would certainly be interested in the effects of long-term (in terms of years) exposure to TV violence. I’m sure that you can see the ethical issues involved in randomly assigning a group of children to watch violent television for several years! Thus, although nonexperimental research designs are not as powerful as experimental designs i.e., do not rule out as many alternative hypothesis (explanations), they provide us with options for pursuing
Random assignment of participants and the manipulation of variables are absent in correlational research because the events of interest have already occurred or are naturally occurring. The interest is in determining how measures on one variable are related to measures on another variable. Often, in psychology, the two measures are behavioral measures. The correlational approach is sometimes referred to as the study of individual differences because emphasis is placed on differences among individuals. For example, assume that we have a distribution of individual scores on one measure (Intelligence Test Scores—Test 1) and a distribution of individual scores on another measure (Final Exam Scores—Test 2). The question asked of these data by a correlational approach is whether differences among individual scores on one variable (Test 1) are related to differences among individual scores on the other variable (Test 2). A statistical procedure called correlational analysis is used to ascertain the extent of the relationship among individual scores on the two variables (tests). This emphasis on individual differences contrasts with an experimental approach where interest is in comparing the average performance of a group in one condition with the average performance of a group in another condition (single-subject designs are an exception). As you may recall from your introductory statistics course, calculating a correlation between two distributions of scores (scores on Test 1 and scores on Test 2) results in a number called a correlation coefficient. The strength of the relationship is indicated by the numerical value of the coefficient and its direction is indicated by a + or - sign. If the individual scores are unrelated (no relationship), the numerical value of the coefficient is 0; if the scores are perfectly related on the two distributions, the numerical value is either a -1.0 or a +1.0. Thus the numerical value of the correlation coefficient may range from a -1.0 to 0 or from 0 to a + 1.0, with variations in between. A positive relationship indicates that individuals scoring high on one distribution also tend to score high on the other distribution and that those scoring low on one tend to score low on the other. Put more simply, as individual scores on one distribution increase, their scores on the other increase (e.g., the more one studies, the higher one’s grade point average). If the relationship is negative, then individuals scoring high on one distribution tend to score low on the other and those scoring low on one tend to score high on the other. Again, put simply, as individual scores on one distribution increase, their scores on the other decrease (e.g., the more one parties, the lower one’s grade point average). Correlational methods are used in virtually every scientific and professional discipline and they serve many purposes. Correlations between variables are often used to make predictions. When measures on two variables are unrelated, i.e., correlation coefficient = 0, knowing an individual's score on one variable is not
at all helpful in predicting his or her score on the other variable. As the correlation becomes greater than zero, the accuracy of predicting the individual's score on one variable, simply by knowing his or her score on the other, increases. And when the correlation is perfect, i.e., +1.0 or -1.0, prediction of an individual's score on one variable from knowing his or her score on the other can be made without error. While working on this chapter, a number of articles appeared in newspaper accounts and in popular magazines dealing with correlational research. Some of these accounts are found in Table 15.1. They give an idea of the variety of problems that can be studied using correlational procedures. We do not describe the results of these reports because we have not read the primary source from which they came, nor have we evaluated the care with which the studies were conducted. After reading the section on Third Variable Problems and Directionality, you may want to return to this table to evaluate the extent to which these two problems may be present in the description found in Table 15.1. You will most likely conclude that many alternative explanations may be offered for the observed relationships.
Although correlational research allows lawful relationships to be discovered that can lead to precise predictions, causal statements can be made only with great risk because these methods lack random assignment, active manipulation, and rigorous control over extraneous factors. Such variables as gender, group membership, racial characteristics, birthplace, and age are historical events over which researchers have little control. They are determined before the researcher arrives on the scene. The measures that are correlated are often personality variables or variables related to the characteristics of people. These
Another example of correlational research that has proven useful relates to diagnostic purposes. After a disorder is observed, a search can be made for other behaviors or conditions that may vary (correlate) with it. The latter is especially the case if the disorder is difficult to detect or to diagnose accurately. If the search is successful and the correlation strong, then both the speed and accuracy of identifying the disorder may be substantially increased. The usefulness and value of a correlational approach are most apparent when studying the effects of events that simply cannot be studied in laboratory settings. This approach may be the only available method when ethical considerations prevent manipulating the phenomena (e.g., abortion, drug use, sexual practices, serious illness, suicide), when the phenomena are impossible to manipulate (e.g., male/female, black/white, temperament), or when studying the effects of natural disasters such as earthquakes, fires, violent storms, etc. Direction of Control and Third Variable Problems Discussion of direction of control and third variable problems will illustrate the difficulties of inferring cause-effect when interpreting correlational data. With correlational research, we usually refer to predictor and criterion variables rather than independent and dependent variables. The measure (or behavior) being predicted is the criterion variable, and the measure (or behavior) from which the prediction is made is the predictor variable. The use of this terminology emphasizes prediction rather than suggesting a cause-effect relationship. However, there are occasions when individuals, be they scientists, writers, or laypeople, come to cause-effect conclusions based on correlational data. They face a risk of drawing false conclusions when doing so. In effect, they must deal with two different problems: the direction of control problem and the third variable problem. We will give examples of both. To infer a cause-effect relationship requires that we specify the direction of control. Assume that variable X and variable Y are highly correlated such that increases in one are associated with increases in the other. Does variable X cause variable Y to vary, or does variable Y cause variable X to vary? With some relationships the answer concerning the direction of control seems obvious but in other instances it can be difficult to specify. Let’s assume that our correlational research on TV violence and aggression showed a positive relationship between these two variables. It would seem to be a natural inference to conclude that exposure to TV violence leads to (causes) more aggressive behavior in children. However, is this necessarily the case? Are there alternative explanations for the observed relationship? Is it not possible that children who are more aggressive, for whatever reason, tend to choose TV shows with more violence? We will give another example of the direction of control problem. Let us say that a high, positive correlation exists between frequency of drug use (variable X) and difficulties in school (variable Y). We
could say that the use of drugs was the cause of experiencing difficulties in school. On the other hand, an equally plausible conclusion is that having difficulties in school caused the individual to use drugs. The fact that in both of these examples we have a high correlation between variable X and variable Y does not help at all in our determining the direction of control. Our risk of coming to a wrong causal conclusion is not reduced. The only way to reduce the risk is to bring additional information to bear on the issue or, when permitted, to attempt an experimental approach and manipulate the important variables. Our problem is intensified in that the risk of error is even greater when we consider possible third variable problems that may be present. For our example with the TV violence study, it is possible that neither of the two variables causes the other. Rather, some other (third) variable may actually cause changes in the two variables that were measured. After giving this some thought, we are sure that you can come up with a potential third variable. In our example that involved drug use, instead of drug use causing school problems, or school problems causing drug use, a third factor (fourth, fifth, etc.) could have caused increases in both drug use and school problems. For example, anxiety, depression, low self-esteem, or conflict within the home could give rise to both increases in drug use and increases in school problems. In this case, then, variables X and Y are related only through some third variable. Figure 15.2 illustrates both the direction of control and third variable issues.
technique for ruling out third variable interpretations seems like an attractive solution. That is, if only individuals who were the same or highly similar on these third variables were assessed, and if the relationship between the two variables of interest still existed, then the third variable considerations could be dispensed with. The thought is a good one, but in practice problems exist; also, matching can be difficult to achieve. One problem is that researchers cannot be sure they have considered the relevant third variables. Some may exist that have not been considered. Another problem is that matching on one variable can sometimes unmatch individuals on other variables. Researchers face a more serious problem when they attempt to match on several variables (e.g., intelligence, education, motivation, class). The problem is simply in getting a sufficiently large sample that has the necessary characteristics. Correlational Ruling Out Factors The correlation between cigarette smoking and cancer (and heart disease) can be used to illustrate an important aspect of correlational research. Had no relationship been found, we could have ruled out causal factors and there would be little interest in pursuing the problem further. Therefore, an important con- tribution of correlational methods may be in a negative sense: In the absence of a relationship, there is no need to devote time to an experimental analysis of the problem to identify the controlling or causal factor. Perhaps an example contrasting the correlational method with the experimental method would be helpful in illustrating why a causal relationship can be made with an experimental approach and not a correlational one. As noted, in a correlational study involving cigarette consumption and incidence of cancer, a positive relationship was found. While most people of sound mind would be alerted to a possible causal link between the two, a strong relationship regarding cause cannot be made. We indicated that it is possible that a third factor may be the cause, e.g., people susceptible to cancer also smoke, but the disease would occur whether or not they smoked. Or we could argue that individuals who smoke also engage in other activities that may be related to cancer and that smoking is not the problem. Or, we could say that individuals with certain dietary habits are susceptible to both smoking and cancer, etc. The tobacco industry today argues that a causal link has not been clearly established in humans. To decide the question, an experiment is necessary. It would not be difficult to design a research project to answer the question of smoking as a cause for illness. However, serious problems of ethics and practicality prevent its implementation. We would randomly select a large number of twelve- to fourteen-year-old male and female students from different geographic areas. We could then randomly assign them to conditions A, B, C, D, and E—corresponding to levels 0, 1/2, 1, 2, and 3 packs of cigarettes a day. We would then follow them up over the years with annual physical examinations and also record specific measures known to be related to tissue problems. We could then determine whether there was a systematic relationship between our independent variable (smoking) and our dependent variable (measures of illness, etc.).
Obviously, the study cannot and should not be done. However, experimental studies have been conducted with nonhuman species. Mice, rats, and dogs have been used to study the relationship between the exposure to tobacco ingredients and cancer. A variety of procedures have been used, such as placing the substances in tobacco on the skin, confining the animals to enclosed rooms where controlled amount of cigarette smoke may be dispersed, and teaching animals to smoke. These studies have established that tobacco is hazardous to a laboratory animal's health. We have stated several times that it is improper to draw cause-effect conclusions from correlational data. To say this is not to say that such a relationship does not exist. It may. To determine its existence requires other research strategies. There are many examples of correlational studies reported in the media that falsely suggest a cause- effect relationship. You are probably familiar with reports of a link between the phase of the moon and “strange” behavior. Box 15.1 illustrates one such report.
behavior in children. As we noted, random assignment of children to groups that view specific levels of TV violence for several years is not ethical. However, we could conduct a correlational study in which children (with assistance from parents) maintain a weekly log of TV shows watched from the time that they are 10 years old until they are 15 years old. Based on these logs, we calculate the mean number of hours of TV violence viewed per week. For the variable that measures aggressive behavior, we obtain the number of disciplinary incidents recorded at their school for the same five-year period. Hypothetical data from 20 students are shown in Table 15.2.
Glancing through the pairs of scores in Table 15.2, whether there is any systematic pattern in the data is difficult to detect, although the one participant with 25 hours/week and 30 incidents does stand out. With correlational data, it is not only helpful, but critical, that the data be graphed in a scatterplot. Figure 15.3 is a scatterplot of the data such that a point on the graph represents each pair of scores. Because the values on the x-axis increase from left to right and the values on the y-axis increase from bottom to top, it is now easy to detect a general pattern in the data such that as the mean hours of TV violence per week increase, the number of disciplinary incidents at school also increases. The presence of the one outlier is also clear on the graph. Because our data are measured on a ratio scale and because our graph suggests a linear relationship, we can calculate a Pearson r correlation coefficient to quantify the relationship (see Figure 10.10 to review the decision tree for measures of relationship). The calculation results in r = .88, a significant positive correlation.
Figure 15.3 Scatterplot that depicts the hypothetical data in Table 15.
It is important to be aware in correlational research that outliers in the data can have dramatic effects on the value of the correlation coefficient and, therefore, the conclusions drawn. For example, if we remove the one outlier point from our hypothetical data, the r value shifts from .88 to .62. Thus, that one participant dramatically increased the strength of the relationship between the two variables. In other cases, the outlier can be such that it dramatically decreases the strength of the relationship. In either case, researchers should always carefully inspect the data that produced outliers to be sure that no errors were
Figure 15.4 Scatterplot that illustrates a curvilinear relationship
Finally, sample size is an issue to consider in correlational research. The power of a correlational analysis is increased by increasing the sample size. With extremely large samples, weak relationships with correlations of .1 or .2 may be statistically significant but not meaningful practically. On the other hand with a small sample size a meaningful relationship might go undetected. The concept of power is an important one (see Chapter 10). Although correlational research represents a powerful nonexperimental design, it is time to turn to several other types of nonexperimental designs.
Ex Post Facto Design
At times, caution must be exercised in deciding whether a study is nonexperimental or experimental. If the determination is not made, an erroneous conclusion may result. A type of study that can masquerade as a genuine experiment is the ex post facto design. Although it appears to be a true experiment because of the way groups are separated and the way the analysis is performed, it is still nonexperimental research and subject to the same limitations discussed earlier with correlational research. It derives its name from the fact that the assignment of participants to levels of the independent variable is based on events that occurred in the past (i.e., after the fact). It mimics an experiment in that comparisons are made between two or more groups of individuals with similar backgrounds who were exposed to different conditions as a result of their natural histories. We then measure the participants on a dependent variable of interest to
determine whether or not statistically meaningful differences exist between the experimental groups. If reliable differences are found, should we conclude that they were due to the historical differences we found in the past records? The answer is no. Note that the ex post facto design uses neither random assignment nor active manipulation of the independent variable. However, the intent of this type of research is precisely that of a true experiment but the problems encountered in drawing conclusions are very different. A few examples will illustrate our point. In Chapter 11, we considered an experimental design in which we, as the researchers, determined which levels of TV violence would be studied and randomly assigned research participants to these levels. We then measured aggressive behavior. A nonexperimental alternative would be to place participants into groups based on how much TV violence they just watched (of their own choosing) and to then measure aggressive behavior. Notice that these two research designs are similar in that the aggressive behavior of groups of participants are being compared, in which the groups differed in terms of exposure to TV violence. We should also mention that the statistical analysis would be similar. However, notice the differences. The latter design is an ex post facto design because participants were not randomly assigned to the groups, i.e., the independent variable was not manipulated by the researcher. Because we lack the control of an experimental design, we must be very cautious with the nature of the conclusion that we draw from the ex post facto design. As we noted with correlational research, direction of control and third variable issues need to be considered. Let’s examine another example. An instructor in a college math course believes that a relationship exists between performance in college math courses and whether students had the "old math" or "new math" techniques in grade school. She decides to do a study to determine whether her observations are indeed accurate ones. She looks up the grade school records of a large number of college students taking her college math course. On the basis of these records she selects fifty students who received "new math" and another fifty students who received "old math" techniques. She then gathers the two groups of students together and gives them a college math proficiency test. Her experimental design would look no different from that of a true experiment. We have what appears to be a "treatment" condition and a comparison or control group (old math)—in this case both of which have already occurred. The “treatment versus comparison group” appears to be the same comparison characterized by a true experiment. But any conclusions drawn from our example must be guarded and weak. Why? Obviously, our participants cannot be assigned randomly, (nor can we manipulate the conditions because we have selected students already). Their assignment to the levels of the independent variable was based on past records.
We could give a personality test (e.g., the Taylor Manifest Anxiety Scale) to a large population of students and then select participants on the basis of their scores. We might select ten students who score very high and ten students who score very low on the anxiety scale. Since high anxious and low anxious participants are our interest and, as it were, our treatment, they cannot be randomly assigned to different groups. Therefore, our independent variable is high versus low anxiety. Our dependent variable is the magnitude of the startle reaction to loud noise. Assume that we find our high anxious participants respond to noise more strongly than do our low anxious participants. Can we conclude that they did so because of anxiety? No, we cannot. We are again plagued by the third variable problem. Because we could not employ random assignment, i.e., our groups were pre-existing groups, it is possible that our participant groups differ in many ways other than anxiety. Their responses could be related to how well they sleep. It may be that high anxious participants sleep less well than others and that any participant, regardless of anxiety, who did not get adequate sleep would perform like high anxious participants. Perhaps it is the case that high anxious participants drink much more coffee than do low anxious participants. If we assume that coffee affects the startle response, then any participants who drink much coffee, anxious or nonanxious, would perform in similar ways. There are any number of other alternative interpretations allowed by this ex post facto design that would not be possible if the research was either a quasi- or a true experiment.
Naturalistic Observation
Naturalistic observations take place under natural conditions or under real-life conditions without any intervention on the part of the researcher. Such observations contrast with those in experimental set- tings where considerable prior control is possible, where events can be manipulated, and the observations may be repeated. When using naturalistic observation, we observe nature without imposing change. However, the observations are carefully planned and systematized. Consequently, the data can be organized in a meaningful way to permit analysis and interpretation. However, the interpretation issues already discussed in this chapter must be considered. Naturalistic observation is the oldest method for the study of behavior or other phenomenon. One of the most accurate sciences (in terms of prediction) is astronomy and it is restricted largely to observation. Ethologists are people who study animal behavior under natural conditions and use natural observation almost exclusively. Jane Goodall's work with primates in Africa has resulted in some fascinating observations regarding their social interactions. Development and personality psychologists have published extremely informative observational studies of children interacting under natural conditions. Jean Piaget's theory of cognitive development in children was based on naturalistic observation. We have been able to
identify migration patterns in fish, fowl, and mammals by tagging studies. Naturalistic observation of primitive cultures has given us insight into the range of variation in human institutions. It must be emphasized that natural observation is not anecdotal and casual but, instead, systematic and carefully planned. The observer must be sufficiently skilled to distinguish between an observation and an interpretation. Many of the principles of good observation were discussed in Chapter 6.
Qualitative Research
All of the research designs discussed thus far have involved the measurement of variables with a subsequent statistical analysis of the values recorded. The application of mathematics to the process of making research conclusions lends a degree of objectivity to the decision-making process. Although there can be differences of opinion regarding the most appropriate statistical analysis, for the most part, a given set of data would lead different researchers to the same conclusion. There is value to this but there is also value to qualitative research , in which the researcher gains insights from more informal and non-numerical observations. In fact, conclusions drawn from such informal observations are often a first step that leads to more rigorous experimental studies to clarify relationships among variables. A good researcher understands that the various types of experimental and nonexperimental designs complement one another when a particular area of behavioral research is studied. Case Study Design With the case study design , one or more individuals are carefully examined over time. Biographical data, interviews, or psychological tests may be components of the case histories. Some examples of case histories are prepared by reconstructing the biography of the individual from memory and records (retrospective). On occasion, a case history may be prospective. It does not rely on memory or records because measurement is taken at periodically planned intervals. There are two important considerations in a case study approach. One is to search for some regularity or patterning to behavior that might suggest some principle around which it is organized. The second involves additional case studies to confirm the previously observed regularity or pattern. Such information may permit generalization to other situations or persons. Case studies often arise when it is impossible or unethical to conduct an experiment. At other times, they are closely related to naturalistic observations. Freud's insights into behavior problems were largely based on case studies of his patients. Piaget's theory of intellectual development stems from his intensive observations of his own three children. Case studies of individuals suffering brain damage have been undertaken to assess the extent to which functions are regained.
