Download Preparing to Make Sampling Choices: A Guide to Sampling Essentials and more Exercises Designs and Groups in PDF only on Docsity!
CHAPTER 1
PreParing to M ake
SaMPling C hoiCeS
INTRODUCTION
The key to good research is preparation, preparation, and preparation. Hence, the key to making good sampling choices is preparation, prepara- tion, and preparation. Sampling may be defined as the selection of a subset of a population for inclusion in a study. If done properly, it can save money, time, and effort, while providing valid, reliable, and useful results. On the other hand, if done poorly, the findings of a study may have little scientific and practical value. In order to increase the likelihood that the findings of a study will have value, preparations should be carried out before making sampling choices. This chapter begins with a brief history of sampling, followed by a descrip- tion of the major steps in selecting a sample. Preparation is the first of these steps. The preparation should include a careful review of the study’s purpose, the nature of the population, the available resources, various research design considerations, and ethical and legal considerations. Guidelines for making these preparations are described in this chapter. What you will learn in this chapter:
- • Milestones in the history of sampling
- • Major steps in selecting a sample
- • Preparations important for making sampling choices
- • Guidelines for preparing to make sampling choices
2 Sampling Essentials
MILESTONES IN THE HISTORY OF SAMPLING
Although sampling probably has always been part of human history, many of the sampling procedures used today have a relatively short history. Govern- ments have long collected population data for taxation, military purposes, and other objectives. Typically, total enumeration was sought. On the other hand, private pollsters tended to use availability sampling such as straw polling. However, by the end of the 19th century, “scientific” procedures for selecting a sample began to surface. U.S. governmental agencies, in particular, began experimenting with these procedures that later became known as probability sampling. Private pollsters, on the other hand, continued to rely on availability sampling until 1936. Critical changes in sampling procedures used by private pollsters came about in 1936 and again in 1948. In 1936, the failure of the Literary Digest to predict the winner of the presidential election led to a movement away from availability sampling to quota sampling. Using availability sampling of mil- lions of respondents, the Literary Digest was successful in predicting the win- ner in each U.S. presidential election that was held between 1916 and 1932. However, it failed to do so in 1936. On the other hand, using a new sampling procedure that later came to be known as quota sampling, pollsters George Gallup, Elmo Roper, Paul Cherington, and Richardson Wood were successful in predicting Franklin D. Roosevelt as the winner in that election. This caused pollsters to pay more attention to quota sampling and less attention to avail- ability sampling (Bryson, 1976; Cahalan, 1989; Katz & Cantril, 1937; Squire, 1988). These sampling procedures are described in detail in Chapter 4. The failure of the Literary Digest’s prediction of the winner of the 1936 presidential election was primarily due to two factors: coverage bias and non- response bias.
- • Coverage bias is the lack of a one-to-one correspondence between the elements in the target population and the elements encompassed by the respondent selection procedures used in a study. Sampling frame bias is the extent to which there are differences between the elements that are listed in the frame and the elements that make up the target population_._ The sampling frame of the Literary Digest consisted of its subscribers, listings in telephone books, and listings of automobile registrants. These lists were not representative of the 1936 voting population_._
- • Nonresponse bias is the extent to which there are significant differences between the respondents and nonrespondents in terms of the variables of
4 Sampling Essentials help the Democratic Party. The Communist Party USA did not run a can- didate for president, but endorsed the Progressive Party’s candidate. This endorsement deflected anti-communism attacks away from the Democratic Party. Many White southerners left the Democratic Party to support the Dixiecrats. This made the Democratic Party more acceptable to Blacks, and they gave it their support. The failure of the major polling companies to predict the winner of the 1948 U.S. presidential election motivated them to move away from quota sampling and incorporate probability sampling into their polling procedures. They joined statisticians in the federal government and academia in endorsing prob- ability sampling. Probability sampling became the dominant sampling proce- dure for estimating population parameters. The major types of probability sampling are described in Chapter 5. Up to today, sampling procedures continued to evolve. To a certain extent, as modes of collecting data changed, sampling procedures changed. During the period of the 1970s through the 1990s, there was a movement from personal interview surveys to telephone surveys. Variants of random digit dialing (RDD) sampling procedures were developed to meet challenges of telephone surveys. As research methods embraced advances in electronic technology, including the use of online surveys, fax machines, and cell phones, sampling procedures were further modified and adjusted. Today, a wide range of nonprobability and probability sampling procedures are used, making sampling choices more chal- lenging than ever before.
MAJOR STEPS IN SELECTING A SAMPLE
One may identify six major steps in selecting a sample (see Figure 1.1): Step 1. Prepare to make sampling choices. Step 2. Choose between taking a census and sampling. Step 3. Choose nonprobability, probability, or mixed-methods sample design. Step 4. Choose the type of nonprobability, probability, or mixed-methods sample design. Step 5. Determine the sample size. Step 6. Select the sample.
Chapter 1 Preparing to Make Sampling Choices 5
Step 1. Prepare to Make Sampling Choices
Specific preparation should be made before making sampling choices. Such preparation should include a careful review of the purpose of one’s study, the nature of the population, available resources, research design considerations, and ethical and legal issues considerations. Guidelines for making these preparations are presented in the next section of this chapter.
Step 2. Choose Between Taking a Census and Sampling
The second step involves choosing between selecting the entire target population (taking a census ) and selecting a subset of the target population (sampling). In making this choice Figure 1.1 Major Steps in Selecting a Sample Step 2. Choose between taking a census and sampling. Step 3. Choose nonprobability, probability, or mixed-methods sample design. Step 5. Determine sample size. Step 6. Select sample. Step 4. Choose the type of nonprobability, probability, or mixed-method sample design. Step 1. Prepare to make sampling choices.
Chapter 1 Preparing to Make Sampling Choices 7
Step 5. Determine the Sample Size
Having chosen a specific type of sampling design to be used to select a sample, the next step involves determining of the number of elements to be selected. Chapter 7 describes factors that should be considered in determining sample size and guidelines for doing so.
Step 6. Select the Sample
The final step in sampling involves implementing one’s sampling choices. The quality of the resulting sample is dependent substantially on the first step: preparing to make sampling choices. Guidelines for preparing to make sam- pling choices are presented below.
GUIDELINES FOR PREPARING
TO MAKE SAMPLING CHOICES
Specific preparation should be made before making sampling choices. Several guidelines may be proposed. In considering the guidelines listed below and oth- ers presented in other chapters, it should be noted that they are not equally important; nor are they absolute. In many cases, their applicability is contin- gent on specific conditions that may or not be present. Often, the researcher must balance competing and conflicting guidelines. Before making sampling choices one should be able to clearly answer such questions as: What are the objectives of the study? How is the target popula- tion defined? What is the nature of the population (i.e., its size, heterogeneity, accessibility, spatial distribution, and destructibility)? What resources are avail- able to conduct the study? What type of research design will be implemented? What ethical and legal issues should be taken into account? Considering ques- tions such as these, guidelines for preparing to make sampling choices may be categorized as:
- • Objectives of the study
- • Definition of the population
- • Nature of the population
- • Availability of resources
- • Research design considerations
- • Ethical and legal considerations
8 Sampling Essentials
Objectives of the Study
Guideline 1.1. Objectives of the study. Prior to making sampling choices, make sure one has a good understanding of the objectives of the study, the importance of the study, and the special needs of the study, if any. There should be a good fit between the objectives of a study and the sam- pling choices a researcher makes. A research study may have only one or a combination of the following objectives: exploration, description, prediction, evaluation, and explanation. Exploratory research targets information seeking to better understand a population, theoretical issues, or methodological issues relating to a study. A study with a descriptive objective seeks to describe the parameters of a population, differences between or among population, or rela- tionships among variables. A study with a prediction objective seeks to predict future parameters of a population, differences between or among populations, or relationships among variables. Evaluation research seeks to determine the need for an intervention, how the need should be addressed, the ongoing prog- ress of an intervention, and the outcome of an intervention. Explanatory research attempts to explain the patterns of population parameters and the relationships among variables. Different objectives of a study may require different sampling choices. Typically, exploratory research does not require a rigorous sample design. A nonprobability sample with a small sample size may suffice. On the other hand, the require- ments of the sample design of a descriptive study may depend on the amount of detail required, the confidence level requirements, and the homogeneity/ heterogeneity of the population. The required precision of one’s predictions may determine the amount of rigor in the sample design of a prediction study. In part, the sample design for an evaluation research project is dependent on the type of evaluation research one is conducting (formative evaluation, pro- cess evaluation, or outcome evaluation). A study with an explanatory research purpose may require a more rigorous sample design than a study that has one of the other purposes. There should be a good fit between the importance of a study and the sam- pling choices that are made. Studies are not equally important. Highly impor- tant studies require a much more rigorous sample design than studies that are not as important. A researcher should have a clear understanding of the impor- tance of a study before making sampling choices. Moreover, there should be a good fit between the special needs of a study, if any, and the sampling choices that are made. The special needs of a study may require the selection of particular population elements, a particular sample size,
10 Sampling Essentials RESEARCH NOTE 1. Example of the Use of Inclusive and Exclusive Criteria in Defining a Target Population Wingood and DiClemente (1998) described the inclusive and exclusive criteria they used in defining their target population in their study of African American women’s noncondom use during sexual intercourse as follows: Inclusion criteria consisted of being a sexually active, heterosexual African American female, 18–29 years of age, residing in the Bayview-Hunter’s Point neighborhood. Exclusion criteria consisted of a history of injection drug use or crack cocaine in the past 3 months. Study participants were recruited using street outreach and media advertisements placed throughout the com- munity. Indigenous African American female field recruiters, familiar with the Bayview-Hunter’s Point neighborhood, approached and screened women at the local unemployment office, the Social Security office, public laundry facilities, beauty salons, grocery stores, health clinics, and the local (AFDC) office to identify women eligible for participation in the study. Source: Wingood & DiClemente, 1998, p. 34. With kind permission from Springer Science+Business Media.
Nature of the Population
Guideline 1.2. Nature of the population. Prior to making sampling choices, one should have a good understanding of the target population, including its content, size, heterogeneity/homogeneity, accessibility, spatial distribution, and destructibility. Content of the Population The content of the population should affect sampling choices. The ele- ments of the population of a research study may be people, things, places, events, situations, or time. The composition of many materials that may be studied in the natural sciences may be assumed to be constant, making any sample essentially identical to any other sample. A sample of blood taken from one’s finger may be considered equivalent to a sample of blood taken from one’s arm. In such situations, generalizing from a single sampled
Chapter 1 Preparing to Make Sampling Choices 11 element may be acceptable. However, different samples of a human population may be extremely unlike each other. Generalizing from a small sample of people may be untenable. Size of the Population The size of a population is a critical factor in making sampling choices. One should have knowledge of the size of the population before making sampling choices. More resources and a larger sample size may be necessary to study a large population than to study a small population. Costs, the amount of time needed to collect the study’s data, management issues, random sampling error, and systematic error are tied to the size of the population, and thereby affect sampling choices. Heterogeneity/Homogeneity of the Population The homogeneity/homogeneity of a population should be considered in making sampling choices. Studies of populations that are relatively homo- geneous require smaller samples than studies of populations that are rela- tively heterogeneous. Before making sampling choices one should determine the homogeneity/heterogeneity of the target population. In conducting a literature review and exploratory research, special attention should be paid to measures of the variability (i.e., standard deviation, variances, etc.) of one’s key variables of interest. A pilot study may be in order to acquire such information. Accessibility of the Population Populations vary in their accessibility. The accessibility of a population will affect the ability of a researcher to successfully implement a sample design, and should be considered in making sampling choices. Segments of the population may be in remote locations, gated communities or buildings, or other inaccessible locations. Some populations are considered to be “hid- den” due to the difficulty to locate them and gain their cooperation. Exam- ples include persons at risk of HIV infection, gang-affiliated adolescents, gays and lesbians who are “in the closet,” injection drug users, sex workers, and the street homeless. Elements of the target population may be inacces- sible because they have neither a postal address nor an e-mail address. In preparing to make sampling choices, one should determine the accessibility
Chapter 1 Preparing to Make Sampling Choices 13 of customers, addresses, telephone numbers, city directory, and a map. A suitable sampling frame may not exist or may not be accessible to the researcher because of privacy regulations or other reasons. Moreover, it may be very time-consuming to develop an appropriate frame or expensive to purchase one from a vendor. A good sampling frame would identify all members of the target population only once, and have no other entries, but also include auxiliary information that may be useful in making sampling choices. A good sampling frame would be complete, accurate, up-to-date, reliable, and convenient to use. In preparing to make sampling choices, one should determine the availability of a good sampling frame, and the resources available to create one if one does not exist. Once obtained or developed, an assessment should be made of any sampling frame bias that may exist.
Research Design Considerations
Guideline 1.4. Research design considerations. Prior to making sampling choices, determine the type of research design, the data collection design, and the data analysis design that will be used. Sampling choices should be made in conjunction with other choices relating to the research design of a study. Most important are choices relating to the following:
- • Type of research design
- • Data collection design
- • Data analysis design Type of Research Design There should be a good fit between the type of research design used in a study and the sampling choices that are made. One should determine the type of research design that will be employed before making sampling choices. Major types of research designs may be classified as follows:
- • Qualitative versus quantitative research designs
- • Nonexperimental versus experimental research designs
- • Cross-sectional versus longitudinal research designs
- • Mixed-methods designs
14 Sampling Essentials Qualitative Versus Quantitative Research Designs. Sampling choices for quali- tative research tend to be different from sampling choices for quantitative research. Qualitative research primarily involves the collection and analysis of non-numerical data, with more attention focused on understanding the nature of the elements selected for study than to generalizing to a target population. It is characterized by in-depth inquiry; immersion into the social setting of that being studied; emphasis on the understanding of the participants’ perspectives; and comprehensive description of the study’s topic. Conversely, quantitative research primarily involves the collection and analysis of numerical data, with more attention focused on generalizing to a target population than understanding the nature of the elements selected for study. Both probability sampling and nonprobability sampling are utilized throughout quantitative research, whereas qualitative research primarily employs nonprobability sampling. Probability sampling might not yield ele- ments of the population that can satisfy the needs of qualitative research. Typically, the qualitative researcher is not interested in estimating population parameters, but rather interested in selecting population elements that are most useful in providing rich information about the topic of the study. These ele- ments may have to be purposefully selected. Typically, in quantitative research a fixed sample size is set prior to data collection. A researcher might set a target sample size so as to yield a specific margin of error or confidence interval within which is expected to include the true population value. On the other hand, in qualitative research, a sequential approach is more common. Using such an approach, a qualitative researcher would continue to sample until data saturation is reached, that is, as new elements are added to the study, no new information or understanding is forthcoming. Nonexperimental Research Versus Experimental Research Designs. Sampling choices for experimental research tend to be different from sampling choices for nonexperimental research. In experimental research , a researcher controls exposure to the key independent variable of a study. The researcher creates variability in the key independent variable. On the other hand, in nonexperi- mental research , a researcher does not control exposure to the key indepen- dent variable of a study. Instead of creating variability in the key independent variable as in experimental research, the researcher measures naturally occur- ring variability in the variable. Although both probability sampling and non- probability sampling are used in nonexperimental research and experimental research, the sample designs tend to be more complex and the sample sizes
16 Sampling Essentials combined with a longitudinal design by systematically adding new elements from the target population to compensate for attrition. A rotating panel design may be used to reduce respondent burden. This involves the use of multiple panels of population elements with each being used a fixed number of times and targeting different variables of interest. Given that these designs require different sampling choices, they require different preparations to enhance their effectiveness. Mixed-Methods Research Designs. A researcher may not limit the study he or she conducts to a single design. A combination of research designs, mixed-methods research designs, may be employed. Much of the literature on mixed-methods designs focus on the mixing of qualitative and quantitative research designs. Other options include the mixing of nonexperimental and experimental research designs, and/or the mixing of cross-sectional and longi- tudinal research designs. The mixing of different designs has implications for the sampling choices that are made. A mixed-methods sample design may be advisable, and a larger or smaller sample size may be required than would be required if a single-method design is used. One may mix different nonprobability sample designs, mix different probability sample designs, or mix nonprobabil- ity and probability sample designs. Mixed-methods sample designs are described in Chapter 6. A research design has three major components or sub-designs: the selection of study participants design, the data collection design, and the data analysis design. These sub-designs must fit well together and with the purpose of a study for a study to be effective. Data Collection Design There should be a good fit between the data collection choices and the sam- pling choices that are made. The different modes of collecting data tend to have different sources of systematic error and costs. The preparation of making sampling choices includes estimating eligibility rates, unit nonresponse rates, item nonresponse rates, and data collection costs. All of these factors should be recognized and taken into account when making sampling choices. Data Analysis Design There should be a good fit between the data analysis choices and the sam- pling choices that are made. The data analysis plans of a research project might require only a few basic analyses. On the other hand, the data analysis design may require complex, multivariate procedures. Moreover, statistical procedures
Chapter 1 Preparing to Make Sampling Choices 17 vary in terms of their sampling requirements. Some statistical procedures require probability sampling. Generally, the more complex the data analysis design, the larger the required sample size. Sampling choices should be made after determination of the data analysis requirements.
Ethical and Legal Considerations
Guideline 1.5. Ethical and legal considerations. Prior to making sampling choices, identify any ethical or legal concerns relating to the research project. There should be a good fit between the ethical and legal concerns and the sampling choices that are made. Concerns relating to informed consent, pri- vacy, anonymity, confidentiality, and professional codes of ethics may make it impractical or impossible to implement certain sample designs. As part of the preparation in making sampling choices, one should make sure one is aware of the relevant ethical and legal regulations. In order to acquire sufficient information to apply the above guidelines, it may be necessary to conduct formative research including a comprehensive literature review and exploratory research. Research Note 1.2 describes the formative research conducted in a study of men who have sex with other men. RESEARCH NOTE 1. Example of Formative Research in Preparing to Make Sampling Choices: A Study of Men Who Have Sex With Men In preparation for the National HIV Behavioral Surveillance (NHBS) study of men who have sex with men (MSM), MacKellar et al. (2007) conducted formative research to acquire information for planning the sample design of the study. They described the formative research conducted as follows: Formative research was conducted to learn about the venues, times, and methods to recruit MSM. To meet these objectives, staff reviewed advertise- ments for MSM in online and print media, interviewed key informants, and (Continued)
Chapter 1 Preparing to Make Sampling Choices 19 SUMMARY Throughout much of human history, total enumeration was considered the only valid way to study populations. Sampling was limited to straw polling, volunteer sampling, and other availability sampling procedures. Quota sampling was introduced in the early 20th century, and took off when pollsters using the technique were successful in predicting the winner of the 1936 U.S. presidential election, whereas pollsters, the Literary Digest in particular, utilizing haphazard polling techniques failed to predict the winner. Toward the end of the 19th century, scholars proposed the use of “scientific” sampling. As pollsters turned to quota sampling, governmental researchers and academicians turned to probabil- ity sampling in their research. Probability sampling became the preferred technique in 1948 when pollsters utilizing quota sampling failed to predict the winner of the U.S. presidential election that year. The sampling process has six major steps: Step 1. Prepare to make sampling choices. Step 2. Choose between taking a census and sampling. Step 3. Choose nonprobability, probability, or mixed-methods sample design. Step 4. Choose the type of nonprobability, probability, or mixed-methods sample design. Step 5. Determine the sample size. Step 6. Select the sample. Proper preparation should be made in the first step so that one has the necessary infor- mation to effectively carry out the subsequent steps. One should clearly understand the objectives of the study, definition of the population, content of the population, size of the population, heterogeneity of the population, accessibility of the population, spatial distri- bution of the population, destructibility of the population, availability of resources, type of research design to be employed, and relevant ethical and legal considerations. REVIEW QUESTIONS
- What are the major milestones in the history of sampling?
- What should contemporary political pollsters learn from the problems political poll- sters had in predicting the winner of the 1936 and the 1948 presidential elections?
- What are the major steps in selecting a sample?
20 Sampling Essentials
- What are the major types of research objectives? Why should the objectives of a study be clarified before making sampling choices?
- What are the characteristics of a good definition of a target population? Give examples.
- In preparing to make sampling choices, why is it important to consider the target population’s (a) content, (b) size, (c) heterogeneity, (d) accessibility, (e) spatial distribu- tion, and (f) destructibility?
- Prior to making sampling choices, what steps should one take in determining the availability of resources to plan and execute a sample design?
- Suppose you were charged with developing a sample design for a study of the prob- lems children who are homeless have in completing their school assignments. Describe in detail the formative research that you would conduct to prepare for constructing your sample design.
- What are the major types of research designs, and how do they differ in terms of the preparation that should be done for making sampling choices?
- How might the sampling requirements of the following pairs of research designs tend to differ from each other? a. Qualitative versus quantitative research b. Experimental versus nonexperimental research c. Cross-sectional versus longitudinal research
- What ethical and legal factors should be considered in making sampling choices?
- It is often difficult to obtain a good sampling frame for a population-based study covering a large geographical area. Would a list of driver’s licenses serve as a good sampling frame? Why or why not? Once you have answered these questions, consider: Lynch, C. F. et al., “The Driver’s License List as a Population-Based Sampling Frame in Iowa” (1994). KEY TERMS Define and give examples of the following concepts: census cohort study coverage bias cross-sectional research design evaluation research experimental research
