Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Community
Ask the community for help and clear up your study doubts
Discover the best universities in your country according to Docsity users
Free resources
Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors
Stepwise versus Hierarchical Regression, 2. Introduction. Multiple regression is commonly used in social and behavioral data analysis (Fox, 1991; Huberty, ...
Typology: Lecture notes
Uploaded on 09/27/2022
1 / 30
This page cannot be seen from the preview
Don't miss anything!
Running head: Stepwise versus Hierarchal Regression Stepwise versus Hierarchical Regression: Pros and Cons Mitzi Lewis University of North Texas Paper presented at the annual meeting of the Southwest Educational Research Association, February 7, 2007, San Antonio.
Introduction Multiple regression is commonly used in social and behavioral data analysis (Fox, 1991; Huberty, 1989). In multiple regression contexts, researchers are very often interested in determining the “best” predictors in the analysis. This focus may stem from a need to identify those predictors that are supportive of theory. Alternatively, the researcher may simply be interested in explaining the most variability in the dependent variable with the fewest possible predictors, perhaps as part of a cost analysis. Two approaches to determining the quality of predictors are (1) stepwise regression and (2) hierarchical regression. This paper will explore the advantages and disadvantages of these methods and use a small SPSS dataset for illustration purposes. Stepwise Regression Stepwise methods are sometimes used in educational and psychological research to evaluate the order of importance of variables and to select useful subsets of variables (Huberty, 1989; Thompson, 1995). Stepwise regression involves developing a sequence of linear models that, according to Snyder (1991), can be viewed as a variation of the forward selection method since predictor variables are entered one at a
positively satanic in their temptation toward Type I errors in this context” (p. 185 ). How are these degrees of freedom incorrectly calculated by software packages during stepwise regression? Essentially, stepwise regression applies an F test to the sum of squares at each stage of the procedure. Performing multiple statistical significance tests on the same data set as if no previous tests had been carried out can have severe consequences on the correctness of the resulting inferences. An appropriate analogy is given by Selvin and Stuart (1966): the fish which don’t fall through the net are bound to be bigger than those which do, and it is quite fruitless to test whether they are of average size. Not only will this alter the performance of all subsequent tests on the retained explanatory model – it may destroy unbiasedness and alter mean-square- error in estimation.” (p. 21) However, as noted by Thompson (1995), all applications of stepwise regression are “not equally evil regarding the inflation of Type I error” (p. 527). Examples include situations with (a) near zero sum of squares explained across steps, (b) small number of predictor variables, and/or (c) large sample size.
Best Predictor Set of a Prespecified Size The novice researcher may believe that the best predictor set of a specific size s will be selected by performing the same s number of steps of a stepwise regression analysis. However, stepwise analysis results are is dependent on the sampling error present in any given sample and can lead to erroneous results (Huberty, 1989; Licht, 1995; Thompson, 1995). Stepwise regression will typically not result in the best set of s predictors and could even result in selecting none of the best s predictors. Other subsets could result in a larger effect size and still other subsets of size s could yield nearly the same effect size. Why is this so? The predictor selected at each step of the analysis is conditioned on the previously included predictors and thus yields a “ situation-specific conditional answer in the context (a) only of the specific variables already entered and (b) only those variables used in the particular study but not yet entered” (Thompson, 1995, p. 528). The order of variable entry can be important. If any of the predictors are correlated with each other, the relative amount of variance in the criterion variable explained by each of the predictors can change “drastically” when the order of entry is changed (Kerlinger, 1986, p. 543). A predictor with a
A colleague of the present author noted that one could also imagine a different type of team being brought together to work on a common goal. For example, a team of the smartest people in an organization might be selected in a stepwise manner to produce a report of cutting edge research in their field. These highly intelligent people might be, for example, Professor B. T. Weight, Professor S. T. Coefficient, Professor E. F. Size, and Professor C. R. Lation. Although these people may be the most intelligent people in the organization, they may not be the group of people who could produce the best possible report if they do not work together well. Perhaps personality conflicts, varying philosophies, or egos might interfere with the group being able to work together effectively. It could be that using an all-possible-subsets approach, or a hierarchical regression approach (see subsequent discussion), would result in a totally different group of individuals since these approaches would also consider how different combinations of individuals work together as a team. This new team might then be the one that would produce the best possible report because they do not have the previously mentioned issues and as a result work together more successfully as a team. (Disclaimer: any
resemblance of these fictional team members to actual people is purely a coincidence.) Replicability Stepwise regression generally does not result in replicable conclusions due to its dependence on sampling error (Copas, 1983; Fox, 1991; Gronnerod, 1006; Huberty, 1989; Menard, 1995; Pedhazur, 1991; Thompson, 1995 ). As stated by Menard (1995), the use of stepwise procedures “capitalizes on random variations in the data and produces results that tend to be idosyncratic and difficult to replicate in any sample other than the sample in which they were originally obtained" (p. 54) and therefore results should be regarded as “inconclusive” (p. 57). As variable determinations are made at each step, there may be instances in which one variable is chosen over another due to a small difference in predictive ability. This small difference, which could be due to sampling error, impacts each subsequent step. Thompson (1995) likens these linear- series decisions to decisions that are made when working through a maze. Once a decision is made to turn one way instead of another, a whole sequence of decisions (and therefore results) are no longer possible. This difficulty of sampling error, and thus the possible impact of sampling error on the analysis, could be
choosing order of variable entry, there is also “no substitute for depth of knowledge of the research problem... the research problem and the theory behind the problem should determine the order of entry of variables in multiple regression analysis” (p. 545). Stated another way by Fox (1991), ”mechanical model- selection and modification procedures... generally cannot compensate for weaknesses in the data and are no substitute for judgment and thought” (p. 21). Simply put, “the data analyst knows more than the computer” (Henderson & Velleman, 1981, p. 391). Hierarchical regression is an appropriate tool for analysis when variance on a criterion variable is being explained by predictor variables that are correlated with each other (Pedhazur, 1997). Since correlated variables are commonly seen in social sciences research and are especially prevalent in educational research, this makes hierarchical regression quite useful. Hierarchical regression is a popular method used to analyze the effect of a predictor variable after controlling for other variables. This “control” is achieved by calculating the change in the adjusted R^2 at each step of the analysis, thus accounting for the increment in variance after each
variable (or group of variables) is entered into the regression model (Pedhazur, 1997). Just a few recent examples of hierarchical regression analysis use in research include:
many software packages in stepwise regression analysis do not correctly reflect the number of statistical tests that have been made to arrive at the resulting model; instead the degrees of freedom are under calculated. Thus, statistical significance levels displayed in hierarchical regression output are correct and statistical significance levels displayed in stepwise regression output are inflated, resulting in inflated chances for Type I errors. Best Predictor Set of a Prespecified Size Hierarchical regression analysis involves choosing a best predictor set interactively between computer and the researcher. The order of variable entry is determined by the researcher before the analysis is conducted. In this manner, decisions are based on theory and research instead of being made arbitrarily, in blind automation, by the computer (as they are in stepwise regression; Henderson & Vellman, 1981). Replicability Like stepwise regression, hierarchical regression is also subject to problems associated with sampling error. However, the likelihood of these problems is reduced by interaction of the researcher with the data. For example, instead of one variable being chosen over another variable due to a small difference in predictive ability, the order
of variable entry is chosen by the researcher. Thus, results from an arbitrary decision that is more likely to reflect sampling error (in the case of stepwise regression) are instead results based on researcher expertise (in the case of hierarchical regression). Of course, remaining sampling error can still be estimated via cross-validation or other techniques. And again, sampling error will be less of an issue the larger the sample size and effect size, and the fewer the predictor variables. Heuristic SPSS Example Stepwise Regression As previously discussed, stepwise regression involves developing a sequence of linear models through variable entry as determined by computer algorithms. A heuristic SPSS dataset has been constructed (Appendix A) and will be analyzed for illustration purposes. Syntax is provided in Appendix B. Stepwise regression was used to regress mother’s education level (ma_ed), father’s education level (fa_ed), parent’s income (par_inc), and faculty interaction level (fac_int) on years to graduation (years_grad). Inspection of correlations between the variables (Table 1) reveal (a) that mother’s education, parent’s income, and faculty interaction are all highly correlated with years to
Table 2 Stepwise Regression Summary Table Model Sum of Squares df Mean Square F Sig. 1 Regression 27.829 1 27.829 178.356 0. Residual 12.171 78 0. Total 40.000 79 2 Regression 31.535 2 15.767 143.423 0. Residual 8.465 77 0. Total 40.000 79 3 Regression 34.992 3 11.664 177.027 0. Residual 5.008 76 0. Total 40.000 79 4 Regression 36.723 4 9.181 210.146 0. Residual 3.277 75 0. Total 40.000 79 5 Regression 36.659 3 12.220 277.951 0. Residual 3.341 76 0. Total 40.000 79 Model Summary Model R R Square Adjusted R Square Std. Error of the Estimate 1 0.834 0.696 0.692 0. 2 0.888 0.788 0.783 0. 3 0.935 0.875 0.870 0. 4 0.958 0.918 0.914 0. 5 0.957 0.916 0.913 0. a Predictors: (Constant), Interaction with Faculty b Predictors: (Constant), Interaction with Faculty, Mothers Education Level c Predictors: (Constant), Interaction with Faculty, Mothers Education Level, Parents Income d Predictors: (Constant), Interaction with Faculty, Mothers Education Level, Parents Income, Fathers Education Level e Predictors: (Constant), Mothers Education Level, Parents Income, Fathers Education Level f Dependent Variable: years_grad Second, the predictor variable that has the highest R with the criterion variable, faculty interaction (fac_int),
is the first variable entered into the analysis. However, the final model of the analysis (model 5/e) does not include the faculty interaction variable. Thus, stepwise regression egregiously results in a model that does not include the predictor variable that has the highest correlation with the criterion variable. Because the significance tests displayed in the output of the stepwise regression analysis do not approximate the probability that the resulting model will actually represent future samples, another method is needed to estimate replicability. Double cross-validation is performed to achieve this objective. The resulting double cross-validation coefficients are 0.999. Upon initial reflection, these findings may seem quite high, but in consideration of the unusually elevated R in these analyses (0.954 & 0.961), the findings are not so surprising. Had the R values been lower or had a larger number of predictor variables been included in the analysis, smaller double- cross validation coefficients would have been expected. Hierarchical Regression The dataset utilized to illustrate some of the concepts involved with stepwise regression can also be used to demonstrate hierarchical regression. Variable selection for the hierarchical regression analysis will be based on
Second, the model summary provides (a) the change in R^2 that occurred as a result of including the additional predictor variable (fac_int) in the model and (b) the statistical significance of the change in R^2. In the example provided, the additional variable only produced a very small change in R^2 and this change was not statistically significant. If the dataset had been actual data instead of fabricated data, the change in explained variance of years to graduation by level of student/faculty interaction would be expected to be larger and statistically significant. Table 3 Hierarchical Regression Summary Table Model Sum of Squares df Mean Square F Sig. 1 Regression 36.659 3 12.220 277.951 0. Residual 3.341 76 0. Total 40 79 2 Regression 36.723 4 9.181 210.146 0. Residual 3.277 75 0. Total 40 79 Model Summary Model R
Square Adjusted R Square Std. Error of the Estimate R Square Change Sig. F Change 1 0.957 0.916 0.913 0.210 0.916 0. 2 0.958 0.918 0.914 0.209 0.002 0. a Predictors: (Constant), Parents Income, Fathers Education Level, Mothers Education Level b Predictors: (Constant), Parents Income, Fathers Education Level, Mothers Education Level, Interaction with Faculty
Again, the adjusted R^2 would indicate that sampling error does not have much impact on the present scenario, probably because of the high effect size and the small number of predictor variables. If the effect size were lower and/or the number of predictor variables increased, the adjusted R^2 would probably provide a larger theoretical correction for these issues, and this correction could be further examined by cross-validation or other techniques. Conclusion Selecting the appropriate statistical tool for analysis is dependent upon the intended use of the analysis. As Pedhazur (1997) stated, Practical considerations in the selection of specific predictors may vary, depending on the circumstances of the study, the researcher’s specific aims, resources, and frame of reference, to name some. Clearly, it is not possible to develop a systematic selection method that would take such considerations into account. (p. 211) This rationale is in conflict with the automated, algorithm based analysis of stepwise regression. Nonetheless, there are still instances where stepwise regression has been recommended for use: in exploratory, predictive research (Menard, 1995). Even in this case, stepwise regression
