class: center, middle, inverse, title-slide # Defining and Detecting Multicollinearity ### STAT 021 with Prof Suzy ### Swarthmore College --- <style type="text/css"> pre { background: #FFBB33; max-width: 100%; overflow-x: scroll; } .scroll-output { height: 70%; overflow-y: scroll; } .scroll-small { height: 50%; overflow-y: scroll; } .red{color: #ce151e;} .green{color: #26b421;} .blue{color: #426EF0;} </style> ## Multicollinearity There are a lot of important c-words in statistics. Especially confusing can be the following: collinearity, correlation, and covariance. But, somewhat intuitively, there is a relationship among these three terms. Recall the definitions of correlation and covariance for any two random variables, say, `\(X\)` and `\(Y\)`: `$$Cov(X,Y) = E[(X -E[X])(Y-E[Y])] = \dots = E[XY] - E[X]E[Y]$$` `$$Cor(X,Y) = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$$` -- Also, recall the fact that .blue[if `\(X\)` and `\(Y\)` are independent, then `\(Cov(X,Y)=0\)` and therefore `\(X\)` and `\(Y\)` are uncorrelated.] **But** if `\(X\)` and `\(Y\)` are uncorrelated, then it is still possible for `\(X\)` and `\(Y\)` to be dependent or independent. --- ## Multicollinearity So now we see that correlation is just a standardized version of the covariance between two variables. (Standardized in the sense that it will always be between the interval `\([-1,1]\)`.) **Q:** What is multicollinearity? -- This is a term specific to MLR models that describes the statistical phenomenon in which two or more predictor variables are highly correlated with each other. In other words, this means that two predictor variables, say, `\(x_1\)` and `\(x_2\)`, are collinear if `\(Cor(x_1, x_2) \approx \pm 1\)`. - This means that we could predict the values of one from the other in a SLR model! - This is **not** a problem for .blue[estimation] or .blue[prediction] with a MLR. - This **is** a problem however that inflates the estimated variances of our regression coefficients and therefore the individual test of significant slope parameters. Thus it can be a problem for .red[inference].^[5] - Outliers can have a big impact on the collinearity of a pair of variables.^[5] --- ## Multicollinearity Example ### Public School SAT data .scroll-output[Again let's investigate the MLR model with four predictor variables that we built to predict SAT scores of public schools. Does it seem like any of the predictor variables might be correlated with each other? ```r SAT_data <- read_table2("Data/sat_data.txt", col_names=FALSE, cols(col_character(), col_double(), col_double(), col_double(), col_double(), col_double(), col_double(), col_double())) colnames(SAT_data) = c("State", "PerPupilSpending", "StuTeachRatio", "Salary", "PropnStu", "SAT_verbal", "SAT_math", "SAT_tot") SAT_data <- SAT_data %>% mutate(prop_taking_SAT = PropnStu/100) %>% select(-PropnStu) SAT_data %>% select(c(PerPupilSpending, StuTeachRatio, Salary, prop_taking_SAT)) %>% plot ``` <!-- --> ] --- ## Multicollinearity Example ### Public School SAT data .scroll-output[Again let's investigate the MLR model with four predictor variables that we built to predict SAT scores of public schools. Does it seem like any of the predictor variables might be correlated with each other? ```r MLR_SAT4 <- lm(SAT_tot ~ PerPupilSpending + StuTeachRatio + Salary + prop_taking_SAT, data = SAT_data) summary(MLR_SAT4) ``` ``` ## ## Call: ## lm(formula = SAT_tot ~ PerPupilSpending + StuTeachRatio + Salary + ## prop_taking_SAT, data = SAT_data) ## ## Residuals: ## Min 1Q Median 3Q Max ## -90.531 -20.855 -1.746 15.979 66.571 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 1045.972 52.870 19.784 < 2e-16 *** ## PerPupilSpending 4.463 10.547 0.423 0.674 ## StuTeachRatio -3.624 3.215 -1.127 0.266 ## Salary 1.638 2.387 0.686 0.496 ## prop_taking_SAT -290.448 23.126 -12.559 2.61e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 32.7 on 45 degrees of freedom ## Multiple R-squared: 0.8246, Adjusted R-squared: 0.809 ## F-statistic: 52.88 on 4 and 45 DF, p-value: < 2.2e-16 ``` ] --- ## Multicollinearity Example ### Public School SAT data We can estimate the correlations among each pair of variables (without risking multiple testing issues): .scroll-small[ ```r cor(SAT_data$PerPupilSpending, SAT_data$StuTeachRatio) ``` ``` ## [1] -0.3710254 ``` ```r cor(SAT_data$PerPupilSpending, SAT_data$Salary) ``` ``` ## [1] 0.8698015 ``` ```r cor(SAT_data$PerPupilSpending, SAT_data$prop_taking_SAT) ``` ``` ## [1] 0.5926274 ``` ```r cor(SAT_data$StuTeachRatio, SAT_data$Salary) ``` ``` ## [1] -0.001146081 ``` ```r cor(SAT_data$StuTeachRatio, SAT_data$prop_taking_SAT) ``` ``` ## [1] -0.2130536 ``` ```r cor(SAT_data$Salary, SAT_data$prop_taking_SAT) ``` ``` ## [1] 0.6167799 ``` Note: The estimated correlation between two variables is not affected by whether or not those variables are standardized because correlation itself is already standardized. ] --- ## Multicollinearity When we see evidence of collinear predictor variables, it's a good idea to re-visit which variables you want to include in the model .red[because] collinearity affects the variance of our predictors and therefore affects the conclusion of the individual t-tests!^[4] **Q:** Do we need to be concerned about multiple testing with interaction terms? with collinearity? -- **A:** We do need to keep an eye on multiple tests issues when we're determining which main effects and interaction effects to include in the model (these are questions of statistical significance). The problem of multiple testing is not a concern however, in estimation/prediction problems therefore it is not a major concern when checking for collinearity (all we're doing is getting estimates for different correlations). **Note:** Interaction terms are structurally collinear, which we can't do much about. --- ## Multicollinearity ### Remedies A couple of simple remedies for collinearity are - Try collecting more data to see if the collinearity is dues to insufficient data; - Reduce the number of variables in your model so that none of them are collinear; - Other regression methods such as *ridge regression* (you don't need to know what this is for my class).^[5] --- ## Multicollinearity ### Public School SAT data Finally we can get a better model for the SAT data by adjusting which variables we include: .scroll-small[ ``` ## ## Call: ## lm(formula = SAT_tot ~ PerPupilSpending + StuTeachRatio + prop_taking_SAT, ## data = SAT_data) ## ## Residuals: ## Min 1Q Median 3Q Max ## -92.284 -21.130 1.414 16.709 66.073 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 1035.474 50.316 20.580 <2e-16 *** ## PerPupilSpending 11.014 4.452 2.474 0.0171 * ## StuTeachRatio -2.028 2.207 -0.919 0.3629 ## prop_taking_SAT -284.912 21.548 -13.222 <2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 32.51 on 46 degrees of freedom ## Multiple R-squared: 0.8227, Adjusted R-squared: 0.8112 ## F-statistic: 71.16 on 3 and 46 DF, p-value: < 2.2e-16 ``` ] --- ## Interaction terms and multicollinearity Broadly speaking, there are two main types of collinearity: 1. Structural multicollinearity which occurs when we create a model term using other terms in the model (e.g. including interaction terms). This is a (unfortunately unavoidable) byproduct of the model that we specify. 2. Data multicollinearity which occurs when the data itself for different predictor variables are highly correlated. Some ways we can address this data-inherent multicollinearity is to collect more data or see if it makes sense to drop a variable from the model. .footnote[Source: https://statisticsbyjim.com/regression/multicollinearity-in-regression-analysis/] --- ## Multicollinearity ### Some things to think about - What are the effects of incorporating interaction terms in your linear model? - What are the effects of severe multicollinearity among some predictor variables in your model? *** ### Reading along in your textbook Chapter 3 Sections 10 and 11 and Chapter 9