18: Advanced Variance
18.1 Adding Complexity to Variance
In applied research, variance analysis becomes more complex as researchers examine multiple independent variables, covariates, or multiple dependent variables simultaneously. Advanced variance tests, such as Factorial ANOVA, ANCOVA, and MANOVA, are powerful tools that allow researchers to investigate interactions between variables, control for extraneous factors, and assess the effects of multiple factors at once. These tests are helpful when researchers explore how more than one independent variable or covariate affects the dependent variable, or when multiple dependent variables are involved. This chapter focuses on the assumptions, interpretations, and applications of these tests and how they can be combined for more complex analyses. Additionally, we will discuss the options available for addressing violations of assumptions and how to handle those violations to maintain the integrity of your analysis.
18.2 Factorial ANOVA
Factorial ANOVA is an extension of the one-way ANOVA that allows researchers to examine the effects of two or more independent variables (factors) on a dependent variable. This test also enables the analysis of interaction effects, where the effect of one factor depends on the level of another factor.
Factorial ANOVA Assumptions
The assumptions for Factorial ANOVA include independence of observations (the groups being compared must be independent), normality (the dependent variable should be approximately normally distributed in each group), and homogeneity of variance (the variance of the dependent variable should be roughly equal across all levels of each factor). Additionally, the dependent variable should be continuous and measured on an interval or ratio scale.
If the assumptions for Factorial ANOVA are unmet, several options exist. For non-normality, a non-parametric alternative like the Kruskal-Wallis test (for comparing more than two groups) can be used. If there is a violation of homogeneity of variance, the Welch’s ANOVA method can be used, as it is a more robust alternative that does not assume equal variances.
How To: Factorial ANOVA
To run the Factorial ANOVA in Jamovi, go to the Analyses tab, select ANOVA, then ANOVA.
-
Move one interval variable into the Dependent Variables box.
-
Move at least two nominal variables into the Fixed Factors box.
- Under Model Fit, select Overall model test.
- Under Effect Size, select η² (eta squared).
- Under the Assumption Checks drop-down, select Homogeneity test, Normality test, and Q-Q plot.
- Under the Estimated Marginal Means drop-down, move all variables in the left box to the Marginal Means box (this will analyze the interaction effects).
- In the Marginal Means box, select the Add New Term button and add each variable as a separate term (this will analyze the main effects).
-
Under Output, select Marginal Means Plots and Marginal Means Tables.
- Optional: If the Factorial ANOVA result is significant, select all variables in the left box and move them to the right box under the Post-Hoc Tests drop-down and check the Cohen’s d box.
TIP: You can use this test to run a Fisher’s One-Way ANOVA to generate effect sizes.
Below is an example of the results generated when the steps are correctly followed.
IMAGE [INSERT NAME OF DATASET]
Interpreting the Results
Phrasing Results: Factorial ANOVA
Use this template to phrase significant results:
- A [number of IVs]-way Factorial ANOVA was conducted to compare [DV] with the interaction between [IV1] and [IV2] (and additional IVs, as applicable).
- A significant model was found (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
- A significant interaction was observed (F([interaction df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
- A main effect was found for [IV1] (F([IV1 df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
- A main effect was also found for [IV2] (F([IV2 df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
Use this template to phrase the post-hoc results:
- A Tukey post-hoc test was conducted to determine the nature of the mean differences between groups of [IV].
- This analysis revealed that pairwise comparisons between [IV group 1] and [IV group 2] (ΔM = [mean difference], p < [approximate p-value], d = [Cohen’s d]) were significantly different.
NOTE: Only include the post-hoc results if the Factorial ANOVA Test produces at least a significant main effect result.
Use this template to phrase non-significant results:
- A [number of IVs]-way Factorial ANOVA was conducted to compare [DV] with the interaction between [IV1] and [IV2] (and additional IVs, as applicable).
- No significant model was found (F([model df], [residuals df]) = [F statistic], p = [p-value]).
- No significant interaction was found (F([interaction df], [residuals df]) = [F statistic], p = [p-value]).
- Additionally, no main effects were found for [IV1] (F([IV1 df], [residuals df]) = [F statistic], p = [p-value]) or [IV2] (F([IV2 df], [residuals df]) = [F statistic], p = [p-value]).
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
TIP: Follow the template to note the main effect results for additional IVs.
TIP: Use a mix of the phrasing language from the templates to match your results if you have a mix of significant and non-significant results within your model.
18.3 ANCOVA (Analysis of Covariance)
ANCOVA is a blend of ANOVA and regression analysis that compares the means of different groups while controlling for the effects of one or more continuous covariates. ANCOVA helps adjust the dependent variable for the influence of covariates, offering a clearer understanding of the relationship between the independent and dependent variables.
ANCOVA Assumptions
The assumptions for ANCOVA include independence of observations (the groups must be independent), normality (the residuals, or differences between observed and predicted values, should be normally distributed), and homogeneity of variance (the variance in the dependent variable should be equal across all groups). There must also be a linear relationship between the dependent variable and the covariates, and the relationship between the covariates and the dependent variable should be the same across all groups (this assumption is tested using an interaction term between the group variable and the covariate).
How To: ANCOVA
To run the ANCOVA in Jamovi, go to the Analyses tab, select ANOVA, then ANCOVA.
-
Move one interval variable into the Dependent Variables box.
-
Move one nominal variable into the Fixed Factors box.
- Move one interval variable into the Covariates box.
- Under Model Fit, select Overall model test.
- Under Effect Size, select η²p (partial eta squared).
- Under the Assumption Checks drop-down, select Homogeneity test, Normality test, and Q-Q plot.
- Under the Estimated Marginal Means drop-down, move all variables in the left box to the Marginal Means box.
-
Under Output, select Marginal Means Plots and Marginal Means Tables.
- Optional: If the ANCOVA result is significant, select all variables in the left box and move them to the right box under the Post-Hoc Tests drop-down and check the Cohen’s d box.
TIP: You can use this test to run a Factorial ANCOVA.
Below is an example of the results generated when the steps are correctly followed.
IMAGE [INSERT NAME OF DATASET]
Interpreting the Results
Phrasing Results: ANCOVA
Use this template to phrase significant results:
- A one-way ANCOVA was conducted to compare [DV] with [IV] while accounting for [CV].
- A significant model was found (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
- [CV] significantly contributed to the model (F([CV df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η²p = [partial eta squared statistic]).
- Additionally, [IV] significantly contributed to the model (F([IV df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η²p = [partial eta squared statistic]).
Use this template to phrase the post-hoc results:
- A Tukey post-hoc test was conducted to determine the nature of the mean differences between groups of [IV].
- This analysis revealed that pairwise comparisons between [IV group 1] and [IV group 2] (ΔM = [mean difference], p < [approximate p-value], d = [Cohen’s d]) were significantly different.
NOTE: Only include the post-hoc results if the Factorial ANOVA Test produces at least a significant main effect result.
Use this template to phrase non-significant results:
- A one-way ANCOVA was conducted to compare [DV] with [IV] while accounting for [CV].
- No significant model was found (F([model df], [residuals df]) = [F statistic], p = [p-value]).
- Additionally, neither [CV] (F([CV df], [residuals df]) = [F statistic], p = [p-value]) nor [IV] (F([IV df], [residuals df]) = [F statistic], p = [p-value]) significantly contributed to the model.
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
TIP: Use a mix of the phrasing language from the templates to match your results if you have a mix of significant and non-significant results within your model.
18.4 MANOVA (Multivariate Analysis of Variance)
MANOVA is an extension of ANOVA used when two or more dependent variables are correlated. It assesses whether the means of multiple dependent variables are equal across the groups defined by the independent variable(s). MANOVA also tests for interaction effects between the independent and dependent variables.
MANOVA Assumptions
The assumptions for MANOVA include independence of observations (the groups should be independent), normality (the dependent variables should be multivariately normally distributed within each group), and homogeneity of variance-covariance (the variance-covariance matrices of the dependent variables should be equal across the groups, which can be tested with Box’s M test). Additionally, the dependent variables should be continuous and measured on an interval or ratio scale, and there should be linearity between the dependent variables.
How To: MANOVA
To run the MANOVA in Jamovi, go to the Analyses tab, select ANOVA, then MANCOVA.
-
Move at least two interval variables into the Dependent Variables box.
-
Move one nominal variable into the Factors box.
-
Under Multivariate Statistics, select Wilks’ Lambda and Pillai’s Trace (needed if you fail the multivariate homogeneity assumption).
- Under Assumption Checks, select Box’s M test, Shapiro-Wilk test, and Q-Q plot of multivariate normality.
TIP: You can use this test to run a Factorial MANCOVA.
NOTE: Jamovi’s MANOVA Test does not calculate post-hoc tests or effect size.
Below is an example of the results generated when the steps are correctly followed.
IMAGE [INSERT NAME OF DATASET]
Interpreting the Results
Phrasing Results: MANOVA
Use this template to phrase significant results:
- A one-way MANOVA was conducted to compare [DV1] and [DV2] with [IV].
- A significant model was found (λ([df1], [df2]) = [Wilks’ Lambda statistic], p < [approximate p-value]).
- The follow-up univariate test for [DV1] was significant (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
- The follow-up univariate test for [DV2] was also significant (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
Use this template to phrase non-significant results:
- A one-way MANOVA was conducted to compare [DV1] and [DV2] with [IV].
- No significant model was found (λ([df1], [df2]) = [Wilks’ Lambda statistic], p = [p-value]).
- The follow-up univariate tests for [DV1] (F([model df], [residuals df]) = [F statistic], p = [p-value]) and [DV2] (F([model df], [residuals df]) = [F statistic], p = [p-value]) were also not significant.
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
NOTE: Replace Wilks’ Lambda (λ) with Pillai’s Trace (V) if you fail the multivariate homogeneity assumption.
18.5 Combining Tests: Factorial MANCOVA
In complex research designs, combining different types of analysis can provide a deeper understanding of the data. For instance, Factorial MANCOVA combines Factorial ANOVA with ANCOVA, allowing researchers to assess the effects of multiple independent variables (factors) while controlling for covariates and multiple dependent variables simultaneously. This combined approach provides a more comprehensive analysis when dealing with complex data that includes interactions, covariates, and multiple outcomes.
18.6 Choosing the Right Test for Your Study
The choice of the correct statistical test depends on the research question, data characteristics, and the complexity of the study design. If the goal is to examine the effects of two or more independent variables on a dependent variable and assess any potential interactions, Factorial ANOVA is the appropriate test. If covariates need to be controlled for while comparing group means, ANCOVA should be used. If there are multiple dependent variables, MANOVA is the test of choice. For more complex designs, Factorial MANCOVA combines both factorial analysis and ANCOVA to analyze multiple dependent variables while controlling for covariates and examining multiple factors.
Chapter 18 Summary and Key Takeaways
In this chapter, we explored several advanced variance tests used in applied research, including Factorial ANOVA, which examines the effects of multiple independent variables on a dependent variable and tests for interaction effects; ANCOVA, which adjusts for the effects of covariates while comparing group means; and MANOVA, which extends ANOVA to multiple dependent variables to assess how groups differ on the overall set of dependent variables. We also discussed Factorial MANCOVA, which combines factorial design with ANCOVA to analyze multiple dependent variables while controlling for covariates. Additionally, we reviewed what to do when assumptions for these tests are not met, such as using Pillai’s Trace for MANOVA when homogeneity of variance-covariance is violated or applying data transformations to address normality issues.
- Factorial ANOVA examines the effects of multiple independent variables and their interactions on a dependent variable.
- ANCOVA adjusts for covariates to provide a clearer comparison of group means.
- MANOVA allows for the analysis of multiple dependent variables simultaneously to determine how independent variables affect the entire set of dependent variables.
- Factorial MANCOVA combines factorial analysis with ANCOVA to analyze multiple dependent variables while controlling for covariates.
- When assumptions are violated, Pillai’s Trace, data transformations, or bootstrapping can be used to achieve more robust results.