18: Advanced Variance
18.1 Adding Complexity to Variance
Basic statistical tests are effective for analyzing simple group differences, but real-world research questions often involve more complexity. Researchers may be interested in how multiple factors interact, how external variables influence outcomes, or how different outcomes are affected simultaneously. In such cases, more advanced methods are required, such as those that can test multiple variables at once, control for confounding influences, and uncover patterns that simpler comparisons might miss. These advanced approaches extend the logic of basic variance analysis, offering a more comprehensive view of the data and allowing for richer interpretation of results.
18.2 Factorial ANOVA
Factorial ANOVA is an extension of One-Way ANOVA that allows researchers to examine the effects of two or more independent variables (factors) on a single dependent variable. In addition to testing the main effects of each factor, it also evaluates interaction effects, which occur when the impact of one factor depends on the level of another.
This method is preferred over running multiple One-Way ANOVAs when a study involves more than one independent variable. Conducting separate tests ignores potential interactions and increases the risk of Type I error due to repeated comparisons. By analyzing all factors within a single model, Factorial ANOVA reveals both individual effects and how variables may interact to influence the outcome. This makes it a more efficient and informative approach for examining complex relationships.
Assumptions
Factorial ANOVA relies on several key assumptions to ensure valid results. The dependent variable should be continuous and measured at the interval or ratio level. The independent variables should consist of two or more categorical groups, and the observations must be independent, meaning that each participant or case appears in only one group. Each group should have approximately normally distributed data, particularly when sample sizes are small. Additionally, the assumption of homogeneity of variances must be met, which means the spread of scores should be similar across all groups.
How To: Factorial ANOVA
To run the Factorial ANOVA in Jamovi, go to the Analyses tab, select ANOVA, then ANOVA.
-
Move one interval variable into the Dependent Variables box.
-
Move at least two nominal variables into the Fixed Factors box.
- Under Model Fit, select Overall model test.
- Under Effect Size, select η² (eta squared).
- Under the Assumption Checks drop-down, select Homogeneity test, Normality test, and Q-Q plot.
- Under the Estimated Marginal Means drop-down, move all variables in the left box to the Marginal Means box (this will analyze the interaction effects).
- In the Marginal Means box, select the Add New Term button and add each variable as a separate term (this will analyze the main effects).
-
Under Output, select Marginal Means Plots and Marginal Means Tables.
- Optional: If the Factorial ANOVA result is significant, select all variables in the left box and move them to the right box under the Post-Hoc Tests drop-down and check the Cohen’s d box.
TIP: You can use this test to run a Fisher’s One-Way ANOVA to generate effect sizes.
Phrasing Results: Factorial ANOVA
Use this template to phrase significant results:
- A [number of IVs]-way Factorial ANOVA was conducted to compare [DV] with the interaction between [IV1] and [IV2] (and additional IVs, as applicable).
- A significant model was found (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
- A significant interaction was observed (F([interaction df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
- A main effect was found for [IV1] (F([IV1 df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
- A main effect was also found for [IV2] (F([IV2 df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
Use this template to phrase the post-hoc results:
- A Tukey post-hoc test was conducted to determine the nature of the mean differences between groups of [IV].
- This analysis revealed that pairwise comparisons between [IV group 1] and [IV group 2] (ΔM = [mean difference], p < [approximate p-value], d = [Cohen’s d]) were significantly different.
NOTE: Only include the post-hoc results if the Factorial ANOVA Test produces at least a significant main effect result.
Use this template to phrase non-significant results:
- A [number of IVs]-way Factorial ANOVA was conducted to compare [DV] with the interaction between [IV1] and [IV2] (and additional IVs, as applicable).
- No significant model was found (F([model df], [residuals df]) = [F statistic], p = [p-value]).
- No significant interaction was found (F([interaction df], [residuals df]) = [F statistic], p = [p-value]).
- Additionally, no main effects were found for [IV1] (F([IV1 df], [residuals df]) = [F statistic], p = [p-value]) or [IV2] (F([IV2 df], [residuals df]) = [F statistic], p = [p-value]).
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
TIP: Follow the template to note the main effect results for additional IVs.
TIP: Use a mix of the phrasing language from the templates to match your results if you have a mix of significant and non-significant results within your model.
18.3 ANCOVA
ANCOVA (Analysis of Covariance) is a combination of ANOVA and regression analysis used to compare the means of different groups while controlling for the effects of one or more continuous covariates. By adjusting the dependent variable for the influence of covariates, ANCOVA provides a clearer understanding of the relationship between the independent and dependent variables.
It is preferred when researchers want to compare group means while accounting for the influence of continuous variables that could affect the outcome. This approach is especially useful when groups differ slightly on baseline characteristics, and those differences need to be statistically controlled. Including covariates improves the precision of group comparisons by reducing unexplained variance in the dependent variable.
Assumptions
ANCOVA relies on several key assumptions. The dependent variable should be continuous and approximately normally distributed within each group. The independent variable should be categorical with two or more levels, and the covariate should be continuous and linearly related to the dependent variable. Observations must be independent. The relationship between the covariate and the dependent variable should be consistent across all levels of the independent variable, an assumption known as homogeneity of regression slopes. Additionally, the variance of the dependent variable should be roughly equal across groups (e.g., homogeneity of variance).
How To: ANCOVA
To run the ANCOVA in Jamovi, go to the Analyses tab, select ANOVA, then ANCOVA.
-
Move one interval variable into the Dependent Variables box.
-
Move one nominal variable into the Fixed Factors box.
- Move one interval variable into the Covariates box.
- Under Model Fit, select Overall model test.
- Under Effect Size, select η²p (partial eta squared).
- Under the Assumption Checks drop-down, select Homogeneity test, Normality test, and Q-Q plot.
- Under the Estimated Marginal Means drop-down, move all variables in the left box to the Marginal Means box.
-
Under Output, select Marginal Means Plots and Marginal Means Tables.
- Optional: If the ANCOVA result is significant, select all variables in the left box and move them to the right box under the Post-Hoc Tests drop-down and check the Cohen’s d box.
TIP: You can use this test to run a Factorial ANCOVA.
Phrasing Results: ANCOVA
Use this template to phrase significant results:
- A one-way ANCOVA was conducted to compare [DV] with [IV] while accounting for [CV].
- A significant model was found (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
- [CV] significantly contributed to the model (F([CV df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η²p = [partial eta squared statistic]).
- Additionally, [IV] significantly contributed to the model (F([IV df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η²p = [partial eta squared statistic]).
Use this template to phrase the post-hoc results:
- A Tukey post-hoc test was conducted to determine the nature of the mean differences between groups of [IV].
- This analysis revealed that pairwise comparisons between [IV group 1] and [IV group 2] (ΔM = [mean difference], p < [approximate p-value], d = [Cohen’s d]) were significantly different.
NOTE: Only include the post-hoc results if the ANCOVA Test produces a significant result.
Use this template to phrase non-significant results:
- A one-way ANCOVA was conducted to compare [DV] with [IV] while accounting for [CV].
- No significant model was found (F([model df], [residuals df]) = [F statistic], p = [p-value]).
- Additionally, neither [CV] (F([CV df], [residuals df]) = [F statistic], p = [p-value]) nor [IV] (F([IV df], [residuals df]) = [F statistic], p = [p-value]) significantly contributed to the model.
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
TIP: Use a mix of the phrasing language from the templates to match your results if you have a mix of significant and non-significant results within your model.
18.4 MANOVA
MANOVA (Multivariate Analysis of Variance) is an extension of ANOVA used when two or more dependent variables are correlated. It assesses whether group differences exist across a combination of dependent variables, rather than evaluating each one separately. MANOVA is especially useful when the dependent variables are conceptually related and when researchers want to understand whether the independent variable(s) influence the overall multivariate outcome. It also allows for the analysis of interaction effects between independent variables.
MANOVA is preferred when a study includes multiple dependent variables that are conceptually or statistically related. Running separate ANOVAs for each dependent variable increases the risk of Type I error and ignores potential relationships between the outcomes. MANOVA evaluates group differences across all dependent variables simultaneously, accounting for their intercorrelations. This provides a more comprehensive understanding of how independent variables affect the combined outcome and allows researchers to detect patterns that might be missed when analyzing each dependent variable in isolation.
Assumptions
MANOVA shares many of the assumptions of other ANOVA tests but includes additional requirements due to the presence of multiple dependent variables. The dependent variables should be continuous and moderately correlated with one another. The independent variable should be categorical with two or more groups, and observations must be independent. Each combination of groups should have multivariate normality, meaning the set of dependent variables is normally distributed within each group. Additionally, MANOVA assumes homogeneity of covariance matrices across groups, which means the relationships among the dependent variables should be similar in each group.
How To: MANOVA
To run the MANOVA in Jamovi, go to the Analyses tab, select ANOVA, then MANCOVA.
-
Move at least two interval variables into the Dependent Variables box.
-
Move one nominal variable into the Factors box.
-
Under Multivariate Statistics, select Wilks’ Lambda and Pillai’s Trace (needed if you fail the multivariate homogeneity assumption).
- Under Assumption Checks, select Box’s M test, Shapiro-Wilk test, and Q-Q plot of multivariate normality.
TIP: You can use this test to run a Factorial MANCOVA.
NOTE: Jamovi’s MANOVA Test does not calculate post-hoc tests or effect size.
Phrasing Results: MANOVA
Use this template to phrase significant results:
- A one-way MANOVA was conducted to compare [DV1] and [DV2] with [IV].
- A significant model was found (λ([df1], [df2]) = [Wilks’ Lambda statistic], p < [approximate p-value]).
- The follow-up univariate test for [DV1] was significant (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
- The follow-up univariate test for [DV2] was also significant (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
Use this template to phrase non-significant results:
- A one-way MANOVA was conducted to compare [DV1] and [DV2] with [IV].
- No significant model was found (λ([df1], [df2]) = [Wilks’ Lambda statistic], p = [p-value]).
- The follow-up univariate tests for [DV1] (F([model df], [residuals df]) = [F statistic], p = [p-value]) and [DV2] (F([model df], [residuals df]) = [F statistic], p = [p-value]) were also not significant.
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
NOTE: Replace Wilks’ Lambda (λ) with Pillai’s Trace (V) if you fail the multivariate homogeneity assumption.
18.5 Combining Tests: Factorial MANCOVA
In complex research designs, combining different types of analysis can lead to a deeper understanding of the data. Factorial MANCOVA integrates elements of both Factorial ANOVA and ANCOVA, enabling researchers to examine the effects of multiple independent variables while simultaneously controlling for covariates and analyzing multiple dependent variables. This method is especially useful when the research involves interactions, continuous control variables, and multiple outcomes that are conceptually or statistically related.
Factorial MANCOVA is particularly useful when researchers anticipate interactions between factors, need to control for pre-existing differences, and aim to evaluate how multiple outcomes are influenced collectively. It is commonly applied in fields where outcomes are multidimensional and influenced by both experimental and contextual variables.
Because this analysis involves multiple assumptions (such as multivariate normality, homogeneity of variance-covariance matrices, and homogeneity of regression slopes) it requires careful data screening and a sufficiently large sample size to ensure reliable results.
Chapter 18 Summary and Key Takeaways
Several advanced variance tests help researchers analyze complex research questions involving multiple variables. Factorial ANOVA examines the effects of two or more independent variables on a dependent variable and tests for interaction effects. ANCOVA adjusts for the influence of continuous covariates while comparing group means, helping isolate the effect of the independent variable. MANOVA extends ANOVA to multiple dependent variables, assessing whether groups differ across a combined set of outcomes. Factorial MANCOVA combines a factorial design with covariate adjustment and multiple dependent variables, offering a comprehensive analysis of complex, multidimensional data. All of these analyses can be conducted in Jamovi using clear menus and options that guide users through model setup, assumption checks, and interpretation.
- Factorial ANOVA examines the effects of multiple independent variables and their interactions on a single dependent variable.
- ANCOVA adjusts for covariates to improve the accuracy of group comparisons.
- MANOVA analyzes multiple dependent variables simultaneously to determine how groups differ on the overall multivariate outcome.
- Factorial MANCOVA combines factorial analysis with covariate control and multiple outcomes for a more comprehensive understanding of group differences.
