Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.
20: Advanced Variance
Chapter 20 Guiding Questions
How do multiple independent variables complicate interpretation?
What does an interaction effect represent conceptually?
How do covariates influence adjusted group comparisons?
What risks arise when advanced models are overinterpreted?
20.1 Adding Complexity to Variance
Basic statistical tests are effective for analyzing simple group differences, but real-world research questions often involve more complexity. Researchers may be interested in how multiple factors interact, how external variables influence outcomes, or how different outcomes are affected simultaneously. In such cases, more advanced methods are required, such as those that can test multiple variables at once, control for confounding influences, and uncover patterns that simpler comparisons might miss. These advanced approaches extend the logic of basic variance analysis, offering a more comprehensive view of the data and allowing for richer interpretation of results.
20.2 Factorial ANOVA
Factorial ANOVA is an extension of One-Way ANOVA that allows researchers to examine the effects of two or more independent variables (factors) on a single dependent variable. In addition to testing the main effects of each factor, it also evaluates interaction effects, which occur when the impact of one factor depends on the level of another.
This method is preferred over running multiple One-Way ANOVAs when a study involves more than one independent variable. Conducting separate tests ignores potential interactions and increases the risk of Type I error due to repeated comparisons. By analyzing all factors within a single model, Factorial ANOVA reveals both individual effects and how variables may interact to influence the outcome. This makes it a more efficient and informative approach for examining complex relationships.
Assumptions
Factorial ANOVA relies on several key assumptions to ensure valid results. The dependent variable should be continuous and measured at the interval or ratio level. The independent variables should consist of two or more categorical groups, and the observations must be independent, meaning that each participant or case appears in only one group. Each group should have approximately normally distributed data, particularly when sample sizes are small. Additionally, the assumption of homogeneity of variances must be met, which means the spread of scores should be similar across all groups.
How To: Factorial ANOVA
To run the Factorial ANOVA in Jamovi, go to the Analyses tab, select ANOVA, then ANOVA.
Move one interval variable into the Dependent Variables box.
Move at least two nominal variables into the Fixed Factors box.
Under Model Fit, select Overall model test.
Under Effect Size, select η² (eta squared).
Under the Assumption Checks drop-down, select Homogeneity test, Normality test, and Q-Q plot.
If the Factorial ANOVA result is significant, under the Post-Hoc Tests drop-down, select all variables in the left box and move them to the right box and check the Cohen’s d box.
Under the Estimated Marginal Means drop-down, highlight and move all variables in the left box to the Marginal Means box (this will analyze the interaction effects).
In the Marginal Means box, select the Add New Term button and add each variable as a separate term (this will analyze the main effects).
Under Output, select Marginal Means Plots and Marginal Means Tables.
TIP: You can use this test to run a Fisher’s One-Way ANOVA to generate effect sizes.
Understanding the Output
The output from the Factorial ANOVA is shown below. The screenshots separate the results for display purposes, but the full output appears in a single Jamovi output window when all test options are selected.
Figure 20.1a. Factorial ANOVA Results with Effect Sizes and Assumption Tests
Figure 20.1b. Factorial ANOVA Results with Post-Hoc Comparisons and Effect Sizes
Figure 20.1c. Factorial ANOVA Results with Post-Hoc Comparisons for the Interaction Effect
Figure 20.1d. Factorial ANOVA Results with Estimated Marginal Means and Confidence Interval Plot
Figure 20.1e. Factorial ANOVA Results with Estimated Marginal Means and Confidence Interval Plot
Figure 20.1f. Factorial ANOVA Results with Interaction Estimated Marginal Means and Confidence Interval Plot
To interpret a Factorial ANOVA in Jamovi, begin with the ANOVA table. The F statistic evaluates whether there are statistically significant differences in the dependent variable associated with each factor and their interaction. The p-value indicates whether the variability explained by a factor is greater than would be expected by chance. If the p-value is below the selected alpha level, you conclude that the factor has a statistically significant effect on the dependent variable.
Next, examine the main effects. Each main effect tests whether group means differ when averaging across the levels of the other factor. A significant main effect indicates that at least one group mean differs from another. However, if an interaction is also significant, main effects should be interpreted cautiously because the effect of one variable depends on the level of the other variable.
Then review the interaction term. The interaction tests whether the relationship between one factor and the dependent variable changes across levels of the second factor. A statistically significant interaction suggests that the combined influence of the two variables is not simply additive. When an interaction is present, interpretation should shift toward examining simple effects and estimated marginal means rather than focusing solely on main effects.
After evaluating statistical significance, examine the effect size (η²). This statistic represents the proportion of variance in the dependent variable explained by each factor and the interaction. As a general guideline, values around .01 indicate a small effect, around .06 a moderate effect, and .14 or higher a large effect. Larger values suggest that the factor explains a greater proportion of variability in the dependent variable.
Next, review the assumption checks. Levene’s test evaluates homogeneity of variances. A non-significant result suggests that group variances are sufficiently similar and that the Factorial ANOVA is appropriate. The Shapiro–Wilk test and Q–Q plot evaluate normality of residuals. A non-significant result and approximately linear Q–Q plot suggest that the normality assumption is reasonably satisfied.
If a main effect or interaction is statistically significant, proceed to the post hoc tests. Tukey-adjusted comparisons examine differences between specific pairs of groups while controlling for Type I error. Significant comparisons identify which groups differ from one another. The mean difference indicates the direction and magnitude of the difference, and Cohen’s d provides a standardized measure of practical significance.
Finally, review the estimated marginal means and their confidence intervals. These adjusted means account for the other variables in the model and are especially important when interpreting interactions. Confidence intervals that show minimal overlap suggest meaningful group differences. Together, these elements provide both statistical and practical insight into how the factors relate to the dependent variable.
Phrasing Results: Factorial ANOVA
Use this template to phrase significant results:
A [number of IVs]-way Factorial ANOVA was conducted to compare [DV] with the interaction between [IV1] and [IV2] (and additional IVs, as applicable).
A significant model was found (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
A significant interaction was observed (F([interaction df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
A main effect was found for [IV1] (F([IV1 df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
A main effect was also found for [IV2] (F([IV2 df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η² = [eta squared statistic]).
Use this template to phrase the post-hoc results:
A Tukey post-hoc test was conducted to determine the nature of the mean differences between groups of [IV].
This analysis revealed that pairwise comparisons between [IV group 1] and [IV group 2] (ΔM = [mean difference], p < [approximate p-value], d = [Cohen’s d]) were significantly different.
NOTE: Only include the post-hoc results if the Factorial ANOVA Test produces at least a significant main effect result.
Use this template to phrase non-significant results:
A [number of IVs]-way Factorial ANOVA was conducted to compare [DV] with the interaction between [IV1] and [IV2] (and additional IVs, as applicable).
No significant model was found (F([model df], [residuals df]) = [F statistic], p = [p-value]).
No significant interaction was found (F([interaction df], [residuals df]) = [F statistic], p = [p-value]).
Additionally, no main effects were found for [IV1] (F([IV1 df], [residuals df]) = [F statistic], p = [p-value]) or [IV2] (F([IV2 df], [residuals df]) = [F statistic], p = [p-value]).
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
TIP: Follow the template to note the main effect results for additional IVs.
TIP: Use a mix of the phrasing language from the templates to match your results if you have a mix of significant and non-significant results within your model.
20.3 ANCOVA
ANCOVA (Analysis of Covariance) is a combination of ANOVA and regression analysis used to compare the means of different groups while controlling for the effects of one or more continuous covariates. By adjusting the dependent variable for the influence of covariates, ANCOVA provides a clearer understanding of the relationship between the independent and dependent variables.
It is preferred when researchers want to compare group means while accounting for the influence of continuous variables that could affect the outcome. This approach is especially useful when groups differ slightly on baseline characteristics, and those differences need to be statistically controlled. Including covariates improves the precision of group comparisons by reducing unexplained variance in the dependent variable.
Assumptions
ANCOVA relies on several key assumptions. The dependent variable should be continuous and approximately normally distributed within each group. The independent variable should be categorical with two or more levels, and the covariate should be continuous and linearly related to the dependent variable. Observations must be independent. The relationship between the covariate and the dependent variable should be consistent across all levels of the independent variable, an assumption known as homogeneity of regression slopes. Additionally, the variance of the dependent variable should be roughly equal across groups (e.g., homogeneity of variance).
How To: ANCOVA
To run the ANCOVA in Jamovi, go to the Analyses tab, select ANOVA, then ANCOVA.
Move one interval variable into the Dependent Variables box.
Move one nominal variable into the Fixed Factors box.
Move one interval variable into the Covariates box.
Under Model Fit, select Overall model test.
Under Effect Size, select η²p (partial eta squared).
Under the Assumption Checks drop-down, select Homogeneity test, Normality test, and Q-Q plot.
Under the Estimated Marginal Means drop-down, move all variables in the left box to the Marginal Means box.
Under Output, select Marginal Means Plots and Marginal Means Tables.
Optional: If the ANCOVA result is significant, select all variables in the left box and move them to the right box under the Post-Hoc Tests drop-down and check the Cohen’s d box.
TIP: You can use this test to run a Factorial ANCOVA.
Understanding the Output
The output from the ANCOVA is shown below. The screenshots separate the results for display purposes, but the full output appears in a single Jamovi output window when all test options are selected.
Figure 20.2a. ANCOVA Results with Effect Sizes, Assumption Tests, and Q–Q Plot
Figure 20.2b. ANCOVA Results with Post-Hoc Comparisons with Effect Sizes
Figure 20.2c. ANCOVA Results with Estimated Marginal Means with Confidence Interval Plot
To interpret an ANCOVA in Jamovi, begin with the ANCOVA table. The F statistic for the independent variable evaluates whether group means on the dependent variable differ after statistically controlling for the covariate. The p-value indicates whether these adjusted group differences are statistically significant. If the p-value is below the selected alpha level, you conclude that the independent variable has a significant effect on the dependent variable after accounting for the covariate.
Next, examine the covariate. The F test for the covariate determines whether the covariate is significantly related to the dependent variable. A significant result indicates that the covariate explains meaningful variability in the dependent variable and that adjusting for it improves the accuracy of the group comparisons. In other words, ANCOVA removes variance associated with the covariate before evaluating differences between levels of the independent variable.
Then review the effect sizes (partial η²). Partial eta squared represents the proportion of variance in the dependent variable uniquely explained by each predictor in the model, after controlling for the other variables. As a general guideline, values around .01 suggest a small effect, around .06 a moderate effect, and .14 or higher a large effect. Larger values indicate stronger practical importance.
After evaluating statistical significance and effect sizes, review the assumption checks. Levene’s test assesses homogeneity of variances across levels of the independent variable. A non-significant result suggests this assumption is satisfied and ANCOVA is appropriate. The Shapiro–Wilk test and Q–Q plot assess normality of residuals. A non-significant Shapiro–Wilk test and approximately linear Q–Q plot suggest the residuals are reasonably normally distributed.
If the independent variable includes more than two levels and its adjusted effect is statistically significant, proceed to the post hoc comparisons. Tukey-adjusted comparisons examine pairwise differences between adjusted group means while controlling for Type I error. Significant comparisons indicate which specific groups differ after controlling for the covariate. The mean difference shows the direction and magnitude of the adjusted difference, and Cohen’s d provides a standardized measure of practical significance.
Finally, examine the estimated marginal means. These are the adjusted group means for each level of the independent variable after controlling for the covariate. They provide the clearest interpretation of group differences in ANCOVA because they represent means that have been statistically adjusted for the covariate. Confidence intervals indicate the precision of these adjusted estimates, and minimal overlap between intervals suggests meaningful differences between groups.
Phrasing Results: ANCOVA
Use this template to phrase significant results:
A one-way ANCOVA was conducted to compare [DV] with [IV] while accounting for [CV].
A significant model was found (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
[CV] significantly contributed to the model (F([CV df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η²p = [partial eta squared statistic]).
Additionally, [IV] significantly contributed to the model (F([IV df], [residuals df]) = [F statistic], p < [approximate p-value]), with a [size] practical effect (η²p = [partial eta squared statistic]).
Use this template to phrase the post-hoc results:
A Tukey post-hoc test was conducted to determine the nature of the mean differences between groups of [IV].
This analysis revealed that pairwise comparisons between [IV group 1] and [IV group 2] (ΔM = [mean difference], p < [approximate p-value], d = [Cohen’s d]) were significantly different.
NOTE: Only include the post-hoc results if the ANCOVA Test produces a significant result.
Use this template to phrase non-significant results:
A one-way ANCOVA was conducted to compare [DV] with [IV] while accounting for [CV].
No significant model was found (F([model df], [residuals df]) = [F statistic], p = [p-value]).
Additionally, neither [CV] (F([CV df], [residuals df]) = [F statistic], p = [p-value]) nor [IV] (F([IV df], [residuals df]) = [F statistic], p = [p-value]) significantly contributed to the model.
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
TIP: Use a mix of the phrasing language from the templates to match your results if you have a mix of significant and non-significant results within your model.
20.4 MANOVA
MANOVA (Multivariate Analysis of Variance) is an extension of ANOVA used when two or more dependent variables are correlated. It assesses whether group differences exist across a combination of dependent variables, rather than evaluating each one separately. MANOVA is especially useful when the dependent variables are conceptually related and when researchers want to understand whether the independent variable(s) influence the overall multivariate outcome. It also allows for the analysis of interaction effects between independent variables.
MANOVA is preferred when a study includes multiple dependent variables that are conceptually or statistically related. Running separate ANOVAs for each dependent variable increases the risk of Type I error and ignores potential relationships between the outcomes. MANOVA evaluates group differences across all dependent variables simultaneously, accounting for their intercorrelations. This provides a more comprehensive understanding of how independent variables affect the combined outcome and allows researchers to detect patterns that might be missed when analyzing each dependent variable in isolation.
Assumptions
MANOVA shares many of the assumptions of other ANOVA tests but includes additional requirements due to the presence of multiple dependent variables. The dependent variables should be continuous and moderately correlated with one another. The independent variable should be categorical with two or more groups, and observations must be independent. Each combination of groups should have multivariate normality, meaning the set of dependent variables is normally distributed within each group. Additionally, MANOVA assumes homogeneity of covariance matrices across groups, which means the relationships among the dependent variables should be similar in each group.
How To: MANOVA
To run the MANOVA in Jamovi, go to the Analyses tab, select ANOVA, then MANCOVA.
Move at least two interval variables into the Dependent Variables box.
Move one nominal variable into the Factors box.
Under Multivariate Statistics, select Wilks’ Lambda and Pillai’s Trace (needed if you fail the multivariate homogeneity assumption).
Under Assumption Checks, select Box’s M test, Shapiro-Wilk test, and Q-Q plot of multivariate normality.
TIP: You can use this test to run a Factorial MANCOVA.
NOTE: Jamovi’s MANOVA Test does not calculate effect size or additional post-hoc tests.
Understanding the Output
The output from the MANCOVA is shown below.
Figure 20.3. MANCOVA Results with Univariate Tests and Assumption Tests
To interpret a MANCOVA in Jamovi, begin with the Multivariate Tests table. Statistics such as Pillai’s Trace and Wilks’ Lambda evaluate whether the independent variable has a statistically significant effect on the combined set of dependent variables. These tests examine group differences across the multivariate outcome space rather than looking at each dependent variable separately. The p-value indicates whether the independent variable significantly influences the dependent variables collectively. If the p-value is below the selected alpha level, you conclude that there is a statistically significant multivariate effect.
Among the multivariate statistics, Pillai’s Trace is generally considered the most robust, especially when assumptions may be violated, while Wilks’ Lambda is commonly reported and widely recognized. When both statistics lead to the same conclusion, confidence in the multivariate finding is strengthened.
If the multivariate test is statistically significant, proceed to the Univariate Tests. These follow-up analyses examine each dependent variable individually to determine where the differences occur. The F statistic and associated p-value indicate whether the independent variable significantly affects each outcome variable separately. These tests help clarify which specific dependent variables contribute to the overall multivariate effect.
Next, review the assumption checks. Box’s M test evaluates the homogeneity of covariance matrices across groups. A non-significant result suggests that the assumption of equal covariance matrices is satisfied. If Box’s M is significant, interpretation should be more cautious, and Pillai’s Trace is typically preferred because it is more robust to violations of this assumption.
The Shapiro–Wilk multivariate normality test and the multivariate Q–Q plot assess whether the joint distribution of the dependent variables approximates multivariate normality. A non-significant Shapiro–Wilk test and a Q–Q plot showing points reasonably aligned with the reference line suggest that the assumption of multivariate normality is satisfied and MANCOVA is appropriate.
Phrasing Results: MANOVA
Use this template to phrase significant results:
A one-way MANOVA was conducted to compare [DV1] and [DV2] with [IV].
A significant model was found (λ([df1], [df2]) = [Wilks’ Lambda statistic], p < [approximate p-value]).
The follow-up univariate test for [DV1] was significant (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
The follow-up univariate test for [DV2] was also significant (F([model df], [residuals df]) = [F statistic], p < [approximate p-value]).
Use this template to phrase non-significant results:
A one-way MANOVA was conducted to compare [DV1] and [DV2] with [IV].
No significant model was found (λ([df1], [df2]) = [Wilks’ Lambda statistic], p = [p-value]).
The follow-up univariate tests for [DV1] (F([model df], [residuals df]) = [F statistic], p = [p-value]) and [DV2] (F([model df], [residuals df]) = [F statistic], p = [p-value]) were also not significant.
TIP: Replace the content inside the brackets with your variables and results, then remove the brackets.
NOTE: Replace Wilks’ Lambda (λ) with Pillai’s Trace (V) if you fail the multivariate homogeneity assumption.
20.5 Combining Tests: Factorial MANCOVA
In complex research designs, combining different types of analysis can lead to a deeper understanding of the data. Factorial MANCOVA integrates elements of both Factorial ANOVA and ANCOVA, enabling researchers to examine the effects of multiple independent variables while simultaneously controlling for covariates and analyzing multiple dependent variables. This method is especially useful when the research involves interactions, continuous control variables, and multiple outcomes that are conceptually or statistically related.
Factorial MANCOVA is particularly useful when researchers anticipate interactions between factors, need to control for pre-existing differences, and aim to evaluate how multiple outcomes are influenced collectively. It is commonly applied in fields where outcomes are multidimensional and influenced by both experimental and contextual variables.
Because this analysis involves multiple assumptions (such as multivariate normality, homogeneity of variance-covariance matrices, and homogeneity of regression slopes) it requires careful data screening and a sufficiently large sample size to ensure reliable results.
Chapter 20 Summary and Key Takeaways
Several advanced variance tests help researchers analyze complex research questions involving multiple variables. Factorial ANOVA examines the effects of two or more independent variables on a dependent variable and tests for interaction effects. ANCOVA adjusts for the influence of continuous covariates while comparing group means, helping isolate the effect of the independent variable. MANOVA extends ANOVA to multiple dependent variables, assessing whether groups differ across a combined set of outcomes. Factorial MANCOVA combines a factorial design with covariate adjustment and multiple dependent variables, offering a comprehensive analysis of complex, multidimensional data. All of these analyses can be conducted in Jamovi using clear menus and options that guide users through model setup, assumption checks, and interpretation.
Factorial ANOVA examines the effects of multiple independent variables and their interactions on a single dependent variable.
ANCOVA adjusts for covariates to improve the accuracy of group comparisons.
MANOVA analyzes multiple dependent variables simultaneously to determine how groups differ on the overall multivariate outcome.
Factorial MANCOVA combines factorial analysis with covariate control and multiple outcomes for a more comprehensive understanding of group differences.
