19: Basic Variance

19.1 Understanding Variance

Understanding variance and comparing group differences are fundamental components of statistical analysis in quantitative research. These comparisons are at the heart of many research questions, from testing interventions to evaluating policy impacts. Variance tests help determine whether the means or distributions of two or more groups differ significantly, considering how much variability exists within each group. These tests are essential for evaluating experimental conditions, treatment effects, or any situation where identifying meaningful differences between populations is important.

19.2 One-Sample t-Test

The One-Sample t-Test is used to assess whether the mean of a single sample differs significantly from a known or hypothesized population mean. It is often applied when researchers want to compare a sample’s average to a theoretical value or standard.

Assumptions

The One-Sample t-Test relies on a few key assumptions to ensure valid results. First, the data should be continuous and measured at the interval or ratio level. Second, the observations must be independent, meaning that the value of one observation does not influence another. Third, the distribution of the sample data should be approximately normal, especially when the sample size is small.

Understanding the Output

The output from the One-Sample t-Test is shown below. The screenshots separate the results for display purposes, but the full output appears in a single Jamovi output window when all test options are selected.

To interpret a one-sample t-test in Jamovi, begin with the t-test table. The t statistic evaluates whether the sample mean differs significantly from the specified test value. The degrees of freedom reflect the sample size minus one. The p-value indicates whether the observed difference between the sample mean and the test value is statistically significant. If the p-value is below the selected alpha level, you conclude that the sample mean differs significantly from the hypothesized value. The mean difference shows the size of that difference in the original measurement units.

Next, examine the effect size, typically reported as Cohen’s d. This statistic reflects the magnitude of the difference independent of sample size. As a general guideline, values around 0.20 indicate a small effect, around 0.50 indicate a medium effect, and 0.80 or higher indicate a large effect. Effect size helps determine practical importance beyond statistical significance.

Then review the normality test (Shapiro–Wilk). The W statistic and corresponding p-value assess whether the data significantly deviate from a normal distribution. A non-significant p-value suggests that the normality assumption has not been violated. Because the one-sample t-test assumes approximate normality, this check helps determine whether the test results are appropriate to interpret.

Next, examine the descriptive statistics, including the sample size, mean, median, standard deviation, and standard error. These values provide context for understanding the central tendency and variability of the data. Comparing the mean and median can also offer insight into potential skewness.

Finally, review the Q–Q plot. Each point represents an observed value compared to what would be expected under normality. If the points closely follow the reference line, the normality assumption appears reasonable. Substantial departures from the line, especially at the tails, may indicate non-normality.

19.3 One-Sample Wilcoxon Rank Test

The One-Sample Wilcoxon Signed-Rank Test is a non-parametric test used to determine whether the median of a sample differs from a specific hypothesized value. It is typically applied when the data are not normally distributed and when a researcher wants to compare the sample median to a theoretical value.

Assumptions

This test serves as a non-parametric alternative to the One-Sample t-Test, which compares a sample mean to a hypothesized mean but assumes normality. The Wilcoxon Test does not require that assumption, making it more appropriate for skewed or ordinal data.

Understanding the Output

The output from the One-Sample Wilcoxon Signed-Rank Test is shown below.

To interpret a One-Sample Wilcoxon Rank Test in Jamovi, begin with the test table. The Wilcoxon W statistic evaluates whether the sample median differs significantly from the specified test value. The p-value indicates whether that difference is statistically significant. If the p-value is below the selected alpha level, you conclude that the central tendency of the sample differs from the hypothesized value.

Next, review the mean difference reported in the table. This value reflects the difference between the sample’s central value and the test value in the original measurement units. While the Wilcoxon Test is based on ranks rather than means, this difference provides helpful context for interpreting the magnitude of the shift.

Then examine the effect size, reported as the rank biserial correlation. This statistic reflects the strength of the difference independent of sample size. Values closer to 0 indicate a weaker effect, whereas values closer to 1 indicate a stronger effect. Effect size helps determine the practical importance of the result beyond statistical significance.

Review the descriptive statistics, including the sample size, mean, median, standard deviation, and standard error. Because the Wilcoxon Test focuses on the median, particular attention should be given to the median when interpreting the direction and size of the difference.

Finally, examine the accompanying plot, which visually displays the sample’s central value and variability relative to the test value.

19.4 Independent Samples t-Test

The Independent Samples t-Test is used to compare the means of two independent groups to determine whether there is a statistically significant difference between them. It is commonly applied in research contexts where the goal is to evaluate whether group differences on a continuous outcome are likely due to chance or reflect a meaningful distinction.

Assumptions

The Independent Samples t-Test relies on several key assumptions to produce valid results. First, the dependent variable should be continuous and measured at the interval or ratio level. Second, the two groups being compared must be independent, meaning the observations in one group are not related to those in the other. Third, the data in each group should be approximately normally distributed, particularly when the sample sizes are small. Lastly, the test assumes homogeneity of variances, (e.g., the variability of scores in each group) should be roughly equal.

Understanding the Output

The output from the Independent Samples t-Test is shown below. The screenshots separate the results for display purposes, but the full output appears in a single Jamovi output window when all test options are selected.

To interpret an Independent Samples t-Test in Jamovi, begin with the t-test table. The t statistic evaluates whether the means of the two groups differ significantly. The degrees of freedom reflect the combined sample sizes of the groups, and the p-value indicates whether the observed mean difference is statistically significant. If the p-value is below the selected alpha level, you conclude that the group means differ significantly. If the p-value is above the alpha level, there is insufficient evidence to conclude that a difference exists.

Next, review the mean difference and the standard error of the difference. The mean difference shows how far apart the two group means are in the original measurement units. The standard error reflects the variability of that difference and influences the size of the t statistic.

Then examine the effect size, typically reported as Cohen’s d. This statistic reflects the magnitude of the difference independent of sample size. As a general guideline, values around 0.20 indicate a small effect, around 0.50 indicate a medium effect, and 0.80 or higher indicate a large effect. Effect size helps determine practical importance beyond statistical significance.

Before interpreting the test results, confirm the assumptions. The Shapiro–Wilk test assesses whether the data are approximately normally distributed. A non-significant p-value suggests that the normality assumption has not been violated. Levene’s test evaluates whether the variances of the two groups are equal. A non-significant p-value indicates that the assumption of equal variances is reasonable. If this assumption is violated, an alternative version of the t-test, such as Welch’s t-test, should be used.

Next, review the group descriptives, including sample size, mean, median, standard deviation, and standard error for each group. These values provide context for understanding the direction and size of the difference. Comparing the group means reveals which group has the higher average, while the standard deviations indicate variability within each group.

Finally, examine the plot, which visually displays the group means and their confidence intervals. Overlapping confidence intervals may suggest that differences are small, while clearly separated intervals may indicate stronger evidence of a difference.

19.5 Mann-Whitney U Test

The Mann–Whitney U Test is a non-parametric statistical test used to determine whether there is a significant difference in the distributions of two independent groups. It compares the ranks of values rather than the raw data, making it especially useful when working with ordinal data or continuous data that are not normally distributed. This test evaluates whether one group tends to have higher or lower values than the other, without assuming a specific distribution.

Assumptions

The Mann–Whitney U Test is a non-parametric alternative to the Independent Samples t-Test, used to assess whether two independent groups differ significantly when assumptions such as normality are not met.

Understanding the Output

The output from the Mann-Whitney U Test is shown below.

To interpret a Mann–Whitney U Test in Jamovi, begin with the test table. The Mann–Whitney U statistic evaluates whether the distributions of the two groups differ. Unlike the independent samples t-test, this test is based on ranked data rather than means. The p-value indicates whether the difference between groups is statistically significant. If the p-value is below the selected alpha level, you conclude that the groups differ significantly in their distributions. If it is above the alpha level, there is insufficient evidence to conclude that a difference exists.

Next, review the mean difference reported in the table. Although the Mann–Whitney U Test is based on ranks, this difference provides context for the direction and magnitude of the separation between groups in the original measurement units.

Then examine the effect size, reported as the rank biserial correlation. This statistic reflects the strength and direction of the difference between groups. Values closer to 0 indicate a weaker association, while values farther from 0 indicate a stronger association. The sign indicates direction, showing which group tends to have higher values. As a general guideline, values around .10 suggest a small effect, around .30 suggest a moderate effect, and .50 or higher suggest a large effect.

Next, review the group descriptives, including the sample size, mean, median, standard deviation, and standard error for each group. Because the Mann–Whitney U Test focuses on differences in central tendency based on ranks, particular attention should be given to the medians when comparing groups.

Finally, examine the plot, which visually displays the group means and confidence intervals. Visual comparison of the groups can help illustrate the magnitude and direction of the difference.

19.6 One-Way ANOVA

The One-Way ANOVA (Analysis of Variance) is a statistical test used to compare the means of three or more independent groups to determine whether at least one group mean is significantly different from the others. Instead of conducting multiple Independent Samples t-Tests for each pair of groups, which increases the risk of making a Type I error due to repeated testing, One-Way ANOVA evaluates all group means simultaneously using a single test. It does this by analyzing the ratio of variability between groups to the variability within groups. A significant result indicates that not all group means are equal, prompting further investigation through post hoc tests to identify which specific groups differ.

Assumptions

The One-Way ANOVA relies on several key assumptions. The dependent variable should be continuous and measured at the interval or ratio level. The groups being compared must be independent, meaning that each observation belongs to only one group. The data within each group should be approximately normally distributed, especially if sample sizes are small. Additionally, the assumption of homogeneity of variances must be met. This means that the variance of the dependent variable should be roughly equal across all groups.

Understanding the Output

The output from the One-Way ANOVA Test is shown below. The screenshots separate the results for display purposes, but the full output appears in a single Jamovi output window when all test options are selected.

To interpret a one-way ANOVA in Jamovi, begin with the ANOVA table. The F statistic evaluates whether there are overall mean differences among the groups. The associated p-value indicates whether the variability between group means is large enough, relative to the variability within groups, to conclude that at least one group differs significantly from the others. If the p-value is below the chosen alpha level, you reject the null hypothesis that all group means are equal.

Next, review the group descriptives. These provide the sample size, mean, standard deviation, and standard error for each group. Examining the means allows you to see the pattern of differences across groups, while the standard deviations and standard errors help you evaluate the spread and precision of those estimates.

Before interpreting the ANOVA results further, examine the assumption checks. The Shapiro–Wilk test assesses normality of the residuals. A non-significant result suggests that the assumption of normality has not been violated. Levene’s test evaluates homogeneity of variances. A non-significant result indicates that the variability across groups is sufficiently similar, supporting use of the standard Fisher’s ANOVA model. If Levene’s test is significant, the assumption of equal variances is violated. In that case, you should use Welch’s ANOVA, which adjusts for unequal variances.

If the overall ANOVA is statistically significant, move to the post hoc tests. The Tukey post hoc test compares each pair of group means while controlling for Type I error across multiple comparisons. For each pair, review the mean difference and adjusted p-value. Significant pairwise comparisons identify which specific groups differ from one another. The direction of the mean difference indicates which group has the higher average. When Welch’s ANOVA is used, the appropriate post hoc procedure is the Games–Howell test, as it does not assume equal variances.

Finally, examine the plot, which visually displays the group means and their confidence intervals. Visual inspection helps reinforce the numerical results and clarify the magnitude and direction of the differences.

19.7 Kruskal-Wallis Test

The Kruskal–Wallis Test is a non-parametric statistical test used to determine whether there are significant differences between the distributions of three or more independent groups. It works by ranking all values across groups and comparing the average ranks to assess whether one or more groups tend to have higher or lower values. When the test indicates a statistically significant difference, post hoc comparisons are needed to identify which specific groups differ from one another.

Assumptions

The Kruskal–Wallis Test is a non-parametric alternative to the One-Way ANOVA. It is particularly well-suited for continuous data that are not normally distributed, such as those that are skewed or contain outliers.

After selecting the appropriate Kruskal–Wallis Test options, run the analysis. Jamovi will generate the results in the Results panel, including the test statistic, p-value, and effect size. Your output should resemble the image shown below.

Understanding the Output

To interpret a Kruskal–Wallis Test in Jamovi, begin with the test table. The chi-square (χ²) statistic evaluates whether there are differences in the distributions across the groups. Unlike the One-Way ANOVA, this test is based on ranked data rather than raw means. The p-value indicates whether at least one group differs significantly from the others. If the p-value is below the selected alpha level, you conclude that there is a statistically significant difference among the groups.

Next, examine the effect size (ε²). This statistic reflects the proportion of variability in the ranked outcome that is associated with group membership. Values closer to 0 indicate a small effect, while values closer to 1 indicate a stronger group effect. As a general guideline, values around .01 suggest a small effect, around .06 a moderate effect, and .14 or higher a large effect.

If the overall Kruskal–Wallis Test is statistically significant, proceed to the pairwise comparisons. Dunn’s test compares each pair of groups while controlling for Type I error. The Bonferroni-adjusted p-values should be used when determining significance, as they correct for multiple comparisons. Significant pairwise comparisons indicate which specific groups differ from one another. The sign of the z statistic reflects the direction of the difference in ranks between the two groups.

19.8 Using Multiple t-Tests Instead of ANOVA

When comparing three or more groups, some researchers conduct multiple Independent Samples t-Tests rather than using a single One-Way ANOVA. While each t-test may appear straightforward, performing several tests on the same dataset increases the probability of committing a Type I error. Each additional comparison raises the likelihood of identifying a statistically significant difference due to chance alone. ANOVA was developed specifically to compare multiple group means within one analytical framework while controlling overall error rates. Failing to use ANOVA in this context reflects a misunderstanding of how error accumulates across repeated testing.