3.13 Choosing a test

The right test follows from three questions: what kind of outcome, how many groups or what kind of predictor, and are the units paired or independent. The answer is rarely a single test; it is a test plus an effect size plus an interval.

Code: style used in this chapter 
import matplotlib.pyplot as plt
from matplotlib.axes import Axes

from statly import style

style.use()

3.13.1 A map of the tests

Code 
TREE = [
    (
        "A number",
        style.BLUE,
        [
            ("one group against a fixed value", "one-sample t  /  Wilcoxon signed-rank (3.4)"),
            ("two separate groups", "Welch's t  /  Mann-Whitney (3.4)"),
            ("the same units twice", "paired t  /  Wilcoxon signed-rank, sign test (3.4)"),
            ("three or more groups", "ANOVA, Welch's ANOVA  /  Kruskal-Wallis (3.5)"),
            ("two factors", "two-way ANOVA (3.5), regression (4.2)"),
            ("another number", "correlation, regression  /  Spearman, Kendall (3.6, 4.1)"),
        ],
    ),
    (
        "Yes / no, or categories",
        style.CORAL,
        [
            ("one share against a value", "z-test for a share  /  binomial test (3.1)"),
            ("two separate groups", "two-proportion z-test  /  Fisher's exact test (3.4)"),
            ("the same units twice", "McNemar's test (3.4)"),
            ("several groups or categories", "chi-square test  /  Fisher's exact test (3.5)"),
            ("against a given distribution", "chi-square goodness of fit (3.5)"),
            ("with numeric predictors", "logistic regression (4.4)"),
        ],
    ),
    (
        "A count",
        style.GOLD,
        [
            ("with or without predictors", "Poisson, negative binomial regression (4.6)"),
        ],
    ),
    (
        "Time until an event",
        style.GREEN,
        [
            ("two or more groups", "log-rank test (9.4)"),
            ("with predictors", "Cox regression (9.5)"),
        ],
    ),
    (
        "Values in time order",
        style.PURPLE,
        [
            ("is it stationary?", "Dickey-Fuller, KPSS (9.1)"),
            ("is anything left in the residuals?", "Ljung-Box (9.2)"),
        ],
    ),
]


def plot_tree(ax: Axes) -> None:
    """Outcome kinds on the left, designs in the middle, tests on the right, joined by lines."""
    leaves = sum(len(branches) for _, _, branches in TREE)
    y = leaves
    for outcome, color, branches in TREE:
        top, bottom = y, y - len(branches) + 1
        middle = (top + bottom) / 2
        ax.text(
            0,
            middle,
            outcome,
            ha="left",
            va="center",
            fontsize=10,
            fontweight="bold",
            color=color,
        )
        for design, test in branches:
            ax.plot([0.27, 0.32], [middle, y], color=color, lw=1, alpha=0.6)
            ax.text(0.33, y, design, ha="left", va="center", fontsize=9)
            ax.plot([0.645, 0.67], [y, y], color=color, lw=1, alpha=0.6)
            ax.text(0.68, y, test, ha="left", va="center", fontsize=9, color=color)
            y -= 1
        y -= 0.6
    ax.set(xlim=(0, 1.24), ylim=(y, leaves + 1))
    ax.set_axis_off()


fig, ax = plt.subplots(figsize=(10, 7.4))
plot_tree(ax)
plt.show()
Figure 3.13.1: From the kind of outcome (left), through the design (middle), to the usual test and its alternative without the normal assumption (right, after the slash). Chapter numbers in brackets.

Three more questions sharpen the choice:

  • Are the units independent? If they come in clusters or in time order, none of the simple tests is valid as it stands (Chapter 3.8).
  • What is the question about? Means (t-test, ANOVA), which group tends to be higher (rank tests), whole distributions (Kolmogorov-Smirnov), or spreads (Levene).
  • How much data? With small samples and skewed data, prefer exact tests, rank tests or the bootstrap; with large samples, almost everything is significant and effect sizes matter more.

3.13.2 Many tests are one model

Most of the tests on the map are special cases of regression (Part 4). A t-test is a regression on a 0/1 group indicator; ANOVA is a regression on several indicators; a correlation test is a regression slope test; the chi-square test for a 2 × 2 table matches a logistic regression with one yes/no predictor. Seeing them as models has two advantages: you can add more predictors (adjust for confounders, Chapter 4.2), and you get estimates with intervals, not only p-values.

Test The same as
one-sample t-test regression with only an intercept
two-sample t-test (Student’s) regression on one 0/1 predictor
one-way ANOVA regression on group indicators
two-way ANOVA regression on two factors and their interaction
correlation test slope test in a simple regression
chi-square test of a table logistic or Poisson regression on the categories
log-rank test Cox regression on a group indicator

3.13.3 How to report a test

A test result is incomplete without the size of the effect and its uncertainty. Report, in this order:

  1. the estimate in meaningful units (a difference, a ratio, a correlation),
  2. its confidence interval,
  3. the test with its statistic, degrees of freedom and p-value,
  4. an effect size where the units are not meaningful on their own.

For example: Home owners spent on average 159 more per month on their credit cards than renters (95% interval 6 to 312; Welch’s t = 2.1, df = 54, p = 0.04; Cohen’s d = 0.5). A reader can see the direction, the size, how uncertain it is, and how surprising it would be if nothing were going on.

3.13.4 In one picture

  • Choose the test from the outcome, the design and independence.
  • Each test has a rank or exact alternative for small, skewed or ordinal data.
  • Most tests are regression models in disguise.
  • Report the estimate, interval, test and effect size, not just “p < 0.05”.

Next: Part 4, regression, where tests and intervals meet models.