Applied Bayesian Data Analysis

Insights on hierarchical models

David Tolpin, david.tolpin@gmail.com

Recap: hierarchical model

\begin{aligned} \tau & \sim \mathrm{HyperPrior} \\ \theta_i & \sim \mathrm{Prior}(\tau)\, \forall i \\ y_{ij} & \sim \mathrm{Conditional}(\theta_i)\, \forall i,j \end{aligned}

Recap: multiple marbles in boxes

\begin{aligned} p_0 & \sim \mathrm{Beta}(\alpha, \beta) \\ k_i & \sim \mathrm{Binomial}(n_i, p_0) \\ \mathit{blue}_{ij} & \sim \mathrm{Bernoulli}(\frac {k_i} {n_i}) \end{aligned}
  • $\tau$ — $p_0$
  • $\theta$ — $k_i$
  • $y$ — $blue_{ij}$

Recap: eight schools

\begin{aligned} \mu_0 & \sim \mathrm{Uniform}(-\infty, \infty) \\ \sigma_0 & \sim \mathrm{Uniform}(0, \infty) \\ \mu_i & \sim \mathrm{Normal}(\mu_0, \sigma_0) \\ y_i & \sim \mathrm{Normal}(\mu_i, \sigma_i) \end{aligned}
  • $\tau$ — $\mu_0$, $\sigma_0$
  • $\theta$ — $\mu_i$
  • $y$ — $y_i$, $\sigma_i$

Recap: tadpoles in tanks

\begin{aligned} \sigma_0 & \sim \mathrm{Exponential}(1) \\ \alpha_0 & \sim \mathrm{Normal}(0, 1.5) \\ \alpha_i & \sim Normal(\alpha_0, \sigma_0) \\ k_i & \sim \mathrm{Binomial}(n_i, \mathrm{logistic}(\alpha_i)) \end{aligned}
  • $\tau$ — $\alpha_0$, $\sigma_0$
  • $\theta$ — $\alpha_i$
  • $y$ — $k_i$

Posterior distributions

  • Parameter posterior $p(\theta|y)$
  • Hyperparameter posterior $p(\tau|y)$
  • Predictive posterior $p(\hat y|y)$

Parameter posterior

$p(\theta|y)$

  • All $\theta_i$ depend on all $y_{ij}$.
  • $\theta_i$ describes the $i$th group.

Hyperparameter posterior

$p(\tau|y)$

  • $\tau$ connected to $y$ through $\theta$.
  • $\tau$ describes similarities (and differences) among groups.

Predictive posterior

$p(\hat y|y)$ or $p(\hat y_i|y)$

  • All $\hat y_i$ depend on all $y_{ij}$.
  • $\hat y$ for a new group vs. $\hat y_i$ for an existing group.

Simulating predictive posterior

Simulating $\hat y$

  • sample $\tau$ (inference)
  • sample $\theta|\tau$ (simulation)
  • sample $\hat y|\theta$ (simulation)

Simulating $\hat y_i$

  • $\tau$ is not used
  • sample $\theta_i$ (inference)
  • sample $\hat y_i|\theta_i$ (simulation)

Prior

  • More posteriors, but only one prior — on $\tau$
  • Usually weak:
    • More data than in separate model.
    • Less influence than in pooled model.
  • Often represents mixed effects:
    • Fixed effect — common among groups.
    • Random effect — variability between groups.

Skyscrapers?

For want of a nail

For want of a nail, the shoe was lost,
For want of a shoe, the horse was lost,
For want of a horse, the rider was lost,
For want of a rider, the message was lost,
For want of a message, the battle was lost,
For want of a battle, the war was lost,
For want of a war, the kingdom was lost,
For want of a nail, the world was lost.

Also, This is the House that Jack Built

Skyscrapers?

  • Can we have more than one level of hierarchy?

  • Marbles

    • Marbles in a factory, $K$ factories
    • Marbles in a bag from each factory, $L_k$ bags at the $k$th factory
    • Marbles in a box from each bag, $M_{kl}$ boxes
    • Marbles in a draw of 3 from each box, $N_{klm}$ draws ...

Skyscrapers?

  • But what is the prior for parameters
    • of $\mathrm{Beta}(\alpha, \beta)$
    • of the prior over parameters of the prior ...

    • Mazes?

    • Sometimes, there is more than one hierarchy
    • Salary:
      • Men vs. women
      • Tel Aviv vs. Haifa vs. Jerusalem vs. Beer Sheva
      • Hitech vs. goverment vs. academia
    • Cross-classified multilevel models

    Cross-cassified

    \begin{aligned} \tau_{1:N_c} & \sim \mathrm{HyperPrior} \\ \theta_i|c_1, c_2, ... & \sim \mathrm{Prior}(\phi(\tau_{c_1}, \tau_{c_2}, ...)) \\ y_i & \sim \mathrm{Conditional}(\theta_i) \end{aligned}
    $\phi(\cdot)$ is sometimes a linear combination
    (but not always!): $$\mathit{salary}|w, h, a\sim \mathcal{N}\left(\mu + \mu_w + \mu_h + \mu_a, \sqrt{\sigma^2 + \sigma_w^2 + \sigma_h^2 + \sigma_a^2}\right)$$

    Readings

    1. Bayesian Data Analysis — chapter 5.
    2. Statistical rethinking — chapter 13.
    3. Probabilistic Models of Cognition — chapter 12.

    Hands-on

    • Marbles
    • Reed frogs
    • 8 schools