Homework 1 - Bayesian learning

Binomial trials, two competing priors

1 Model and likelihood

Assume that we observed the data \(D_n=\{0,0,1,0,1,0\}\), denoted here by \(x_1,\ldots,x_n\), and that we would like to model them as independent and identically distributed Bernoulli trails with sucess probability equal to \(\theta \in (0,1)\). Therefore \[ p(x_i|\theta) = \theta^x_i(1-\theta_i)^{1-x_i}, \qquad x_i=0,1. \]

1. Show that \(y_n=\sum_{i=1}^n x_i \sim \mbox{Binomial}(n,\theta)\), where \[ p(y_n|\theta) = \frac{n!}{y_n!(n-y_n)!}\theta^{y_n}(1-\theta)^{n-y_n}, \qquad y_n=0,1,\ldots,n. \] In the above example, \(n=5\) and \(y_n=2\). The likelihood function is, therefore, \[ L(\theta) \propto \theta^{y_n}(1-\theta)^{n-y_n}, \qquad \theta \in (0,1). \] The normalized likelihood (transforming the likelihood into a density function of \(\theta\)), exists in this example.

2. Show that the normalized likelihood is a Beta distribution with parameters \(y_n+1\) and \(n-y_n+1\).

2 Prior distributions

We will entertain two prior specifications:

1. \(\theta|M_1 \sim Beta(\alpha_0,\beta_0)\), and
2. \(\theta|M_2 \sim N_{(0,1)}(\mu_0,\sigma_0^2)\).

We will use the following hyperparameters: \[ \alpha_0 = 1.667 \ \ \ \mbox{and} \ \ \ \beta_0 = 15, \] and \[ \mu_0 = 0.1 \ \ \ \mbox{and} \ \ \ \sigma_0^2 = (0.2)^2. \]

3. Plot both prior densities of \(\theta\), \(p(\theta|M_1)\) and \(p(\theta|M_2)\), along with the normalized likelihood in the same figure. Comment their similarities and/or differences. Recall that the density of a truncated normal random variable \(x\) with location parameters \(\mu\), scale parameter \(\sigma\), and truncation interval \({\cal A}=(a,b)\), is given by \[ p_{TN}(x|\mu,\sigma,{\cal A}) = \frac{\phi\left(\frac{x-\mu}{\sigma}\right)}{\Phi(\frac{b-\mu}{\sigma})- \Phi(\frac{a-\mu}{\sigma})} \] for \(x \in {\cal A}\) and \(-\infty < a < b < \infty\).

3 Posterior distributions

One might want to compare both models by computing posterior quantities, such as \(E(\theta|M_i)\) and \(V(\theta|M_i)\), for \(i=1,2\), and the Bayes factor, i.e. \(B_{12}=Pr(y_n|M_1)/Pr(y_n|M_2)\).

3.1 Model \(M_1\)

The Binomial model and the Beta prior for \(\theta\) conjugate: \[ \theta|n,y_n,M_1 \sim Beta(\alpha_0+y_n,\beta_0+n-y_n), \] such that \(E(\theta|n,y_n,M1)\) and \(V(\theta|n,y_n,M1)\) are easily calculated.

3.2 Model \(M_2\)

The Binomial model and prior 2 for \(\theta\) do not conjugate! Therefore, there is no closed-form analytical solutions to the posterior quantities or the prior predictive.

\[ p(\theta|n,y_n,M_2) = \frac{\left[p_{TN}(\theta|\mu_0,\sigma_0^2,{\cal A}=(0,1))\right]\left[\frac{n!}{y_n!(n-y_n)!}\theta^{y_n}(1-\theta)^{n-y_n}\right]}{p(y_n|M_2)} = \frac{g(\theta)}{p(y_n|M_2)}, \] where \[ p_{TN}(\theta|\mu_0,\sigma_0^2,{\cal A}=(0,1)) = \frac{(2\pi\sigma_0^2)^{-1/2}\exp\left\{-\frac{1}{2\sigma^2_0}(\theta-\mu_0)^2\right\}}{\int_0^1 (2\pi\sigma_0^2)^{-1/2}\exp\left\{-\frac{1}{2\sigma^2_0}(\gamma-\mu_0)^2\right\}d\gamma}. \]

4 Prior predictives

4.1 Model \(M_1\)

4. Show that the prior predictive under model \(M_1\) is \[ Pr(y_n|M_1) = \frac{n!}{y_n!(n-y_n)!}\frac{\Gamma(\alpha_0+\beta_0)}{\Gamma(\alpha_0)\Gamma(\beta_0)} \frac{\Gamma(\alpha_0+y_n)\Gamma(\beta_0+n-y_n)}{\Gamma(\alpha_0+\beta_0+n)}. \] When \(\alpha_0=\beta_0=1\) (Uniform prior for \(\theta\)), it follows that \[ Pr(y_n|M_1) = \frac{n!}{y_n!(n-y_n)!} \frac{\Gamma(1+y_n)\Gamma(1+n-y_n)}{\Gamma(2+n)}=\frac{1}{n+1}, \qquad y_n=0,1,\ldots,n. \]

4.2 Model \(M_2\)

The prior predictive under model \(M_2\) is not analytically available in closed-form: \[ p(y_n|M_2)=\int_0^1 g(\theta)d\theta, \] and needs to be computed numerically, either deterministically (Reimann, quadrature or Laplace approximations, to name a few ones) or via Monte Carlo integration. Notice that the prior is truncated normal, so it needs to be normalized by the normal distribution in the interval \((0,1)\).

5. Use a simple Reimann approximation to obtain \(p(y_n|M_2)\). Now you have all the ingredients to compute the Bayes factor, \(B_{12}\).