1 Model and likelihood
2 Prior distributions
3 Posterior distributions
- 3.1 Model $M_1$
- 3.2 Model $M_2$
4 Prior predictives
- 4.1 Model $M_1$
- 4.2 Model $M_2$

1 Model and likelihood

Assume that we observed the data $D_n=\{0,0,1,0,1,0\}$ , denoted here by $x_1,\ldots,x_n$ , and that we would like to model them as independent and identically distributed Bernoulli trails with sucess probability equal to $\theta \in (0,1)$ . Therefore $p(x_i|\theta) = \theta^x_i(1-\theta_i)^{1-x_i}, \qquad x_i=0,1.$

Show that $y_n=\sum_{i=1}^n x_i \sim \mbox{Binomial}(n,\theta)$ , where $p(y_n|\theta) = \frac{n!}{y_n!(n-y_n)!}\theta^{y_n}(1-\theta)^{n-y_n}, \qquad y_n=0,1,\ldots,n.$ In the above example, $n=5$ and $y_n=2$ . The likelihood function is, therefore, $L(\theta) \propto \theta^{y_n}(1-\theta)^{n-y_n}, \qquad \theta \in (0,1).$ The normalized likelihood (transforming the likelihood into a density function of $\theta$ ), exists in this example.
Show that the normalized likelihood is a Beta distribution with parameters $y_n+1$ and $n-y_n+1$ .

2 Prior distributions

We will entertain two prior specifications:

$\theta|M_1 \sim Beta(\alpha_0,\beta_0)$ , and
$\theta|M_2 \sim N_{(0,1)}(\mu_0,\sigma_0^2)$ .

We will use the following hyperparameters: $\alpha_0 = 1.667 \ \ \ \mbox{and} \ \ \ \beta_0 = 15,$ and $\mu_0 = 0.1 \ \ \ \mbox{and} \ \ \ \sigma_0^2 = (0.2)^2.$

Plot both prior densities of $\theta$ , $p(\theta|M_1)$ and $p(\theta|M_2)$ , along with the normalized likelihood in the same figure. Comment their similarities and/or differences. Recall that the density of a truncated normal random variable $x$ with location parameters $\mu$ , scale parameter $\sigma$ , and truncation interval ${\cal A}=(a,b)$ , is given by $p_{TN}(x|\mu,\sigma,{\cal A}) = \frac{\phi\left(\frac{x-\mu}{\sigma}\right)}{\Phi(\frac{b-\mu}{\sigma})- \Phi(\frac{a-\mu}{\sigma})}$ for $x \in {\cal A}$ and $-\infty < a < b < \infty$ .

3 Posterior distributions

One might want to compare both models by computing posterior quantities, such as $E(\theta|M_i)$ and $V(\theta|M_i)$ , for $i=1,2$ , and the Bayes factor, i.e. $B_{12}=Pr(y_n|M_1)/Pr(y_n|M_2)$ .

3.1 Model $M_1$

The Binomial model and the Beta prior for $\theta$ conjugate: $\theta|n,y_n,M_1 \sim Beta(\alpha_0+y_n,\beta_0+n-y_n),$ such that $E(\theta|n,y_n,M1)$ and $V(\theta|n,y_n,M1)$ are easily calculated.

3.2 Model $M_2$

The Binomial model and prior 2 for $\theta$ do not conjugate! Therefore, there is no closed-form analytical solutions to the posterior quantities or the prior predictive.

$p(\theta|n,y_n,M_2) = \frac{\left[p_{TN}(\theta|\mu_0,\sigma_0^2,{\cal A}=(0,1))\right]\left[\frac{n!}{y_n!(n-y_n)!}\theta^{y_n}(1-\theta)^{n-y_n}\right]}{p(y_n|M_2)} = \frac{g(\theta)}{p(y_n|M_2)},$ where $p_{TN}(\theta|\mu_0,\sigma_0^2,{\cal A}=(0,1)) = \frac{(2\pi\sigma_0^2)^{-1/2}\exp\left\{-\frac{1}{2\sigma^2_0}(\theta-\mu_0)^2\right\}}{\int_0^1 (2\pi\sigma_0^2)^{-1/2}\exp\left\{-\frac{1}{2\sigma^2_0}(\gamma-\mu_0)^2\right\}d\gamma}.$

4 Prior predictives

4.1 Model $M_1$

Show that the prior predictive under model $M_1$ is $Pr(y_n|M_1) = \frac{n!}{y_n!(n-y_n)!}\frac{\Gamma(\alpha_0+\beta_0)}{\Gamma(\alpha_0)\Gamma(\beta_0)} \frac{\Gamma(\alpha_0+y_n)\Gamma(\beta_0+n-y_n)}{\Gamma(\alpha_0+\beta_0+n)}.$ When $\alpha_0=\beta_0=1$ (Uniform prior for $\theta$ ), it follows that $Pr(y_n|M_1) = \frac{n!}{y_n!(n-y_n)!} \frac{\Gamma(1+y_n)\Gamma(1+n-y_n)}{\Gamma(2+n)}=\frac{1}{n+1}, \qquad y_n=0,1,\ldots,n.$

4.2 Model $M_2$

The prior predictive under model $M_2$ is not analytically available in closed-form: $p(y_n|M_2)=\int_0^1 g(\theta)d\theta,$ and needs to be computed numerically, either deterministically (Reimann, quadrature or Laplace approximations, to name a few ones) or via Monte Carlo integration. Notice that the prior is truncated normal, so it needs to be normalized by the normal distribution in the interval $(0,1)$ .

Use a simple Reimann approximation to obtain $p(y_n|M_2)$ . Now you have all the ingredients to compute the Bayes factor, $B_{12}$ .

LS0tCnRpdGxlOiAiSG9tZXdvcmsgMSAtIEJheWVzaWFuIGxlYXJuaW5nIgpzdWJ0aXRsZTogIkJpbm9taWFsIHRyaWFscywgdHdvIGNvbXBldGluZyBwcmlvcnMiCmF1dGhvcjogIkhlZGliZXJ0IEZyZWl0YXMgTG9wZXMiCmRhdGU6ICIxLzIzLzIwMjIiCm91dHB1dDoKICBodG1sX2RvY3VtZW50OgogICAgdG9jOiB0cnVlCiAgICB0b2NfZGVwdGg6IDIKICAgIGNvZGVfZG93bmxvYWQ6IHllcwogICAgbnVtYmVyX3NlY3Rpb25zOiB0cnVlCi0tLQoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSkKYGBgCgojIE1vZGVsIGFuZCBsaWtlbGlob29kCgpBc3N1bWUgdGhhdCB3ZSBvYnNlcnZlZCB0aGUgZGF0YSAkRF9uPVx7MCwwLDEsMCwxLDBcfSQsIGRlbm90ZWQgaGVyZSBieSAkeF8xLFxsZG90cyx4X24kLCBhbmQgdGhhdCB3ZSB3b3VsZCBsaWtlIHRvIG1vZGVsIHRoZW0gYXMgaW5kZXBlbmRlbnQgYW5kIGlkZW50aWNhbGx5IGRpc3RyaWJ1dGVkIEJlcm5vdWxsaSB0cmFpbHMgd2l0aCBzdWNlc3MgcHJvYmFiaWxpdHkgZXF1YWwgdG8gJFx0aGV0YSBcaW4gKDAsMSkkLiAgVGhlcmVmb3JlCiQkCnAoeF9pfFx0aGV0YSkgPSBcdGhldGFeeF9pKDEtXHRoZXRhX2kpXnsxLXhfaX0sIFxxcXVhZCB4X2k9MCwxLgokJAoKYSkgU2hvdyB0aGF0ICR5X249XHN1bV97aT0xfV5uIHhfaSBcc2ltIFxtYm94e0Jpbm9taWFsfShuLFx0aGV0YSkkLCB3aGVyZQokJApwKHlfbnxcdGhldGEpID0gXGZyYWN7biF9e3lfbiEobi15X24pIX1cdGhldGFee3lfbn0oMS1cdGhldGEpXntuLXlfbn0sIFxxcXVhZCB5X249MCwxLFxsZG90cyxuLgokJApJbiB0aGUgYWJvdmUgZXhhbXBsZSwgJG49NSQgYW5kICR5X249MiQuICBUaGUgbGlrZWxpaG9vZCBmdW5jdGlvbiBpcywgdGhlcmVmb3JlLAokJApMKFx0aGV0YSkgXHByb3B0byBcdGhldGFee3lfbn0oMS1cdGhldGEpXntuLXlfbn0sIFxxcXVhZCBcdGhldGEgXGluICgwLDEpLgokJApUaGUgbm9ybWFsaXplZCBsaWtlbGlob29kICh0cmFuc2Zvcm1pbmcgdGhlIGxpa2VsaWhvb2QgaW50byBhIGRlbnNpdHkgZnVuY3Rpb24gb2YgJFx0aGV0YSQpLCBleGlzdHMgaW4gdGhpcyBleGFtcGxlLiAgCgpiKSBTaG93IHRoYXQgdGhlIG5vcm1hbGl6ZWQgbGlrZWxpaG9vZCBpcyBhIEJldGEgZGlzdHJpYnV0aW9uIHdpdGggcGFyYW1ldGVycyAkeV9uKzEkIGFuZCAkbi15X24rMSQuCgojIFByaW9yIGRpc3RyaWJ1dGlvbnMKV2Ugd2lsbCBlbnRlcnRhaW4gdHdvIHByaW9yIHNwZWNpZmljYXRpb25zOgoKMS4gJFx0aGV0YXxNXzEgXHNpbSBCZXRhKFxhbHBoYV8wLFxiZXRhXzApJCwgYW5kCjIuICRcdGhldGF8TV8yIFxzaW0gTl97KDAsMSl9KFxtdV8wLFxzaWdtYV8wXjIpJC4KCldlIHdpbGwgdXNlIHRoZSBmb2xsb3dpbmcgaHlwZXJwYXJhbWV0ZXJzOgokJApcYWxwaGFfMCA9IDEuNjY3IFwgXCBcIFxtYm94e2FuZH0gXCBcIFwgXGJldGFfMCA9IDE1LAokJAphbmQgCiQkIApcbXVfMCA9IDAuMSBcIFwgXCBcbWJveHthbmR9IFwgXCBcIFxzaWdtYV8wXjIgPSAoMC4yKV4yLgokJAoKYykgUGxvdCBib3RoIHByaW9yIGRlbnNpdGllcyBvZiAkXHRoZXRhJCwgJHAoXHRoZXRhfE1fMSkkIGFuZCAkcChcdGhldGF8TV8yKSQsIGFsb25nIHdpdGggdGhlIG5vcm1hbGl6ZWQgbGlrZWxpaG9vZCBpbiB0aGUgc2FtZSBmaWd1cmUuICBDb21tZW50IHRoZWlyIHNpbWlsYXJpdGllcyBhbmQvb3IgZGlmZmVyZW5jZXMuICBSZWNhbGwgdGhhdCB0aGUgZGVuc2l0eSBvZiBhIHRydW5jYXRlZCBub3JtYWwgcmFuZG9tIHZhcmlhYmxlICR4JCB3aXRoIGxvY2F0aW9uIHBhcmFtZXRlcnMgJFxtdSQsIHNjYWxlIHBhcmFtZXRlciAkXHNpZ21hJCwgYW5kIHRydW5jYXRpb24gaW50ZXJ2YWwgJHtcY2FsIEF9PShhLGIpJCwgaXMgZ2l2ZW4gYnkKJCQKcF97VE59KHh8XG11LFxzaWdtYSx7XGNhbCBBfSkgPSBcZnJhY3tccGhpXGxlZnQoXGZyYWN7eC1cbXV9e1xzaWdtYX1ccmlnaHQpfXtcUGhpKFxmcmFje2ItXG11fXtcc2lnbWF9KS0KXFBoaShcZnJhY3thLVxtdX17XHNpZ21hfSl9CiQkCmZvciAkeCBcaW4ge1xjYWwgQX0kIGFuZCAkLVxpbmZ0eSA8IGEgPCBiIDwgXGluZnR5JC4KCiMgUG9zdGVyaW9yIGRpc3RyaWJ1dGlvbnMKCk9uZSBtaWdodCB3YW50IHRvIGNvbXBhcmUgYm90aCBtb2RlbHMgYnkgY29tcHV0aW5nIHBvc3RlcmlvciBxdWFudGl0aWVzLCBzdWNoIGFzIAokRShcdGhldGF8TV9pKSQgYW5kICRWKFx0aGV0YXxNX2kpJCwgZm9yICRpPTEsMiQsIGFuZCB0aGUgQmF5ZXMgZmFjdG9yLCBpLmUuCiRCX3sxMn09UHIoeV9ufE1fMSkvUHIoeV9ufE1fMikkLgoKIyMgTW9kZWwgJE1fMSQKVGhlIEJpbm9taWFsIG1vZGVsIGFuZCB0aGUgQmV0YSBwcmlvciBmb3IgJFx0aGV0YSQgY29uanVnYXRlOgokJApcdGhldGF8bix5X24sTV8xIFxzaW0gQmV0YShcYWxwaGFfMCt5X24sXGJldGFfMCtuLXlfbiksCiQkCnN1Y2ggdGhhdCAkRShcdGhldGF8bix5X24sTTEpJCBhbmQgJFYoXHRoZXRhfG4seV9uLE0xKSQgYXJlIGVhc2lseSBjYWxjdWxhdGVkLiAgCgojIyBNb2RlbCAkTV8yJApUaGUgQmlub21pYWwgbW9kZWwgYW5kIHByaW9yIDIgZm9yICRcdGhldGEkIGRvIG5vdCBjb25qdWdhdGUhClRoZXJlZm9yZSwgdGhlcmUgaXMgbm8gY2xvc2VkLWZvcm0gYW5hbHl0aWNhbCBzb2x1dGlvbnMgdG8gdGhlIHBvc3RlcmlvciBxdWFudGl0aWVzIG9yIHRoZSBwcmlvciBwcmVkaWN0aXZlLgoKJCQKcChcdGhldGF8bix5X24sTV8yKSA9IFxmcmFje1xsZWZ0W3Bfe1ROfShcdGhldGF8XG11XzAsXHNpZ21hXzBeMix7XGNhbCBBfT0oMCwxKSlccmlnaHRdXGxlZnRbXGZyYWN7biF9e3lfbiEobi15X24pIX1cdGhldGFee3lfbn0oMS1cdGhldGEpXntuLXlfbn1ccmlnaHRdfXtwKHlfbnxNXzIpfSA9IFxmcmFje2coXHRoZXRhKX17cCh5X258TV8yKX0sCiQkCndoZXJlCiQkCnBfe1ROfShcdGhldGF8XG11XzAsXHNpZ21hXzBeMix7XGNhbCBBfT0oMCwxKSkgPSBcZnJhY3soMlxwaVxzaWdtYV8wXjIpXnstMS8yfVxleHBcbGVmdFx7LVxmcmFjezF9ezJcc2lnbWFeMl8wfShcdGhldGEtXG11XzApXjJccmlnaHRcfX17XGludF8wXjEgKDJccGlcc2lnbWFfMF4yKV57LTEvMn1cZXhwXGxlZnRcey1cZnJhY3sxfXsyXHNpZ21hXjJfMH0oXGdhbW1hLVxtdV8wKV4yXHJpZ2h0XH1kXGdhbW1hfS4KJCQKCiMgUHJpb3IgcHJlZGljdGl2ZXMKCiMjIE1vZGVsICRNXzEkCgpkKSBTaG93IHRoYXQgdGhlIHByaW9yIHByZWRpY3RpdmUgdW5kZXIgbW9kZWwgJE1fMSQgaXMKJCQKUHIoeV9ufE1fMSkgPSBcZnJhY3tuIX17eV9uIShuLXlfbikhfVxmcmFje1xHYW1tYShcYWxwaGFfMCtcYmV0YV8wKX17XEdhbW1hKFxhbHBoYV8wKVxHYW1tYShcYmV0YV8wKX0KXGZyYWN7XEdhbW1hKFxhbHBoYV8wK3lfbilcR2FtbWEoXGJldGFfMCtuLXlfbil9e1xHYW1tYShcYWxwaGFfMCtcYmV0YV8wK24pfS4KJCQKV2hlbiAkXGFscGhhXzA9XGJldGFfMD0xJCAoVW5pZm9ybSBwcmlvciBmb3IgJFx0aGV0YSQpLCBpdCBmb2xsb3dzIHRoYXQKJCQKUHIoeV9ufE1fMSkgPSBcZnJhY3tuIX17eV9uIShuLXlfbikhfQpcZnJhY3tcR2FtbWEoMSt5X24pXEdhbW1hKDErbi15X24pfXtcR2FtbWEoMituKX09XGZyYWN7MX17bisxfSwgXHFxdWFkIHlfbj0wLDEsXGxkb3RzLG4uCiQkCgoKIyMgTW9kZWwgJE1fMiQKVGhlIHByaW9yIHByZWRpY3RpdmUgdW5kZXIgbW9kZWwgJE1fMiQgaXMgbm90IGFuYWx5dGljYWxseSBhdmFpbGFibGUgaW4gY2xvc2VkLWZvcm06CiQkCnAoeV9ufE1fMik9XGludF8wXjEgZyhcdGhldGEpZFx0aGV0YSwKJCQKYW5kIG5lZWRzIHRvIGJlIGNvbXB1dGVkIG51bWVyaWNhbGx5LCBlaXRoZXIgZGV0ZXJtaW5pc3RpY2FsbHkgKFJlaW1hbm4sIHF1YWRyYXR1cmUgb3IgTGFwbGFjZSBhcHByb3hpbWF0aW9ucywgdG8gbmFtZSBhIGZldyBvbmVzKSBvciB2aWEgTW9udGUgQ2FybG8gaW50ZWdyYXRpb24uICBOb3RpY2UgdGhhdCB0aGUgcHJpb3IgaXMgdHJ1bmNhdGVkIG5vcm1hbCwgc28gaXQgbmVlZHMgdG8gYmUgbm9ybWFsaXplZCBieSB0aGUgbm9ybWFsIGRpc3RyaWJ1dGlvbiBpbiB0aGUgaW50ZXJ2YWwgJCgwLDEpJC4KCmUpIFVzZSBhIHNpbXBsZSBSZWltYW5uIGFwcHJveGltYXRpb24gdG8gb2J0YWluICRwKHlfbnxNXzIpJC4gIE5vdyB5b3UgaGF2ZSBhbGwgdGhlIGluZ3JlZGllbnRzIHRvIGNvbXB1dGUgdGhlIEJheWVzIGZhY3RvciwgJEJfezEyfSQuCgo=

Homework 1 - Bayesian learning

Binomial trials, two competing priors

Hedibert Freitas Lopes

1/23/2022

1 Model and likelihood

2 Prior distributions

3 Posterior distributions

3.1 Model $M_1$

3.2 Model $M_2$

4 Prior predictives

4.1 Model $M_1$

4.2 Model $M_2$

Homework 1 - Bayesian learning

Binomial trials, two competing priors

Hedibert Freitas Lopes

1/23/2022

1 Model and likelihood

2 Prior distributions

3 Posterior distributions

3.1 Model M1M_1

3.2 Model M2M_2

4 Prior predictives

4.1 Model M1M_1

4.2 Model M2M_2

3.1 Model $M_1$

3.2 Model $M_2$

4.1 Model $M_1$

4.2 Model $M_2$