Advanced Bayesian Statistical Learning

HW3: Gibbs and Metropolis steps

1 Model

The following example was taken from Chapter 6 (Metropolis-Hastings algorithms) of our book Gamerman and Lopes (2006) Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference.

The variable \(x_i\) is used to explain \(y_i\) on a nonlinear Gaussian regression set up: \[ y_i|x_i,\alpha,\beta,\theta,\sigma^2 \sim N(y_i,\alpha+\beta x_i/(\theta+x_i),\sigma^2). \] The observed data is data=\(\{x,y\}\), where \(x=(0.02,0.02,0.06,0.06,0.11,0.11,0.22,0.22,0.56,0.56,1.10,1.10)\) and \(y=(76,47,97,107,123,139,159,152,191,201,207,200)\).

2 Non-informative prior

The prior for parameters \((\alpha,\beta,\theta,\sigma^2)\) are as follows: \[ p(\alpha,\beta,\theta,\sigma^2) \propto \frac{1}{\sigma^2}, \] which represents a non-informative prior. Therefore, the joint posterior is \[ p(\alpha,\beta,\theta,\sigma^2|\mbox{data}) \propto \frac{1}{\sigma^2} \prod_{i=1}^n \left(\frac{1}{\sigma^2}\right)^{1/2}\exp\left\{-\frac{1}{2\sigma^2} \left(y_i-\alpha-\beta z_i(\theta)\right)^2 \right\}, \] where \(z_i(\theta) = x_i/(\theta+x_i)\) is explictly a function of \(\theta\). Posterior inference is approximated by a Markov chain Monte Carlo scheme that cycles through the full conditions of the four parameters, \(\alpha\), \(\beta\), \(\theta\) and \(\sigma^2\).

3 Full conditionals of \(\alpha\)

1. Show that the full conditional distribution of \(\alpha\) is as follows: \[ \alpha|\beta,\theta,\sigma^2,\mbox{data} \sim N\left[ {\bar y}-\beta {\bar z}(\theta),\sigma^2/n \right], \] where \({\bar y}=\sum_{i=1}^n y_i/n\) and \({\bar z}(\theta)=\sum_{i=1}^n z_i(\theta)/n\).

4 Full conditional of \(\beta\)

2. Show that the full conditional distribution of \(\beta\) is as follows: \[ \beta|\alpha,\theta,\sigma^2,\mbox{data} \sim N\left[\frac{\sum_{i=1}^n (y_i-\alpha)z_i(\theta)}{\sum_{i=1}^n z_i^2(\theta)}, \frac{\sigma^2}{\sum_{i=1}^n z_i^2(\theta)}\right] \]

5 Full conditional of \(\sigma^2\)

3. Show that the full conditional distribution of \(\sigma^2\) is as follows: \[ \sigma^2|\alpha,\beta,\theta,\mbox{data} \sim IG\left[\frac{n-3}{2},\frac{ \sum_{i=1}^n (y_i-\alpha-\beta z_i(\theta))^2. }{2}\right] \] # Implementing the MCMC algorithm
4. Implement the MCMC scheme and verify the trace plots for mixing and convergence of the chains. Also, plot the autocorrelation functions of the sequences of simulated draws. Finally, plot the histograms of the marginal posterior distributions for each one of the four parameters. The parameter \(\theta\) is the only one whose full conditional distribution does not belong to any known family of distributions, so we will use a Metropolis-Hastings within our MCMC scheme to sample from \(\theta\) given \((\alpha,\beta,\sigma^2)\) and the data. For the random-walk Metropolis proposal we will use \(\theta^* \sim N(\theta^{old},\tau^2)\), where \(\tau=0.01\) is the tuning variance of the proposal. Aditionally, we will start our MCMC scheme with \(\alpha^{(0)}=50\), \(\beta^{(0)}=170\), \(\theta^{(0)}=0.13\) and \(\sigma^2=126\). Run your MCMC for a burn-in period of \(M_0=10000\) draws and then keep the next \(M=10000\) draws from posterior summaries and inference.

6 Posterior predictive analysis

5. For \(x_{new} \in \{0.02,0.03,\ldots,1.10\}\), obtain MCMC approximations to $ \[ E(y_{new}|x_{new},\mbox{data}) = \int E(y_{new}|x_{new},\alpha,\beta,\theta)p(\alpha,\beta,\theta|\mbox{data})d\alpha d\beta d\theta, \] where \[ E_{MC}(y_{new}|x_{new},\mbox{data}) = \frac{1}{M} \sum_{i=1}^M E(y_{new}|x_{new},\alpha^{(i)},\beta^{(i)},\theta^{(i)}) = \frac{1}{M} \sum_{i=1}^M \left(\alpha^{(i)}+\beta^{(i)} \frac{x_{new}}{\theta^{(i)}+x_{new}}\right), \] where \((\alpha^{(i)},\beta^{(i)},\theta^{(i)})\) are the MCMC-based posterior draws from \(p(\alpha,\beta,\theta|\mbox{data})\). Plot the scatterplot of \(x_i\) vs \(y_i\) (observed data) and include in the picture the pairs \((x_{new},E(y_{new}|x_{new},\mbox{data}))\). Notice the nonlinear relationship.