Binomial vs Negative Binomial

1 Binomial

Let \(x\) be the number of successes on \(n\) (independent) trials with \(\theta\) as the probability of success of a trial, denoted by \(x|n,\theta \sim Binomial(n,\theta)\) and probability density function \[ p(x|n,\theta) = {n \choose x}\theta^x(1-\theta)^{n-x}, \qquad\qquad x=0,1,\ldots,n, \] with \[ E(x|n,\theta) = n\theta \ \ \ \mbox{and} \ \ \ V(x|n,\theta) = n\theta(1-\theta). \]

1.1 Prior and posterior for \(\theta\)

Suppose we use a simple (non-informative) uniform prior for \(\theta\), i.e.\(\theta \sim U(0,1)\). Therefore \[ p(\theta|x,n) \propto \theta^x(1-\theta)^{n-x}, \] which is the kernel of a Beta distribution with parameters \(x+1\) and \(n-x+1\).

Recall that when \(\theta \sim Beta(a,b)\), then \[ p(\theta|a,b) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}\theta^{a-1}(1-\theta)^{b-1}, \] with \(E(\theta|a,b)=a/(a+b)\) and \(V(\theta|a,b)=ab/((a+b)^2(a+b+1))\).

In our case, \(a=x+1\) and \(b=n-x+1\),so \[ E(\theta|x,n) = \frac{x+1}{n+2} \qquad \mbox{and} \qquad V(\theta|x,n) = \frac{(x+1)(n-x+1)}{(n+2)^2(n+3)}. \]

1.2 Prior predictive

The prior predictive is given by \[ p(x|n,\theta) = \int_0^1 {n \choose x}\theta^x(1-\theta)^{n-x}d\theta = {n \choose x}\frac{\Gamma(x+1)\Gamma(n-x+1)}{\Gamma(n+2)}. \]

2 Negative Binomial

Let \(y\) be the number of failures before \(m\) successes based on independent Bernoulli trials. Therefore the total number of trials is \(n=m+y\). The random variable \(y\), given \(m\) and \(\theta\), what we call the Negative Binomial with parameters \(m\) and \(\theta\), denoted by \(y|m,\theta \sim NB(m,\theta)\) with probability density function \[ p(y|\theta,m) = {m+y-1 \choose y}\theta^m(1-\theta)^y, \qquad y=0,1,2,\ldots, \] with \[ E(y|m,\theta) = \frac{m(1-\theta)}{\theta} \qquad \mbox{and} \qquad V(y|m,\theta) = \frac{m(1-\theta)}{\theta^2}. \]

2.1 Prior and posterior for \(\theta\)

Suppose, like with the Binomial case, we use a simple (non-informative) uniform prior for \(\theta\), i.e.\(\theta \sim U(0,1)\). Therefore \[ p(\theta|x,n) \propto \theta^m(1-\theta)^y, \] which is the kernel of a Beta distribution with parameters \(m+1\) and \(y+1\). \[ E(\theta|y,m) = \frac{m+1}{m+y+2} \qquad \mbox{and} \qquad V(\theta|y,m) = \frac{(m+1)(y+1)}{(m+y+2)^2(m+y+3)}. \]

2.2 Prior predictive

The prior predictive is given by \[ p(y|m,\theta) = \int_0^1 {m+y-1 \choose y} \theta^m(1-\theta)^y d\theta = {m+y-1 \choose y} \frac{\Gamma(m+1)\Gamma(y+1)}{\Gamma(m+y+2)}. \]

3 Data

Let us assume we observed the following data \[ \{0,1,1,0,0,1,0,1,1,1\} \] For this dataset, the values of \((n,x)\) and \((m,y)\) for the Binomial and Negative Binomial, respectively, are \[ (n,x) = (10,6) \qquad \mbox{and} \qquad (m,y)= (6,4). \]

3.1 Posterior distributions

\[\begin{eqnarray*} p(\theta|x,n) &\propto& \theta^x(1-\theta)^{n-x} = \theta^6(1-\theta)^4\\ p(\theta|y,m) &\propto& \theta^m(1-\theta)^y = \theta^6(1-\theta)^4, \end{eqnarray*}\] so both posterior are exactly the same. Basically, for a given dataset of 0/1 Bernoulli trials, it does not matter of the data were collected under the assumption of \(n\) iid draws or \(m+y\) iid draws until \(m\) successes are collected. Both likelihood are the same up to constants, therefore the posteriors are identical. This is an instance of the Likelihood Principle. Check my PhD lecture, where I review sufficiency principle, conditionality principle and likelihood principle, among other topics: https://hedibert.org/wp-content/uploads/2017/04/principlesofdatareduction.pdf,

3.2 Bayes factor

The Bayes factor is the ration of the predictive densities:

\[ B = \frac{p(x|n,\theta)}{p(y|m,\theta)} = \frac{{10 \choose 6}\frac{\Gamma(7)\Gamma(5)}{\Gamma(12)}} {{9 \choose 4} \frac{\Gamma(7)\Gamma(5)}{\Gamma(12)}} = \frac{{10 \choose 6}}{{9 \choose 4}} = \frac{210}{126} = 1.666667. \] In words, for the observed data, the Binomial model is slightly better than the Negative Binomial, despite the fact that both models lead to identical posterior distributions for \(\theta\).