Course: BAYESIAN LEARNING  MPE-2024
Professor: Hedibert Freitas Lopes - www.hedibert.org (hedibertfl@insper.edu.br)

Teaching assistant: Luiza Tuler Veloso (luizatv@insper.edu.br)

Syllabus

Homework assignments: 

• HW1

Additional examples (to be) discussed in class: 

• Class of April 23rd 2024:
  • Markowitz's minimum variance portfolio: Static vs Time-varying covariance estimation
  • Salaries for Professors: Bayesian linear regression
• Class of April 30ht 2024:
• Class of May 7th 2024:
• Class of May 14th 2024:
• Class of May 21th 2024:
• Class of May 28th 2024:
• Class of June 4th 2024:
• Class of June 11th 2024:
• Class of June 18th 2024:
• Class of June 25th 2024:

Course notes (+ R code & references)

• Bayesian ingredients  
  • Bayesian update: a toy example
  • Flipping one of three coins three times and observing three heads (R code)
  • Chapter 1 of "Bayes' Rule: A tutorial Introduction to Bayesian Analysis (Slide)
  • Tiago Mendonca's shiny for the physicists example
  • Phisycists A, B, C and D: Normal model and 4 priors

• Bayesian computation
  • Learning the number of degree of freedom of a Student's t  (Rmarkdown)
  • Bayesian regression with the normal-gamma (NG) prior
  • A bit of Monte Carlo simulation and integration
  • Back to the physicists example
  • Normal vs skew-normal distributions: an exercise in Bayesian learning
  • The potential fallacy of using the prior as proposal in SIR algorithms
  • Bayesian learning of correlation in bivariate normal data via SIR algorithm
  • Smith and Gelfand (1992) Bayesian Statistics without Tears: A Sampling-Resampling Perspective. The American Statistician, 46(2), 84-88.

• Bayesian linear regression
  • CAPM regression
  • Return on education:  Gaussian linear model with conjugate prior
  • Boston housing data: ML and Bayesian inference
  • Motorcycle example
  • Regressing weight on height: SIR for Gaussian and Student's t models
  • Gibbs sampler in action: Bayesian linear regression
  • iid Bernoulli or logit regression or probit regression? (Rmarkdown)
  • Bayesian Poisson regression model versus i.i.d. Poisson model
  • Another example of Poisson regression
  • A few R packages for Bayesian inference in linear models
  • Stan/rstan: Bayes Sparse Regression

• Bayesian classification via logistic regression
  • Sparse logistic regression for the spam/ham dataset (data)
  • Marketing campaigns of a Portuguese banking institution (data)
  • Sparse logistic regression: comparison of regularization and Bayesian implementations
  • Gelman, Jakulin, Pittau and Zu (2008) A weakly informative default prior distribution for logistic and other regression models, AOAS, 2(4), 1360-1383.
  • Polson, Scott and Windle (2013) Bayesian Inference for logistic models using Pólya-Gamma latent variables, JASA, 108, 1339-1349.

• Other important modeling structures
  • Factor models (Additional material)
    • Indice de Desenvolvimento Humano
    • US market (data)
    • Bayesian factor models
    • Reducao de ruido com PCA e FA
  • Finite mixture of distributions
• Treed models
  • Classification and regression trees (CART)
    • CART step-by-step for the Boston housing data
    • CART for the motorcycle data
    • ROC and AUC
  • Bayesian CART
  • Bootstrap aggregating  (bagging)
    • Ilustracao do bootstrap
  • Boosting (weak/stronger learners)
  • Random forests
    • Predicao da covid-19 com dados publicos do Hospital Israelita Albert Einstein/SP
  • Bayesian additive regression trees (BART) + (Rob's notes) + (a list of slides/examples/people/papers)
• Latent Dirichlet Allocation (LDA) + short list of slides and papers
  • Example 1 & 2: Twitter & BBC
  • Example 3: Dickens, Wells, Verne or Austen?
• Neural Networks + short list of slides and papers

Additional supporting material

• Stan/rstan for posterior inference: Hamiltonian MC (HMC) methods - by Hedibert Lopes (February 2021)
• MC and MCMC: Key References - by Hedibert Lopes (February 2021)
• R packages for Bayesian linear regression - by Hedibert Lopes (February 2020)
• R packages for Bayesian Econometrics - by Hedibert Lopes (March 2014)
• CRAN Task View on Bayesian Inference (July 2023)
• Mathematics for Machine Learning - by Deisenroth, Faisal and Ong (2020)
• Conceitos e analises estatisticas com R e JASP - by  Luis Anunciação (September 2021)
• Data Science, Marketing and Business by Pedro Fernandes & Paulo Marques (October 2019)
• Aprendizado de Máquina: Uma Abordagem Estatística (by Rafael Izbicki & Tiago Mendonça)
• Estatística e Ciência de Dados (by Pedro Morettin & Julio Singer)

Course: ADVANCED BAYESIAN ECONOMETRICS PhD-2024
Professor: Hedibert Freitas Lopes - www.hedibert.org

Objective: The end of the course goal is to allow the student to critically decide between a Bayesian, a frequentist or Bayesian-frequentist compromise when facing real world problems in the fields of micro- and macro-econometrics and finance, as well as in quantitative marketing, strategy and business administration.  With this end in mind, we will visit well known Bayesian issues, such as prior specification and model comparison and model averaging, but also study regularization via Bayesian LASSO, Spike-and-Slab and related schemes, “small n, large p” issues, Bayesian statistical learning via additive regression trees, random forests, large-scale VAR and (dynamic) factor models.

Course description: Basic ingredients: prior, posterior, and predictive distributions, sequential Bayes, conjugate analysis, exchangeability, principles of data reduction and decision theory.  Model criticism: Bayes factor, computing marginal likelihoods, Savage-Dickey ratio, reversible jump MCMC, Bayesian model averaging and deviance information criterion.  Modern computation via (Markov chain) Monte Carlo methods: Monte Carlo integration, sampling-importance resampling, Gibbs sampler, Metropolis-Hastings algorithms.  Mixture models, Hierarchical models, Bayesian regularization, Instrumental variables modeling, Large-scale (sparse) factor modeling, Bayesian additive regression trees (BART) and related topics, Dynamic models, Sequential Monte Carlo algorithms, Bayesian methods in microeconometrics, macroeconometrics, marketing and finance.

• Part I Bayesian ingredients: i) Inference: likelihood, prior, predictive and posterior distributions; ii) Model criticism: Marginal likelihoods, Bayes factor, model averaging and decision theory; and iii) Computation: An introduction (Markov chain and sequencial) Monte Carlo methods.
• Part II Multivariate models: i) Large-scale vector autoregressive models; ii) Factor models and other dimension reduction models; and iii) Time-varying high-dimensional covariance models.
• Part III Modern Bayesian statistical learning: i) Mixture models and the Dirichlet process: handling non-Gaussian models; ii) Regularization: sparsity via shrinkage and variable selection; iii) Large vector-autoregressive and factor models: combining sparsity and parsimony; iv) Classification and support vector machines; v) Regression trees and random forests; and vi) Latent Dirichlet allocation: Text as data, text mining.

Take-home midterm exam: To be added 

Homework assignments: To be added

Paper presentations:  To be added

Examples developed in class:  To be added

LECTURE NOTES

PART I: Bayesian ingredients

1. Basic Bayes
2. Exchangeability
3. Principles of data reduction
4. More on estimators
5. Decision theory
6. Bayesian model criticism (pages 1-6 & 32-34)
7. Additional reading material:
   • Chapter 2 of Gamerman and Lopes (2006) - Compact, but easy to read.
   • Chapters 2-4 of Migon, Gamerman and Louzada (2014) - Integrates classical and Bayesian inference.
   • Chapter 1 and 2 of Gelman et al. (2013) - Application-oriented.
   • Chapter 4 (Sections 4.1-4.4) of Berger (1985) – More technical.
   • van de Schoot, R., Depaoli, S., King, R. et al. Bayesian statistics and modelling. Nat Rev Methods Primers 1, 1 (2021). 
8. Discussion about p-values

PART II: Bayesian Computation

1. Monte Carlo (MC) methods
2. Markov chain: a brief review
3. Markov chain Monte Carlo (MCMC) algorithms
4. Hamiltonian Monte Carlo: A toy example
5. Stan/rstan for posterior inference: Hamiltonian MC (HMC) methods
6. MC and MCMC: Key References
7. More on Bayesian model criticism

PART III: Bayesian Learning

1. Fundamentos de Aprendizagem Estatística + R code + MC exercise
2. Multiple linear regression: selection, shrinkage, sparsity
   • Motorcycle example
   • Slides from the 2015 School of Time Series and Econometrics tutorial
   • Hahn, He and Lopes (2018) Gaussian linear regression with arbitrary sparsity + slides of a talk + R package bayeslm
   • Fava and Lopes (2021) The illusion of the illusion of sparsity + slides of a talk + UFPE webinar (62' and forward)
   • van Erp, Oberski and Mulder (2019) Shrinkage priors for Bayesian penalized regression, Journal of Mathematical Psychology, 89, 31-50.
   • Michael Betancourt's Bayes Sparse Regression (stan/rstan example)
3. Classification: logistic regression and discriminant analysis
   • Sparse logistic regression for the spam/ham dataset (data)
   • Marketing campaigns of a Portuguese banking institution (data)
   • Sparse logistic regression: comparison of regularization and Bayesian implementations
   • Gelman, Jakulin, Pittau and Zu (2008) A weakly informative default prior distribution for logistic and other regression models, AOAS, 2(4), 1360-1383.
   • Polson, Scott and Windle (2013) Bayesian Inference for logistic models using Pólya-Gamma latent variables, JASA, 108, 1339-1349.
4. Bayesian factor analysis (BFA)
   • Bayesian factor analysis in R: a simple illustration (additional routines)
   • Factor models: an annotated bibliography (2003)
   • Lopes (2014)
5. Principal components analysis (PCA), PCA-based and FA-based regressions
6. Classification and regression trees (CART)
7. Bayesian CART
8. Bootstrap aggregating (bagging)
   • Bootstrap illustration
   • CART step-by-step for the Boston housing data
   • CART for the motorcycle data
9. Bayesian additive regression trees (BART)
10. Latent Dirichlet Allocation (LDA)
    • Twitter + BBC + Dickens, Wells, Verne or Austen? + The cat in the hat by Dr. Seuss
11. Neural Networks

Complementary material to PART III

1. Boosting (weak/stronger learners)
2. Random forests
3. Bayesian instrumental variables
4. General linear and hierarchical models
5. Limited dependent variable models
6. Finite mixture of distributions
7. Spatial models
8. P.Richard Hahn's top 25 books on Statistics, Causal Inference, Statistical Computing, Machine Learning and Data Science

Bibliography: Bayesian econometrics

1. Zellner (1971) An Introduction to Bayesian Inference in Econometrics
2. Goel and Iyngar (1992) Bayesian Analysis in Statistics and Econometrics
3. West and Harrison (1997) Bayesian Forecasting and Dynamic Models (2nd edition)
4. Dorfman (1997) Bayesian Economics Through Numerical Methods
5. Bauwens, Lubrano and Richard (2000) Bayesian Inference in Dynamic Econometric Models
6. Koop (2003) Bayesian Econometrics
7. Geweke (2005) Contemporary Bayesian Econometrics and Statistics
8. Lancaster (2004) Introduction to Modern Bayesian Econometrics
9. Rossi, Allenby and McCulloch (2005) Bayesian Statistics and Marketing
10. Prado and West (2010) Time Series: Modeling, Computation and Inference
11. Geweke, Koop and Van Dijk (2011) The Oxford Handbook of Bayesian Econometrics
12. Greenberg (2013) Introduction to Bayesian Econometrics
13. Herbst and Schorfheide (2015) Bayesian Estimation of DSGE Models
14. Chan, Koop, Poirier and Tobias (2019) Bayesian Econometric Methods (2nd edition)
15. Broemeling (2019) Bayesian Analysis of Time Series
16. Bernardi, Grassi and Ravazzolo (2020) Bayesian Econometrics

Bibliography: Bayesian statistics

1. Berger (1985) Statistical Decision Theory and Bayesian Analysis
2. Bernardo and Smith (2000) Bayesian Theory
3. Gelman and Hill (2006) Data Analysis Using Regression and Multilevel/Hierarchical Models
4. Robert (2007) The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation
5. Hoff (2009) A First Course in Bayesian Statistical Methods
6. Carlin and Louis (2009)  Bayesian Methods for Data Analysis (3rd edition)
7. Gelman, Carlin, Stern, Dunson, Vehtari and Rubin (2016) Bayesian Data Analysis
8. Migon, Gamerman and Louzada (2015) Statistical Inference: An Integrated Approach (2nd edition)
9. Reich and Ghosh (2019) Bayesian Statistical Methods
10. Held and Sabanes-Bove (2020) Likelihood and Bayesian Inference: With Applications in Biology and Medicine

Bibliography: Bayesian computation

1. Gilks, Richardson and Spiegelhalter (1995) Markov Chain Monte Carlo in Practice
2. Doucet, de Freitas and Gordon (2001) Sequential Monte Carlo Methods in Practice
3. Robert and Casella (2004) Monte Carlo Statistical Methods (2nd edition)
4. Gamerman and Lopes (2006) MCMC: Stochastic Simulation for Bayesian Inference, Second Edition
5. Marin and Robert (2007) Bayesian Core: A Practical Approach to Computational Bayesian Statistics
6. Albert (2009) Bayesian Computation with R
7. Brooks, Gelman, Jones and Meng (2011) Handbook of Markov Chain Monte Carlo
8. Givens and Hoeting (2012) Computational Statistics (2nd edition)
9. Marin and Robert (2014) Bayesian Essentials with R (complete solution manual)
10. Turkman, Paulino and Mueller (2019) Computational Bayesian Statistics: An Introduction
11. McElreath (2020) Statistical Rethinking: A Bayesian course with Examples in R and STAN
12. Chopin and Papaspiliopoulos (2020) An Introduction to Sequential Monte Carlo

Bibliography: (Bayesian) statistical learning

1. Bishop (2006) Pattern Recognition and Machine Learning
2. Hastie, Tibshirani and Friedman (2008) The Elements of Statistical Learning, 2nd edition
3. Murphy (2012) Machine Learning: A Probabilistic Perspective
4. Barber (2012) Bayesian Reasoning and Machine Learning
5. James, Witten, Hastie and Tibshirani (2013) An Introduction to Statistical Learning
6. Hastie, Tibshirani and Wainwright (2015) Statistical Learning with Sparsity
7. Efron and Hastie (2016) Computer Age Statistical Inference: Algorithms, Evidence and Data Science
8. Fernandez and Marques (2018) Data Science, Marketing and Business
9. Izbicki & Santos (2020) Aprendizado de máquina: uma abordagem estatística

Bibliography: Classical Monte Carlo papers

• Metropolis and Ulam (1949) The Monte Carlo method. JASA, 44, 335-341.
• Metropolis, Rosenbluth, Rosenbluth, Teller and Teller (1953) Equation of state calculations by fast computing machines. Journal of Chemical Physics, 21, 2087-1092.
• Hastings (1970) Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57, 97-109.
• Peskun (1973) Optimum Monte Carlo sampling using Markov chains. Biometrika, 60, 607-612.
• Besag (1974) Spatial Interaction and the Statistical Analysis of Lattice Systems. JRSS-B, 36, 192-236.
• Kirkpatrick, Gelatt and Vecchi (1983) Optimization by Simulated Annealing. Science, 220 (4598), 671-680.
• Geman and Geman (1984) Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Analysis and Machine Intelligence, 6, 721-741.
• Pearl (1987) Evidential reasoning using stochastic simulation of causal models. Artificial intelligence, 32, 245-257.
• Tanner and Wong (1987) The Calculation of Posterior Distributions by Data Augmentation. JASA, 82, 528-540.
• Geweke (1989) Bayesian Inference in Econometric Models Using Monte Carlo Integration. Econometrica, 57, 1317-1339.
• Gelfand and Smith (1990) Sampling-Based Approaches to Calculating Marginal Densities. JASA, 85, 398-409.
• Casella and George (1992) Explaining the Gibbs Sampler. The American Statistician, 46, 167-174.
• Gilks and Wild (1992) Adaptive Rejection Sampling for Gibbs Sampling. Applied Statistics, 41, 337-348.
• Smith and Gelfand (1992) Bayesian Statistics without Tears: A Sampling-Resampling Perspective. The American Statistician, 46, 84-88.
• Chib and Greenberg (1995) Understanding the Metropolis-Hastings algorithm. The American Statistician, 49, 327-335.

Course: Advanced Econometrics MPE-2024
Professor: Hedibert Freitas Lopes - www.hedibert.org

Objective: The main objective of this course is to introduce basic aspects of microeconometrics, macroeconometrics, statistical learning, and Bayesian learning that are necessary for the master's degree program.

Course description:Regression with endogeneity, regression with measurement error, instrumental variables, potential outcomes, Neyman-Rubin model, selection bias, reverse causality, and omitted variables, panel data, hierarchical models, fixed effect, and random effect, difference-in-differences methods, ARIMA models; long memory; unit root, GARCH models, and stochastic volatility. Vector autoregressive models, factorial models with stochastic volatility, and multivariate models with time-varying parameters, logistic regression; performance metrics for classification, training and testing data; cross-validation, bias-variance trade-off, prior, posterior, and predictive distributions; sequential Bayes and conjugate analysis, Monte Carlo methods.

• Regression with endogeneity; instrumental variables; potential outcomes.
• Neyman-Rubin model; selection bias; reverse causality, and omitted variables.
• Panel data and hierarchical models; difference-in-differences methods.
• ARIMA; long memory; integration/co-integration.
• State space models; modeling variance; multivariate time series.
• Logistic regression; training and testing samples; validation; bias-variance trade-off.
• Prior, posterior, and predictive distributions; sequential Bayes; conjugate analysis.
• Monte Carlo methods; Markov Chain Monte Carlo; sequential Monte Carlo.

Teaching assistant: To be announced 

Take-home midterm exam: To be announced

Homework assignments: To be added

1. HW1
2. HW2
3. HW3 
4. HW4 
5. HW5 

Basic bibliography

• Mostly Harmless Econometrics: An Empiricist's Companion (Angrist and Pischke, 2009)
• Analysis of Financial Time Series, 3rd Edition (Tsay, 2010)
• An Introduction to Statistical Learning (James, Witten, Hastie and Tibshirani, 2023) – https://www.statlearning.com
• Introduction to Bayesian Econometrics (Greenberg, 2013)

Additional bibliography

• Introduction to Econometrics, 3^rd edition (Stock and Watson, 2010)
• Introductory Econometrics: A Modern Approach (Wooldridge, 2012)
• Time Series Analysis (Hamilton, 1994)
• Aprendizado de Máquina: Uma Abordagem Estatística (Izbicki and Mendonça, 2020) - http://www.rizbicki.ufscar.br/ame/
• Estatística e Ciência de Dados (Morettin and Singer, 2021) -https://www.ime.usp.br/~pam/cdadosf3.pdf
• Introduction to Modern Bayesian Econometrics (Lancaster, 2004)
• Bayesian Econometric Methods, 2a edição (Chan, Koop, Poirier and Tobias)
• Bayesian Statistics and Marketing (Rossi, Allenby and McCulloch, 2005)
• Time Series: Modeling, Computation, and Inference (Prado and West, 2010)

Paper presentations: To be announced 

TEACHING MATERIAL: To be added

PART I: Microeconometrics 

• Regression with endogeneity; instrumental variables; potential outcomes.
• Neyman-Rubin model; selection bias; reverse causality, and omitted variables.
• Panel data and hierarchical models; difference-in-differences methods.

PART II: Macroeconometrics 

• ARMA and long memory.
• Integration and cointegration.
• State space models.
• Time-varying variance.
• Multivariate time series.

PART III: Statistical learning 

• Logistic regression.
• Training and testing samples.
• Validation.
• Bias-variance trade-off.

PART IV: Bayesian learning

• Prior, posterior, and predictive distributions.
• Sequential Bayes and conjugate analysis.
• Bayesian decision.
• Monte Carlo & Markov Chain Monte Carlo.