A curated list of resources for my custom-designed Master's degree in Statistics with a concentration in Machine Learning at Louisiana State University.
Yeah, it's a lot! I wanted to learn a variety of interesting topics with connections to Machine Learning, so I ended up taking a ton of classes.
-
- Computational Linear Algebra
- Advanced Statistical Computing
- Bayesian Analysis
- Multivariate Statistics
- Statistical Machine Learning
- Deep Learning
- Network Analysis and Causal Inference
- Advanced Statistical Modelling
- Probabilistic Graphical Models
- Longitudinal Data Analysis
- Survival and Reliability Analysis
- Databases
Probably one of the top 3 most relevant areas of study for statisticians. You could get away with little knowledge in Abstract Algebra, but you really need to master Linear Algebra. Go beyond the usual algebraic matrix calculations - try to get a deep, almost philosphical view on concepts like projectivity, spectral decomposition, sparse matrices, etc.
- Video Lectures: Matrix Methods in Data Analysis, Signal Processing, and Machine Learning - by Gilbert Strang (MIT), accompanying notes
- Linear Algebra - Paul Dawkins (Cornell University)
- MIT OpenCourseWare Lectures on Linear Algebra as Jupyter Notebooks - Juan Klopper
- Introduction to Applied Linear Algebra - Stephen Boyd (Stanford University), Lieven Vandenberghe (UCLA)
- Introduction to vectors and tensors, Vol 1: linear and multilinear algebra - Ray M Bowen, C. C. Wang
- Introduction to vectors and tensors, Vol 2: vector and tensor analysis - Ray M Bowen, C. C. Wang
I HATED this subject as an undergrad - I don't recall why though :) Now, I can barely imagine computational statistics without some approximate or numerical solutions. Almost all statistical software provide numerical solutions to all probability distributions, simulation and optimization problems. Again over here, go beyond the usual undergrad numerical topics - at least try numerical implementations/solutions to eigen-problems, smoothing and optimization, splines, random number generation, and Monte Carlo Simulations. This subject also provides good practice in computing and design in general.
- A Concise Introduction to Numerical Analysis - Douglas N. Arnold (University of Minnesota)
- Introduction to Numerical Analysis - Kyle Mandli (Columbia University)
- Computational Numerical Analysis - J. M. McDonough (University of Kentucky)
- Numerical Analysis for Engineers - Douglas Wilhelm Harder
Personally, this is the most exciting field in applied math that I have encountered so far. The overarching theme is the relationship between space and time variables to rates of change. I took this class because of its deep connections with uncertainty quantification, optimization, network analysis, image analysis and stochastic DEs (most notably, the Black-Sholes in Options Pricing).
- Notes on Partial Differential Equations - John K. Hunter (University of California at Davis)
- Computational Numerical PDEs - Kyle Mandli (Columbia University)
- COOL Applications - by Jamie Sethian (UC - Berkeley)
Well, this section is self explanatory. I will recommend you thoroughly understand the following concepts: Transformations, Independency and Copulas, Convergence, Sufficiency, Parameter Estimation and Inference, and the Exponential Family of Distributions. Dr. Luis Escobar of LSU has amazing lecture notes on these topics.
- Statistical Inference - Casella and Berger
- Stat 414/415 - Penn State
- Introduction to Probability - Dimitri P. Bertsekas, John N. Tsitsiklis (MIT)
- A Short Introduction to Probability - Dirk P. Kroese (University of Queensland)
- Probability: Theory and Examples - Rick Durrett (Duke University)
- Probability and Statistics Cookbook - Matthias Vallentin (UC Berkeley)
Here are the bitter pills that you just have to swallow :)
- MIT OpenCourseWare Lectures on Calculus - G. Strang
- An Introduction to Real Analysis - John K. Hunter (University of California at Davis)
- Introduction to Real Analysis - William F. Trench (Trinity University, Texas)
- Basic Analysis: Introduction to Real Analysis - Jiří Lebl
- Elementary Real Analysis - Thomson, Bruckner
- Lecture Notes in Real Analysis - Eric T. Sawyer (McMaster University)
- Real Analysis - C. McMullen
- Real Analysis for Graduate Students - Richard F. Bass
- Modern Real Analysis - William P. Ziemer (Indiana University)
- Mathematical Analysis Vol I - Elias Zakon
- An Introduction to Measure Theory - Terence Tao (UCLA)
- Lecture Notes in Measure Theory - Christer Borell
- A Crash Course on the Lebesgue Integral and Measure Theory - Steve Cheng
- Measure Theory - John K. Hunter (University of California at Davis)
- Measure and Integration - Dietmar A. Salamon (ETH Zürich)
- Stochastic Processes (Advanced Probability II) - Cosma Shalizi (Carnegie Mellon University)
- Modelling with Stochastic Processes - Alexandre Bouchard-Côté (University of Bristish Columbia)
- Stochastic Models and Simulation - Deena Schmidt(University of Nevada, Reno)
This module is a MUST. Enough said!
- Convex Optimization - Stephen P. Boyd (Stanford University)
- Convex Optimization - Ryan Tibshirani (Carnegie Mellon University)
This was mostly a slightly refined version of undergrad statistical methods with little bits of design of experiments. This class, I believe, was designed for grad students with little to no background in math/stats. For folks with some stats background, I highly recommend the resources below especially Dr. Cosma Shalizi's book, which is hands down, the best introductory stats book I have ever read.
- Advanced Data Analysis from an Elementary Point of View - Cosma Shalizi (Carnegie Mellon University)
- Statistical Methods: The Geometric Approach - Alan Agresti (University of Florida)
- Applied Linear Models - Kutner et al.
- Design of Experiments - Penn State
- TA Resource: Applied Statistics with R - David Dalpiaz (University of Illinois - Urbana Champaign)
- GREAT TA Resource: Statistics in Action with R
This module is almost as important as Linear Algebra.
Some of these are more 'pseudo' than actual statistical computing, but very useful content all the same.
- Numerical Recipes: The Art of Scientific Computing - and codes site -William H. Press
- Statistical Computing Methods - Mark Irwin (Harvard University)
- Statistical Computing - Ryan Tibshirani (Carnegie Mellon University)
- Statistical Computing - Cosma Shalizi (Carnegie Mellon University)
This stand-alone class on regression is one of the best classes I took at LSU. The professor, Dr. Brian Marx, effortlessly walks you through the rationale behind the nitty-gritties of regression and generalized modelling in general. You do not want to miss his classes. Alan Agresti's book below is invaluable!
- Regression: Models, Methods and Applications - B.D. Marx et al.
- Introduction to Linear Regression Analysis 5th Edition - Montgomery et al.
- The Truth About Linear Regression - Cosma Shalizi (Carnegie Mellon University)
- Foundations of Linear and Generalized Linear Models - Alan Agresti (University of Florida)
- Extending the Linear Model with R
- Example: https://cran.r-project.org/web/packages/BradleyTerryScalable/vignettes/BradleyTerryScalable.html
Again Dr. Brian D. Marx does a masterful job at teaching this class. This class introduces the concepts of Generalized Linear Modelling, with the latter half of the class focusing mainly on the analysis of categorical/discrete data.
- Categorical Data Analysis - Alan Agresti (University of Florida)
- Resources for Alan Agresti's Books
- Analysis of Categorical Data with R - Christopher R. Bilder (University of Nebraska-Lincoln)
- Analysis of Discrete Data - Penn State
- Generalized Linear Models - Mark Irwin (Harvard University)
- Some Projects
Interest in this field after my Fall 2019 summer internship.
Current reading materials is Generalized Linear and Nonlinear Models for Correlated Data: Theory and Applications Using SAS.
Best taken after after the modules on Generalized Linear Models. I found Dr. Robert Nau's notes on Time Series Analysis particularly useful. Dr. Shalizi does an equally good job.
- Statistical Forecasting: Regression and Time Series - Robert Nau (Duke University)
- Data over Space and Time - Cosma Shalizi (Carnegie Mellon University)
- Forecasting: Principles and Practice - by R. J. Hyndman and G. Athanasopoulos
- Forecasting at Scale - Prophet - Facebook see good example
- Time Series Analysis - Standford University
- Time Series Analysis - GitHub R Codes Repo
- Little Book of R for Time Series
- Interesting Examples from Peter Laurinec
- Example: Time Series with GAM
- Resource: Time Series Classification News, Algorithms and Sample Codes Archive - University of California, Riverside
- Paper: Resampling strategies for imbalanced time series forecasting - Moniz et al. 2017.
- Paper: The Impact of Bootstrap on Time Series Data
- Example: Time Series Cross Validation
- Example: Time Series Cross Validation
Just follow this amazing group, Event and Pattern Detection Laboratory - Carnegie Mellon University and this NYU Professor, Daniel B. Neill, (his other link)
See below a selection of the most relevant links to me from this exhaustive list
- Outlier Analysis - Aggarwal, Charu C.
- Outlier Ensembles: An Introduction - Aggarwal, Charu C., Sathe, Saket
- Anomaly Detection: A Tutorial
- Tutorial: Novel Machine Learning Methods for Public Health and Disease Surveillance
- Tutorial: Efficient Discovery of Heterogeneous Treatment Effects via Anomalous Pattern Detection
- Blog: Anomaly.io
| Name | Language | Pitch | License |
|---|---|---|---|
| Twitter's AnomalyDetection | R | AnomalyDetection is an open-source R package to detect anomalies which is robust, from a statistical standpoint, in the presence of seasonality and an underlying trend. | GPL |
| Linkedin's luminol | Python | Luminol is a light weight python library for time series data analysis. The two major functionalities it supports are anomaly detection and correlation. | Apache-2.0 |
| Donut | Python | Donut is an unsupervised anomaly detection algorithm for seasonal KPIs, based on Variational Autoencoders. | - |
| NASA's Telemanom | Python | A framework for using LSTMs to detect anomalies in multivariate time series data. Includes spacecraft anomaly data and experiments from the Mars Science Laboratory and SMAP missions. | custom |
| banpei | Python | Outlier detection (Hotelling's theory) and Change point detection (Singular spectrum transformation) for time-series. | MIT |
| CAD | Python | Contextual Anomaly Detection for real-time AD on streaming data (winner algorithm of the 2016 NAB competition). | AGPL |
This content is complementary to Time Series Anomaly Detection.
- Paper: Bayesian Online Changepoint Detection - by Ryan Prescott Adams, David J.C. MacKay with some inspired examples like 1, 2 and 3
- Paper: A Survey of Methods for Time Series Change Point Detection - Samaneh Aminikhanghahi and Diane J. Cook
- Example: Detection of Changes in Signals
- Paper: Detecting Correlation Changes in Multivariate Time Series: A comparison of four non-parametric change point detection methods - Jedelyn Cabrieto
Maybe the MVP in the entire masters program. With the current era of 'big data', there is unnecessarily high focus on prediction, but prediction in itself is sometimes useless without control. Randomized experiments is one way to test competing ideas or solutions, and thus control predictions or outcomes. The professor of this course, Dr. David C. Blouin is a seasoned expert in experimental design. I recommend taking this class after taking the linear modelling and or data mining classes because this class (DoE) introduces an almost artistic yet scientific paradigm to problem solving which you would not appreciate if taken prior to taking the latter classes. This class is project-heavy - roughly 2 projects every week - which helps in honing this almost artistic skill.
Although not necessary, I recommend you take this class simultaneously with an introductory course in Causal Inference.
- Design of Experiments - Dean et al.
- Desgin of Experiments - Stanford University
- Design and Analysis of Experiments - GitHub Repo
- Elements of Causal Inference: Foundations and Learning Algorithms - Peters et al.
- Tutorial: Online Experiments for Computational Social Science - Eytan Bakshy and Sean J. Taylor
- Tutorial: Propensity Score Analysis with R and a GitHub Repo
- Lecture Notes: Causal Modeling, Especially Graphical Causal Models - Cosma Shalizi(CMU)
- Lecture Notes: Estimating Causal Models - Cosma Shalizi(CMU)
- A Crash Course in Causality: Inferring Causal Effects from Observational Data
- Paper: Social Influence in Social Advertising: Evidence from Field Experiments - Eytan et al. 2012.
One critique of this class is that there is little emphasis on testable problem formulation - the statistician is relegated to only churning out design solutions (that is, model specification and analysis), and not appropriate problem formulation or experimentation. The book below offers approaches to optimal experimentation for a given problem or purpose.
- Book: Optimum Experimental Designs, with SAS - Anthony Atkinson, Alexander Donev, and Randall Tobias
Basically numerical methods to linear algebra
Although one of the most important skills a modern statistician needs to acquire, this field is barely teased in graduate statistics programs in the US. You can think of the techniques discussed in this module as machine translations or implementations of statistical algorithms. And provides answers or introduces questions like how computers solve statistical problems. Sampling and/or Resampling techniques, the EM and other deterministic algorithms, and the whole gamut of [Markov Chain] Monte Carlo algorithms including the rather famous Gibbs Sampler, and some popular machine learning optimization algorithms like Stochastic Gradient Descent are discussed. Concepts on convergence and numerical optimization are further explored. Ideas on cloud computing and parallel computing are also introduced.
Best taken in parallel with the Bayesian Analysis and Advanced Statistical Modeling modules, and after the Generalized Linear Models modules. Having a good background in both Linear Algebra and Real Analysis will for sure make life a little easier in this module.
The Computing for Data Science provides a practical introduction to modern techniques for computing with data, teaching advanced use of the R system and exploring connections to other environments such as C, python, Java, and databases. It also provides a hands-on practice on some computing paradigms, including parallel, cluster and map/reduce style computation.
- Advanced Statistical Computing with Python and accompanying repo - Chris Fonnesbeck (Vanderbilt University)
- Computing for Data Science - John Chambers and Balasubramanian Narasimhan (Stanford University)
- Computational Statistics with Python - Duke University
- Book: Introducing Monte Carlo Methods with R - Christian Robert and George Casella
- Open Source Book: Advanced Statistical Computing with R - Roger D. Peng (John Hopkins University)
- Book: An Introduction to Scientific Programming and Simulation Using R. - Jones, Owens; Maillardet, Robert & Robinson, Andrew. (2011).
I struggled in this class partly because I had a terrible load of 18 credits that semester - which is A LOT for graduate shool. I spent the subsequent semester vacation reviewing the material and I found it surprisingly coherent and accessible. I found reading materials from other schools and practicing the coding very useful.
I have come to like it so much that I am even considering a lifetime of research in Bayesian frameworks.
I would strongly recommend the following materials first before any advanced material:
- Statistical Rethinking - Richard McElreath
- Bayesian Data Analysis - Aki Vehtari
As always, it is good to look up the similar content from other schools or programs:
- Bayesian Methods and Modern Statistics - Course GitHUb Repo - Rebecca Steorts (Duke University)
- Bayesian Methods and Modern Statistics - Course Website - Rebecca Steorts (Duke University)
- Applied Bayesian Analysis - Brian Reich (North Carolina State University)
- Bayesian Inference - Brian Reich (North Carolina State University)
- Think Bayes - Allen B. Downey
- Book: A First Course in Bayesian Statistical Methods - Peter D. Hoff
- Book: Bayesian Computation with R by Jim Albert and Book's site with R Scripts, Notebooks and Blogposts
- Book: Introducing Monte Carlo Methods with R - Christian Robert and George Casella
- Probabilistic Programming and Bayesian Methods for Hackers - GitHub Repo - Cameron Davidson-Pilon
- Bayesian Modelling with Python
- Probabilistic Programming and Bayesian Methods for Hackers - Website - Cameron Davidson-Pilon
- Bayesian Modeling and Inference - Michael Jordan(UC - Berkeley)
- Learning Bayes
I took this class after the GLM modules - it kind of made this class at LSU is somewhat redundant. I suggest you read on the freely available NC State class on Nonlinear Models for Univariate and Multivariate Response as a supplement.
This course is extremely vital if you are interested in any high dimensional statistics or manifold learning in general. So I suggest you look elsewhere for relevant materials.
- Applied Multivariate Methods for Data Analysts - Dallas E. Johnson
- A First Course in Structural Equation Modeling, 2nd ed. - Tenko et al.
- Applied Multivariate Analysis - Penn State
- Nonlinear Models for Univariate and Multivariate Response - Marie Davidian (NC State University)
With the recent 'Data Science' craze there is a litany of dumbed-down materials on SL. Although good for beginners, these are woefully lacking for statisticians hoping to engineer new methodologies, improve or critique current techniques.
Dive into these as much as you can - they are the best materials on Introduction to Statistical Machine Learning out there. Take key note of concepts regarding RE-presentation of data, and projectivity.
- Advanced Topics in Statistical Modelling - Bin Li (Louisiana State University)
- Statistical Machine Learning R. Tibshirani and L. Wasserman (Carnegie Mellon University)
- Data Mining Cosma Shalizi (Carnegie Mellon University)
- Book: An Introduction to Statistical Learning with Applications in R - Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani
- Book: The Elements of Statistical Learning: Data Mining, Inference, and Prediction - Trevor Hastie, Robert Tibshirani and Jerome Friedman
- Book: Computer Age Statistical Inference - Bradley Effron and Trevor Hastie
- MOOC: Outlier Detection Algorithms in Data Mining and Data Science
- Blog Post: Winning Solutions - Kaggle Competitions
Stanford Univeristy has made this topic and all its allied topics amazingly accessible, and freely available online. Just burry yourself in these 4 classes on Deep Learning:
- CS230 Deep Learning
- CS231n: Convolutional Neural Networks for Visual Recognition
- CS224n: Natural Language Processing with Deep Learning
- STATS 385: Analyses of Deep Learning
I found these other materials also useful:
- Machine Learning - Stanford by Andrew Ng in Coursera (2010-2014)
- Deep Learning with Python - Francois Chollet
- Deep Learning by Yoshua Bengio, Ian Goodfellow and Aaron Courville (05/07/2015)
- Neural Networks and Deep Learning by Michael Nielsen (Dec 2014)
- Artificial Intelligence: A Modern Approach
- Artificial intelligence and machine learning: Topic wise explanation
- MOOC: Advanced Deep Learning with Keras by Zach Mayer
Discovering this area of statistics has been more than fulfilling - it bridges several disciplines like Philosophy (although more subtly), Behavorial Psychology and Economics, Geometric Topology and Probabilistic Modelling. I have dedicated most of my weekends perusing as much material on these topics as I can.
My current reading materials or courses are based on the following:
- Statistical Network Analysis - Cosma Shalizi (Carnegie Mellon University)
- Advanced Statistical Network Models - Cosma Shalizi (Carnegie Mellon University)
- Social and Information Network Analysis - Jure Leskovec (Stanford University)
- Machine Learning with Graphs - Jure Leskovec (Stanford University)
- Networks, Crowds, and Markets: Reasoning About a Highly Connected World - David Easley and Jon Kleinberg
- Statistical Analysis of Network Data with R - Eric D. KolaczykGábor Csárdi
- Bayesian Networks: With Examples in R - Marco Scutari, Jean-Baptiste Denis
- Bayesian Network in R with Applications in Systems Biology - Radhakrishnan NagarajanMarco ScutariSophie Lèbre.
- Data Science and Complex Networks: Real Case Studies with Python - Guido Caldarelli and Alessandro Chessa
- A Crash Course in Causality: Inferring Causal Effects from Observational Data
- Lecture Notes: Causal Modeling, Especially Graphical Causal Models - Cosma Shalizi(CMU)
- Lecture Notes: Estimating Causal Models - Cosma Shalizi(CMU)
- A Crash Course in Causality: Inferring Causal Effects from Observational Data
- Example: Exponential Random Graph Modelling
This module is a revisit of Generalized Linear Modelling and Statistical Machine Learning that emphasize model building skill and intuition with relatively advanced statistical rigor and computation. Empirical model selection techniques (e.g. basis degree, effective degrees of freedom) and regularization, and extensions to linear regression via basis functions and kernels are further explored. The relationship between randomness and determinism is also further probed.
- Advanced Topics in Statistical Modelling - Bin Li (Louisiana State University)
- Chaos, Complexity, and Inference - Cosma Shalizi (Carnegie Mellon University)
- Statistical Methods for Analysis With Missing Data - Marie Davidian(NC State University)
- Example: Preprocessing Data with Caret
- Paper: Flexible Smoothing with B-Splines and Penalties - Paul H. C. Eilers and Brian D. Marx
- Paper: Importance Sampled Learning Ensembles - Friedman and Popescu
- Paper: Gradient Directed Regularization - Friedman and Popescu
- Paper: Predictive Learning vis Rule Ensembles - Friedman and Popescu
- Paper: Kernels and Ensembles: Perspectives on Statistical Learning - Mu Zhu
- Book: Generalized Additive Models: An Introduction with R, Second Edition - Simon N. Wood
- Simon Wood's Website
- MOOC: Nonlinear Models with GAM
- Example: An introduction to Generalized Additive Models with R
- Example: Generalised Additive Mixed Models
- Paper: Convex Regression With Interpretable Sharp Partitions - Ashley Petersen, Noah Simon, Daniela Witten
- Tutorial: Information Filtering for arXiv.org: Bandits, Exploration vs. Exploitation, and the Cold Start Problem - Peter Frazier (Cornell University)
- Probabilistic Graphical Models - Stefano Ermon (Stanford University)
- Probabilistic Graphical ModelsProbabilistic Graphical Models - Eric Xing (Carnegie Mellon University)
I took this class simultaneously with Dr. Kevin S. McCarter's Multivariate Analysis.
- Longitudinal Data Analysis - Marie Davidian(NC State University)
- Applied Multivariate and Longitudinal Data Analysis
- Tutorial: Propensity Score Matching
Spontaneous and somewhat hobby-like reading on:
I have also been trying to learn how to model survival data with Deep Learning algorithms by reading some of the papers from the link below.
soon...
soon...
List incomplete...
- Mining Massive Datasets - Stanford University Online (Lagunita)
- Data Intensive Scientific Computing - edX (University of Notre Dame)
- Convolutional Neural Networks for Text - Graham Neubig (Carnegie Mellon University)
- Big Data Analytics Using Spark - edX (UC San Diego)
- The Ultimate Hands-On Hadoop - Tame your Big Data - Udemy
- High-performance Computing for Reproducible Genomics - edX (Harvard University)
- Data Visualization and D3.js - Udacity, and a ton of Tutorials
- CS50's Web Programming with Python and JavaScript - edX (Harvard University)
soon...
soon...