Foundational ML reads
A recent recommendation thread listed classic textbooks for a rigorous ML foundation — notably 'Information Theory, Inference, and Learning Algorithms' and 'The Elements of Statistical Learning' — as core reads for practitioners (x.com). The posts framed these books as complementary: one covers probabilistic and information‑theoretic framing, the other delivers statistical learning techniques and model selection detail (x.com).
Machine learning is a way to teach computers from examples, and two older textbooks still anchor many rigorous reading lists: David MacKay’s *Information Theory, Inference, and Learning Algorithms* and *The Elements of Statistical Learning* by Trevor Hastie, Robert Tibshirani, and Jerome Friedman. (inference.org.uk) (hastie.su.domains) MacKay’s book was published by Cambridge University Press in September 2003 as a 640-page text, and its table of contents runs from probability and entropy to coding theory, neural networks, Monte Carlo methods, and clustering. (inference.org.uk 1) (inference.org.uk 2) Hastie, Tibshirani, and Friedman first published *The Elements of Statistical Learning* in 2001, then released a second edition in February 2009; the authors’ site lists the corrected 12th printing from January 2017 and describes the book as 745 pages. (hastie.su.domains 1) (hastie.su.domains 2) (hastie.su.domains 3) The two books teach different parts of the same subject. MacKay starts with uncertainty, compression, and Bayes’ rule — the math for updating beliefs from evidence — while *The Elements of Statistical Learning* organizes prediction problems around tools such as linear methods, trees, boosting, support vector machines, and model assessment. (inference.org.uk) (link.springer.com) That split helps explain why both keep resurfacing in recommendation threads even after the rise of deep learning courses and software-first tutorials. One book gives a probabilistic view of learning; the other gives a catalog of statistical methods for fitting models and checking whether they generalize beyond the training data. (inference.org.uk) (link.springer.com) MacKay also made his text unusually accessible for a graduate-level book: the full PDF has long been available from his site, along with chapter files and solutions pages. (inference.org.uk 1) (inference.org.uk 2) The Stanford-hosted site for *The Elements of Statistical Learning* likewise offers a free PDF with Springer’s permission, plus chapter-by-chapter resources and notes on revisions in the second edition. (hastie.su.domains) (hastie.su.domains) Neither book is a beginner’s first stop. MacKay assumes comfort with probability and calculus, and *The Elements of Statistical Learning* says its emphasis is on concepts rather than mathematics but still covers topics at a level aimed at statisticians and technically trained readers. (inference.org.uk) (link.springer.com) For readers who want a stricter foundation than prompt engineering guides or framework tutorials, these books still map the field’s older core: how to measure uncertainty, how to fit predictors, and how to tell when a model is learning signal instead of noise. (inference.org.uk) (link.springer.com)
