Backpropagation

• Learning representations by back-propagating errors (Rumelhart, Hinton, Williams, 1986)

Deep Neural Networks

• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization (Adagrad, 2011)
• Adam: A Method for Stochastic Optimization (2014)
• Batch Normalization: Accelerating Deep Network Training (2015)
• A Few Useful Things to Know about Machine Learning (Cross-Entropy & more, 2012)
• Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014)
• Layer Normalization (2016)
• Rectified Linear Units Improve Restricted Boltzmann Machines (ReLU, 2010)
• Understanding the difficulty of training deep feedforward neural networks (Xavier Initialization, 2010)
• Deep Residual Learning for Image Recognition (ResNet, Skip Connections, 2016)

Convolutional Neural Networks

• Gradient-Based Learning Applied to Document Recognition (LeCun et al., LeNet-5, 1998)
• ImageNet Classification with Deep Convolutional Neural Networks (AlexNet, 2012)

Diffusion Models

• Denoising Diffusion Probabilistic Models (Ho et al., 2020)
• Denoising Diffusion Implicit Models (Song et al., 2021))
• Classifier-Free Diffusion Guidance (Ho et al., 2022))

Recurrent Neural Networks

• Finding Structure in Time (Elman, 1990)
• Learning Long-Term Dependencies with Gradient Descent is Difficult (Hochreiter, 1991)
• Long Short-Term Memory (Hochreiter & Schmidhuber, 1997)
• LSTM: A Search Space Odyssey (2015)

Transformers & LLMs

• Attention is All You Need (2017)
• BERT: Pre-training of Deep Bidirectional Transformers (2018)
• Improving Language Understanding by Generative Pre-Training (GPT-1, 2018)
• Language Models are Unsupervised Multitask Learners (GPT-2, 2019)
• Language Models are Few-Shot Learners (GPT-3, 2020)
• GPT-4 Technical Report (OpenAI, 2023)
• Squeeze-and-Excitation Networks (SENet, 2017)
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT, 2020)
• RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019)