Prehistory — From Neural Networks to the TransformerYoshua Bengio, Patrice Simard, Paolo FrasconiYoshua Bengio
Learning long-term dependencies with gradient descent is difficult
Learning long-term dependencies with gradient descent is difficult — IEEE TNN 5(2) 157–166
Why it matters
Confirmed in theory and experiments that recurrent neural networks struggle to learn long-term dependencies, motivating the development of later architectures
Source — Primary sources first; secondary reports are labelled “Press”
Pressen.wikipedia.orghttps://en.wikipedia.org/wiki/Vanishing_gradient_problemAdded in Week 35, 2026 · See that week