Paul Werbos Proposes Backpropagation to Train Neural Networks

In a major advancement in AI and Machine Learning research, Paul Werbos proposed using backpropagation to train neural networks at the “System Modeling and Optimization” conference in New York. The method was initially introduced by Seppo Linnainmaa in 1970 for automatic differentiation in complex systems.

Werbos’s adaptation of this algorithm marked a significant advancement in neural network training, enabling the efficient adjustment of network weights by propagating errors backward through the layers.

Jürgen Schmidhuber describes it as “essentially an efficient way of implementing Leibniz’s chain rule for deep networks.”

Backprop's roots go back further, though, to control theory: Henry J. Kelley published a precursor in 1960 for computing optimal flight paths, followed by Arthur E. Bryson and Stuart Dreyfus. Werbos’s own 1974 thesis included a preliminary discussion of applying it to neural networks.

This breakthrough became foundational to modern deep learning, influencing the development of widely used AI frameworks like TensorFlow and PyTorch.