Alexey Ivakhnenko and Valentin Lapa Train Multilayer Networks

The story of deep learning's ascendance in AI research over decades is a long and twisting tale. But it's remarkable how quickly its theory and potential were recognized.

In their 1965 book Cybernetic Predicting Devices, Soviet scientists Alexey Ivakhnenko and Valentin Lapa set out a way to train networks with many layers. The approach, known as the Group Method of Data Handling (GMDH), grew a network one layer at a time. Each unit fitted a simple curve to its inputs and any unit that did poorly on fresh test data was dropped. Ivakhnenko likened it to plant breeding.

In 1971, at the Institute of Cybernetics in Kyiv, Ivakhnenko described a GMDH network eight layers deep. Jürgen Schmidhuber’s 2015 survey of deep learning calls such networks “perhaps the first” deep learning systems of the multilayer perceptron type. No big deal.