Conclusion: outside of a very brief period in which pre-training with Unsupervised Learning was shown to be helpful, Deep Learning has largely been about hardware brute force, and learning how to use brute force to solve problems.

Terms I need to learn more about

• Pattern Deformations

• Hessian-free learning

• Batch Normalisation (Thanks A Breitman)

• Competing Units

Neural networks have been around for a long time. What exactly changed between the 80s and now? Why the resurgence of late?

Reading this review paper (page 4),

1. Shallow NN-like models with few such stages have been around for many decades if not centuries, and models with several successive nonlinear layers of neurons date back at least to the 1960s and 1970s
2. Training of deeper architectures (ANNs with more layers, including RNNs) only became feasible through the use of unsupervised learning techniques in the 2000s
3. Training of deeper architectures without unsupervised learning became possible later

I'm a bit puzzled by (3), because it's not clear where it differs from (1). There are 3 candidates:

1. More data
2. More computation
3. Better techniques

More data

I think this is actually conflated with More computation below. Without fast computation, you can't get through enough data.

More computation

Page 23 of the reference - in 2010, a new MNIST record was set using backpropagation and pattern deformations, both of which are decades old.

these results seemed to suggest that advances in exploiting modern computing hardware were more important than advances in algorithms

Better techniques

Unsupervised learning followed by supervised learning

It seems that there was a year or two where this was popular, and then it just went away.

Vanishing / Exploding Gradients

LSTMs solve this, and are a recent invention. ReLU activations do not explode but have other problems.

Proper Initialization

(Thanks to A Breitman for mentioning this)

When they are not suffering from vanishing / exploding gradients, activation functions can be too linear. Reference

I looked into the topic a bit more, and found this exchange, which I think makes sense to me.

From The Diamond Age:

That the highest levels of the society received news written with ink on paper said much about the steps New Atlantis had taken to distinguish itself from other phyles.

Now nanotechnology had made nearly anything possible, and so the cultural role in deciding what should be done with it had become far more important than imagining what could be done with it. One of the insights of the Victorian Revival was that it was not necessarily a good thing for everyone to read a completely different newspaper in the morning; so the higher one rose in the society, the more similar one's Times became to one's peers'.

News has an important social function, and it's often more important to read the same news as your peers than it is to read the news that is the most interesting to you.