The first rule of Improvisional Comedy the Yes, And rule, compels one to

accept what the other person has created ("Yes") and then add something to it ("And")

This is something that a skilled conversationist is good at. It poses trouble for the discussion of critical analyses however, because such discussions involve acts of disagreement ("No" - the negation of something the other person said) and of refocusing the discussion are to reduce incidental and nonessential complexity.

Being able to disagree productively and reach consensus regarding topic definitions is difficult, and something I've only managed to achieve with a handful of people who are all much more emotionally mature than I am. That's really mostly other people figuring that I care much more about the topic than they do, however, and hardly an ideal resolution to the issue.

Ideally, in the face of disagreement, you would simply go up one layer in the abstraction stack, agree on the approach, and then figure out the specific bit of disagreement. By social convention, you start with the premise that both parties are rational, after which the search for the mistaken logic (which will be clear when identified) or for the differing premise can commence.

Hardly ever happens that way. Usually there's too much ego involved, and enough entanglement with social status happens that it's difficult to talk about the topic in isolation.

Conclusion: The pollution now (December) is from China, the pollution in November was locally generated.

Earlier this year, there was a bout of posts about whether the air pollution in Taiwan came from China. The air is apparently pretty bad right now, and this time I think it's clear that it's from China. I just pulled this screenshot from waqi.info:

 

Observe that the pollution in Yilan Country, southeast of Taipei, is not that different from Taipei itself. This is what you would expect if the pollution came from far away. Compare this to the map from November (http://www.thenewslens.com/post/243421/):

 

Notice the huge disparity in homogeneity. Also, if I remember correctly, back then the wind was also unusually still. I think based on these facts, I would conclude that the pollution in November was locally generated and the pollution now is from a far away source, probably China. (Disregard the low pollution directly to the West, the wind doesn't come from there. Also, there could be a time lag, so even the pollution levels in the direction of the wind source at the same point in time are not immediately relevant.)

If I had all the historical data in a csv, could probably do a better analysis than two snapshots, if anyone could point me to a source I'd be happy to do that.

Conclusion: outside of a very brief period in which pre-training with Unsupervised Learning was shown to be helpful, Deep Learning has largely been about hardware brute force, and learning how to use brute force to solve problems.

Terms I need to learn more about

• Pattern Deformations

• Hessian-free learning

• Batch Normalisation (Thanks A Breitman)

• Competing Units

Neural networks have been around for a long time. What exactly changed between the 80s and now? Why the resurgence of late?

Reading this review paper (page 4),

1. Shallow NN-like models with few such stages have been around for many decades if not centuries, and models with several successive nonlinear layers of neurons date back at least to the 1960s and 1970s
2. Training of deeper architectures (ANNs with more layers, including RNNs) only became feasible through the use of unsupervised learning techniques in the 2000s
3. Training of deeper architectures without unsupervised learning became possible later

I'm a bit puzzled by (3), because it's not clear where it differs from (1). There are 3 candidates:

1. More data
2. More computation
3. Better techniques

More data

I think this is actually conflated with More computation below. Without fast computation, you can't get through enough data.

More computation

Page 23 of the reference - in 2010, a new MNIST record was set using backpropagation and pattern deformations, both of which are decades old.

these results seemed to suggest that advances in exploiting modern computing hardware were more important than advances in algorithms

Better techniques

Unsupervised learning followed by supervised learning

It seems that there was a year or two where this was popular, and then it just went away.

Vanishing / Exploding Gradients

LSTMs solve this, and are a recent invention. ReLU activations do not explode but have other problems.

Proper Initialization

(Thanks to A Breitman for mentioning this)

When they are not suffering from vanishing / exploding gradients, activation functions can be too linear. Reference