Exploring Neural Networks with Activation Atlases
Discussion and Review
keywords: neural networks, feature visualization
Starting with an Analogy: The effect of a global view versus a local one
While the 26 letters in the alphabet provide a basis for English, seeing how letters are commonly combined to make words gives far more insight into the concepts that can be expressed than the letters alone. Similarly, activation atlases give us a bigger picture view by showing common combinations of neurons.
Research Problem
How neural networks outperform more traditional approaches to machine learning and historically? (但是个人觉得这篇文章并没有直接触及到解决这个问题的方法，而是提供了更多的思路)
Research Thread
| Step | Question | Solution |
|---|---|---|
| 1 | Original Question: What have these networks learned that allows them to classify images so well? | Visualize individual neurons |
| 2 | Natually: Neurons don’t work in isolation | Visualize simple combinations of neurons (pretty natural) |
| 3 | Further: What combinations of neurons should we be studying? | Visualize activations, the combination of neurons firing in response to a particular input |
Challenges
Unfortunately, visualizing activations has a major weakness — it is limited to seeing only how the network sees a single input. Because of this, it doesn’t give us a big picture view of the network. When what we want is a map of an entire forest, inspecting one tree at a time will not suffice.
Contributions
- In this article we introduce activation atlases to this quiver of techniques. (An example is shown at the top of this article.)
- Broadly speaking, we use a technique similar to the one in Karpathy’s CNN codes, but instead of showing input data, we show feature visualizations of averaged activations.
- By combining these two techniques (Methodology), we can get the advantages of each in one view — a global map seen through the eyes of the network.
To Be Continued
- Review on Current Methods
- Methodology
- Experiment Design and Results
- Limitations, Research Opportunies, and Future Work
Personal Notes
- 个人觉得这篇文章的一大亮点在于一个棒球图使神经网络将灰鲸转为预测大白鲨，按照原文说“reveal high-level misunderstandings in a model that can be exploited”。
- 研究成果的交互性展示。
- 算法解释性意义：越来越多的研究强调transparency和Interpretablity。不无道理。如果无法解释，我们凭什么去相信算法的结果呢？尤其是在社会科学领域，诸如司法领域。即便是自动驾驶，如果缺失了解释性而仅有准确度，人们对自动驾驶的信任也很难提高。换句话说，可解释性可以加快自动驾驶的普及。