This article introduces Hinton's very candid sharing of his academic career, the future of deep learning and his research experience.
The academic road seems to be staggering, but Geoffrey Hinton has become the last laugher. He is known as the "Godfather of Deep Learning" and has won the highest honor in the computer field, the "Turing Award".
Hinton was born in a scientific "rich family" in the UK, but the academic career and rough life he has experienced in his life are rich and bizarre.
His father, Howard Everest Hinton, was a British entomologist, and his mother, Margaret, was a teacher. They were both communists. His uncle is the famous economist Colin Clark, who invented the economic term "gross national product". His great-great-grandfather is the famous logician George Boole, whose Boolean algebra invented laid the foundation of modern computer science.
Influenced by a strong family background of scientists, Hinton has independent thinking ability and tenacity since he was a child, and shoulders the burden of inheriting the family's honor. His mother gave him two choices, "either to be a scholar or to be a failure." He had no reason to choose to lie flat, and even though he struggled a lot in college, he still completed his studies.
In 1973, at the University of Edinburgh in the UK, he studied for a Ph.D. in artificial intelligence under the tutelage of Langer Higgins, but at that time almost no one believed in neural networks, and his tutor advised him to give up researching this technology. The doubts around him were not enough to shake his firm belief in neural networks. In the following ten years, he successively proposed the backpropagation algorithm and the Boltzmann machine, but he would have to wait for decades before deep learning Usher in a big explosion, when his research will be widely known.
After graduating from Ph.D., Hinton's life also experienced hardships. He and first wife Ros (molecular biologist) Go to the United States and get a teaching position at Carnegie Mellon University, however, due to dissatisfaction with the Reagan administration and when artificial intelligence research is basically supported by the US Department of Defense, they went to Canada in 1987, and Hinton began at the University of Toronto. He teaches in the School of Computer Science and conducts research in the Machine and Brain Learning Program at CIFAR, the Canadian Institute for Advanced Study.
Unfortunately, his wife Ros died of ovarian cancer in 1994, leaving Hinton alone to raise their two young children, who he adopted with ADHD and other learning disabilities . He later remarried to his current wife, Jackie , an art historian , but a similar blow loomed. Jackie also suffered from cancer a few years earlier.
He himself also suffers from severe lumbar spine disease, which prevents him from sitting down like a normal person. He has to stand and work most of the time in his daily life. Therefore, he also rejects flying because he is required to sit upright when taking off and landing. It also restricted him from going to other places to make academic reports.
From left to right are Ilya Sutskever, Alex Krizhevsky and Geoffrey Hinton.
After nearly half a century of technical perseverance and life honing, finally, in 2012, the AlexNet proposed by him and his students Alex Krizhevsky and Ilya Sutskever shocked the industry. It has shaped the field of computer vision and launched a new golden age of deep learning.
Also at the end of 2012, he and these two students established the trio company DNN-research and sold it to Google for a "sky-high price" of US$44 million. He also changed from a scholar to a Google vice president and Engineering Fellow .
In 2019, Hinton, an AI professor who did not come from a computer science class, won the Turing Award together with Yoshua Bengio and Yann LeCun.
After weathering the storm, the 74-year-old "Godfather of Deep Learning" is still fighting in the front line of AI research. He is not afraid of doubts from other scholars, and will frankly admit those judgments and predictions that have not been realized. In any case, he still believes that ten years after the rise of deep learning, this technology will continue to release its energy, and he is also thinking and looking for the next breakthrough point.
So where did his strong belief in neural networks come from? In today's doubts about deep learning "hitting the wall", how does he view the development of AI in the next stage? What message does he have for the younger generation of AI researchers?
Recently, in The Robot Brains Podcast hosted by Pieter Abbeel, Hinton shared his academic career, the future of deep learning and research experience very frankly, as well as the inside story of auctioning DNN-research. The following is his account.
Hinton, who was 8 years old,
had the most profound impact on me from the education I received as a child. My family is not religious, my father is a communist, but considering private schools are better for science education, at age 7 he insisted on sending me to an expensive Christian private school where all but me Children believe in God.
As soon as I got home, my family said religion was bullshit. Of course, it may be because I have a strong sense of self, I don’t believe it myself, realize that believing in God is wrong, and I have developed a habit of questioning others. Of course, many years later, they did find out that their original beliefs were wrong, and realized that God may not really exist.
However, if I tell you now to have faith, faith is important, it may sound ironic, but we do need to have faith in scientific research so that even if people say you are wrong, you will be on the right path Keep going.
In the 1970s, the study of neural networks "lonely brave"
My educational background is rich. In my freshman year at Cambridge University, I was the only student who took physics and physiology at the same time, which laid a certain foundation of science and engineering for my later scientific research career.
However, I was not very good at mathematics, so I had to give up studying physics. However, I was curious about the meaning of life, so I turned to study philosophy. After I got a certain grade, I began to study psychology.
In my last year at Cambridge I had a hard time and was not happy, so I dropped out as soon as I passed my exams to work as a carpenter. In fact, I prefer being a carpenter to anything else.
In high school, after class during the day, I would go home and do some woodworking, and that was my happiest moment. Slowly, I became a carpenter, but after about six months or so, I found that carpenters earn too little money to survive, although carpenters need to do far more than they appear on the surface. Renovation is much easier and quicker to pay, so while working as a carpenter, I will also do part-time decoration work. Unless you're an advanced carpenter, you can't make as much money as a carpenter.
It wasn't until one day that I met a really good carpenter that I realized I wasn't in the business. A coal company asked the carpenter to make a door for a dark and damp basement. In view of the special environment, he arranged the wood in the opposite direction to counteract the deformation of the wood due to moisture expansion, which I had never thought of before. The way. He can also cut a piece of wood into squares with a handsaw. He explained to me: If you want to cut the wood into squares, you have to line up the saw and the wood with the room.
At that time, I felt that I was too far behind him, so I thought maybe I should go back to school to study artificial intelligence.
Later, I went to the University of Edinburgh to study for a Ph.D. in neural networks under the famous Professor Christopher Longute-Higgins. When he was in his 30s, he figured out the structure of borohydrides and almost won the Nobel Prize for it, which is really amazing. Until now, I still don't know what he is studying, but I only know that it is related to quantum mechanics. The factual basis of this research is "the rotation of the identity operator is not 360 degrees, but 720 degrees."
He used to be very interested in the relationship between neural networks and holograms, but after I arrived at the University of Edinburgh, he suddenly lost interest in neural networks, mainly because he read Winograd (American computer scientist) was completely convinced after the paper, thinking that the neural network has no development prospects, but should be converted to symbolic artificial intelligence, and that paper had a great influence on him.
In fact, he didn't agree with my research direction and wanted me to do some research that was more likely to be awarded, but he was a good person and still told me to stick to my own direction, and never stopped me from studying neural networks.
In the early 1970s, everyone around me questioned me. Both Marvin Minsky and Seymour Papert said that the future of neural networks was bleak, so why persist? Honestly, I feel lonely.
In 1973, I gave my first talk to a group on how to do real recursion with neural networks. In the first project, I found that if you want a neural network to draw a graph, divide the graph into parts, and those parts of the graph can be drawn by similar neural hardware, then the neural center that stores the entire graph is The position, orientation and size of the overall figure need to be kept in mind.
If the neural network that is drawing the graph suddenly stops working, and you want to use another neural network to continue drawing the graph, then you need somewhere to store this graph and the progress of the work, and then you can continue to draw the work. The difficulty now lies in how to make the neural network realize these functions. Obviously, just copying neurons is not enough, so I want to design a system to adapt and record work progress in real time through fast weights . In this way, by restoring the relevant state (state) , you can continue to complete the task. So, I created a set of neural networks to achieve true recursion
by reusing the same neurons and weights to perform recursive calls (as used for advanced calls) . However, I'm not good at presentations, so it felt like maybe no one understood what I was talking about.
They say, why do recursion in neural networks when you can use Lisp recursion. What they don't know is that unless neural networks can implement things like recursion, there's a whole bunch of things they can't solve. Now, it's an interesting question again, so I'm going to wait another year until this question becomes a 50-year-old relic before I write a research paper on fast weights.
At that time, not everyone was against neural networks. If you go back to the 1950s, researchers such as von Neumann and Turing still believed in neural networks. They were all very interested in the way the brain works, especially Turing, who believed in the strengthening of neural networks. Training, which also makes me very confident in my research direction.
It is a pity that they died young. If they could live a few more years, their wisdom would be enough to influence the development of a field. Britain may have already made a breakthrough in this area, and the status quo of artificial intelligence may be very different.
From a pure academic to a Googler
去Google工作的主要原因是，我的儿子患有残疾，我得为他挣钱。
2012年，我觉得在Coursera上讲课能挣到很多钱，所以就开设了神经网络相关课程。早期的Coursera软件并不好用，加上我自己并不太擅长操做软件，因此我时常感到烦躁。
最初我与多伦多大学达成了一项协议，如果这些课程能赚到钱的话，那么大学会把到手的钱分一部分给讲课老师。虽然他们没有明确说具体的分成比例，但有人说是对半分，我也就欣然接受了。
在录课过程中，我曾要求过学校帮我录制视频，但他们却反问我，「你知道制作视频有多贵吗？」我当然知道，因为我自己一直在制作视频，校方还是没有提供任何支持。然而在我开课之后 （当时我已经骑虎难下了） ，教务长在没有咨询我和其他任何人的情况下就单方面决定学校会拿走所有的钱，而我则一分钱也拿不到，这就完全违反了当初的协议。
他们让我好好录课，并说那本就是我教学工作的一部分，但那实际上并不属于我的教学范畴，而只是基于我之前做过的相关讲座的课程。因此，我在后续的教学工作中再也没有用过Coursera。那件事让我很生气，甚至开始考虑是否要从事其他的职业。
就在此时，突然有很多公司向我们抛出了橄榄枝，愿意赞助一大笔经费，或者支持我们创立一家公司，这说明还是有很多公司对于我们的研究内容很感兴趣。
鉴于州政府已经给过我们一笔研究经费，我们也不再想赚外快，还是把精力放在自己的研究上。但那次学校骗我赚钱的经历不禁让我萌生想多赚点钱的想法，所以后来把成立不久的DNN-research拍卖了。
这桩买卖发生在2012年12月的NIPS （神经信息处理系统大会） 期间，会议在塔霍湖边的一个娱乐场所举办，地下室里灯光闪耀，一群光着膀子的赌徒在烟雾缭绕的房间里尽情高呼，「你赢了25000，这些都是你的」......与此同时，楼上进行拍卖一家公司。
当时就像在演电影，与社交媒体上看到的情形一模一样，真的很棒。我们之所以拍卖公司，是因为我们完全不知道自身的价值，所以我就咨询了一个知识产权方面的律师，他说，现在有两个办法：一是直接雇一名专业的谈判员去和那些大公司谈判，但这可能会遇到不愉快；二是发起一场竞拍。
据我所知，像我们这样的小公司进行拍卖在历史上还是第一次。最终我选择通过Gmail进行竞拍，因为那年夏天我一直在Google工作，我知道他们不会随意窃取用户的邮件，即使到现在，我还是这样认为的。但对于我们这一决定，微软表现出不满。
拍卖过程如下：参与竞拍的公司必须通过Gmail将他们的报价发给我们，然后我们再将其连同Gmail的时间戳发送给其他参与者。起拍价为50万美元，然后有人出价100万美元，看到竞价不断上涨时，我们真是太高兴了，同时也意识到我们的价值远比预想的要高。当竞价达到一定程度时 （当时我们认为是天文数字了） ，我们更倾向于在Google工作，于是叫停了拍卖。
来Google工作是一个正确的选择，到现在我在这儿工作了九年。等我在这里工作满十年，他们应该会给我颁个奖，毕竟在这儿工作这么久的人屈指可数。
相比其他公司，人们都更喜欢在Google工作，我也一样。我喜欢这家公司的主要原因是Google Brain团队很棒。我更专注于研究如何构建大型学习系统和研究大脑的工作机制，Google Brain不仅有研究大型系统所需要的丰富资源，还能跟众多优秀人才交流学习。
I'm one of those straight guys, and Jeff Dean is a smart guy, and he's a pleasure to be around. He wanted me to do some basic research and try to come up with new algorithms, which is what I like to do. I am not good at managing large teams. In contrast, I am more happy to improve the accuracy of language recognition by one percent. Bringing a new revolution in this field is what I have always wanted to do.
The Next Big Thing in Deep Learning
深度学习的发展取决于，在拥有海量数据和强大算力的大型网络中做随机梯度下降，基于此，一些想法得以更好地生根发芽，比如随机失活 （dropout） 和现在的很多研究，但这一切离不开强大算力、海量数据以及随机梯度下降。
经常有人说深度学习遇到了瓶颈，但事实上它一直在不断向前发展，我希望怀疑论者能将深度学习现在不能做的事写下来。五年后，我们会证明深度学习能做到这些事。
当然，这些任务必须经过严格定义。比如Hector Levesque （多伦多大学计算机系教授） 是一个典型的AI人士，他本人非常优秀。Hector制定了一个标准，即Winograd句子，其中一个例子是，「奖杯不适合放在手提箱中，因为它太小了；奖杯不适合放在手提箱里，因为它太大了。」
如果你想把这两句翻译成法语，必须明白在第一种情况下，「它」指的是手提箱，而在第二种情况下，「它」指的是奖杯，因为它们在法语中是不同的性数 （genders） ，而且早期的神经网络机器翻译是随机的，所以当机器把上述句子翻译成法语时，机器无法正确识别性数。但这种情况一直在改进，至少Hector给神经元下了一个非常明确的定义，指出神经元可以做什么。虽然做的并不完美，但这样至少比随机翻译要好得多。我希望怀疑论者能提出更多类似的质疑。
我认为，深度学习这种非常成功的范式将继续保持繁荣：即根据一些目标函数的梯度来调整大量的实值参数，但我们很可能不会使用反向传播机制来获得梯度，而目标函数可能会更加局部和分散。
I personally guess that the next big AI event must be the learning algorithm of the spiking neural network. It can solve the discrete decision of whether to make a pulse, and the continuous decision of when to make a pulse, so that the pulse time can be used for interesting calculations, which is actually difficult to do in a non-spiking neural network. I haven't been able to study the learning algorithm of spiking neural network in depth before, which is a great regret in my research career.
I don't intend to study AGI, and I try to avoid defining what AGI is, because there are various problems behind the AGI vision, and general artificial intelligence cannot be achieved just by expanding the number of neurons or neural connections with parameters.
AGI envisions a human-like intelligent robot that is as smart as a human being. I don't think intelligence will necessarily develop this way, but rather that it develops more symbiotically. I think maybe we'll design intelligent computers, but they won't be as conscious as humans. If their purpose is to kill other humans, they probably have to be self-aware, but hopefully we don't go in that direction.
Believe in research intuition, driven by curiosity
每个人的思维方式都有所不同，我们不一定了解自己的思维过程。我喜欢按直觉行事，更倾向于在做研究时运用类比，我认为，人类推理的基本方式是基于在大向量中利用正确的特征来进行类比，我本人也是这样做研究的。
我经常在电脑上对某一研究反复进行试验，来看看哪些有用，哪些没用。弄清事物的数学底层逻辑和进行基础研究确实很重要，进行一些论证也很有必要，但这些不是我想做的事。
做一个小测试：假如现在NIPS会议上有两场讲座，一场是关于用一种全新、聪明和优雅的方法来证明一项已知的结论；另一场则是关于一种新的、强大的学习算法，但算法背后的逻辑暂时无人知晓。
如果你必须在这两场讲座中选择一场去听讲座，你会做何选择？相比第二场讲座，第一场可能更容易被人们所接受，大家似乎更好奇证明已知事物的新方法，但我会去听第二场，毕竟在神经网络领域，几乎所有的进步都源于人们在进行数学推演时瞬间萌生的直觉，而非常规推理。
那么你是否要相信自己的直觉？我有一个标准——要么你有敏锐的直觉，要么干脆没有。如果没有敏锐的直觉，那做什么都没关系；但如果有敏锐的直觉，那应该相信直觉，去做你认为对的事。
当然，敏锐的直觉源自你对世界的理解以及大量的辛劳付出。当你在同一件事上积累了大量经验，就会产生直觉。
我患有轻微的狂躁抑郁症，所以一般会游走在两种状况之间：适当的自我批评会让我非常有创造力，而极度自我批评会让我产生轻度抑郁。但我认为这样比仅有单一情绪的效率更高。当你感到烦躁时，你只要忽视那些显而易见的问题，并且确信一些有趣的、激动人心的东西正等你去发现，继续前进。当你面对问题感到措手不及时，一定要坚持下去，理清思路，仔细斟酌想法的好坏。
由于有这样的情绪交替，我经常会告诉大家，我弄清大脑的工作机制了，可过段时间，我又失望地发现之前的结论是错误的，但事情就应该是这样发展的，正如William Blake的那两句诗，「将快乐和忧伤编织，披在我神圣的心上」。
我认为科研工作的本质也是如此，如果你不会因为成功而感到兴奋，也不会因为失败而感到沮丧，那算不上真正意义上的研究者。
研究生涯里，尽管有时会觉得自己完全摸不着一些算法的门道，但我还从未真正感到迷茫和毫无希望。在我看来，无论最终结果如何，总有值得去做的事情。优秀的研究人员总是有很多想做的事情，只是苦于没有多余的时间。
在多伦多大学任教时，我发现计算机科学专业的本科生都很优秀，而很多辅修计算机科学的认知科学专业的本科生也表现得相当出色，这一部分同学并不擅长技术，但他们仍然把研究做得很好，他们热爱计算机科学，非常想弄清人类的认知如何形成，有着源源不断的兴趣。
像Blake Richards （蒙特利尔神经学研究所助理教授） 这样的科学家，他们很清楚自己想解决什么问题，然后就只管朝着这个方向前行。现在，很多科学家都不知道自己到底想做什么。
回头看，我觉得年轻人要找到自己感兴趣的方向，而不是单纯地学些技术。在自身兴趣的驱动下，你会主动去掌握一些应有的知识来寻找你想要的答案，这比盲目地学习技术更重要。
现在想想，我年轻时就应该再多学一点数学知识，这样做线性代数就会容易很多。
数学时常让我感到绝望，导致很难读懂一些论文，尤其要弄懂那一大堆符号，真是一项莫大的挑战，所以我并没有读太多论文。关于神经科学方面的问题，一般我会向Terry Sejnowski （计算神经学教授） Please, computer science questions, I will ask graduate students to explain to me. When I need to use mathematics to prove whether a certain research is feasible, I can always find a suitable method.
The idea of making the world a better place by doing research is great, but I enjoy exploring the upper limit of human creativity more. I really want to understand how the brain works. I believe we need some new ideas, such as through pulses. The learning algorithm of a neural network understands how the brain works.
In my opinion, the best research work is done by a large group of graduate students and given ample resources. Scientific research requires youthful vigor, continuous motivation, and a strong interest in research.
You have to be driven by curiosity to do the best basic research. Only then will you have the motivation to ignore the obvious obstacles and anticipate what you will achieve. If it is a general study, creativity is not the most important thing.
It's always a good idea to figure out what a bunch of smart people are working on, and then you can do something different. If you have already made some progress in a certain field, you don't need other new ideas, you just need to dig deep into the existing research to be successful. But if you want to work on some new ideas, like building large hardware, that's also great, although the road ahead can be a bit bumpy.