[Editor's note] In the brilliant galaxy of open source and artificial intelligence, Jia Yangqing's name is particularly dazzling. Because of Professor Trevor Darrell's sentence "Do you want to spend more time writing a graduation thesis that everyone probably doesn't care about, or write a framework that everyone will use in the future?", student Jia Yangqing plunged into the world of creating Caffe. Caffe has become Jia Yangqing's masterpiece, and Jia Yangqing's open source and AI journey will go further.
Interview | Liu Ren, Chairman of Cloud Computing Technology, author of "Knowledge Heroes", founder of DoNews
Author | Li Xinxin Liu Ren Zhou Yang Editor | Tang Xiaoyin
Produced | "New Programmer" editorial department
June 2013, Berkeley University. 28-year-old Jia Yangqing (see Figure 1) is writing Decaf (the predecessor of Caffe). Three months later, Dr. Yangqing Jia graduated. At this moment, he is working with Professor Thomas Griffith of the Berkeley Department of Psychology to study a psychological topic—how humans form the concept of "categories" during their personal growth. In the research, Jia Yangqing used a probabilistic framework to express human behavior, but the characteristics of human behavior extracted from images are weak, and it is difficult to draw complete conclusions.
Figure 1 Jia Yangqing during the Berkeley period
One day, Jia Yangqing saw that a paper "Advances in neural information processing systems", which won the first place in the 2012 ILSVRC competition, mentioned the deep learning AlexNet model, using Convolutional Neural Network (Convolutional Neural Network, CNN for short) technology, defeated Other non-neural network algorithms are used, and only two GPUs can replace Google's previous 10,000 CPU solution. One machine is worth 10,000 machines, and the error rate is reduced from 25% to 15%. This paper hit the water with one stone and shocked the industry. Previously, the neural network has not been optimistic about the industry.
Jia Yangqing was inspired and thought about applying the CNN feature extraction technology in the thesis to his current psychology project research. So Jia Yangqing approached Alex Krizhevsky (one of the authors of the AlexNet model) and asked him if he could share the source code of AlexNet? Alex responded like this: Sorry, I started a company and am starting a business. Due to intellectual property issues, I cannot directly provide the code, but you can ask me anytime when you encounter problems during the research process.
Just at this time, Jia Yangqing received a K20 GPU from NVIDIA's academic donation program (see Figure 2). For students, "GPU is very expensive!" So, Jia Yangqing, who was writing his graduation thesis, started to save A machine, at idle time, reproduces the AlexNet framework to extract features from images.
Figure 2 The earliest GPU as a framework
At this time, Jia Yangqing took the subway every day to go to Google for an internship. Sitting on the subway seat, Jia Yangqing turned on the computer, spread it on his legs, and continued to write the frame as needed. As a GPU beginner, everything is difficult at the beginning, but Jia Yangqing is addicted to it. "Writing code may be almost as addictive as playing games." "The time I spend on programming has gradually increased from 20%, 40%, 80%...." Google, Jia Yangqing drinks several cups of coffee every day, "This is not good", Jia Yangqing named the frame he was writing Decaf, in order to alert himself and quit coffee.
At this time, Jia Yangqing's task list includes: 1. Doctoral thesis; 2. Psychology research topics; 3. Looking for a job; 4. Reproducing the AlexNet framework. Compared with the first three tasks, Jia Yangqing spent more time on the last one, and the time for writing the thesis was severely squeezed. He asked his advisor Trevor Darrell (computer scientist, professor at Berkeley) for advice, and the advisor only asked one question: "Do you want to spend more time writing a graduation thesis that everyone probably doesn't care about? A framework that can be used?"
Encouraged by the tutor's words, Jia Yangqing put the thesis aside and plunged into Decaf. "The mentor can always teach me to distinguish between priorities."
After Jia Yangqing wrote the scaffolding and "run it" on a small scale, he put Decaf in the Berkeley group and let the students try it out. Everyone thought it was "quite easy to use". Evan Shelhamer, Jonathan Long, Jeff Donahue, Sergio Guadarrama and Jia Yangqing hit it off and decided to form a "core group" to develop Decaf together outside of daily scientific research and engineering work. Soon, the "little gang" reproduced the AlexNet model.
Decaf needs to be trained based on cuda-convnet, but through Decaf verification, deep learning features can use learning paradigms for in-depth experiments. Everyone thinks that it is better to develop a complete deep learning framework and make it a general and clean AI. Framework tools.
Because of the fast speed of the GPU, Jia Yangqing thought about changing Decaf to Caffe, and the students in the Berkeley group preferred the name Caffe. As a result, Decaf changed its name to Caffe. Two months later, Caffe was finished. Jia Yangqing specially applied for a fee with the instructor, bought an ice drip coffee machine, and put it in the research room. Since then, Evan Shelhamer often made coffee for everyone, which caused Jia Yangqing, who wanted to quit coffee, "in the end, he couldn't quit coffee." Everyone simply changed the email address for external consultation work to caffe-coldpress.
Figure 3 Jia Yangqing’s earliest computer host used to develop Caffe
Jia Yangqing is faced with a difficult problem: which method should be used to release Caffe? Is it to set up a startup company like Alex Krizhevsky and operate it commercially? Or as a program library, purely to support scientific research? Or open source? Jia Yangqing was indecisive, and the opinions of other Caffe developers also had their own tendencies, and they couldn't agree.
Jia Yangqing intends to open source Caffe. He thought that in the past few months, if there was a public deep learning framework that could obtain the details of the code and algorithm, he would not have to waste energy to reproduce it. Moreover, "When I was a student, deep down in my heart was the desire to 'make something and put it on open source'", "Most of the codes I used in Berkeley were open source", "Only by putting the market If it grows bigger, everyone may have a cake to eat", "Open source will not erase personal technical capabilities", "Anyway, I can still afford to buy coffee! What more can I ask for."
How to get everyone to agree? "This is much harder than writing Caffe". Jia Yangqing decided to talk to the core development students one by one. Some students are easy to talk about, while others are hard to argue with. Jia Yangqing was in a hurry, and blurted out angrily, "This is the framework I wrote, so I should have the right to make decisions!" We all agree that Caffe is open source.
In December 2013, Caffe was put on GitHub and officially open sourced. Alex Krizhevsky was also very happy to hear the news. Jia Yangqing's mentor suggested that Berkeley University be written in the description of Caffe, and Jia Yangqing was also very happy. "Caffe is open sourced in the name of Berkeley University. Everyone is very proud and feels that they are honoring their alma mater."
When he was an intern at Google, Jia Yangqing got an official offer from Google and only waited to join the job after graduation. Jia Yangqing, who no longer wanted to look for a job, let go completely, and simply stopped writing his graduation thesis, and the psychology topic he originally studied was nothing.
Caffe began to attract users and developers. With the aura of Caffe, Jia Yangqing met many people in the industry. Two months later, Jia Yangqing accidentally received an email from Nvidia Corporation. Nvidia proposed to provide computing resources for the Berkeley Research Institute and send engineers to work with Jia Yangqing on framework optimization to improve the application stability of Caffe. Jia Yangqing agreed to cooperate, and he liked the advantages of Nvidia on the system side. In the following year, Caffe, drawing on the strength of all parties, was on the road of accelerated development.
While working at Google, Jia Yangqing continued to maintain Caffe with his Berkeley colleagues. Jia Yangqing began to redesign some structures of Caffe to make it more modular and more adaptable to deployment in various environments. Evan Shelhamer, who has experience in the open source community, led the cooperation with all parties; Jeff Donahue helped Pinterest build a deep learning system; Jonathan Long provided Caffe with many new features including the Python interface... In terms of community building, GitHub and The caffe-users mailing group together form a loose, free organization, relying on the spontaneous management of Caffe users.
When working on Caffe, Jia Yangqing experienced a complete project experience from 0 to 1. "Caffe should be regarded as my first C++ project." Overall, "it has been a great exercise for me, from team development, how to promote, and how to get feedback. , How to improve the process, I have personally experienced every link.” In May of the second year after leaving Berkeley, Jia Yangqing finally completed his doctoral dissertation.
Small town youth go to Tsinghua University
In 1984, Jia Yangqing was born in Shangyu County, Shaoxing. Both parents are middle school Chinese teachers. When he was 1 year old, Jia Yangqing loved listening to stories, and his mother often told him stories with picture books. At the age of 3, Jia Yangqing knew two to three hundred characters, and he often held a book and read it fascinated. When he was 5 years old, his parents took him to Xinhua Bookstore. Jia Yangqing chose "Andersen's Fairy Tales". His mother asked in surprise: Can you understand the words in the book? Jia Yangqing nodded.
Jia Yangqing's family lives on campus, peaceful and peaceful, with a regular life. At 6:00, my parents got up to go to morning study, and Jia Yangqing also got up. In the sixth grade of elementary school, Jia Yangqing transferred from his parents' school to Shangyu Central School. In addition to being curious about the new environment, Jia Yangqing also felt a little inferior. He studied harder and wanted to prove himself through grades. In the senior high school entrance examination, Jia Yangqing ranked third in Shangyu District and was admitted to Chunhui Middle School.
In the second year of junior high school, he was popular with computers. Even though the school had a lab, Jia Yangqing's parents still bought him a Pentium II for more than 7,000 yuan. Jia Yangqing fiddled with the machine based on his feelings, tried various software, and played minesweeper games... Once, Jia Yangqing went to a classmate's house to play, saw the classmate using the mouse to click on the graphical interface on the computer screen, and demonstrated it to him Jia Yangqing thought it was fun to learn programming by himself. After returning home, he groped to write a small program in BASIC—input a year number in the box, and the corresponding zodiac sign of the year would be displayed on the screen.
The computer made Jia Yangqing feel that "a new thing can be created." "Very happy." But Jia Yangqing listened to his parents and still focused on his studies for the goal of the college entrance examination. In junior high school, the Chinese teacher taught the students to write literary reviews. Jia Yangqing chose "On the Description and Art of Poetry in "The Story of the West Chamber"" as a topic. The teacher was surprised and asked Jia Yangqing's mother: Is it too early for the children to read "The Story of the West Chamber"? His mother said, "Look, it doesn't matter." Parents teach Chinese, and Jia Yangqing has a lot of literature books at home, so he often randomly pulls out a book from his parents' bookshelf and flips through it. He took a copy of Qian Zhongshu's "Talking about Art" and flipped through it. Of course, "I couldn't understand it, so I put it back." Jia Yangqing read "History of Western Literature" and "Homer's Epic" and felt that "the outside world and my life The town is different." Jia Yangqing read "Sonnets" and found that Western poetry pays attention to rhyme, which is similar to ancient Chinese poetry. "It's very interesting that they can confirm each other..." Jia Yangqing has always been interested in literature, but he chose science. "Compared with liberal arts, science can go a little further with your own efforts." Or maybe, it was influenced by the general environment that everyone said at the time that "study mathematics, physics and chemistry well, and you will not be afraid to travel all over the world".
In high school, Jia Yangqing won two first prizes in the national physics and chemistry competitions, and the second prize for comprehensive ability in English. Mathematics is not outstanding, parents bought him "Human Online" CD-ROM, a lot of questions. "For questions that I don't know how to do, I will buy a whole book of practice questions to do until I am familiar with the application." Jia Yangqing's math performance gradually improved, and he won the second prize in the National Mathematical League.
In 2002, Jia Yangqing made a mistake on the last question of the physics test paper for the college entrance examination, and lost 27 points! As a result, I scored 686 points. Jia Yang's clear goal is the Department of Computer Science of Tsinghua University. After discussing with the teachers of the Admissions Office of Tsinghua University, to be on the safe side, he filled in "Department of Automation, Tsinghua University" on his first choice. When Jia Yangqing opened the notice and saw the words "Tsinghua University is the pride of your life" written on the cover, he was very happy.
Start artificial intelligence research
The Tsinghua Library opened at 8:00 in the morning, and after 7:00, Jia Yangqing and a few close classmates were standing in front of the door and waiting in line for self-study. There were 27 students in the class, and Jia Yangqing quickly ranked among the top again. In the four years of undergraduate studies, apart from studying, Jia Yangqing's life is still studying. After class, Jia Yangqing took calculus as a research topic and did "Jimidovich Mathematical Analysis Exercises".
The Tsinghua School of Information is composed of the Department of Automation, the Department of Computer and the Department of Electronics. The three departments study the same basic subjects, and the Department of Computer Science focuses on theory and software. "The Department of Automation essentially does two things—burning boilers and driving elevators. To burn boilers, the temperature must be raised rapidly, and after reaching a certain high temperature, the temperature must be kept stable. This is the problem to be solved by control theory. To drive an elevator Same thing."
In his junior year, Jia Yangqing took the course "Pattern Recognition and Intelligent System" taught by Professor Zhang Changshui. Jia Yangqing suddenly realized that what fascinates him most about artificial intelligence is to break through the inherent experience and explore the way for the possibility. "There is something to do!" And "for the algorithm of artificial intelligence, we can only say how much accuracy is used to identify certain things, and many problems are neither known nor solved." Jia Yangqing said to "Machine Learning" aroused great interest. "It is interesting and meaningful to let machines do things for people automatically and liberate people from low-level and repetitive labor." In the thesis reading class, Jia Yangqing found "Science" magazine (Science) and read aloud Geoffrey Hinton "Reducing the Dimensionality of Data with Neural Networks" article. After class, Jia Yangqing looked for information related to "neural network" and taught himself. He learned about concepts such as "Boltzmann machine"... Although at that time, in the field of artificial intelligence, "neural network" was at a low ebb.
On the eve of graduation, Jia Yangqing made a course design based on his interest-identifying a single car in heavy traffic. If you can identify how many individual cars there are, you can determine the congestion situation on the current road section. Jia Yangqing and his classmates stood on each overpass on the Fourth Ring Road, took many photos of vehicles passing under the bridge with their cameras, and marked them by hand. "There was no deep learning at that time. I used the classic method of computer vision to see how far I can recognize vehicles." "I think this problem is fun and challenging, so I will explore it. A correct method is meaningless.”
In July 2006, Jia Yangqing, a "good student", graduated with a bachelor's degree and entered the school to study for a master's degree without examination. Jia Yangqing followed Professor Zhang Changshui to major in pattern recognition and intelligent systems, and officially embarked on the road of artificial intelligence research.
5 international papers by one person
July 2008, Chicago, International Conference on Computer Science. The weather was hot, and Jia Yangqing, who was wearing short sleeves, was shivering from the extremely cold air conditioner in the conference hall. Jia Yangqing was also uneasy. He had to go on stage repeatedly and give 5 international papers in English, 4 of which were in fields he was unfamiliar with, while the audience were professionals from all over the world.
The papers of 5 students in the laboratory were shortlisted, but only Jia Yangqing got the US visa. In desperation, Jia Yangqing had no choice but to speak on his behalf. The instructor encouraged him: "Don't worry, if you speak badly, no one will remember you. You are not that important. Just speak at ease." "If you screw it up, people won't remember you; Not bad!" At the academic conference, colleagues in Europe and the United States were willing to take the initiative to show their ethos, which had a great impact on Jia Yangqing. In domestic laboratories, everyone is generally busy doing their own scientific research. Europeans and Americans very much hope that their scientific research can be seen by more people. They take the initiative to find someone to talk to, set up a roll-up banner next to the podium, and strive to promote their projects. When they talked about what they were doing, they beamed, their eyes lit up, and they beamed with pride. Jia Yangqing was deeply infected. "I learned to communicate with others the ideas behind my research. At that time, we generally lacked this ability."
In the summer of 2009, 25-year-old Jia Yangqing graduated from Tsinghua University and applied for doctoral students from more than a dozen foreign universities in one go. Among them, the computer department of the University of California, Berkeley provided a full scholarship, and the school in California was also very suitable for Jia Yangqing. At that time, the research of artificial intelligence was in the initial stage of groping for application scenarios. Speech recognition, machine translation, object recognition... The employment direction was limited to the field of algorithms, and the work of data scientists and data mining had not yet become a separate recruitment category.
Google Internship
At the Berkeley seminar, a colleague from a major factory came to Jia Yangqing and said: "We like Caffe very much. There are unit tests in the code! Many times, the code written by the researcher is really hard to read, but you write Not bad!" This is due to Jia Yangqing's internship at Google, and he developed a good habit when learning to write code.
Every year from late May to August, Berkeley has a summer vacation, and most of the students will go to large factories for internships in order to cultivate practical industrial experience. In the summer of 2011, Jia Yangqing was an intern at the NEC laboratory and was exposed to sparse coding (Sparse Coding) for the first time. Jia Yangqing made a perceptual domain algorithm that automatically learns each feature, and obtained the best at that time on the CIFAR dataset. the accuracy rate.
During the summer vacation of the second year of Ph.D., Google invited Jia Yangqing to the company for an interview, and then stayed for an internship. Google's internship is divided into two directions: product engineering and research institute. Jia Yangqing is in the research institute, and his mentor is Chinese Han Mei (currently the dean of Ping An Technology Silicon Valley Research Institute). Jia Yangqing is doing research on image recognition and video understanding on visual images. Together with the image search team, he is working on accurate recognition models to improve the accuracy of recognition. Later, the work he participated in was integrated into the personal photo album of Google Photos.
Being in Google, Jia Yangqing felt the extreme efficiency of tens of thousands of engineers working in a distributed manner. Google's engineering practice process system is sound, and the process of writing code is fixed by specifications: when writing a program, the unit test method should be written next to it, so that it is convenient for others to modify and test; it is required to write documents in a standard format. In a short period of time, Jia Yangqing's ability to write code was rapidly improved. "On the one hand, I have benefited from learning from many open source software; on the other hand, I think curiosity is the universal standard for measuring learning. When you see good code, try it yourself, write it a few times, and constantly Improve." "Everyone has curiosity, just like children dare to eat anything because they have no experience in eating, and the marginal benefit of trying to eat new things is large. The joy of the delicious world is greater than the pain of occasionally eating mud." "As we get older, there are more and more empirical data to learn from, so we need to replace economics with philosophy, and use faith to force ourselves into exploratory actions, instead of always scientifically and rationally learning deeply from experience and selecting the best. "
As an intern at Google, Jia Yangqing "learned things while improving his life". The food in the Google cafeteria is rich, and there is a dessert cake called "Ten Pounds", which means that after staying at Google for a year, the weight will increase by 10 catties! Jia Yangqing used Google's internship salary to buy a new car. The good working habits developed at Google are left in the code of Caffe.
In 2013, Jia Yangqing graduated from Berkeley with a doctorate in computer science.
From TensorFlow to PyTorch
Jia Yangqing went to Spain for a meeting and suffered from a bad cold. He went to the pharmacy late at night to buy medicine. He didn’t understand Spanish, so he turned on Google Translate’s camera translation, scanned the shelves row by row, and found ibuprofen.
In an emergency, using the functions developed by myself solved the problem, "it feels amazing". The OCR algorithm acquired by Google is relatively simple and cannot recognize complex fonts and texts. It is recognized on the cloud, and the speed is slow. Jia Yangqing and the author worked together to implement the deep learning OCR model on the mobile phone for the first time.
In 2013, Jia Yangqing joined Google Brain (in April 2023, Google Brain and DeepMind merged into Google DeepMind), and two years later, Jia Yangqing became a member of the founding team of TensorFlow. "Most of the authors of the first-generation framework of TensorFlow are still at Google, and the second-generation framework is in-depth and complete. It is widely adopted by Google's products." TensorFlow was open sourced by Google, and once became a project with a very high number of Stars on GitHub. In Google, Jia Yangqing found a personal "comparative" advantage. "I started with AI research and had a common language with scientists; when I was doing engineering, the engineers thought I was good at writing code." "I was able to allow both sides to communicate and collaborate."
In 2016, Jia Yangqing joined Facebook (now Meta). Facebook needs to build an AI base platform that supports all products such as advertising, feeds, search recommendations, image recognition, natural language processing, and mixed reality. Jia Yangqing's 4-person team is small but efficient. On the basis of Caffe, develop Caffe2. When Caffe2 was released, "augmented reality and virtual reality" suddenly became popular, and it took only 2 months for Caffe2 to be embedded in the mobile terminal. Zuckerberg was very happy, and personally posted a post, officially announcing the artist style conversion function (see Figure 3). This is the first time a deep learning network has been deployed on more than 1 billion mobile phones.
Figure 3 Zuckerberg personally announced the function of artist style conversion
While Jia Yangqing was working on Caffe2, PyTorch led by the Facebook Artificial Intelligence Research Institute in New York was successful. In 2018, under the leadership of Jia Yangqing, the backend of Caffe2, the frontend of PyTorch, and the standard of ONNX synthesized a complete framework, named PyTorch1.0. "If TensorFlow is like a large and complex combine harvester, PyTorch is more like a flexible and convenient bicycle."
Q&A Record
Liu Ren: What is the breakthrough of machine translation?
Jia Yangqing: In the past, machine translation relied on the establishment of grammatical rules, but the syntactic differences between the two languages are huge, and it is difficult to define grammatical rules exhaustively by handwriting. Now, use the neural network, and then use the Internet to collect data, train the neural network, and gradually optimize and improve its accuracy.
Liu Ren: What is the difference between the rule method and the neural network method?
Jia Yangqing: When programmers write rules, they will fall into endless rules and enter the swamp of rules. There are exceptions beyond exceptions, which is an endless problem. Use the neural network to solve the problem in a vague way, and then gradually improve the prediction accuracy through data training.
Liu Ren: What is the difference between deep learning and reinforcement deep learning?
Jia Yangqing: Intensive deep learning means how to return the benefits or penalties obtained in the future to the present.
Liu Ren: In which direction will artificial intelligence compete with humans for jobs?
Jia Yangqing: It is a good thing that simple and repetitive human labor is replaced. When people are freed, they can think about more possibilities. Da Vinci has a very famous painting called "Virgin of the Rocks". The main body of the Virgin was painted by Da Vinci, and the flowers and stones in the background were painted by Da Vinci's assistants. Da Vinci also needs assistant. Many painters today draw the main body, and then ask an assistant to fill in the background. He can also let the AI fill in the background. Like da Vinci's assistant, AI can make painters more efficient.
Liu Ren: Where is the technological gap between China and the US?
Jia Yangqing: Curiosity. Our best people are chasing execution, solving specific problems. Europeans and Americans prefer to create new things. I have to admit that Americans have been able to get enough food and clothing for a long time, so they play more and more, and they can always play something new. We have to pursue in many aspects, and this matter cannot be rushed.
About Open Source Heroes
"Open Source Heroes" is an exclusive interview column jointly produced by CSDN and "New Programmers", initiated by Jiang Tao, founder & chairman of CSDN, and founding partner of Geekbang Venture Capital, led by Liu Ren, a famous Chinese IT reporter and founder of DoNews. Through the open source and open CSDN writing class, in-depth interviews with the world's top open source figures, to deliver more and higher quality information to developers, to find resonance and dance in the open source world through their life and growth.
Since the launch of the column, we have interviewed You Yuxi, the author of Vue, Zhang Wensong, the founder of LVS, Wu Fengguang, the guardian of the Linux kernel, Gong Min, the first Linux person in China, Cui Baoqiu, the founder of Xiaomi’s open source, and Jia Yangqing, the author of Caffe. If you want us to report Open source characters, welcome to submit Issues or PR on GitCode.
Address: https://gitcode.net/programmer_editor/Open-Source-Heroes
Recommended reading:
