Author: Tsinghua University teacher Liu Zhiyuan
Background note: ACL 2020 submission deadline approaching, intensive discussions with the students, the argument which invested in research ideas for a chance to hit ACL. Ten years of research experience from their own point of view, a study to determine how good a bad idea, and ideas come from these studies, for starters, is indeed a challenge. So, simply save a little of this essay, to share some experiences and ideas and hope for just entering the field of NLP new classmates. Unfortunate more than a mistake, please correct me.
Wong Kar-wai film "great master" in the contest section of the classic plot, president of the Palace of Ip Man said, "Today we are better than martial arts, than the idea." In fact, also the soul of an outstanding research achievements good idea or ideas (idea). The popular computer field the word "IDEA is cheap, show me the code", also shows the importance for the practice of computer science terms, good or bad idea also depends on its actual performance. Good research idea came here to talk about next.
What is considered good idea
2015, I wrote a small piece of ridicule on microblogging:
ML faction located in the United States of Triangle Hill, Wu Xueqi years before giving birth, first name implicitly into rivers and lakes the door upright, entry door with three sets of martial arts, said: plus circle graph model, neural nets and floor, plus regular optimization goals. There are nursery rhymes for the card: ML Getting skilled work, writing will not make up.
To 2018, I continued the short:
No multi-year, the sudden emergence of the North Church DL, repair learns within and outside training the neural network, numerous miscellaneous, said door, said attention, said memory, said confrontation, said enhanced. By ImageNet battle Megatron martial arts, no one can rear Alpha dog a closer. Dan furnace building every family moment, everyone is busy with alchemy, the disciples gathered, attached by many people, there is a tendency to dominate the arena. There are nursery rhymes for the card: big data left hand, right hand Nvidia, will top every alchemy busy.
There is mention of the graph model plus laps, neural networks and floor, plus regular optimization goals, neural networks doors, attention, memory, etc., are some innovative ideas to improve model performance, it is widely used and delivered a major task NLP paper, perhaps because they are used in different NLP tasks and published, somewhat fatigued and lack of deeper innovative ideas, sense of shame by some friends and scholars as "irrigation", seems not a good idea.
So what is it a good idea? I understand this "good" word, meaning there are at least two levels.
Development point of discipline "good"
Academic research is to explore the nature of the unknown, a quest for answers to open questions. So from the perspective of promoting development of the subject, the study criteria to judge what is good idea, first in a "new" word.
There is a saying in the past, there is a curse artificial intelligence, artificial intelligence who is resolved (or have solutions) part, no longer is considered to represent "human intelligence." Computer vision, natural language processing, machine learning, artificial intelligence is also listed as the main reason why the robot direction, perhaps because they have not been resolved, and can still stands for "human intelligence" dignity. And we want to carry out innovative research is to propose new ideas to solve these problems. This is one of the "new", can be reflected in the proposed new questions and tasks, explore new idea, put forward a new algorithm technology, new tools systems.
Ensuring the "new" on the basis of research ideas is good, then look at it to help promote the development of the subject much. The reason why such a deep learning has a prominent influence, it lies in artificial intelligence for natural language processing, an important direction for each speech recognition, computer vision, etc. have had a revolutionary impact, completely changed the structure of the non-signal (voice, image, text) technology roadmap semantic representation.
Practice point of view of "good"
That was not the idea as long as enough "new" like it? Is not the newer the better? I think it should not yet. Because, can only be done out of ideas to be eligible to be analyzed is good. Therefore, from a practical point of research, also we need to consider research ideas can be realized and verifiability.
Realizability, embodied in the idea of whether there is sufficient mathematical or machine learning tools to support implementation. Verifiability, whether embodied in the idea of a suitable set of data and widely accepted evaluation criteria. Many scientists think the reason why folk are not academic recognition, because these ideas can be realized and often a lack of verifiability, only stay in the abstract paper, just some vague concept.
Good research ideas come from
Good idea or bad, is not black and white dichotomy, but rather like a spectrum as a continuous distribution varies with time, people and should. The process of technological development in the field of computer both accumulation, but also the transition of singularities, quantitative accumulation will produce qualitative change, eat bread third full, but also because the previous two buns bottoming.
Now academic research has become a highly specialized profession, there is a huge group of researchers. "Publish or Perish", who is engaged in the academic profession (such as professors, researchers, graduate students) must do things balanced, not every job requires researchers are "Nobel Prize" or "Turing Award" class was worth publishing. As long as some boost to the development of research, it is worth publishing out to help fellow forward. Lu said: genius is not live and grow in the deep forest wilderness of the monster, is a genius can make people grow produce, long out of the sterile, so there is no such people, there is no genius . This huge group of researchers is the genius of the masses foundation for growth. Meanwhile, also in the new academic conduct innovative research training, and constantly honed ability to find good ideas, Lu Xun said: even genius born when the first sound of the crying, and children are also the same as usual, never is a good poem.
Well, good research ideas come from? I summarize, we must first have the ability to distinguish between good and bad research ideas , which requires in-depth comprehensive understanding of the history and current situation where the direction of research, is a comprehensive mastery of specific subject literature. Most people are good at learning the animal, the idea of completely different periods of the existing research literature can be used as learning objects, by understanding the impact on the development of the subject after they made - embodied quoted in the paper, and so on all aspects of academic evaluation of the situation - - evaluation model for study of the idea of good and bad. It is difficult to list all perfectly Tiaofenlvxi feature vectors to distinguish good and bad ideas, but the human brain's ability to learn strong, if given sufficient input data, you can automatically determine the model of learning in neural networks, Kam ancient knowledge today, around corners, and maybe that is often said that the academic insight.
Students have done some research experience, just read their own literature research, new ideas or not particularly high. This is because, when the idea of reading are the research questions have been completed, they alone can not inspire new ideas. How to generate new ideas? I summarize, there are three basic ways possible:
Practices Act. That task has been achieved in the study of the best algorithms, analyzing results, for example, found that the computational complexity is particularly high, particularly slow convergence training, or find an error sample of the algorithm showed obvious regularity, can inspire you improvement of the existing algorithms. Now the latest algorithm on the Leaderboard many natural language processing tasks, is to have targeted improved by analyzing a sample error algorithm [1].
Analogy. Research is about to make an analog connection issues with other tasks, the latest research on effective ideas other similar tasks, algorithms or tools that migrate through reasonable conversion, applied to the current research questions come up. For example, the neural network had attentional mechanisms in machine translation with great success, attention was mainly based on word level, then our group of Linyan Kai and Chen Shiqi proposed the establishment of a sentence-level attention to solve the remote supervision of the training data relation extraction labeling noise problem [2], which is a kind of analogy approach.
Combination method. The upcoming new research questions into several sub-problems have been well resolved by the best practices of these sub-problems organically, build solutions for new research questions. For example, pre-trained language model merging knowledge map we propose, it is to BERT and TransE and other existing algorithms integrate newly established model [3].
As the highest state of martial arts is no stroke win a trick, a good idea is not rigidly adhere to the study of more than one path, it is often based on a deep understanding of the researchers on the study of the problem on a comprehensive wealth of research experience and ingenuity to produce "epiphany" results. This is probably hard to glimpse avenue for starters, you need to start from the basics, after extensive research and practical training, in order to have a sense of sexually explicit.
In research practice, in addition to understanding history by reading a lot of literature, through in-depth insight into the thinking generated summary, there is an essential job, and that is actively opening up academic exchanges and cooperation consciousness . Different areas of research ideas and the exchange of results of the collision, both provide a new source of innovative ideas, but also for the "analogy" and "insight" provided the opportunity. You can look at history to know, artificial intelligence is proposed, that is, mathematics, computer science, product control theory, information theory, brain science and other interdisciplinary integration. The depth of the field of study of the origin of popular, Parallel Distributed 1980's Processing (PDP), also computer science, cognitive brain science, psychology, biology researcher co-operation of the product. The following is published in 1986 masterpiece "Parallel Distributed Processing: Explorations in the Microstructure of Cognition" the cover of the first volume.
Author in the foreword is so to speak their cooperation process in the initial six-month period, they meet up twice a week to discuss research progress.
We expected the project to take about six months. We began in January 1982 by bringing a number of our colleagues together to form a discussion group on these topics. During the first six months we met twice weekly and laid the foundation for most of the work presented in these volumes.
The list of members of the PDP Research Group provides in the book, 40 years later I still marvel at the height of the cross-agency, cross-cross-disciplinary features. So, specifically recommended the students in scientific research training, under the premise of focusing research questions remain active academic exchanges consciousness, whether it is listening to lectures report, attend an academic conference, or elective courses have consciously widened the breadth of academic exchanges, not only mingle with fellow small, seemingly more academic research partners in the field of eight pole could not beat. With rich experience in research, will be more strongly felt, the more large span crossing the academic report, the more make you feel more inspired to produce more excited to make their research ideas.
Beginners should be how do
And read the paper, writing papers, experimental design compared to other sectors, how to generate good research idea, it is a less rule-based links, is difficult to summarize the available follow a fixed paradigm. Like Xiaomaguohe, we need a lot of training practices, to build up their research experience. However, for starters, there are still a few simple principles can refer feasible.
The paper can be published value, depending on its existing Delta and most directly related to the workplace. Most of our research work are all on the basis of previous work on advancing. Newton said: If I have more important than others farther, it is because I stood on the shoulders of giants. In my opinion, judge a paper study the idea of value, it is to see which of standing on the shoulders of giants, and on this basis, and how far up. In turn, until you are ready to start a study, at the time of the formation of research ideas, perhaps to prepare first clear stand on the shoulders of giants which, by what means and plans to go further. Delta has the most direct correlation between the work and determine the value of this research idea how much.
Both pick the fruit and gnawing on a bone. It is generally easier to think of the idea of the study, called the Low Hanging Fruit (fruit falls). Fruit falls off easily, but also many people who pick and choose pick the child are likely to suffer from the idea of a collision. For example, pre-trained language model in 2018, led to a major breakthrough BERT, in 2019 appeared in a lot of improvements, which cross-modality pre-training model as an example, just a few months to hang out over arxiv.org pre-training model image and text from six different teams fusion [4]. Put yourself to think, to study cross-modal pre-training model is easier to think of a direction, you will need to have the ability to predict, know for sure there will be many teams also carry out research in this area at the same time in the world, then if you choose the field, we must do more in-depth features and more, have their own unique contribution to the job. In contrast, those difficult issues, less people are willing to touch, dive down to Kenyinggutou, also a good choice, of course, but is at risk do not come out, or do not get too much out of concern risk . Students need according to their characteristics, experience and needs, taking into account picking the fruit gnawing on a bone and two types of research ideas.
Note the theme coherence of a number of research work. Training students often continue for several years, need to pay attention to the theme coherence of a number of studies before and after work, to ensure that the internal logic of unity. Need to consider, on a personal resume in Personal Statement apply to go abroad, or in all kinds of awards shows, the results of these studies can be aggregated together to tell the overall goal of their own to carry out these studies, the overall vision. Objectively speaking, the field of artificial intelligence research fast-paced, technology replacement fast, it also tends to leave a small, fast track results. I have the School of Business, Social Sciences friends, they often need a study or even years for more than a year; high-performance computing, computer networks direction of the study period is relatively long. This small run of artificial intelligence characteristics determine many students even graduate, also published a number of papers, let alone graduate students. In this case, it is particularly needed in the study of the topic, pay attention to continuity of care and the relationship before and after work. Several studies work together, in the end is fragmented not speak to each other, or in a single big goals, especially the ability to reflect the overall situation and the layout of the study. For example, the following figure is a doctoral dissertation when our group painted deposit ultra-graduate school in 2018 "for social computing network learns" section to set, as a whole than "Several studies have important social computing," and so there is no inherent association wording to Some more convincing. Of course, for beginners, I wanted to start a five-year research program clearly impossible. But would like to, or not to think, the result was different.
Pay attention to grasp and summarize research and trends, because of the occasion. 2019 know almost on the question: "2019 in the field of NLP, which valued promising job with limited resources personal / team can do?" I replied as follows:
I feel, the industry began to engage in group of questions, indicating that one of the major open problem has been solved or less, such as speech recognition, face recognition, etc., over the past 20 years there have gradually been widespread commercial use. See recent BERT, GPT-2, I understand that the ability to play more depth study of large-scale data fitting to the extreme, in depth technical route learning basic premise mature, large companies have powerful computing capability, the Natural you can get more data, model bigger and better fitting effect.
Mature high-tech entered the commercial competition, there will be roughly in line with the development of the law of Moore's Law. BERT and other training now seems out of reach, but with the development of factors such as the popularity of computing power, maybe a few years, everyone can easily BERT training and GPT-2, we will in the same starting line, the attention has turned to the next challenging problem.
So better to think ahead, what the problem is data-driven technology can not solve. NLP and AI in difficult tasks, such as common sense knowledge and reasoning, complex and inter-modal contextual understanding, explain the intelligent, are not a viable solution, I personally do not optimistic about data-driven approach can solve. A higher level of association, creativity, insight and other cognitive abilities, is not even a side not met. These are precisely the direction of visionary researchers should begin concern.
We need to see the different research and trends over time. Grasp these dynamics and trends, we will be able to make the results of research of interest to the community. Otherwise, even though research has not changed, but simply a few years ago or a few years later submission, the result will be different. For example, in 2013 word2vec published in 2014-- word for the study and research between 2016, it is relatively easy to get hired ACL, EMNLP and other meetings; but in the 2017-2018 years, the word of the meeting expressed ACL and other related learning work is relatively rare.
The final supplement
This short article is mainly for beginners wish, introduce some novelty experience in the process and considerations, I hope you take some detours. But read the literature, deep thinking, receiving rejections continue to improve the plight of the food or eat. Academic and research papers published, for individuals may mean higher salaries and scholarships, but its ultimate aim is to promote the development of a real discipline. So, do stand the test of academic research, the key to the "true" and "new", we need to always abide by and assiduous.
Famous historian, Mr. Bingdi alumni have mentioned the famous mathematician Chia-Chiao Lin sentence asked in his autobiography "Histories see the world six years": "The important thing is to do no matter what line, do not do so on the subject of the second . "specific to each area, what is the number itself is the subject matter of opinion, in fact, point to more heart" truth "attitude.
