Source: Zhuanzhi Wechat ID: Quan_Zhuanzhi
Are you an AI researcher at an academic institution? Do you worry that you won’t be able to keep up with the current pace of AI development? Do you feel like you don’t have (or have very limited) access to the computing and human resources needed for AI research breakthroughs? You are not alone ; We feel the same way. A growing number of AI academics can no longer find the resources to remain competitive on a global scale. It's a recent phenomenon, but it's accelerating, with private companies investing vast computing resources in cutting-edge AI research. Here, we discuss how academic researchers can remain competitive. We also briefly discuss what universities and the private sector can do to improve the situation if they choose . This is not an exhaustive list of strategies, and you may not agree with all of them, but it helps to start a discussion. These strategies include:
-
-
give up?
-
try to expand
-
downsizing
-
reuse and remake
-
analyze rather than synthesize
-
reinforcement learning
-
Small models without computation
-
Work in a specialized application area or field
-
Addressing issues of little current concern
-
Try things that "shouldn't" work
-
Do something that "looks bad"
-
Start a business; spin it off!
-
Cooperate or jump ship!
-
How Can Large Industry Players Help?
-
How universities can help
-
https://www.zhuanzhi.ai/paper/f5b56758eda2c2fd84f5b70a9c567822
As someone who did AI research at university, you developed a complex relationship with corporate AI research organizations such as DeepMind, Open AI, Google Brain, and Meta AI. Whenever you see a paper like this that trains some sort of giant neural network model to do something that you're not even sure a neural network can do, pushes state-of-the-art beyond a doubt, and reconfigures your thinking, You will have conflicting emotions. On the one hand: very impressive. Glad you pushed the boundaries of artificial intelligence. On the other hand: how can we possibly keep up? As an AI academic leading a lab with a few PhD students and (if you're lucky enough) a few postdocs, maybe only a few dozen GPUs in your lab, This kind of research is simply impossible to do.
To be clear, this has not always been the case. Just a decade ago, if you had a decent desktop computer and an internet connection, you had everything you needed to compete with top researchers. Breakthrough papers back then were usually done by one or two people doing all the experiments on their normal workstations. This is especially worth pointing out to those who have entered the research field within the past decade, for whom enormous computational resource demands are taken for granted.
If there's one thing learned from deep learning, it's that scaling works . From the ImageNet [15] competition and its various champions, to ChatGPT, Gato [13], and most recently GPT-4 [1], we see that more data and computation can lead to quantitative and often even qualitative better results. (This very recent list of AI milestones may be out of date by the time you read this.) Of course, learning algorithms and network architectures have improved, but these improvements are only really useful in the context of large-scale experiments. (Sutton speaks of a "bitter pill", referring to the fact that simple approaches that scale well always win when more computing resources are available [18].) Today, academic researchers cannot achieve this scale. We understand that the gap between the amount of computing available to the average researcher and the amount of computing needed to remain competitive grows every year. This largely explains why many AI researchers in academia harbor resentment toward these companies. It's one thing to have healthy competition with your peers, but it's another to compete with those who have the resources to easily do things you'd never be able to accomplish, no matter how good your idea. When you’ve been working on a research topic for a while, and DeepMind or OpenAI decide to work on the same thing, you might feel the same way the small-town grocer feels when Walmart sets up shop next door. This is sad, because we like to believe that research is an open and collaborative enterprise where everyone gets recognized for their contributions, no?
So, if you're just a professor with limited team size and limited computing resources, how do you stay relevant in the face of an onslaught of well-funded research companies ? This is a question that has plagued us and many of our colleagues for years. Recent events, where models like GPT-4 are surprisingly powerful, yet surprisingly closed source and lack public details, have made this question even more urgent. We heard from multiple researchers at all levels, both in person and via social media, that they were concerned about doing meaningful research with scarce resources and unfair competition from big tech companies. First let's be clear: we're both safe. We have tenure-track professorships, and we've grown fairly rapidly in academia, partly because we've systematically pushed the boundaries of AI in video games. While we clearly care about continuing relevant AI research, we are writing this primarily for our younger colleagues who may wonder which career path to choose. Is it worth trying to go into academia, join a big tech company, or start a startup? Is a career in AI a good idea, or is it better to be a plumber? Should you be a cog in the machine, or a rebel? (It's usually easier to be a rebel when you have nothing to lose, either early in your career or when you get tenure.) While one can be very skilled, has this brilliant battle to remain competitive been over? lost? Are we here to obey our laws? This paper is written partly as serious advice, partly as emotional encouragement, but most importantly to open up a discussion to improve our standing as academics before the battle is completely lost. We don't want to prevent the development of AI technologies (even if we could); instead, we want to discuss strategies that will enable as many people as possible to participate in this journey.
If you're an AI scholar desperate for your options, here's a list of some ideas. These options are in no particular order. We're not making any particular recommendations here either . At the end of the document, however, we discuss what big tech companies and universities can do to improve the situation. There, we make some specific recommendations.
2. Give up!
Giving up is always an option. Not to give up doing research, but to give up doing truly influential and breakthrough things. Publish papers in mid-level journals and conferences, and there are still many technical details and side issues that can be explored . Note: (1) this approach is best for people who already have a stable position, and you don't really care about promotion; (2) when you decide to pursue a research career, it's not really what you want to do, right? Forcing yourself to adjust your research agenda because of this intense competition is akin to adjusting research to some arbitrary priority of a funding agency such as the European Commission or the US National Science Foundation. Going for the latter at least might get you some funding for your lab, which, in turn, could help you collaborate with some talented AI researchers and PhD students. It is worth noting that we both consider ourselves very lucky in that we have coordinated or been involved in a number of small and large research projects that have allowed us to support our research agenda and helped us (in part) secure our positions .
3 try to expand
It is an admirable sentiment to stand against overwhelming competition. If the extension works, let's implement it in a university lab! Let's challenge the windmill (GPU fan)! The most obvious problem is access to CPU and GPU. Let's say you get $50,000 in cloud computing funding from somewhere and start running your big experiment. But that's very little compared to what it takes to train something like GPT-3. A recent OpenAI agent capable of making diamond drafts in Minecraft required training for 9 days on 720 V100 GPUs [2]; this equates to hundreds of thousands of dollars for one experiment. Not even the prestigious ERC (EU) or NSF (US) grants can support such high levels of investment. However, spending $50,000 on cloud computing gets you more computing power than a bunch of gaming PCs, so you can scale a little. At least for that experiment. But as we know, most experiments don't work on the first try. For every large experiment we see, we spend months or even years prototyping, proof-of-concept, debugging, parameter tuning, and failed attempts. You need constant access to this computing power. The less obvious problem is that you need the right team to build scalable experimental software, which is not easily compatible with academic work structures. Most members of a typical academic research laboratory in computer science are PhD students who need to complete their studies within a few years, do independent project work, publish multiple first-author papers, and find employment after graduation. A large AI project usually means that most members of the team need to work on the same project for many months or years, and only one of them can be the first author of the paper. The team may also include some people who perform "mundane" software engineering tasks that are critical to the success of the project, but are not considered AI research per se. The structure required for successful large-scale projects is simply not compatible with the structure of academia.
4 Downsizing
A popular way to get around the problem is to focus on simple and representative (toy) problems that will theoretically demonstrate the advantages of a new method, or demonstrate the relative advantages of a new method. For example, a recent paper [17] on Behavior Transformers demonstrates the strength of the approach on a toy navigation task that requires only a simple multi-layer perceptron to solve. A similar approach was later adopted in [11]. However, the two studies could be impactful because they demonstrate the capabilities of the algorithms on popular gaming and robotics benchmark problems that require large models and computationally intensive training. In [10], we observed the same pattern again: a case was presented in a toy (gambling) setting, but the influence, arguably, came from the algorithm's demonstration of relative advantage. A downside of this approach is that people will be drawn to high-resolution beautiful colors and take a real car navigating the road more seriously than a toy car, although the challenges may be the same. Therefore, you will get less media exposure. Also, there are some domains, such as language, that are difficult to scale down further.
5 Reuse and remake
One of the core reasons AI has been able to develop so rapidly over the past decade is because researchers have shared their code and models with the scientific community. In the past, model sharing and code accessibility were neither a norm nor a priority for AI researchers. Having access to pre-trained large models such as variants of ImageNet [15], ViT [4] or GPT [?] can help you save time and effort as you can take most of them and make your own fine-tuning for specific problems. Arguably, one needs to assume that the representations of these large models are general enough to provide good performance for downstream tasks with limited training. Unfortunately, fine-tuning and post-hoc analysis on large models is often not enough to provide good performance.
6 Analysis rather than synthesis
Another approach is to analyze publicly available pretrained models. While this may not directly contribute to new functionality, scientific progress can still be made. The current situation is that we have publicly available excellent text and image generative models, but our understanding of them is not sufficient. In fact, we barely fully understand them. Let's face it: transformers are not something intuitive to anyone, and the scale of the training data these models are trained on is almost incomprehensible in itself. There is a lot of work to be done, such as probing them in creative ways, and developing visualizations and conceptual devices to help us understand them. One can approach analysis with different mindsets. It is useful to try to find and describe specific circuits and mechanisms that have been learned and can help us (well, others with resources) create better models in the future. But one can also play the role of a pest who is constantly looking for a way to crack it! That's valuable, no matter what those trying to profit from the big models say.
7 Reinforcement Learning! no data!
One might be less data-hungry and approach AI problems through the lens of (online) reinforcement learning (RL). Following the RL path may allow you to bypass all issues related to data availability, analysis, storage, and processing; however, it does not necessarily reduce the computational effort required. In fact, even the most efficient RL methods are considered computationally heavy because the exploration process is inherently expensive. Furthermore, shaping reward functions often involves forms of black magic (informal) or practical wisdom (more formal). That said, researchers often need to repeatedly run lengthy experiments with different types of rewards (and other hyperparameters) to achieve breakthrough results. Therefore, it is ultimately necessary to reduce the complexity of the problem. The point is, if you want to move away from large datasets, unless you're working on simple (toy) problems or specialized domains, you're probably still facing large computations; the next section is devoted to the latter strategy.
8 miniatures! No calculations!
Another effective strategy is to compromise on model size to save computation. In many cases, you may want or need a smaller model. Imagine the smallest model capable of solving a problem or completing a task. This is especially important for real world applications. Domains such as gaming, IoT, and self-driving cars could allow AI to be deployed next to the end user and next to user-generated data, i.e. at the edge of the network. This is often referred to as edge AI [8], where AI applications can run on devices in the physical world when memory requirements are low and inference happens quickly. Neuroevolution and neural architecture search [8] and knowledge distillation [5, 9] approaches are just a few of the approaches available for edge AI. Note that in addition to learning more from smaller models, one can also try to learn more from less data [6]. Following this research path could lead to major breakthroughs in our understanding of the inner workings of models. Working with small AI models makes analysis easier and improves the explainability of what the models do. Deploying models on-device helps address privacy concerns. Importantly, it supports the Green AI research initiative [16], which promotes inclusive AI that considers (and tries to minimize) its environmental footprint. Clearly, there are limits to what small models can do, but we think the importance of this research direction will grow dramatically in the coming years.
9 Work in a specialized application field or field
It is a fairly effective strategy to choose a niche but established area of research, perhaps outside the immediate area of industry interest, to try to innovate within that field. Often, taking your idea to a whole new field and testing it is a successful strategy, but less often the results have a big impact outside of that field. There are many examples of niche fields that eventually become dominated by a few dedicated researchers. We're mostly taking this strategy right now, with AI in games as our primary scientific community, because very few big companies are serious about modern game AI. Think of how the field of video games infiltrated and dominated the robotics and computer vision research communities in the early '00s (eg the IJCAI and AAAI conference series at the time). Consider the dominance of neural network and deep learning methods in the support vector machine and regression modeling investment community (e.g. the NeurIPS conference a decade ago). Also think about how reinforcement learning and deep learning have changed the core principles of multi-agent learning and cognitive/emotional modeling, for example in the communities represented at AAMAS, ACII and IVA conferences. A core disadvantage of this strategy is that it is difficult to get your paper accepted in the most influential large venues in AI (such as NeurIPS, AAAI, ICML and IJCAI). Your paper and its results may not end up gaining widespread attention. However, it is very possible to create your own community and have your own publishing conference.
10 Solve problems that few care about today
Focusing on an established market segment or application area is a relatively safe strategy, while looking for a market segment or application that does not yet exist has certain risks. Basically, focus on problems whose importance almost no one sees, or approaches that no one sees as promising. One approach is to look for areas in which people have not yet seriously applied AI. A good idea is to study a field that is neither timely nor "sexy". The bet here is that this particular application domain will become important in the future, either for its own merit or because it enables other things. Both of us have gone this route. Fifteen years ago, programmatic content generation for games was a very niche topic that we all helped bring to the research community [20, 23]; Approaches to (deep) reinforcement learning become more important [14, 19]. Reinforcement learning research is a core AI field with thousands of papers published each year, making this once relatively little-known topic all the more important. This high-risk, high-reward mentality can lead to a lonely path, but it can pay off handsomely in the long run. So look around and talk to people who are not AI researchers. What problem domains do you see little application of AI that AI researchers don't seem to understand or care about? Might someone care about these fields in the future? If so, you may need to dig deeper in one of these areas.
11 Things to try that "shouldn't" work
Another relative advantage of small academic teams is the ability to try things that "shouldn't work," meaning they aren't supported by theoretical or experimental evidence. The dynamic in large industrial research labs is often such that researchers are motivated to try things that might work; otherwise, money is lost. In academia, failure can be just as instructive and rewarding as success, with less risk overall. Many important inventions and ideas in the field of AI come from trying the "wrong" things. In particular, all of deep learning arose from researchers doggedly studying neural networks despite good theoretical reasons why they shouldn't be effective.
12. Do something that "looks bad"
The larger and more important a company is, the more likely it is to be constrained by ethics and ideas. Any company is ultimately accountable to its shareholders, and shareholders can easily fire a CEO if they believe the company has suffered "reputational damage." So big companies try to avoid doing anything that looks bad. To get around this, big companies sometimes fund startups to do more experimental work that can go wrong (think Microsoft and OpenAI). But even such drama has limits, as bad PR can come back like a tidal wave in San Francisco Bay. You have nothing to lose as an individual researcher who does not have a position or already has a security position. You can do crazy things you want. You are bound only by the law and your own personality. In no way do we believe that you should be doing unethical research. Anyway, try to do the right thing. But what you find distasteful may be very different from what a group of overeducated, mostly liberal white engineers on the American coast find distasteful. FusionExcel's PR department, ethics committee, and board of directors embrace a very specific set of values. But the world is big, full of different people, different cultures. So there's a big opportunity to do research that these tech companies wouldn't even if they could do. As an example of a project that took advantage of this opportunity, one of us worked on a project to critically examine the orthodoxy of "neutral English" in the current writing support system by creating an autocompletion system, the language model Say you write in the voice of Chuck Tingle, famous author of the absurd sci-fi political satire gay erotica. Our guess is that this project will not be approved for publication by Amazon or Google. Another example is this paper. Likewise, you may find that you stray from the Big Tech culture about nudity, sex, rudeness, religion, capitalism, communism, law and order, justice, equality, welfare, representation, history, reproduction, violence, or other topics consensus. Since all AI research takes place in and is influenced by cultural and political contexts, consider deviation from the norm as an opportunity. If you can't do the research they can't do, do the research they won't do.
13 Start a business; spin off!
It should be obvious by now that academia somehow, paradoxically, limits academic AI research. Even if one manages to acquire multi-million projects at scale, this covers only a fraction of the human and computing resources required for modern AI research. A popular option among AI scientists is to spin their ideas out of university labs and start a company that will gradually translate AI research into a range of commercially standard services or products. Both authors participated in this process by co-founding modl.ai [12] and learned a lot from it. There are many benefits to being part of the applied AI field. In principle, you have access to rich data for practical applications that you were not able to obtain before. Additionally, your AI algorithms are tested in challenging commercial standard applications and must operate in real-world environments. In the end, you usually get more computing power, and if the startup scales, you get a growing human pool. However, this process is far from ideal as there are several limiting factors to consider. First, not all research ideas are directly applicable to entrepreneurial business models. Your best research idea might be brilliant at understanding the world, or at least get published in a high-profile venue, but that doesn't mean it can easily be made into a product. Second, many of the brilliant results achieved in labs today may have a long runway until they turn into some sort of business case. Most startups do development rather than research because the runway is shorter and you need to have a functioning product, preferably with some market traction, before the next funding round in two years or so. Third, even if you get some investment, it doesn't mean you have an unlimited computing budget. Seed funding is usually in the multi-million dollar range, which doesn't equip you to conduct OpenAI-level experiments, especially since you need to pay your employees real salaries (not PhD stipends). Fourth, not every AI scholar enjoys such a risk. At the end of the day, most academics agree early on about their priorities when choosing an academic career path.
14 Cooperate or jump ship!
If none of the above options work for you, and you still want to innovate with large-scale methods that require a lot of data to train, you can always partner with someone who has the compute and the data. There are several ways to continue this approach. Universities located near leading AI companies have a relative advantage, as local social networking and face-to-face meetings make collaboration easier. Researchers at remote universities can still form collaborations through research visits, internships and internships as part of joint research projects. More radically, some established AI professors have decided to dedicate some, if not all, of their research time to industrial partners, or even move entire labs there. The results of such partnerships, internships or laboratory transfers can be dramatic [21, 22]. At first glance, this might seem like the best way forward for AI academics, however, 1) the intellectual property generated is not always publishable, and 2) not all are able or willing to work in AI industry labs. Some even argue that innovation should be driven by public institutions with the support of industry, not the other way around. Universities are therefore responsible for preserving (some or all) the talented AI researchers (scholars and students) they produce and the intellectual property they generate. Otherwise, AI education and research will eventually become redundant in university settings. Next, let's take a closer look at this relationship and outline ways in which industrial companies and universities might help each other.
15 How Can Large Industry Players Help?
It's not clear that big corporations with well-funded AI labs really want to help alleviate the situation. Individual researchers and managers may be concerned about a downturn in academic AI research, but companies care about the bottom line and shareholder value, and having a competitive academic research community may or may not be in their best interest. For the most part, however, large private sector actors do care about this issue, and there is a lot they can do. At the most basic level, an open-source model, including weights and training scripts, can be of great help. This allows academic AI researchers to study trained models, fine-tune them, and build systems around them. That still puts academic researchers at a disadvantage when it comes to training new models, but it's a start. To their credit, some large industrial research organizations regularly release their most capable models publicly. Other organizations are not doing this, and they are rightfully blamed for not doing so. The next step in improving the situation is to collaborate with academia. As mentioned earlier (see Section 14), some large institutions do this frequently, primarily by accepting current PhD students as interns, allowing them to work on a large scale. Some companies offer joint appointments to certain academic researchers, and some even provide occasional research grants. All of this is good, but more could be done. In particular, scholars can initiate collaborations by proposing work that they will accomplish together. Taking this a step further, private companies that genuinely want to help bridge the gap between academia and industry can choose to collaborate openly: publish their plans, submit code, models, and development updates to public repositories, and allow academics to contribute freely. That's not how most companies work, and usually they have very good reasons for keeping it secret. On the other hand, letting academics contribute to your code and training for free can be very rewarding.
16 How can the University help?
While industry may be willing to help, the main initiative should come from universities looking to drive innovation. Notably, some of the most influential papers in the wider field of AI involve university departments. These papers are co-authored by researchers who collaborate with or participate in the company's work. Successful examples already exist [21, 22, 3], but universities need to do more to enable such collaborations. Indeed, there are many ways for academic institutions to initiate and facilitate collaborations with industry. Universities can also help faculty and staff navigate the changing competitive landscape by encouraging and allowing them to be more adventurous. The comparative advantage of academic researchers in AI lies in more high-stakes exploration, so the incentive structure of universities must change to accommodate this. For example, it is unreasonable to expect consistent publications at top conferences such as NeurIPS and AAAI; large, well-funded industrial research labs have a great advantage in producing such papers. Likewise, funding structures reward safe and incremental research on hot topics; this appears to be an inherent feature of the way grant applications are evaluated, and is unlikely to change no matter how funding agencies use words like “disruptive.” The type of research favored by the most traditional (closed) funding mechanisms, primarily the type of research for which academic researchers cannot compete with industry. Therefore, universities should avoid making funding a condition of hiring and promotion. If universities are serious about motivating faculty to play to their competitive advantage, they should reward experimentation and failure, and promote high-risk, high-reward funding schemes and research initiatives. As a result, funding agencies are likely to jump on the bandwagon and spend more on basic and blue-sky research.
17 Epilogue
We have several purposes in writing this letter.
First, to share our concerns with other AI researchers in the hopes of finding a common cause (and collective remedy?) as a community. Second, based on our own experience and discussions in academic and industrial AI venues where we participate or organize Provide a set of guidelines. And third, foster an open dialogue to solicit input to develop strategies that might be more effective for all of us. Arguably, the list of strategies we have discussed at the end is far from exhaustive of all possible possibilities; however, we believe that they are the seeds of a conversation that, in our view, is very timely
