Yoshua Bengio and Geoffrey Hinton, along with 22 other leading AI scholars and experts, propose a policy and governance framework designed to address the growing risks associated with AI.
Yoshua Bengio, Geoffrey Hinton and Yann LeCun jointly won the 2018 ACM AM Turing Award for their outstanding contributions to deep neural networks. They are often hailed as the "Godfather of AI" and the "Father of Deep Learning".
The document proposed by Bengio and Hinton this time states that companies and governments should allocate one-third of their AI R&D budgets to AI safety, and emphasizes the urgency of seeking specific research breakthroughs to support AI safety efforts. Calls for special action from large private companies developing artificial intelligence, as well as government policymakers and regulators.
Commensurate with their investments in AI capabilities, companies and governments should allocate at least one-third of their AI R&D budgets to ensuring safe and ethical use.
There is an urgent need for governments to fully understand the development of artificial intelligence. Regulators should require model registration, whistleblower protection, incident reporting, and monitoring of model development and supercomputer use.
Regulators should have access to advanced AI systems before they are deployed to assess their dangerous capabilities, such as autonomous replication, breaking into computer systems, or enabling the widespread spread of pandemic pathogens.
The government should also hold developers and owners of “frontier AI,” the name given to state-of-the-art artificial intelligence, legally responsible for reasonably foreseeable and preventable harm caused by their models.
Governments must be prepared to license certain AI development, suspend development in response to worrisome capabilities, enforce access controls, and require strong information security measures against nation-state hackers until adequate protections are in place.
The document focuses on the risks posed by companies that are developing autonomous artificial intelligence, or systems "capable of planning, acting in the world and pursuing goals."
It states that the cutting-edge GPT-4 models provided by Open AI will soon be used to browse the web, design and perform chemical experiments, and utilize software tools, including other AI models. Software programs like AutoGPT were created to automate such AI processes, allowing AI processing to continue without human intervention. But they believe there is a huge risk that these autonomous systems will run out of control and there is no way to control them.
If we build highly advanced autonomous AI, we risk creating systems that pursue undesirable goals. Malicious actors may intentionally embed harmful targets. Furthermore, no one currently knows how to reliably combine AI behavior with complex values. Even well-intentioned developers can inadvertently build AI systems that pursue unintended goals—especially if they neglect expensive safety testing and human oversight in order to win AI competitions.
The document also calls for accelerated research breakthroughs to address some of the technical challenges in creating artificial intelligence with safety and ethical goals:
- Supervision and honesty: More capable AI systems are better able to exploit weaknesses in supervision and testing—for example, by producing false but convincing outputs;
- Robustness: AI systems behave unpredictably in new situations (under distribution shifts or adversarial inputs);
- Explainability: AI decisions are opaque. Until now, we have only been able to test large models through trial and error. We need to learn to understand their inner workings;
- Risk Assessment: Frontier AI systems can develop unforeseen capabilities that are only discovered during training or even after deployment. Better assessment is needed to detect hazardous capabilities earlier;
- Addressing emerging challenges: More capable future AI systems may exhibit failure modes that have so far only been seen in theoretical models. For example, AI systems might learn to feign compliance or exploit weaknesses in safety goals and shutdown mechanisms to advance specific goals.
But unlike Bengio and Hinton, Yann Lecun believes that current artificial intelligence risks do not require such urgent measures.
More details can be found here .