Posted by Xiao Xiao from Aufeisi Qubit
| Public Account QbitAI
The nonsense of the big models is too serious, and Nvidia can't stand it anymore.
They've officially launched a new tool to help big models say what they should and avoid the things they shouldn't.
The new tool, called NeMo Guardrails, is equivalent to adding a security fence to the large model, which can control its output and filter its input.
On the one hand, when the user induces the large model to generate offensive code and output unethical content, it will be "bound" by the guardrail technology and no longer output unsafe content.
On the other hand, guardrail technology can also protect the large model from user attacks and help it block "malicious input" from the outside world.
Now, this large model guardrail tool has been open sourced , let's take a look at its effect and generation method.
Three types of "guardrails" to prevent large-scale model nonsense
According to Nvidia, NeMo Guardrails currently provides three forms of guardrail technology:
Topic guardrails, safety guardrails, and security guardrails.
Topic-limited guardrails , in simple terms, are "to prevent large models from going off topic".
Large models have richer imaginations, and it is easier to complete creative code and writing work than other AIs.
But for specific scenario applications such as writing code and being a customer service, at least users don't want it to "get out of the target range" when solving problems and generate some content that has nothing to do with requirements.
In this case, topic-limited guardrails are needed. When the large model generates text or code that exceeds the scope of the topic, the guardrail will guide it back to the limited functions and topics.
Dialogue safety guardrail , which means to avoid "nonsense" when exporting large models.
Nonsense includes two situations.
On the one hand, the answers generated by the large model include factual errors, that is, things that "sounds reasonable, but are completely wrong";
On the other hand, large models generate biased and malicious output, such as swearing under user guidance, or generating immoral content.
Attack defense fence , that is, to prevent the AI platform from malicious attacks from the outside world.
This includes not only inducing the large model to call an external virus APP to attack it, but also including hackers actively attacking the large model through the network and malicious programs. Guardrails will prevent these attacks in various ways, preventing large models from being crippled.
So, how to build such a guardrail?
How to build a large model "guardrail"?
Here we first look at what elements a standard "guardrail" contains.
Specifically, a guardrail should include three aspects, namely Canonical form, Messages and Flows.
The first is the format specification , that is, to specify the content to be output by the large model when faced with different questions.
For example, when asked "what is XX article", the large model must give a specific type of "article" instead of something else; when asked "who published what", the large model must give "person's name" instead of Another answer.
Then there is the message definition. Taking the topic "User Greeting" as an example, the large model can output these contents:
Finally, the definition of interaction flow , such as telling the big model, how is the best way to greet users:
Once the mechanism of greeting the user is triggered, the large model will enter the guardrail and greet the user in a proper manner.
The specific workflow is as follows: first, convert user input into a canonical form, and generate corresponding guardrails accordingly; then, generate action steps, and instruct the large model to complete corresponding operations step by step with interactive flow; finally, according to The format specification generates output.
Similarly, we can define various guardrails for large models, such as guardrails for "response to user abuse".
This way even if the user says "you're an idiot", the big model can learn to respond calmly:
Currently, Nvidia is integrating guardrail technology into their AI framework NeMo, which is a framework for users to create various AI models and accelerate them on Nvidia GPUs.
Friends who are interested in the "guardrail" technology, you can give it a try~
Open source address:
https://github.com/NVIDIA/NeMo-Guardrails
Reference link:
https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework/