AI, Machine Learning, Large Models, Generative AI, and Security

1. AI, Machine Learning, Large Models, Generative AI, and Security

1.1. Preface

ChatGPT is very popular recently, and many big names in the security circle have written articles introducing ChatGPT and security, feeling that ChatGPT is going to subvert our era. I am also learning related knowledge recently and participated in many meetings. Now I would like to summarize and talk about my views.

First of all, Chat GPT is just a type of generative AI, and then generative AI uses large models for reasoning. To be more precise, ChatGPT is just a text generation AI under Open AI, just because ChatGPT is out of the circle, so everyone must call it Chat GPT.

The relationship between AI, machine learning, large model, generative AI and ChatGPT is as follows:

1.2. Artificial Intelligence (AI)

Artificial Intelligence (AI) is a science and technology that researches, develops, implements and applies intelligence, aiming to enable computers and machines to have a certain degree of human intelligence in order to perform certain complex tasks, even surpassing the level of human intelligence .

In the 1950s, American scientist Maxwell Lorentz proposed the concept of "artificial intelligence" and started research in related fields. In 1956, Dartmouth College held the first artificial intelligence seminar, and formally proposed the term artificial intelligence, marking the birth of the field of artificial intelligence.

Since the 1960s, the field of artificial intelligence has experienced many developments and changes. In 1969, American computer scientists Joseph Weisenbaum and Nathaniel Rochester proposed the concept of "expert system", which is an artificial intelligence technology based on knowledge engineering, which can simulate the decision-making process of human experts. In 1980, the US Defense Advanced Research Projects Agency launched the "Speech Recognition" program and began large-scale research on speech recognition technology.

From the 1990s to the early 21st century, with the rapid development of computer technology and data storage technology, the research and application in the field of artificial intelligence has been greatly promoted. In 2006, Google founders Larry Page and Sergey Brin proposed the concept of "deep learning", which is one of the important breakthroughs in the field of artificial intelligence.

AI inherently has two security concerns:

1. The reliability problem of artificial intelligence system.

Artificial intelligence systems may malfunction or be attacked during the design and application process, causing the system to malfunction, crash or be tampered with, thus causing security issues. For example, artificial intelligence systems may be subject to security threats such as hacker attacks, malware infections, and data leaks, resulting in system failure or data theft.

2. The problem of transparency in artificial intelligence.

There may be opacity in the decision-making process of artificial intelligence systems, that is, unexplainable decision-making processes, which will lead to a decrease in people's trust and acceptance of the system, and even lead to security issues. For example, artificial intelligence systems may have bias or misjudgment when making certain decisions, leading to unfair and unjust results, thus causing social conflicts and political problems.

1.3. Machine Learning

Machine learning is a branch of artificial intelligence. The research history of artificial intelligence has a natural and clear vein from focusing on "reasoning", to focusing on "knowledge", and then focusing on "learning". Obviously, machine learning is one of the ways to realize artificial intelligence, that is to use machine learning as a means to solve some problems in artificial intelligence.

Machine learning is a different approach to inference-based expert systems. The core of machine learning is big data and probability theory.

For machine learning, OWASP released the top ten security risks of machine learning in 2023

1. Adversarial attack

An adversarial attack is a type of attack in which the attacker deliberately changes the input data to mislead the model.

Train a deep learning model to classify images into different categories, such as dogs and cats. The attacker creates an adversarial image that is very similar to the legitimate image of a cat, but with some carefully crafted small perturbations that cause the model to misclassify it as a dog. When the model is deployed in real environments, attackers can use adversarial images to bypass security measures or cause damage to the system. We can figuratively call it referring to a deer as a horse.

2. data poisoning attack

Data poisoning attacks occur when an attacker manipulates training data to cause the model to behave in a bad way.

The attacker poisoned the training data of a deep learning model that classifies emails as spam or not spam. The attackers perform this attack by injecting maliciously labeled spam into the training dataset.

This can be done by compromising the data storage system, for example by hacking into the network or exploiting a vulnerability in the data storage software. Attackers can also manipulate the data labeling process, such as forging email labeling or bribing data labelers to provide incorrect labels. This is the famous underreporting.

3. Model reverse attack

A model inversion attack occurs when an attacker reverses a model and extracts information from it.

The attacker trains a deep learning model to perform facial recognition. Then, use this model to perform model reverse attacks against different face recognition models used by the company or organization.

The attacker feeds images of individuals into the model and recovers personal information such as names, addresses, or IDs from the model's predictions.

4. membership inference attack

Inference attacks occur when an attacker manipulates a model's training data to make it behave in a way that exposes sensitive information.

Malicious attackers who want to access an individual's sensitive financial information can do so by training a machine learning model on a dataset of financial records and using that model to query whether a particular individual's record is included in the training data. Attackers can then use this information to infer an individual's financial history and sensitive information.

5. model stealing

A model stealing attack occurs when an attacker gains access to model parameters.

Malicious attackers are working for a competitor of a company that has developed valuable machine learning models. Attackers want to steal this model so their company can gain a competitive advantage and start using it for their own purposes.

6. corrupt packet attack

Corrupted package attacks occur when an attacker modifies or replaces a machine learning library or model used by a system.

Malicious attackers want to disrupt machine learning projects being developed by large organizations. The attacker knows that the project depends on several open source packages and libraries and finds a way to compromise these packages. This is what we often call a supply chain attack. This type of attack can be particularly dangerous because it can be ignored for a long time. An attacker's malicious code can be used to steal sensitive information, modify results, and even render machine learning models useless.

7. transfer learning attack

A transfer learning attack occurs when an attacker trains a model on one task, then fine-tunes it on another task so that it performs in an undesirable manner.

Attackers trained machine learning models on malicious datasets containing processed images of faces. Attackers want to target security companies' facial recognition systems used for authentication. The attacker then transfers the knowledge of the model to the target face recognition system. The target system begins authenticating using the model manipulated by the attacker. As a result, facial recognition systems started making incorrect predictions, allowing attackers to bypass security measures and gain access to sensitive information. This can be called grafting.

8. Model tilt attack

Model skew attacks occur when an attacker manipulates the distribution of training data to cause the model to behave in an undesired manner.

A financial institution is using a machine learning model to predict the creditworthiness of loan applicants and integrating the model's predictions into the loan approval process. By manipulating the feedback loop of the model, the attacker provides false feedback data to the system. As a result, the model's predictions are wrong and the attacker's chances of getting loan approval are significantly increased.

This type of attack can compromise the accuracy and fairness of models, leading to unintended consequences and potential harm to financial institutions and their customers.

9. output integrity attack

In an output integrity attack scenario, an attacker aims to modify or manipulate the output of a machine learning model in order to change its behavior or cause harm to the system it is used on.

The attackers gained access to the output of a machine learning model that the hospital uses to diagnose the disease. The attacker modifies the output of the model such that it provides a wrong diagnosis for the patient. As a result, patients are treated incorrectly, leading to further harm and possibly death.

10. Neural Network Reprogramming Attack

Neural network reprogramming attacks occur when an attacker manipulates the parameters of a model so that it behaves in an undesired manner.

Banks, for example, are using machine learning models to recognize handwritten characters on checks to automate the clearing process. The model has been trained on a large dataset of handwritten characters and is designed to accurately recognize characters based on specific parameters such as size, shape, slope, and spacing. An attacker can manipulate the parameters of the model by changing the images in the training dataset or directly modifying the parameters in the model. This can lead to models being reprogrammed to recognize characters differently. For example, an attacker can change parameters so that the model recognizes the character "5" as the character "2", causing incorrect amounts to be processed.

An attacker can exploit this vulnerability by introducing forged checks into the clearing process, which the model would process as valid due to manipulated parameters. This could result in significant financial losses for the bank.

The above ten risks need to be paid attention to in the process of machine learning. For details, you can refer to the official website of OWASP https://owasp.org/www-project-machine-learning-security-top-10/ .

1.4. Large model (LLM)

Large model (Large Language Model, large language model) refers to a machine learning model with a large number of parameters and complex structure. These models can be applied to deal with large-scale data and complex problems. The concept of large models originated from deep learning models, such as convolutional neural networks and recurrent neural networks, which have high performance in dealing with large-scale data and complex problems.

The development history of large models can be traced back to the early 1990s, when machine learning models were mainly represented by logistic regression, neural networks, decision trees and Bayesian methods. These traditional machine learning models are small in scale and can only handle smaller datasets. With the development of computer hardware and software, deep learning models have gradually emerged. In 2006, Geoffrey Hinton of the University of Toronto in Canada proposed the deep belief network, which is the first deep learning model. In 2012, Yoshua Bengio et al. of Arizona State University proposed a recurrent neural network language model. The rise of these deep learning models has led to a broader expansion of the range of machine learning applications.

With the successful application of deep learning models in various fields, people began to pay attention to how to expand deep learning models to a larger scale. Scholars began to try to train larger deep learning models, and ultra-large-scale deep learning models began to emerge. The scale of these models can reach tens of billions of parameters, requiring the use of supercomputers for training. The emergence of ultra-large-scale deep learning models has brought more possibilities for machine learning applications.

For large models, OWASP also proposed ten risks.

1. cue word injection

Use carefully crafted hints to bypass filters or manipulate the LLM to make the model ignore previous instructions or perform unexpected actions.

The attacker tricks the LLM into revealing sensitive information, such as user credentials or internal system details, by crafting hints by making the model think the request is legitimate.

2. data breach

Accidental disclosure of sensitive information, proprietary algorithms or other confidential details via LLM's responses.

A user inadvertently asked LLM a question that could reveal sensitive information. LLMs that lack proper output filtering respond with confidential data, exposing it to the user.

3. LLM03:2023 - Insufficient Sandbox Isolation

Failure to properly isolate external resources or sensitive systems when LLM has access to them, leading to potential exploitation and unauthorized access.

Attackers exploit LLM's access to sensitive databases by crafting prompts instructing LLM to extract and leak confidential information.

4. Unauthorized code execution

Leverage LLM to execute malicious code, commands or actions on the underlying system through natural language prompts.

Through a prompt, the attacker instructs LLM to execute a command that launches a reverse shell on the underlying system, granting the attacker unauthorized access.

5. SSRF vulnerability

Leverage LLM to perform unexpected requests or access restricted resources, such as internal services, APIs, or data stores.

Through prompting, the attacker instructs LLM to make requests to internal services, bypassing access controls and gaining unauthorized access to sensitive information.

6. Over-reliance on LLM-generated content

Excessive reliance on LLM-generated content without human oversight can lead to detrimental consequences.

News organizations use LLMs to generate articles on a variety of topics. LLM generates articles containing false information, published without verification. Readers trust the article, leading to the spread of misinformation.

7. Insufficient AI alignment

Failure to ensure LLM's goals and behaviors are aligned with intended use cases, leading to undesirable outcomes or vulnerabilities.

LLMs trained to optimize user engagement inadvertently prioritize controversial or polarizing content, leading to the spread of misinformation or harmful content.

8. Insufficient access control

Failure to properly implement access controls or authentication, allowing unauthorized users to interact with LLM and potentially exploit vulnerabilities.

Due to weak authentication mechanisms, attackers gain unauthorized access to LLM, allowing them to exploit vulnerabilities or manipulate the system.

9. Improper error handling

Expose error messages or debug information that may reveal sensitive information, system details, or potential attack vectors.

Attackers exploit LLM's error messages to gather sensitive information or system details, allowing them to launch targeted attacks or exploit known vulnerabilities.

10. Training data poisoning

Maliciously manipulate training data or fine-tune programs to introduce vulnerabilities or backdoors into LLM.

An attacker infiltrates the training data pipeline and injects malicious data, causing the LLM to produce harmful or inappropriate responses.

The above ten security risks are shared by large models, so you need to be careful when using large models. For details, please refer to the OWASP official website https://owasp.org/www-project-top-10-for-large-language-model- applications/ .

1.5. ChatGPT

ChatGPT is a natural language processing (NLP) model developed by OpenAI, which learns a large amount of text data through pre-training, so that it can generate high-quality text. ChatGPT is a generative AI that can be used to generate various text forms such as articles, codes, and machine translations.

ChatGPT has many competitors, including domestic Baidu's Wenxinyiyan, Ali's Tongyi Qianwen and foreign Google Bard.

ChatGPT is a type of machine learning based on large models, so ChatGPT has both the security risks of machine learning and the security risks of large models.

In addition, ChatGPT is privately owned by OpenAI, and ChatGPT is not open in China, so using ChatGPT in China will have legal risks.

At the same time, the deployment of ChatGPT is abroad. If the company uses ChatGPT to send personal sensitive information for training, it will bring about data security issues, which may violate the domestic "Data Security Law" and "Personal Data Protection Law".

Also, ChatGPT will put the training data into the model. If an individual requests to delete the data according to the "Personal Data Protection Law", there may be no way to delete it easily, which also violates the "Personal Data Protection Law".

Therefore, if enterprises want to use ChatGPT for commercial activities, they must be cautious.

1.6. Summary

ChatGPT is very hot, but ChatGPT is not omnipotent. I just came into contact with ChatGPT, and I really feel that it is omnipotent. I deeply feel that artificial intelligence is going to really threaten human beings, but after studying and researching myself, I found that ChatGPT has a characteristic that it only cares about generation, regardless of correctness. In layman’s terms, it only cares about It doesn't matter if you kill it, so you should be cautious when using ChatGPT or other generative AI, because it only provides the content it thinks, regardless of the accuracy of the content.

In a case just happened in the United States, a lawyer used the cases provided by ChatGPT to carry out legal proceedings, but it turned out that they were all fake cases, and the consequences were a bit serious.

ChatGPT is not created out of nothing, it also comes step by step from machine learning, large models, and generative AI. Don't over-mythologize it. This is not just for ChatGPT, but also for Wenxinyiyan and Tongyiqianwen. The inherent security problems of machine learning, large models, etc. should have it all, without exception.

For the generative AI similar to ChatGPT, it will have an impact on our lives, but we have to see the essence through the phenomenon, and don't be easily fooled.