"Little Lessons in Hidden Words" The role of LDP and CDP in federated learning for privacy and robustness

The content of this sharing is an article included in NDSS: "Local and Central Differential Privacy for Robustness and Privacy in Federated Learning". This paper mainly analyzes the privacy and robustness of LDP and CDP in federated learning. The sharing around this paper will be divided into the following 4 parts:

motivation and method
background knowledge
Attack method and defense method against FL
Summarize

1. Motivation and method

1. Motivation:

Existing defense methods can only protect one of the two dimensions of privacy and robustness, so two questions are raised:

Can the above two dimensions be protected at the same time?
How to measure the trade-off between protection effect and model usability?

2. Method and achieved effect:

Summarizes the two dimensions of FL, that is, the existing attacks and defense methods of robustness and privacy, and analyzes the protection effect of LDP and CDP on the robustness and privacy of FL. The feasibility of the Intuition lies in : LDP is sample-level, CDP is participant-level, which can reduce the influence of 'poisonous data or gradients' in training to varying degrees, and can provide varying degrees of privacy protection. In addition, extensive experimental comparisons are performed on real datasets to verify the effects of LDP and CDP.

2. Background knowledge

1. Federated Learning

Image source: https://air.tsinghua.edu.cn/info/1008/1312.htm

Federated learning was first proposed by Google. The main idea is that each data site iterates the training model locally, obtains the gradient of the model, and then submits the gradient aggregation to the central server, and sends the aggregated gradient to each data site. In order to protect the gradient of the model and resist malicious attacks, the central server will adopt different aggregation methods or homomorphic encryption technologies to improve security and robustness.

2. Differential privacy

Differential privacy provides a way to quantify the degree of privacy leakage, and its standard definition is:

In the field of machine learning, it can be used to add noise to gradients containing private information when sending or aggregating gradients:

3. Differential privacy + federated learning

Common places to add noise are:

Loss Function: Modify the loss function of the model
Model: Add noise to the trained model parameters
Gradients: Noise the gradient during model training

At present, the most used method is to add noise to the gradient. Modifying the loss function requires analyzing the convergence of the model, and adding noise to the final model may lead to serious loss of accuracy.

There are two common ways to add noise, based on DP-SGD [1]

LDP: local differential privacy. That is, each party locally adds noise to the gradients that have not yet been aggregated

CDP: central differential privacy. That is, the central server adds noise to the aggregated gradient

The above two noise adding methods are to add noise according to the second norm of the gradient when calculating the gradient.

3. Attack and defense methods against FL

Robustness

1. Attack method

Poisoning attack (note that the adversary of this attack assumes only client), which can be classified as follows

Random: Construct some random samples/gradients, or misclassified samples to reduce the Acc of the final model
Target (Backdoor): Construct a specific sample/gradient so that the classification result of a specific sample is a specified value

This article is aimed at Target, the Backdoor backdoor attack.

In the backdoor attack [2,3], the model parameters use [failed to save the external link image, the source site may have an anti-leeching mechanism, it is recommended to save the image and upload it directly (img-ZmyxZvJk-1683356837955) (null)], backdoored version Use [External link picture transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the picture and upload it directly (img-Nt9jBuQ0-1683356837842)(null)],

Use the model-replacement method to realize the attack during aggregation:

So there are:

2. Defense method

Byzantine-robust Defense: Krum, Trimmed Mean...
The defense related to Bzyzantine is not analyzed here, and the claim is that these solutions do not provide privacy-related protection
Sun et al.[2] proposed Norm Bounding and Weak DP to defend against attacks
Norm Bounding: By limiting the gradient, the influence of the malicious attacker's gradient on model training is weakened

Weak DP: Similar to CDP, but does not consider Total Privacy Budget, which will lead to too much noise added, affecting the usability of the model
Small noise, large privacy loss

3. Experimental results

Experimental setup: select the experiment on two data sets as an example

EMNIST- 5-layer CNN. 2400 clients
CIFAR10 – ResNet18 100 clients

The metrics evaluated are Main Task Accuracy and Backdoor Accuracy. The former represents the availability of the model (the higher the availability, the better), the latter represents the ability to resist attacks (the lower the probability of Backdoor success).

Setting 1: Using the attack method in [2], there is only one attacker in each iteration

Figure 3 shows that if no defense measures are added, although the Main Task Accuracy is high, the corresponding Backdoor Accuracy is also high, indicating that the backdoor attack is very effective.

Both Norm Bounding and Weak DP can bring a certain degree of resistance, but Norm Bounding will still lead to a higher Backdoor Accuracy after training for many rounds.

LDP and CDP have significant resistance to backdoor attacks, but the increase in epsilon will lead to a decrease in utility, and the Main Task Accuracy is lower than Weak DP.

Setting 2: Increase the number of attackers in each iteration, and the attacker can choose whether to implement LDP defense according to the protocol

Compared with Norm Bounding and Weak DP, LDP and CDP can play a good defensive effect, but they also lead to a decline in utility. It is worth noting that only 10% of the attackers in the LDP defense need not add noise, which can achieve a stronger attack effect than no defense. This is because the gradient without DP has a greater impact on the aggregated model gradient.

From the perspective of robustness alone, LDP and CDP have no significant advantages. In addition, CDP has security assumptions for the server, while LDP has weak defense against attacks (assuming the client is the attacker)