Information leakage of federated learning

Sorting out federated learning and information leakage issues in federated learning

1 Federated Learning

The term federated learning was introduced by McMahan et al. in "Communication-efficient learning of deep networks from decentralized data". The basic structure of federated learning is: n clients (such as mobile devices, etc.) collaborate to train the model under the coordination of a federated server (such as a service provider), and the client's training data is stored locally and not directly provided to the server.

1.1 Composition of federated learning framework

The federated server coordinates the collaborative training process by iterating client selection, local training, and global aggregation until training converges.

Client selection:
The total number of clients is m. In each round of training, n clients are selected to participate in training. This process can improve the training efficiency and performance of the model while protecting user privacy . In federated learning, clients participating in training are usually divided into multiple groups or sets. By selecting a specific set of clients for training, you can better control the distribution and diversity of the data, thereby improving the model's generalization ability . In addition, since different clients may have different computing capabilities and bandwidth limitations, filtering clients can also effectively reduce communication overhead and speed up training . It should be noted that when performing client filtering, privacy protection issues must be fully considered . You should try to avoid operations such as selecting clients that contain sensitive information or using clear text to transmit data to reduce the risk of user privacy leaks .

Local training:
The client will receive a notification that round t is selected, and download the parameters (weights) and training program of the global model from the federated learning server . Each selected client updates the global model locally using local training data by executing the training program . The updated gradient is grad G = ∂ l ( X , y , θ ) ∂ θ grad\ G=\frac{\partial l(X,y,\theta)}{\partial \theta}
$$gradG=\frac{\partial l(X,y,i}{\partial \theta }$$
Among them, X and y represent the training data divided into batches and the corresponding labels, $l(\cdot )$ represents the loss function.

Trainer : Often called "client code" or "client model". It is a code provided by the server to all participants. It implements the local update logic of the federated learning algorithm, including steps such as data loading, model optimization, and uploading new parameters. Different federated learning frameworks may have different implementations of client code, but generally they all have the following basic components:

• Data loading: Client code needs to be able to load data from local storage and convert it into the format required for model training (such as tensors or sequences).
• Model definition: The client code should define the structure and parameters of the local model . The parameters are only definitions and have no specific values.
• Optimizer definition: In order to optimize the local model, the client code also defines an optimizer to calculate gradients and update local model parameters.
• Training loop: Finally, the client code needs to define a training loop to iteratively update the local model multiple times. After each round of training is completed, the client code also needs to upload local model parameters to the server for aggregation.

The training program does not contain the weight parameters of the global model, but is defined without specific values. Its task is only to perform the training of the local model and upload the updated parameters. Usually, the global model parameters are sent to the client at the beginning of each round of training , and the locally updated parameters are uploaded to the server for aggregation. Therefore, the client does not need to download the training program in every training round , but only needs to download it from the server once at the beginning of the first round of training . Once clients successfully download the training program and initialize the local model, they can reuse the program in multiple training rounds to perform local model training and parameter upload operations. This approach can significantly reduce network bandwidth and communication costs, while also improving training efficiency and performance.
It should be noted that in some cases, the client may need to periodically download new versions of the training program to ensure that it can keep pace with the latest code on the server. For example, if the server undergoes a major upgrade or modification to the algorithm or model architecture, or if a security vulnerability or bug is discovered, the client may be required to update its local training program. At this time, the client can obtain the latest version of the training program by sending a request to the server or using some automatic update mechanism.

Global aggregation:
After receiving local updates from the n clients selected in each round, the federated learning server aggregates these updates and updates the global model, and enables the next round of learning.

1.2 Common federated learning models

Different federated learning models mainly differ in the following aspects:

• Model architecture: The model used in federated learning can be any type of machine learning model, including neural networks, decision trees, etc. Different model architectures may have an impact on the training effect and communication overhead of the model.
• Participant/client selection: Different federated learning models may adopt different participant selection strategies, such as random selection, selection based on participant contribution, etc. The strategy selected by the participants will have an impact on the convergence speed and accuracy of the model.
• Global aggregation method: Different federated learning models may use different global aggregation methods, including weighted average, FedAvg, FedProx, etc. The choice of global aggregation method will also have an impact on the convergence speed and accuracy of the model.
• Security guarantee: Federated learning involves data sharing among multiple participants, so security issues are also an important consideration. Different federated learning models may adopt different security measures, such as differential privacy technology, encryption technology, etc.
• Data preprocessing: In federated learning, since the local data of each participant may differ, data preprocessing is required. Different federated learning models may use different data preprocessing methods, such as local update, global update, etc.

Therefore, if you want to develop a privacy and security defense strategy for real-life application scenarios, you need to consider the impact of these differences on attacks.

Summary of common federated learning models
Reserved space

Paper publication status of classic federated learning models:

FedAvg (Federated Averaging): This is one of the most common federated learning models. It trains the global model by updating the model parameters locally and then uploading the local model parameters to the server for averaging. This algorithm was originally proposed by the Google Research team in a paper published in 2017, titled "Communication-Efficient Learning of Deep Networks from Decentralized Data". The paper was published at the International Conference on Machine Learning (ICML).

FedProx (Federated Proximal): The FedProx model uses a penalty function method to ensure that the updates of local model parameters are not too far away from the global model parameters. Technology proposed in 2018 by CMU and IBM. The paper is titled "Federated Learning with Proximal Gradient Descent" and was published at the NIPS (Neural Information Processing Systems) conference.

FedOpt (Federated Optimization): This model uses global model parameters to initialize the local model, improves the local model through multiple local optimizations, and finally uploads the local model parameters to the server for aggregation. It was proposed in a paper published by Tencent AI Lab in 2019. The paper is titled "FedOpt: A Pragmatic and Flexible Framework for Federated Optimization" and was published at the ICML conference.

FedNova (Federated Novel Aggregation): This model uses second-order gradient information to calculate step size and direction on each device, allowing for better local model convergence. In 2020, a paper titled "FedNova: Federated Learning with Novel Aggregation Algorithms" was proposed by researchers at the University of Washington and published at the ICML conference.

FedMA (Federated Model-Agnostic Meta-Learning): This model provides a meta-learning framework that allows devices to train on a small amount of local data and quickly adjust global model weights to adapt to new tasks. Proposed by researchers at the University of California, Los Angeles in 2020, the paper is titled "Federated Meta-Learning for Recommendation" and was published at the AAAI (Association for the Advancement of Artificial Intelligence) conference.

2 Information leakage attack on federated learning

综述部分参考：
[1] Xuefei Yin, Yanming Zhu, and Jiankun Hu. 2021. A Comprehensive Survey of Privacy-preserving Federated Learning: A Taxonomy, Review, and Future Directions. ACM Comput. Surv. 54, 6, Article 131 (July 2022), 36 pages. https://doi.org/10.1145/3460427

2.1 Attack

2.1.1 Background information of the attack

potential attacker

Internal actors : Participating clients or central servers . In a client-server architecture, a central server controls the entire training process by receiving and sending local training updates from and back to client participants, so both clients and servers have access to intermediate training updated values ​​(such as weights or gradients) ) and the final model.
External actors : model consumers or eavesdroppers . Model consumers have two ways to detect private data, one is to obtain access to the entire model weights, and the other is to obtain access query results provided by the platform API, so they can detect privacy through the final model or its query results. data. An eavesdropper steals intermediate training updates (weights or gradients) or the final model by intercepting communications between participants and the server.

Malicious client :
obtains intermediate training updates from the aggregator to detect private information of other client datasets;
sends deliberately designed training updates to the aggregator to detect special private data of other client datasets;
conducts inference attacks, If the final model is used, privacy will be leaked.
——Defense: Mitigated by designing specific training protocols, such as selecting a subset of clients in each round

Malicious servers : poses a high risk of privacy leaks (csDLG)
Since the server maintains intermediate updates and final models, this information can be exploited for reconstruction attacks (iDLG);
private data of a specific client can be detected by distinguishing updates to its model ;
Participating clients can be selected and specified in each round of training to explore the privacy of the training data set.
——Defense: Limit and quantify the server's ability to reconstruct the client's local training data set

Malicious model consumers : May pose a high risk of privacy leakage.
They can easily access the final model or query results, and therefore can use this information to conduct information inference attacks. For example, a malicious model consumer can use this information to generate Generate representative samples, determine whether samples are used for model training, obtain features of training data, or generate training samples and labels.
——Defense: - - - - - - - - - - - - -

Eavesdropper : Can cause serious leakage of sensitive information and create a high risk of privacy leakage.
Detect private information by stealing intermediate training updates or final models transmitted between client participants and servers, such as reconstructed training samples (DLG).
——Defense: It can be defended through encryption mechanisms (homomorphic encryption, secret sharing or multi-party secure computation).

Attack type

Active attacks : attacks designed to change system resources or affect system operation
. Upload special gradients to learn separable representations.
Control certain clients to strategically adjust training data to make the global model rise or fall in a special model.

Passive attack : An attack that aims to exploit information or learn from the system without modifying it.
Passive black-box attack: For example, in a service platform, it is assumed that the attacker can only access the query results, but not the model parameters or intermediate training updates. Passive white-
box Attack: Suppose an attacker has access to intermediate training updates, model parameters, and query results

The risk of passive black-box attacks is usually limited because the attacker can only use the query results to detect sensitive information, while passive white-box attacks can often reveal more sensitive information of the training data set.

This blog mainly studies passive white-box attacks, so the risks of active attacks will not be elaborated here.

stage of leakage

Training phase : It mainly includes calculating local gradients on the client, aggregating the global model on the server, passing intermediate updates between the client and the aggregator, and publishing the final model to the client. The update information mainly includes local gradients and local model weights. , aggregate gradients or model weights, and the final model. Participants or eavesdroppers can conduct attacks at this stage. Malicious actors can collect aggregated model updates and conduct privacy attacks to infer the data privacy of other participants; eavesdroppers may transfer local updates to selected clients ( For example, local gradients or model weights) are sent to the server for model aggregation, or when the server sends aggregated updates back to the client, eavesdropping on the update information passed between the actor and the aggregator.

Inference stage : mainly involves the method of providing query services to consumers. The risk of privacy leakage is mainly related to the final model, which is released to participants or provided as a service platform

where the leak occurs

In federated learning, three types of important data need to be transmitted between participants and aggregators: weight updates, gradient updates, and final models.

In the FL framework based on gradient/weight update, the client downloads the aggregated model from the server, performs local training based on local training data, and then sends gradients/weights to the server, and the server performs aggregation, and the cycle repeats until the end of training. In deep network models, gradients are typically computed by backpropagating the loss from the training dataset throughout the network.

The gradient is related to the label and sample characteristics of the local training data, which means that the gradient contains part of the label information and original information, so the gradient may lead to serious privacy leakage risks . Privacy attacks based on gradient updates proposed by DLG and iDLG have been successfully used to extract training samples and labels, where DLG leaks private training samples and labels by minimizing the loss between the attack and the true gradient.

The purpose of privacy attacks

Privacy attacks often aim to infer sensitive information about a training data set

Inference about class representation aims to generate representative samples that are not real data instances of the training data set but can be used to study sensitive information about the training data set. Membership
inference aims to determine whether a data sample has been used for model training.
Training data attributes Inference aims to infer attribute information about the training data set.
Inference aims to reconstruct the original training data samples and corresponding labels (DLG, iDLG).

What I'm mainly working on is inference of training samples and labels:

DLG is an optimization algorithm that can successfully obtain training inputs and labels over several rounds of training. First, a virtual data set consisting of virtual samples and labels is randomly generated, and then based on the virtual data set, the virtual gradient is obtained through forward and reverse calculations. This algorithm iteratively optimizes by minimizing the loss between the virtual gradient and the real gradient. Virtual samples and labels, and finally the attacker obtains training samples and labels. This type of attack does not require prior knowledge about the training set.

2.1.2 Focused attack methods: gradient depth leakage & gradient inversion attack

The attack algorithms used in the paper " Protect Privacy from Gradient Leakage Attack in Federated Learning " (hereinafter referred to as GradDefense) are deep gradient attack (DGA) and gradient inverting attack (GIA)

• deep gradient attack (DGA) 参考文献：L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” Advances in Neural Information Processing Systems (NeurIPS), vol. 32, pp. 14 774–14 784, 2019.
• gradient inverting attack (GIA) 参考文献： J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting gradients - how easy is it to break privacy in federated learning?” Advances in Neural Information Processing Systems (NeurIPS), pp. 16 937–16 947, 2020

The attack algorithms used in the paper " Soteria: Provable Defense against Privacy Leakage in Federated Learning from Representation Perspective " (hereinafter referred to as Soteria) are DLG attack and Gradient Similarity (GS) attack

• DLG attack 参考文献：Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems, 2019
• Gradient Similarity (GS) attack 参考文献：Jonas Geiping, Hartmut Bauermeister, Hannah Droge, and Michael Moeller. Inverting gradients–how easy is it to break privacy in federated learning? In Advances in Neural Information Processing Systems, 2020

The attack algorithms used in the paper " More than Enough is Too Much: Adaptive Defenses against Gradient Leakage in Production Federated Learning " (hereinafter referred to as OUTPOST) are gradient leakage attacks DLG and csDLG. When applied, a mechanism for matching increments from updates is used.

• DLG 参考文献：L. Zhu, Z. Liu, , and S. Han, “Deep Leakage from Gradients,” in Advances in Neural Information Processing Systems (NeurIPS), 2019.
• csDLG 参考文献：J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting Gradients — How Easy Is It to Break Privacy in Federated Learning?” Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 16 937–16 947, 2020.

It can be seen from the references that DGA = DLG, GIA = GS = csDLG, which means that these three papers use the same two attack strategies, which are
collectively referred to as DLG and csDLG below.

DLG gradient depth leak

A commonly accepted approach in distributed machine learning and federated learning is to share data gradients . Multi-party data are jointly modeled through shared gradient information , that is, modeling is performed without the original data leaving the database.
But even if the gradient is shared instead of the original data, privacy and security cannot be achieved. The gradient is related to the label and sample characteristics , which means that the gradient contains part of the label information and original information . The DLG attack is through the gradient in the neural network. Information to reverse the original data and labels.

DLG is a gradient-based feature reconstruction attack. The attacker receives gradient updates ▽ W t , k \triangledown W_{t,k} from other participants k in round t$▽W_{t,k}$, the purpose is to steal the training set (xt, k, yt, k) of participant k from the shared information (x_{t,k}, y_{t,k})$(x_{t,k},y_{t,k})$。

The algorithm first initializes a fake image/fake sample with the same resolution as the real image and a fake label with a probability representation, and then inputs these fake samples and fake labels into the existing model, and then obtains the fake model gradient. The goal of optimization is to minimize the distance between true and false gradients, and use the distance between true and false gradients to optimize false samples and false labels, so that the false samples and false labels are consistent with the real sample labels. Note that for most federated learning applications (my understanding is the training program mentioned above), the model architecture F() and weights Wt are shared by default.

How to know the resolution of the real image to generate a fake image? How do you know which categories fake labels are specifically divided into? Can the above questions be completed only after knowing the data set?

See flow chart below

In the flow chart, the author circles the variables that need to be updated with bold borders. Normal participants compute ▽ W \triangledown W by using the private training set$▽W$ toupdate model parameters$W$ , that is, updating the parameter W to minimize the loss function, andthe malicious attackerminimizes the distance between the true and false gradientsbyupdating the false samples and false labels

The attacker's goal function $$x^{\prime \ast },y^{\prime \ast }=\underset{x^{\prime },y^{\prime }}{\mathrm{argmin}}\Vert \nabla W^{\prime }-\nabla W{\Vert }^2=\underset{x^{\prime },y^{\prime }}{\mathrm{argmin}}\Vert \frac{\partial l(F({\mathbf{x}}^{\prime },W),y^{\prime })}{\partial W}-\nabla W{\Vert }^2$$

See pseudo code below

Thesis experiment: Given an image containing an object, image classification aims to determine the category of the item. The power of the DLG attack is first evaluated on the modern CNN architecture ResNet [18] and images from MNIST [15], CIFAR-100 [16], SVHN [17] and LFW [9]. Note that two changes were made here: (1) for the model architecture, all ReLU operators were replaced with Sigmoid, and the step size in CONV was removed, since our algorithm requires the model to be twice differentiable (2) for Image Labels, Instead of optimizing discrete categorical values ​​directly, we randomly initialize a vector of shape N × C, where N is the batch size and C is the number of classes, and then softmax the output as the categorical labels for optimization and DLG attacks.

How to know the batch size N and the number of categories C?

Method: Randomly generate a false input sample and false label of the same size as the real data, then input these false samples and false labels into the existing model, and then obtain the false model gradient. The goal is to generate fake gradients with the same gradient as the original model, so that the fake samples and fake labels are consistent with the real sample labels.

How to know the size of this real data? It is provided to the attacker as prior knowledge in the paper, but can it be obtained in a real attack? The size issue to be studied
here includes the size of individual data, such as the number of pixels in length and width of the image, and the batch size. In the paper these are all prior knowledge

csDLG gradient inversion attack

Attacker: Malicious server
- Since the server maintains intermediate updates and final models, this information can be used to perform reconstruction attacks.
Attack type: Passive white-box attack
- Assuming that the attacker has access to intermediate training updates, the model parameters
are leaked: Training phase
Where leakage occurs: weight update/gradient update
- mainly includes calculating local gradients on the client, aggregating the global model on the server, passing intermediate updates between the client and the aggregator, and publishing the final model to the client , the update information mainly includes local gradients, local model weights, aggregated gradients or model weights.
The purpose of the privacy attack: inference of training samples and labels
- aiming to reconstruct the original training data samples and the used labels

This paper also mentioned that parameter gradients carry important privacy information about the input, and even running multi-image federated average learning under real architecture cannot guarantee data privacy for all users. In each batch of a hundred images, there will still be a few that can be recovered.

The recovery mentioned here is not all recovery, it is mentioned that the number of recovered pictures is smaller.
What is multi-image?
First observe the Federated Gradient Descent algorithm (Federated SGD):
$$i^{k+1}=i^k-t\sum _{i=1}^N{\nabla }_{}{\mathcal{L}}_{i^k}(x_i,y_i)$$
Each user calculates part of the gradient descent locally, and then sends the updated parameters back to the server. Finally, the information about (label y under the training data) will be blurred through the gradient information collection of some local examples. This is called multi-image setting

Denied the previous attack model DLG, pointing out that the assumptions put forward are strong and conducive to data reconstruction, but not realistic. The loss function used is the optimized argmin ⁡ x ∥ ∇ of the Euclidean matching term $$\underset{x}{\mathrm{argmin}}\Vert {\nabla }_{}{L}_{}(x,y)-{\nabla }_{}{L}_{}(x^{\ast },y){\Vert }^2$$
Minimize this function to recover the original input image x from the transmitted gradient information.

This paper proposes to decompose the gradient parameters into norm size and direction (refer to my blog for what gradient is )
. Size: represents the local optimality of the measured data point relative to the current model.
Direction: quantification of the angle between two data points. The change in prediction as one data point moves toward another data point

Let us assume that there is an infinitesimal (indefinitely constructive, infinitesimal)
$$\underset{x\in [0,1{]}^n}{\mathrm{argmin}}1-\frac{⟨{\nabla }_{}{\mathcal{L}}_{}(x,y),{\nabla }_{}{L}_{}(x^{\ast },y)⟩}{\Vert {\nabla }_{}{\mathcal{L}}_{}(x,y)\Vert \Vert {\nabla }_{}{L}_{}(x^{\ast },y)\Vert }+\alpha TV\left(x\right)$$
where TV(x) represents the TV regularization parameter to reduce the noise of the reconstructed image

Summary: Differences and Connections

Both attacks attempt to solve the optimization problem

By the target gradient ∇ ∗ = ∂ L grad ( F w ( x ∗ ) , y ∗ ) received from participating client k at some step t ∂ w \nabla^{*} = \frac{\partial \mathcal{L ${\nabla }^{\ast }=\frac{\partial {\mathcal{L}}_{grad}(F_w(x^{\ast }),y^{\ast })}{\partial w}$, the attacker can restore the input with labels $(x_t^k,y_t^k)$ (e.g. pixel images or sentences in tags during this training step)

Attacker
1. Randomly generate inputs and labels with the same size as the target data $(x_0^{\prime },y_0^{\prime })$ (initial dummy data and labels)
2. By inputting dummy data and labels into the model shared between client participants$F_w$, generate virtual gradient ${\nabla }_0^{\prime }=\frac{\partial {\mathcal{L}}_{grad}(F_w(x_0^{\prime }),y_0^{\prime })}{\partial w}$
3. Calculate and update virtual data $x^{\prime \ast },y^{\prime \ast }={\mathrm{argmin}}_{x^{\ast },y^{\ast }}\mathbb{D}({\nabla }^{\prime },{\nabla }^{\ast })$

Among them, the distance $D({\nabla }^{\prime },{\nabla }^{\ast })$is a differentiable function calculated through virtual data, which can be theL2 distance proposed in DLG$\Vert {\nabla }^{\prime }-{\nabla }^{\ast }{\Vert }^2$ , or it can bethe cosine distance 1 proposed in csDLG$1-\frac{⟨{\nabla }^{\prime },{\nabla }^{\ast }⟩}{\Vert {\nabla }^{\prime }\Vert \Vert {\nabla }^{\ast }\Vert }$

2.2 Defense

Privacy-preserving Machine Learning (PPML) aims to use privacy protection technology in machine learning to protect data privacy. Privacy-preserving Federated Learning (PPFL) can be regarded as a part of PPML. A special case. There are three common PPML methods, namely: PPML based on homomorphic encryption (HE), PPML based on secure multi-party computation (SMC), and PPML based on differential privacy (DP).

In the field of computer security, privacy is defined as: “Ensuring that information about individuals operating or affecting them may be collected and stored, and by whom or to whom that information may be disclosed.” Privacy protection mechanisms aim to achieve data utility while ensuring that original information is not disclosed to other individuals or groups. Three types of general privacy protection mechanisms are encryption technology, perturbation technology, and anonymity technology . Homomorphic encryption and secure multi-party computation are encryption technologies, and differential privacy is a perturbation technology.

PPFL is a clever combination of FL and privacy protection mechanisms. The main challenge is how to balance the trade-off between data privacy and data utility when applying privacy protection mechanisms to the FL framework.

上述来自参考文献
[1] Xuefei Yin, Yanming Zhu, and Jiankun Hu. 2021. A Comprehensive Survey of Privacy-preserving Federated Learning: A Taxonomy, Review, and Future Directions. ACM Comput. Surv. 54, 6, Article 131 (July 2022), 36 pages. https://doi.org/10.1145/3460427

Some existing privacy-protecting machine learning defense mechanisms can be divided into the following types:
differential privacy (DP), multi-party computation (MPC), data compression, and homomorphic encryption.

Differential Privacy (DP)
Secure Federated Transfer Learning" by Shuang Song et al. (2020) - Published in the Proceedings of the IEEE Symposium on Security and Privacy (SP) Network and Information Security Class A Conference

"DP-FedAvg: Robust and Communication-Efficient Federated Learning with Differential Privacy" by Tian Li et al. (2020) - Published in the Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS). Network and Information Security Category A Meeting

Multi-Party Computation (MPC)

P. Mohassel and Y. Zhang, “Secureml: A system for scalable privacy-preserving machine learning,” in Proceedings of IEEE Symposium on Security and Privacy (SP), 2017, pp. 19–38

K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan,
S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in proceedings of ACM SIGSAC Conference on Computer and Communications Security (CCS), 2017, pp. 1175–1191.

Homomorphic Encryption (HE)
C. Zhang, S. Li, J. Xia, W. Wang, F. Yan, and Y. Liu, "Batchcrypt: Efficient homomorphic encryption for cross-silo federated learning," in Proceedings of USENIX Annual Technical Conference (ATC), 2020, pp. 493–506. Computer Architecture/Parallel and Distributed Computing/Storage Systems Class A Conference

Y. Aono, T. Hayashi, L. Wang, S. Moriai et al., “Privacy-preserving deep learning via additively homomorphic encryption,” IEEE Transactions on Information Forensics and Security (TIFS), vol. 13, no. 5, pp. 1333–1345, 2017. Network and Information Security Category A Journal

The above are not specific defense measures against gradient leakage attacks, but general defense measures. These mechanisms either result in unacceptable computational overhead or significantly reduce accuracy.

Targeted Defense (TD)
was recently proposed (the main comparison algorithm in one of Mr. Li’s references, the GD paper)
J. Sun, A. Li, B. Wang, H. Yang, H. Li, and Y. Chen, “Provable defense against privacy leakage in federated learning from representation perspective,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 9311–9319 Artificial Intelligence Class A Conference

2.2.1 Soteria

Soteria is specifically proposed to target gradient leakage attacks. In order to reduce the quality of the reconstructed data, and based on their finding that privacy leakage mainly comes from the data representation embedded in the fully connected (FC) layer, a refined design of perturbing the data representation in the FC layer with shared gradients is proposed.

2.2.2 GradDefense

GradDefense shows that the original data can still be recovered from the remaining gradients by silencing the perturbed layer, thus proposing a stronger defense by measuring the sensitivity of perturbing all layers that share gradients.

2.2.3 OUTPOST

OUTPOST is a variation of the gradient perturbation method in the defense method mentioned above.

This paper re-evaluates the information leakage threats considered in previous research and proposes several differences from reality: 1.
In actual production, shared model updates are not equal to shared gradients ;
2. Even if direct sharing is assumed Without the gradient, even if one or several images are reconstructed (this is proposed by GradDefense, 100 images may be able to reconstruct several of them, but not all), in actual production, a lot of local calculations are performed on the client and a lot of work is done. Whether these images can indeed be recognized by people with little communication (and therefore little information that can be intercepted) ;
3. Explicitly initialize the model (wide distribution of weights) , using an untrained neural network will make the model and Shared gradients are more fragile and easier for attackers to perform image reconstruction .

What is explicit initialization of wide distribution model weights?

The two defense measures mentioned above, Soteria and GradDefense, are designed against gradient inversion attacks (including DLG, csDLG, iDLG, etc.), but the feasibility of reconstructing DLG attack data has not been proven before applying the defense, especially In a FL production environment using FedAvg.

In this article, the intensity of the attack is made close to the actual attack intensity, which is called a gradient leak attack . This attack is much weaker than the above gradient inversion attack , so it can be more lightweight when designing the defense mechanism.

Defense mechanism OUTPOST
inspiration : models with narrower distribution weights and more local SGD update steps will effectively weaken potential attacks.
Design : biased perturbation of gradients based on the degree of dispersion and information content of model weights in different local update steps.
Whether to perturb: Use the probability threshold to decide whether to perturb at the current step; the threshold attenuates with the local update step and time. How to perturb:
Evaluate the leakage risk of the current local model through the variance statistics of the model weights of each layer; the risk determines the Fisher range , add Gaussian noise to each gradient layer of the current step according to the Fisher information matrix

The paper proposes that existing gradient inversion attacks cannot work effectively in production FL, and some implicit assumptions are not valid. Therefore, the paper re-evaluates gradient leakage attacks in production FL.

The gap between the assumptions of the current study and the actual setup in production FL is shown in the following table

When this paper experiments, it only considers matching the increment from the update, rather than the approximate gradient from the update. And in actual production, FL itself has the ability to resist gradient leak attacks, such as multiple local iterations and more complex gradient descent optimizers.