Deep understanding of federated learning - vertical federated learning

Category Catalog: "In-depth Understanding of Federated Learning" General Catalog

---

Assume that the data provider for federated learning is $A$ and$B$ , the third party is$C$ , then the steps of vertical federated learning are as follows:

1. Encrypted sample alignment is performed at the system level, and non-crossing users will not be exposed at the enterprise perception level.
2. Align samples for model encryption training:
   • CC by third party$C$ to$A$ and$B$ sends the public key to encrypt the data that needs to be transmitted;
   • $A$ and$B$ respectively calculates the intermediate results of features related to itself and encrypts the interaction.
   • $A$ and$B$ calculates their respective encrypted gradients and adds masks and sends them to$C$ , while owning the class standard side, calculate the encrypted loss and send it to$C$
   • $C$ decrypts the gradient and loss and passes it back to$A$ and$B$
   • $A$ and$B$ remove the mask and update the model

We take ridge regression as an example to illustrate the training process of longitudinal federated learning. Suppose there is a data set { xi A } ( i ∈ DA ) \{x_i^A\}(i\in D_A){ xiA​}(i∈DA​) 和数据集 { x i B , y i B } ( i ∈ D B ) \{x_i^B, y_i^B\}(i\in D_B) { xiB​,yiB​}(i∈DB​) , where$B$ is the data owner with the class label. We take linear regression as an example to illustrate the training process of vertical federated learning:

1. $A$ and$B$ initialize the model parameters${Th}_A$和${Th}_B$For example: $$\underset{{Th}_A,{Th}_B}{\min }\sum _i({Th}_A{x}_i^A+{Th}_B{x}_i^B-y_i{)}^2+\frac{}{2}(||{\Theta }_A|{|}^2+||{\Theta }_B|{|}^2)$$
2. 令$u_i^A={Th}_A{x}_i^A$sumui $u_i^B={Th}_B{x}_i^B$, then the original objective function is homomorphically encrypted ( $[[\cdot ]]$表示同态加密）可表示为：$$\begin{array}{rl}[[L]]& =[[\sum _i(u_i^A+u_i^B-y_i{)}^2+\frac{}{2}(||{\Theta }_A|{|}^2+||{\Theta }_B|{|}^2)]]\\ & =[[\sum _i(u_i^A{)}^2+\frac{}{2}||{\Theta }_A|{|}^2]]+[[\sum _i((u_i^B-y_i{)}^2)+\frac{}{2}||{\Theta }_B|{|}^2]]+2\sum _i([[u_i^A]](u_i^B-y_i))\end{array}$$
3. 我们令：$$\begin{array}{rl}[[L_A]]& =[[\sum _i(u_i^A{)}^2+\frac{}{2}||{\Theta }_A|{|}^2]]\\ [[L_B]]& =[[\sum _i((u_i^B-y_i{)}^2)+\frac{}{2}||{\Theta }_B|{|}^2]]\\ [[L_{AB}& =2\sum _i([[u_i^A]](u_i^B-y_i))]]\\ [[d_i]]& =[[u_i^A]]+[[u_i^B-y_i]]\end{array}$$则$[[L]]=[[L_A]]+[[L_B]]+[[L_{AB}]]$
4. 计算梯度：$$\begin{array}{rl}[[\frac{\partial L}{\partial {\Theta }_A}]]& =\sum _i[[d_i]]x_i^A+[[\lambda {Th}_A]]\\ [[\frac{\partial L}{\partial {\Theta }_B}]]& =\sum _i[[d_i]]x_i^B+[[\lambda {Th}_B]]\end{array}$$

The data provider is $A$ and$B$ and the third party are$C.$ The training steps of vertical federated learning are as follows:

\qquad\quad	The data provider is $A$	The data provider is $B$	The third party is $C$
Step $1$	Initialization parameter ${Th}_A$	Initialization parameter ${Th}_B$	Create an encryption key pair and send the public key to the data provider as $A$ and$B$
Step $2$	Calculation $[[u_I^A]]$ Sum$[[L_A]]$ and sent to data provider$B$	Calculation $[[u_I^B]]$、$[[d_i]]$ and$[[L]]$ and$[[d_i]]$ Sent to data provider$A$ , will$[[L]]$ Send to third party$C$
Step $3$	Initialization mask $R_A$，计算$[[\frac{\partial L}{\partial {\Theta }_A}]]+[[R_A]]$ and send it to the third party$C$	Initialization mask $R_B$，计算$[[\frac{\partial L}{\partial {\Theta }_B}]]+[[R_B]]$ and send it to the third party$C$	Decrypt $L$并发送$\frac{\partial L}{\partial {\Theta }_A}+R_A$To data provider $A$，发送$\frac{\partial L}{\partial {\Theta }_B}+R_A$To data provider $B$
Step $4$	Update parameters ${Th}_A$	Update parameters ${Th}_B$

During the whole process, the participants do not know the data and characteristics of the other party, and after the training, the participants only get the model parameters of their own side, that is, the half model. Since each participant can only obtain the model parameters related to itself, prediction requires collaboration between both parties:

1. Third party $C$ sends the user ID to be predicted to the data provider$A$ and$B$
2. Data provider $A$ and$B$ minute calculation$u^A$和$u^B$ and perform homomorphic encryption to obtain$[[u^A]]$Sum$[[u^B]]$
3. Data provider $A$ and$B$ homomorphically encrypted$[[u^A]]$Sum$[[u^B]]$sent to third party$C$
4. Third party $C$ calculates the predicted value after homomorphic encryption$[[y]]=[[u^A]]+[[u^B]]$and then decrypt to get the predicted value$y$

References:
[1] Yang Qiang, Liu Yang, Cheng Yong, Kang Yan, Chen Tianjian, Yu Han. Federated Learning [M]. Electronic Industry Press, 2020 [2] WeBank, FedAI. Federated Learning White Paper V2.0
. Tencent Research Institute, etc., 2021