In-depth understanding of federated learning - classification of federated learning

Category: General Catalog of "In-depth Understanding of Federated Learning"

In practice, island data has different distribution characteristics. According to these characteristics, we can propose corresponding federated learning schemes. Below, we will classify federated learning based on the distribution characteristics of island data.

$D_i$ held by each data owner $D_{i}$ It can be represented by a matrix. Each row of the matrix represents a user, and each column represents a user characteristic. At the same time, some datasets may also contain labeled data. If you want to build a predictive model of user behavior, you must have labeled data. We can call the user feature $X$ , call the label feature $Y.$ _ that needs to be predicted $Y$ ; in the field of marketing, the label is the user's purchase desire $Y$ ; in the field of education, it is the degree to which students have mastered knowledge, etc. User characteristic $X$ tagged $Y$ constitutes the complete training data $(X, Y)$ . However, in reality, such a situation is often encountered: the users of each data set are not completely the same, or the characteristics of the users are not completely the same. Specifically, taking federated learning with two data owners as an example, the data distribution can be divided into the following three situations:

User features of two datasets $(X_1, X_2, \cdots)$ has a large overlap, while users $(U_1, U_2, \cdots)$ with a smaller overlap
Users of two datasets $(U_1, U_2, \cdots)$ has a large overlap, while user features $(X_1, X_2, \cdots)$ with a smaller overlap
Users of two datasets $(U_1, U_2, \cdots)$ overlaps with user features $(X_1, X_2, \cdots)$ parts are relatively small.

In order to deal with the above three data distribution situations, we divide federated learning into horizontal federated learning, vertical federated learning and federated transfer learning, as shown in the following figure:
Taxonomy of Federated Learning

Horizontal Federated Learning

In the case that the user characteristics of the two data sets overlap more but the user overlap is less, we divide the data set according to the horizontal direction (that is, the user dimension), and take out the part of the data that has the same user characteristics but not exactly the same users. train. This approach is called horizontal federated learning. For example, there are two banks in different regions. Their user groups come from their respective regions, and their mutual intersection is very small. However, their businesses are similar, so the recorded user characteristics are the same. At this point, you can use horizontal federated learning to build a joint model. In 2017, Googie proposed a data joint modeling scheme for Android phone model update: when a single user uses an Android phone, the model parameters are continuously updated locally and uploaded to the Android cloud, so that each data with the same feature dimension A federated learning scheme in which the owner builds a joint model.

The steps of horizontal federated learning are as follows:

Participants each download the latest model from the server
Each participant uses local data to train the model, and the encrypted gradient is uploaded to the server, and the server aggregates the gradient of each participant to update the model parameters
The server returns the updated model to each participant
Each participant updates its own model
Repeat steps 1~4 until the model converges or meets expectations

In traditional machine learning modeling, the data required for model training is usually collected in a data center, and then the model is trained to make predictions. In horizontal federated learning, it can be regarded as sample-based distributed model training, distributing all data to different machines, each machine downloads the model from the server, then uses the local data to train the model, and then returns the parameters that need to be updated to the server. The server aggregates the returned parameters on each machine, updates the model, and feeds back the latest model to each machine. In this process, each machine has the same and complete model, and the machines do not communicate or depend on each other. When predicting, each machine can also predict independently. This process can be regarded as a sample-based distributed model training. . Google initially adopted a horizontal federation approach to solve the problem of Android phone end users updating models locally.

Vertical Federated Learning

In the case that the users of the two data sets overlap more but the user features overlap less, we split the data set vertically (that is, the feature dimension), and take out the part of the data that has the same users but not completely the same user features. train. This approach is called vertical federated learning. For example, there are two different institutions, one is a bank in a certain place, and the other is an e-commerce company in the same place. Their user groups are likely to contain most of the residents in the area, so the intersection of users is relatively large. However, since banks record users' income and expenditure behaviors and credit ratings, while e-commerce companies keep users' browsing and purchase histories, their user characteristics overlap less. Vertical federated learning is a federated learning that aggregates these different features in an encrypted state to enhance model capabilities. At present, many machine learning models such as logistic regression model, tree structure model and neural network model have gradually been confirmed to be able to build on this federation system.

Federated Transfer Learning

In the case where there is little overlap between users and user features in the two datasets, instead of splitting the data, we can use transfer learning to overcome the lack of data or labels. This method is called federated transfer learning. For example, there are two different institutions, one is a bank in China, and the other is an e-commerce company in the United States. Due to geographical restrictions, the intersection of the user groups of the two institutions is very small. At the same time, due to the different types of trifoliate plants, only a small part of the data characteristics of the two overlap. In this case, in order to carry out effective federated learning, transfer learning must be introduced to solve the problem of small unilateral data and few labeled samples, so as to improve the effect of the model.

References:
[1] Yang Qiang, Liu Yang, Cheng Yong, Kang Yan, Chen Tianjian, Yu Han. Federated Learning [M]. Electronic Industry Press, 2020 [2] WeBank, FedAI.
Federated Learning White Paper V2.0. Tencent Research Institute, etc., 2021