Federated Learning with Non-IID Data Paper Notes
Original translation reference: https://blog.csdn.net/GJ_007/article/details/104632718
The thesis has verified through experiments that in non-IID data, the model trained by FedAvg algorithm will reduce the accuracy.
It can be seen from the figure that the accuracy of the model trained by FedAvg algorithm in non-IID has decreased significantly, but it has little effect on the accuracy of IID data.
In order to explore the reasons, the original text uses two different degrees of distribution data: non-IID1 and non-IID2. It can be found from the figure that the accuracy rate of non-IID2 decreases less than that of non-IID1, so the reason is speculated because of the different data distribution.
The accuracy of the model trained by the FedAvg algorithm is affected by the skewness of the data distribution.
Research method: use the same initialization value to train the model and observe the difference in the obtained weights
Define weight differences
It can be seen from the above figure that the weight difference obtained by FedAvg and SGD in the IID data is not large, and the trend of the polyline is observed. The weight difference obtained after the next few rounds of updates is also not large. In the right figure, it can be observed that the difference in weights obtained by FedAvg and SGD is large, and according to the trend of the polyline, it can be seen that this difference will become larger and larger.
The original text uses the EMD method to calculate the difference between the data distribution
Derive the formula:
Therefore, the weight difference after the mth synchronization mainly comes from the difference between the m-1th and the actual total distribution
And proposed that only sharing 5% of the global data can improve the accuracy rate by 30%
The original article finally mentioned that global shared data is a different data set from the client, so it is not sensitive to privacy.
However, the original text does not mention where the cloud shared data comes from and why it is not sensitive to privacy.
Did not read
Welcome everyone to comment and put forward your own views haha
