The author shared the link: [ICLR 2023] Federated Learning for Spatiotemporal Heterogeneous Data_哔哩哔哩_bilibili
Link to the original paper: https://arxiv.org/abs/2205.10920
Introduction to the report: As a distributed learning paradigm for privacy protection, federated learning proposes to collaboratively train the neural network model on the device side in a way that the data does not move and the model moves. However, data with heterogeneous distribution on the end-to-end side poses severe challenges to federated learning. Although a series of recent studies have alleviated this problem to a certain extent, most of the work only focuses on the difference in data distribution in space, ignoring the spatiotemporal heterogeneous data that will appear with the federated learning process. This report will address the real challenges faced by federated learning and reasoning systems, and propose a theoretical framework for federated learning covering spatiotemporal heterogeneous data; explore the distribution shift of federated learning in deployment scenarios, and propose a robust adaptive data distribution shift Personalized federated learning algorithm.
Challenges faced by federated learning:
1. Communication overhead: limitations and unreliable networks
2. Data heterogeneity: the high degree of heterogeneity of non-independent and identically distributed data (the main focus of the article)
3. System heterogeneity: hardware, power, etc.
4. Privacy Leakage
Data heterogeneity in federated learning: prone to client drift
Due to the existence of heterogeneity, each client will optimize to its own best point, there will be a large number of gradients moving in the opposite direction, and resources will be wasted
Most previous work only considers federated learning on static heterogeneous data
Now: Identifies the flaws of existing work under test time distribution changes, and proposes Federated Test Time Head Ensemble Plus Tuning (FedTHE+), which makes FL models more individualized and robust to various test time distribution changes .
The comparative experiment part shows:
The traditional one will add some regularization terms
Aiming at solving: end-to-end heterogeneously distributed data
Traditional schemes only focus on spatial data distribution differences, ignoring spatiotemporal heterogeneous data in the federated learning process
Innovation:
- Propose a theoretical framework for federated learning covering spatiotemporal heterogeneous data
- Facing the problem of distribution deviation during deployment, an adaptive personalized algorithm for data distribution deviation is proposed
Traditional method: when performing gradient optimization, add a regularization term; this method requires more consideration for local deep learning
First of all: non-independent and identically distributed data can easily lead to poor generalization ability, biased classifier and inconsistent features
Second: client distribution problem
Finally, by adjusting for the difference in the distribution of heterogeneous clients (t-sne visualization), it is found that clients with non-IID have a smaller degree of cosine similarity to the classifier
Problem modeling:
Architecture for robustness evaluation:
