Decentralized federated learning thought

Decentralized federated learning is a decentralized machine learning method that protects user privacy. Compared with centralized federated learning, decentralized federated learning pays more attention to protecting user data privacy, and is also more scalable and robust.

In decentralized federated learning, each device uses local data for model training and sends model updates to surrounding neighbor devices. These neighboring devices can receive updates and conduct model training based on their own local data. This process is repeated multiple times until the models for all devices reach a state of convergence.

Compared with other federated learning methods, decentralized federated learning does not require a centralized server to coordinate communication between devices, thus greatly reducing communication overhead and single point of failure risk.

Algorithm formula
Assume that there are $n$ devices$D_1,D_2,...,D_n$Jointly participate in federated learning tasks. Our goal is to learn a global model $\theta$ , such that each device$i$ are able to use local data for model inference.

In decentralized federated learning, first each device initializes a local model $i_i$. Each iteration consists of the following steps:

Choose a random subset $S_t\subseteq 1,2,...,n$ as a communication device group.
For each device$i\in S_t$Calculate the gradient ∇ θ i J ( θ i ) \nabla_{\theta_i} J(\theta_i) using local data${\nabla }_{i_i}J({\theta }_i)$ , among which$J\left({\theta }_i\right)$ is the local model$i_i$loss function.
For each device $i\in S_t$,if ${\nabla }_{i_i}J\left({\theta }_i\right)$ to all neighboring devices$j\in N_i$, where $N_i$Indicates the device iiThe set of neighboring devices of$i\; .$
for each device$i$ , updated mainland model$i_i$,impose $i_i^{t+1}=i_i^t-the\cdot \frac{1}{|S_t|}{\sum }_{j\in S_t}{\nabla }_{i_j}J({\theta }_j)$ , among which$\eta$ is the learning rate.
Up to the models of all devices$i_1^{t+1},i_2^{t+1},...,i_n^{t+1}$After all updates are completed, enter the next round of iteration.
In decentralized federated learning, each device only communicates with its neighbors, so the communication overhead is small. Furthermore, by randomly selecting groups of devices for communication in each iteration, the randomness and stability of the learning process can be increased. However, since the model update of each device is only based on the information of its neighbor devices, there may be a problem of model divergence or failure to converge.

The main advantages of decentralized federated learning algorithms are better protection of user privacy and are generally more scalable and robust than centralized federated learning.

Here are some references about decentralized federated learning:

《Communication-Efficient Learning of Deep Networks from Decentralized Data》

《Federated Learning: Strategies for Improving Communication Efficiency》

《Towards Federated Learning at Scale: System Design》

《Decentralized Federated Learning: A Segmented Gossip Approach》

《A Comprehensive Survey on Federated Learning》

《Federated Learning with Non-IID Data》

The above papers have carried out more detailed research and introduction on decentralized federated learning, which can be used as a reference for further understanding of this field.