FLMix: A new paradigm of federated learning - the combination of local and global

文章链接：Federated Learning of a Mixture of Global and Local Models

Journal published (conference): ICLR 2021 Conference (top machine learning conference)

This blog studies traditional federated learning from the perspective of ◊ \Diamond 优化函数$◊$ and new federated learning $♣$ difference

1. Background introduction

In an article published by King Abdullah University of Science and Technology, two scholars, Philip and Peter, conducted an early mathematical discussion on the combination of Federated Learning and Mixed Experts (MoE).

What is interesting is that the research motivation of these two scholars is to protect their mobile device data from being exposed, and at the same time, they can also use this data for machine learning. They gave two very simple reasons.

• First, many device users are increasingly sensitive to privacy concerns and prefer their data to never leave their devices.
• Second, moving data from their place of origin to a centralized location is very inefficient in terms of energy and time.
  
  One reason is that it is unsafe, and another reason is that it is inconvenient.

2. Traditional federated learning

To date,FL has emerged as an interdisciplinary field focused on training machine learning models directly on edge devices< a i=3>to solve the problem. In traditional FL framework, each client participates in FL training.

Parameter definition: Number of training customers N; global model structure $M_G$; Global model parameters $\theta (d_1)维$
其中 $i\in {\mathbb{R}}^{d_1}$ and $R^{d_1}\in \mathbb{R}$
FL的学习目标为：
$$◊\underset{\theta \in {\mathbb{R}}^{d_1}}{\min }F(\theta )=\frac{1}{N}\sum _{i=1}^N{f}_i\left(\theta \right)$$
对于每一个$f_i$, due to different data distribution, assume that $The\; data\; distribution\; of\; i$ customers is defined as ${\mathcal{D}}_i$ 则：
$$f_i(\theta )={AND}_{(x,y)\sim {\mathcal{D}}_i}[f(x,\xi )]$$
inside$f_i(\cdot )$is a guest $i$ target lapse function

Solve $F\left(\theta \right)$ The most popular method is FedAvgalgorithm, in its FedAvgsimplest form, that is, when no partial participation, model compression, or stochastic approximation is used, FedAvg Reduced to local gradient descent (LGD). This is an extension of GD that performs multiple gradient steps per device before aggregating.

FedAvg has been proven to be empirically effective, especially for non-convex problems (problems with multiple local minima) . But when the data is heterogeneous, FedAvg's convergence guarantee is poor compared to its non-local counterpart.

Although FL has many theories proving its feasibility, its final result is global. We need to think about, for those individuals with heterogeneous data, Is it necessarily good to use global solutions to solve individual problems?

The answer is no, data heterogeneity is not only solved by designing new training methods $◊$ raises challenges and inevitably questions the utility of such global solutions for individual users. In fact, a global model trained on all data from all devices may be removed from the typical data and usage patterns experienced by individual users to the point of rendering them virtually useless.

---

3. FL new paradigm

This article proposes a new optimization formula for training federated learning models. The standardFLaims to find a single global model from the private data stored on all participating devices. In contrast, the new approachseeks a trade-off between global and local models where each device can learn from its own private data without communication .

This paper develops efficient stochastic gradient descent (SGD) variants to solve the new formulation and demonstrates guarantees on communication complexity. The main contributions of this work includea new paradigm that combines global and local modelsFL, Theoretical properties of the new paradigm, Loop-free local gradient descent (L2GD), Convergence theory of L2GDandInsights on the role of local steps in federated learning. The paper also highlights the potential of local SGD to outperform traditional SGD in terms of communication complexity and the benefits of personalized federated learning.

The new paradigm of training supervised federated learning proposed in this article is as follows:

♣ min ⁡ x 1 , . . . , x n ∈ R d { F ( x ) : = f ( x ) + λ ψ ( x ) } f ( x ) : = 1 n ∑ i = 1 n f i ( x i ) ψ ( x ) : = 1 2 n ∑ i = 1 n ∥ x i − x ‾ ∥ 2 \clubsuit \quad \min_{x_1,...,x_n \in \mathbb{R}^d } \{ F(x): = f(x)+ \lambda \psi (x)\} \\ f(x):=\frac{1}{n}\sum_{i=1}^{n} f_i(x_i) \\ \psi (x) := \frac{1}{2n}\sum_{i=1}^{n} \left \| x_i-\overline{x} \right \| ^2 ♣x1​,...,xn​∈Rdmin​{ F(x):=f(x)+λψ(x)}f(x):=n1​i=1∑n​fi​(xi​)ψ(x):=2n1​i=1∑n​∥xi​−x∥2 inside $l\ge 0$ is a penalty hyperparameter, $x_1,...,x_n\in {\mathbb{R}}^d$ is the local model parameter, $x:=(x_1,x_2,...,x_n)\in R^{nd}$ 并且 $\overline{x}:=\frac{1}{n}{\sum }_{i=1}^n{x}_i$is the average of all local models.

Text introduction $f_i$ obtainable $F$ is a strongly convex function. A convex function is a function whose second derivative is always positive (negative), and the local minimum is the global minimum. For $◊$ has a unique solution. This solution can be expressed as:
$$x(\lambda ):=(x_1(\lambda ),...,x_n(\lambda ))\in R^{nd}$$Adhesion possible $\overline{x}(\lambda ):=\frac{1}{n}{\sum }_{i=1}^n{x}_i\left(\lambda \right)$

---

theoretical logic

The proposed paradigm $♣$'s theory:

• Local models ($l=0$): At this time, the model degenerates into a local model, and only needs to minimize the local loss, that is, solve $$\underset{x_i\in R^d}{\min }{f}_i\left(x_i\right)$$也到说，$x_i(0)$ 仅基于存储在设备 $i$ 上的数据 $D_i$ 的局部模型。该模型可以由设备 $i$ 计算，而无需任何通信。通常情况下，$D_i$ 不够丰富，无法使用此本地模型。为了学习更好的模型，还必须考虑其他客户的数据。然而，这存在沟通成本。
• Mixed models ($\lambda \in (0,\infty )$)：随着 $\lambda$ 的增加，惩罚 $\lambda \psi (x)$ 的效果越来越明显，需要沟通以确保模型不会太不相似，否则惩罚 $\lambda \psi (x)$ 会增大。
• Global model ($\lambda =\infty$)：现在我们来看 $\lambda \to \infty$ 的极限情况。直观上，这种极限情况应该迫使最优局部模型之间是相同的，同时最小化损失 $f$，即让 $\psi (x)\to 0$ 。$$\psi (x):=\frac{1}{2n}\sum _{i=1}^n{\Vert x_i-\overline{x}\Vert }^2$$此时，这种情况有一个特殊的极限解： min ⁡ { f ( x ) : x 1 , . . . , x n ∈ R d , x 1 = ⋯ = x n } \min\{ f(x):x_1,...,x_n\in \mathbb{R}^d ,x_1=\cdots=x_n \} min{ f(x):x1​,...,xn​∈Rd,x1​=⋯=xn​}。可以反证，如果 $\lambda =\infty$ 并且 $x_1=x_2=\cdots =x_n$不成立，那么 $F(x)=\infty$

---

重要假设

对于每一个设备 $i$ ,它的目标函数 $f_i:{\mathbb{R}}^d\to \mathbb{R}$ 是 $L-smooth$ 并且 $\mu -strongly$ 的凸函数。

• $L-smooth$：通常用来描述一个函数的平滑程度。一个函数被称为是 L-smooth 的，如果它的一阶导数（梯度）是 Lipschitz 连续的，即梯度的变化受到了一定的约束。
  如果存在一个常数 $L>0$，使得函数 $f$ 的梯度 $\nabla f(x)$ 对于任意的 $x$ 和 $y$ 满足以下不等式：$$\Vert \nabla f(x)-\nabla f(y)\Vert \le L\Vert x-y\Vert$$$\Vert \cdot \Vert$ is the norm of the vector. This definition shows that the gradient change of the function is affected by $The\; limit\; of\; L$ means that the gradient change at adjacent points on the function surface is bounded.
• $m-strongly$: Describes the degree of curvature of a function, referring to the extent to which a function is more strongly curved than a convex function . If there is a constant $m>0$ , which satisfies the following inequality: $$f(y)\ge f(x)+⟨\nabla f(x),and-x⟩+\frac{\mu }{2}\Vert y-x{\Vert }^2$$ $⟨\cdot ,\cdot$ represents the inner product operation. This inequality shows that the function $f$ Any point $The\; curvature\; at\; x$ is at least $\mu$, that is, the function image is sufficiently curved in the local area.

$L-smooth$ The characteristics of function make the solution in optimization problems more feasible and stable. Because functions with Lipschitz continuous gradients are more likely to converge to the local optimal solution for optimization algorithms such as gradient descent, avoiding oscillation or divergence caused by dramatic gradient changes. Ensure convergence

$m-strongly$ The function has a strict lower bound in the local area. This feature allows the optimization algorithm to converge to the global maximum more quickly. Excellent solution. Accelerate convergence

---

Solution properties

对于 $The\; optimal\; solution\; of\; ♣$ should have the following three characteristics:

We will characterize two functions, local and global $f(x(\lambda ))$ 和 $\psi \left(x\right(\lambda ))$ 视工关于变数 $\lambda$ function.

• 特性一： $\psi \left(x\right(\lambda ))$ Please take advantage of it. $\forall \lambda >0$ 有$$\psi (x(\lambda ))\le \frac{f(x(\infty ))-f(x(0))}{l}$$One step $f\left(x\right(\lambda ))$ Definitely, that's why $f\left(x\right(\infty ))\ge f\left(x\right(\lambda ))$.
  
  Above expression child assertion: Adjudication $\lambda$ 目增大，惩罚项 $\psi \left(x\right(\lambda ))$ The best local model is 0 , this is the best local model $x_i\left(\lambda \right)$ $The\; growth\; of\; \lambda$ is increasingly similar. At the same time, according to the second expression, $f(x(\lambda ))$随 $\lambda$ increases, but does not exceed the optimal global loss of the standardFL formula$f(x(\infty ))$ 。
• 特性二：对于 $\forall \lambda >0$ and $1\le i\le n$ We can get the following optimal local solution representation:$$x_i(\lambda )=\bar{x}(\lambda )-\frac{1}{l}\nabla f_i(x_{i}i\; =\; 1\; n\; \nabla \; f\; i\; (\; x\; i\; (\; \lambda \; )\; )\; =\; 0\; \psi \; (\; x\; (\; \lambda \; )\; )\; =\; 1\; 2\; \lambda \; 2\; |\; |\; \nabla \; f\; (\; x\; (\; \lambda \; )\; )\; |\; |\; 2\; \backslash sum\_\{i=1\}^\{n\}\backslash \; nabla\; f\_i(x\_i(\backslash lambda))=0\; \backslash \backslash \; \backslash psi\; (x(\backslash lambda))=\backslash frac\{1\}\{2\backslash lambda^2\}||\backslash nabla\; f(x(\backslash lambda))\; ||^\; 2$$$$\begin{gathered} \sum _{i=1}^n\nabla f_i(x_i(\lambda ))=0 \\ \psi (x(\lambda ))=\frac{1}{2{\lambda }^2}||\nabla f(x(\lambda ))|{|}^2 \end{gathered}$$Subtract the multiple of the local gradient from the average model to obtain the optimal local model. In the optimal state, the sum of local gradients is always zero. This pair $l=\infty$ is obviously correct, but this is true for $\forall \lambda >0$ Akiaki Tsufuta.
• 特性三: From the best local model $O(1/\lambda )$ P ( z ) : = 1 n ∑ i = 1 n f i ( z ) P(z):=\frac{1}{n} { \textstyle \sum_{i=1}^{n}}f_i(z) Order FLsolution. 's speed is
  $P(z):=\frac{1}{n}{\sum }_{i=1}^n{f}_i\left(z\right)$ , at this time $x(\infty )$ 是 $The\; unique\; minimum\; value\; of\; P$ can be obtained: $$||\nabla P(\bar{x}(\lambda ))|{|}^2\le \frac{2L^2}{l}(f(x(\infty ))-f(x(0)))$$

$♣$ solution $x(\lambda )$ 到纯局部解 $x\left(0\right)$ Japanese chiropractic solution < /span>$x(\infty )$ 的距离是 $\lambda$ function.

---