How to solve the regional problems of distributed machine learning? DLion: Decentralized Distributed Deep Learning in Micro-Clouds Paper Intensive Reading

Original link: DLion (acm.org)

ABS

Due to the popularity of smartphones, edge devices will obtain more and more user data. However, it is unrealistic to collect all user data in the data center for distributed training. This mainly concerns user privacy.

At the same time, if the collected data is directly calculated locally on the user's computer, when the model is too large, the training will be insufficient due to the performance of the edge device.

Due to the above two reasons, the author hopes to propose a distributed architecture based on micro-cloud computing.

1 INTRO

The popularity of edge devices has generated a large amount of data. Building data centers in traditional ways faces two problems:

• Too much data needs to be moved and it is difficult to achieve;
• Regarding user privacy issues, users are not willing to disclose their data.

Federated learning was proposed to solve privacy issues. This method will be trained directly on the edge device. However, when the model is too large, even if the amount of data that the edge device needs to process is small, it will still take a long time. (A relatively large model may contain hundreds of megabytes of parameter size, which means that each sample requires hundreds of megabytes of memory, which is $8GB,6GB$ of memory is very difficult, and this does not take into account the computing power gap between mobile phones and computers)

The performance of micro clouds is far better than that of edge devices. At the same time, using micro clouds can only upload data in local areas, which also protects user privacy to a certain extent.

$Figure1is\; a\; basic\; architecturefor$ distributed computing using micro clouds$:$

There are two major challenges in using micro clouds for computing:

• Performance differences and dynamic variability of micro-clouds: The performance of micro-clouds in different regions may vary greatly. At the same time, because devices in a micro-cloud may be arranged to handle other services, the performance of the same micro-cloud also changes dynamically over time. of;
• Differences and dynamics of network resources: $LAN$ to communicate, while different micro clouds need to go through$WAN$ ,$WAN$ resources are very limited. At the same time, LAN resources will be diluted as the number of nodes in the micro cloud increases, that is, the resources will change dynamically.

Most distributed systems do not take into account the differences and dynamics of resources, which will lead to an increase in training time.

DL ion DLion proposed in this article$DLionis\; mainlydesigned$ to solve the computing problems of distributed systems with remote and dynamic characteristics$.$

$DLion$ uses three key technologies to solve the$above$$problems$$:$

• $Weighteddynamicbatching$ : This technology is used to solve the problem of computing performance heterogeneity.
• $Per-linkprioritizedgradientexchange$ : This technology is used to solve the problem of network remoteness.
• $Directknowledgetransfer$ : This technique is used to improve the accuracy of the model.

The main contributions of this article:

• A distributed system that can solve the problem of remoteness and dynamics is proposed;
• The designed system is concise and flexible, easy to modify, and can be applied to different distributed training;
• Environmental $TensorFlow$ established a prototype of the system while supporting$CPU$ GPUGPU$\_$$GPUversion$ .

2 Background and Motivation

2.1 Distributed Deep Learning

This section introduces machine learning using gradient descent and distributed machine learning using gradient descent.

Machine learning: Calculate gradients to update model parameters;

Distributed machine learning: Each working node calculates its own gradient, and updates the parameters after summarizing.

2.2 Distributed Deep Learning Systems

A framework for implementing distributed machine learning with parameter servers, such as $TensorFlow$和$MXNet$ Both provide$The\; implementation\; of\; PS(Para\; meterServer)$ .

A machine learning framework that implements distributed computing without parameter servers: $Iam,\_Hop,Pragueetal$ .$\_$$\_$

$Figure2$ shows the difference between the two,$\left(a\right)$ is a server with parameters,$\left(b\right)$ is a parameterless server (decentralized)

These frameworks simplify the process of distributed machine learning.

2.3 DL Learning in Micro-Clounds

The overall structure is as $As\; shown\; inFigure3\; ,\; the\; interiorof\; the\; micro\; cloud\; is\; connected\; throughthe$ LAN, and the clouds are connected through the WAN.

2.4 Challenges and Motivation

What is introduced here is $Two\; challenges\; were\; mentioned\; in\; Section\; 1and$ will$not$$be$$repeated$$here$$.$

3 Our Approach: DLion

3.1 Design Goals and Overview

The framework is designed without a central parameter server.

Design goals:

• Maximize data parallelism. Maximizing data parallelism can reduce the time required for model training, and at the same time, it is necessary to minimize the accuracy loss of training.
• Reduce communication time: While reducing communication time between nodes, it is necessary to reduce the accuracy loss of the model as much as possible.
• Improve model accuracy: Reduce negative impacts by sharing data between nodes.

The above three goals correspond in turn to the three key technical points proposed earlier. $Fig\; ure4$ demonstrates the use time of these three key technical points and the entire workflow:

3.2 Weighted Dynamic Batching

First introduce two concepts:

• $LBS$：$Lo\; calBatchSizeThe$ local batch size of each worker node participating in the calculation.
• $GBS$ : The global batch size of the entire distributed system for one round of training, which is the LBS LBSof all nodes.$LBS$ sum.

It should be noted that in the traditional distributed system architecture, the LBS LBS owned by each working node$LBS$ is the same.

This can usually be achieved by adding $LBS$ and the number of nodes to increase$GBS$ , and here we only discuss changing$LBS$ to change$In\; the\; case\; of\; GBS$ , the number of working nodes is fixed to$n$。

Add $GBS$ has advantages and disadvantages:

• Benefits: Equivalent to the increased amount of calculations, the time required for training will be reduced;
• Disadvantages: Increase $GBS$ usually leads to a decrease in final training accuracy.

So increase $GBS$ needs to find a suitable value that can reduce training time as much as possible without losing too much accuracy.

A previous article pointed out that the final convergence accuracy of the model does not necessarily need to be improved by controlling the learning rate, but can also be improved by regularly changing $GBS$ is implemented. This article was inspired by this, so we designed a controller that can automatically adjust, which can reduce the calculation time while almost not losing the accuracy of the model.

When the computing performance of each working node is the same and unchanged, $LBS$ is set to$\frac{GBS}{n}$is reasonable. However, the actual situation is that the performance of each working node is not the same, and the performance of each left node may fluctuate. If $If\; LBS$ is set to equal, then the node that completes first must wait for the node that completes last, which will cause the time required for model calculation to increase.

$The\; weighteddynamicbatching$ method has three components:

• $GBSContr\; oller$ : Automatically control$Increase\; or\; decrease\; in\; GBS$ . The design inspiration for the modified component comes from two findings: increasing GBS GBSat the beginning of training$GBS$ will cause a serious decrease in accuracy; increase$The\; accuracy\; drop\; caused\; by\; GBS$ is relatively low and stable (see$Figure5$ , where the horizontal axis represents the$epochincrease$ GBS$GBS$ , the vertical axis represents the final model accuracy). These two findings prompted the authors to divide the workflow of this component into two stages:$warm-up$和$speed-up$ . In the first stage:$GB{S}_{t+1}=GB{S}_t+C_{warmup}$, when it increases to $The\; increase\; will\; stop\; at\; 1\%$ (to prevent excessive increase from causing a decrease in accuracy). In the second stage:$GB{S}_{t+1}=GB{S}_t\times C_{speedup}$, when it increases to $Stops\; growing\; at\; 10\%$ (existing research finds that it should not be$GBS$ is too large), parameter$C$ needs to be set.

• $LBSController$ : Confirm$After\; GBS$ , this component will determine the LBSof each working node.$LBS$。$The\; design\; concept\; of$$LBS$ is very simple, LBS LBSof nodes with strong computing power$LBS$ will be larger, and the amount of data will be allocated based on the proportion of computing power. (This can ensure that the calculation time of all nodes is close),$Figure6$ showsthe LBS LBSAdjustment of$L$$BS\; .$LBS LBSThe specific calculation formula of$L$$BS$$RCP$代表$relativecomputationpower$ (relative computing power):
  $$LB{S}_i=GBS\frac{RC{P}_i}{{\sum }_{j=1}^{n}RC{P}_j}$$

• $weightedmodelupdatemodule$ : This module is responsible for the aggregation of parameters, nodejjAfter$j$$g_t^j$, node $k$ receives node$The\; aggregation\; process\; after\; j$ will be updated as follows:
  $$\begin{gathered} w_{t+1}^k=w_t^k-the\frac{1}{n}\sum _{j=1}^{n}d{b}_j^k{g}_t^j \\ d{b}_j^k=\frac{LB{S}_j}{LB{S}_k} \end{gathered}$$

3.3 Per-Link Prioritized Gradient Exchange

$Dataqualityassurancemodule$ : This module is responsible for selecting important gradients for update, using$MaxN$ algorithm This algorithm will select$N\%$ of the gradient is updated.

$Figure7$ shows different$The\; impact\; of\; N$ on the accuracy of the model.

$Transmissionspeedassurancemodule$ : This module is used to automatically determineM ax N Max\ N NNin$M$$a$$x$$N\; algorithm$$N.$ _ The determination method is to make it possible to select a larger NNwithout the network environment becoming a bottleneck.$N$ , in this module, node$i$ -directed node$The\; gradient\; size\; that\; j$ can send is evaluated by the following formula:
$$\begin{gathered} \frac{BW\_ne{t}_j}{Iter\_co{m}_i} \\ BW\_ne{t}_{j}Represents\; the\; direct\; available\; bandwidth\; between\; two\; nodes \\ Iter\_co{m}_{i}Represents\; the\; number\; of\; iterations\; that\; node\; ican\; perform \end{gathered}$$$$per\; unit\; time.$$
The core of the above formula is to make the calculation time and communication time equal.

$Figure8$ is node$1$ to node$3$ and node$5.$ The gradient size sent is automatically adjusted with the number of iterations:

3.4 Direct Knowledge Transfer

Since the method introduced above does not use a parameter server and uses an asynchronous method, the parameters owned by each node participating in the training may be different.

The method of this part is to periodically exchange parameters: select the node with the best training effect, and other nodes obtain parameters from this node.

Several problems need to be solved when adopting this method:

• When does the training take place;
• Does it need to be sent to all nodes (this is a relatively large overhead);
• How to aggregate the received parameters, directly replace or average?

In order to solve the above three questions, the author did some experiments.

$Fig\; ure9$ shows the experimental results:

• (a) represents $The\; time\; required\; for\; DKT$ training to complete (reach the same accuracy),$earlyDKT$ stands for early training,$lateDKT$ stands for DKT in the later stages of training, and the author found that$DKT$ can achieve better accuracy. And the above$DKT100iter$ stands for every$It\; is\; performed\; once\; every\; 100$ rounds. It can be seen that if it is performed more frequently, a large amount of network resources will be required, so the training time will increase. If the interval is too long, the convergence speed will decrease because there is no good parameter interaction, resulting in Training takes more time.
• (b) represents the result of sending to different nodes: not used, sending to the node with the worst effect each time, and sending to all nodes each time. It can be seen that only sending to the worst node can achieve a great improvement in accuracy. But for better accuracy, the later implementation still chooses to send it to all nodes.
• Default values: $w_{local}=(1-l)w_{local}+\lambda w_{best}$, with $As\; \lambda$ increases, the best weight proportion increases, which can be seen at$l=$The best effect is when$0.75\; .$

4 Implementation

Here we talk about $How\; DLionis$ implemented? If you are interested, you can read the original$text$$.$

4.1 Key Components and Operations

Some key components are introduced here, which have been introduced previously and will not be repeated here.

4.2 Generic and Flexible DLion

This section explains $DLioncanbe$ easily applied in different$algorithms$$.$

5 Evaluation

5.1 Methodology

5.1.1 Applications and Datasets

slightly.

5.1.2 Experimental Platforms

slightly.

$Table2is$ a test of the bandwidth between different Amazon$regions$$:$

5.1.3 Performance Metrics

• The accuracy a model can achieve at a given time;
• The time required to achieve a certain accuracy;
• The time required for convergence;

5.1.4 Comparison Systems

Compare with some previous methods:

• $Baseline$ does not exchange all gradients, and the initial asynchronous$SGD$ (This is presumed to be asynchronous$SGD$ instead of synchronization is based on the results of the last experiment);
• $Ako$ : You can view the intensive reading of previously published papers -Ako intensive reading of papers
• $Gaia$；
• $Hop$；

5.1.5 Experimental Setup

$Table3$ is the details of each environment. The numbers in the calculation represent the number of cores of each node, and the numbers in the network represent the bandwidth of$each$$node$$:$

No emulation No\ emulation in the picture$Noemulation$ should mean that each node has the same computing resources$.$

5.2 Evaluation Results

5.2.1 System Heterogeneity

This section evaluates the situation on a system composed of nodes with different network resources and computing resources.

$Fig\; ure11$ shows the accuracy that each fixed-time method can achieve on various configurations:

$HeteroSYSA$ is a node with strong computing power and strong network resources, and$He$$HeteroSYSB$ is a node with strong computing resources and weak network resources$.$

5.2.2 System Robustness in Heterogeneous GPU cluster

The model used in this part is larger and the amount of data generated that needs to be exchanged is larger (because in $training\; on$$GPU$ , so a larger model was chosen), that is, the network became an obvious bottleneck.

$Figure12$ is also the accuracy achieved after fixed training time:

5.2.3 Heterogeneous Compute Resources

This section evaluates the situation where the network resources are the same but the computing resources are different.

$Fig\; ure13$ is the accuracy that can be achieved by each method at fixed time in the environment where the network resources of each node are the same:

$FIgure14$ is the time required to achieve a certain accuracy, where$DLion-no-DBWU$ represents not using dynamic batch selection,$DLion-no-WU$ represents no weighted aggregation update:

5.2.4 Heterogeneous Network Resources

This section evaluates the situation where the network resources are different but the computing resources are the same.

$Fig\; ure15$ shows the accuracy that can be achieved with fixed time on different systems:

$Figure16$ is$Max-10$（即$BaselineincreaseM$ ax − 10 Max-10$Max-10$ ) Comparison with other methods (fixed time):

5.2.5 Deviation of Model Accuracy

$Fig\; ure17$ shows the standard deviation of the model accuracy:

5.2.6 Dynamic Resource Changes

The computing resources and network resources of this part of the environment will change dynamically.

$Figure18$ is the experimental result:

$Figure19$ shows the change process of the local batch size of each node during the training process:

$Fig\; ure20$ shows the change process of selecting the updated gradient size during the training process:

5.2.7 Effect on Improving Model Accuracy

This section focuses on the final convergence accuracy and convergence time of the model.

$Figure21$ shows the final convergence accuracy and convergence time of each method:

6 Related Work

The related work involved is as follows:

• Distributed deep learning;
• Resource-aware distributed deep learning;
• federated learning;
• Deep learning inference for edge devices;
• parallel computing;
• Data analysis based on geographical distribution.

7 Conclusion

A distributed machine learning architecture suitable for micro-clouds is proposed. It mainly improves speed through three aspects:

• $Weightedynamicbatching$ : Maximize data parallelism$;$
• $Per-linkprioritizedgradientexchange$ : Reduce communication costs;
• $Directknowledgetransfer$ : Improve model accuracy.

Also using $TensorFlow$ built$Dlion\text{'}sprototype$ .$\_$Compared with the current architecture, it has a good speed improvement and accuracy improvement.