How to solve "STALENESS" in ASGD? Staleness-aware Async-SGD for Distributed Deep Learning gives you an explanation of the paper intensive reading

Original paper: Staleness-aware Async-SGD for Distributed Deep Learning

ABS

Using asynchronous $When$$SGD$ , how to control hyperparameters is an important issue (such as how to adjust the learning rate according to the progress of training). The setting of hyperparameters is important for asynchronous$SGD$ has a very important influence.

This article proposes asynchronous $A\; variant\; of$$SGD$ that adjusts the learning rate based on the gradient delay. And the convergence of the algorithm is theoretically guaranteed.

1 INTRO

Commonly used distributed machine learning methods include $SSGD$ (Simultaneous Stochastic Gradient Descent) and$ASGD$$ASGD$ (Asynchronous Stochastic Gradient Descent).

For $For\; SSGD$ , in each round, all working nodes must wait for the slowest working node to complete the calculation before starting the next round of calculation, which greatly affects the training speed.

And for $For\; A,SGand\; D$ , there is no need to wait for all work nodes to complete the calculation of this round. Instead, no work node completes the calculation. After the work node is updated, the next round of calculation can be directly performed, which leads to new problems: The gradient is out of date, that is, the parameters being calculated by the node may lag behind the latest parameters. This effect is huge. When the number of training rounds is fixed,ASGD ASGDThe training effect of$A$$SG$$D$$SSGD$ is much worse.

The author discovered $The\; degree\; of\; convergence\; of\; ASGDwill$ be greatly affected by hyperparameters (such as learning rate and batch size) and the implementation of the distributed system (such as synchronization protocol, number of nodes), and there is currently a lack of information on setting hyperparameters$.$ to improveASGD ASGDA study on the effects of$A$$SG$$D.$

So the author of this article decided to propose an ASGD ASGD that can automatically adjust the learning rate.$ASGD$ is so good that$ASGD$ avoids the trouble of gradient expiration. In this method, gradient expiration is recorded, and the learning rate is divided by the degree of gradient expiration to obtain the new learning rate. And it is proved that the convergence rate of this method is consistent withSSGD SSGDSame as$SSG$$D.$

Some previous related work can achieve good results on distributed systems with a small number of nodes by reducing the learning rate at an exponential level. However, in large distributed systems, the learning rate will become a special value as the training progresses. Small (close to $0$ ), which will cause the entire system to converge too slowly.

2 System Architecture

This part will describe the design of the entire algorithm. The architecture of the two algorithms is given here:

• $n-softsyncprotocol$ : Able to sense part of the gradient expiration and prevent excessive expiration through soft synchronization;
• $hardsyncprotocol$ : that is,$SSGD$，Work$Baseline$ for comparison.

2.1 Architecture Overview

Description of some parameters:

• $\lambda$ : The number of working nodes;
• $\mu$ : batch size of each node;
• $\alpha$ : learning rate;
• $Epoch$ : iterative version;
• $Timestamp$ : Timestamp, the timestamp will increase every time the parameters are updated. Each gradient will record the current timestamp.
• $t_{i,l}$: working node llThe degree of gradient expiration of$l\; .$If a working node is in$i$ moment sent$The\; gradient\; at\; time\; j$ , which means that the currently sent gradient is expired, then$t_{i,l}=i-j$。

Each working node will perform the following operations in sequence:

• $getMinibatch$ : Get the batch used for calculation in this round.
• $pullWeights$ : Get the weights from the parameter server.
• $calcGradient$ : Calculate the gradient of the current batch based on the weights.
• $pushGradient$ : Send the calculated gradient to the parameter server.

The parameter server will perform the following operations:

• $sumGradient$ : Aggregate the received gradients.
• $applyUpdate$ : Update parameters with aggregated gradients.

2.2 Synchronization Protocols

$hardsyncprotocol$ : During each round of update, the parameter server will execute$sumGradient$ operation, and then$applyUpdateOnly$ when these two operations are completed, the working node can pass$pullWeights$ obtains the parameters of a new round, which means that the worker node must wait for the slowest worker node to complete the task. Although this method is slower, the training accuracy is very high.

$n-softsyncprotocl$ : In this method, the parameter server only needs to receive at least$c=[\frac{}{n}After\; ]$ gradient, you can perform this$Gradient\; aggregation\; operation\; of\; c$ nodes, and then the parameters are updated. At this time, the completed nodes can obtain new parameters and continue training (where$n$ is a hyperparameter if$n=\lambda This$ is similar to the previously proposed$ASGD$ is similar, the learning rate updated by the method in this article changes dynamically). The updated formula is as shown below:

The learning rate is calculated based on the degree of expiration.

2.3 Implementation Details

The method in this article still adopts some synchronization mechanisms, such as $pullWeights$ only when$The\; parameters\; can\; be\; obtained\; only\; after\; pushGradient\; is\; completed.\; This\; can\; ensurethat$ the parameters of the model are inconsistent due to concurrency.

In implementation, a computer may have multiple working nodes. Model parallelism is not used.

2.4 Staleness Analysis

For $hardsyncprotocolThe$ server must receive the gradient of all nodes before updating, so expiration will not occur, that is to say$t=0$。

For different values ​​nnThe degree of expiration that occurs in$n$$l=30$ different$n$ carried outτ \tauCalculation of$\tau$$Figure1$ is shown:

What puzzles me about the above results is that when $n=1$ , this means that the parameter server has only received at least30 30The parameters will not be updated until 30 gradients, which means that at this time it should be the same as$SSGD$$SSGD$ produces the same behavior, i.e.$t=0$ will be true, but the experimental results show that this is not the case, so I guess the 30 30received here$30$ gradients may have two gradients from the same node, but according to2.3 2.3Judging from the description in$2.3$$pullWeights$在$pushGradient$ can only be executed after it is completed. This seems to be a bit of a conflict$.$

Through experiments, the author found that for $n-softsyncprotoclwhere$ τ \tau$\tau$ is mostly equal to$n$ . With this intuitive understanding, we can control$n$ to control the degree of expiration that can be tolerated.

time $i$ , node$The\; learning\; rate\; of\; l$ is determined by the following formula:
$$\begin{gathered} a_{i,l}=\frac{a_0}{t_{i,l}}if{\tau }_{i,l}>0 \\ {a}_{i,l}=a_{0}if\tau {}_{i,l}=0 \end{gathered}$$

3 Theoretical Analysis

This part proves the convergence, but due to my limited ability, I don’t understand it very much ( mainly because I don’t want to read it ).

4 Experimental Results

4.1 Hardware and Benchmark Datasets

This section presents details of the equipment and datasets used. Can be viewed in the original text.

4.2 Runtime Evaluation

$Fig\; ure2$ shows the speed improvement brought by updating without receiving a gradient on two data sets under different numbers of nodes:

It can be seen that the graph shows that the method proposed in this article provides approximately linear acceleration (as the number of nodes increases).

4.3 Model Accuracy Evaluation

$Figure3$ demonstratesCIFAR 10 CIFAR10Training error and test accuracy results on$C$$I$$F$$A$$R$$10\; :\; (the\; two\; pictures\; above\; use\; fixed\; learning\; rates,\; while\; the\; two\; pictures\; below\; use\; dynamically\; updated\; learning\; rates)$

$Figure4$ shows that inImage N et ImageNetResults on$I$$m\; a$$g$$e$$N$$e$$t\; :$

As can be seen from the comparison of the pictures, when the learning rate is dynamically updated according to the degree of expiration, $n-softsync$ andSSGD SSGDThe fit of$SSG$$D\; is\; better.$

5 Conclusion

Contributions of this article:

• Proved $ASGD$ can be compared with$SGD$ has the same convergence speed;
• Quantified the extent to which gradients expire on some datasets. At the same time, a maximum of 30 30 was used in the experiment$With\; 30$ working nodes, it is considered an attempt at a relatively large distributed system.