Text｜Qi Wenbo (flower name: Bai Mo)
Dragonfly Maintainer
Ant Group Software Engineer
Mainly responsible for "P2P-based file and image acceleration system".
This article reads 5005 words for 13 minutes
The latest official version of Dragonfly v2.0.9 has been released! Thanks to the contributors of Dragonfly, and also to the various public cloud teams who silently supported the Dragonfly project. Welcome to visit the d7y.io [1] website to learn more details. The following is a detailed introduction to the updates brought by the v2.0.9 version.
Function
1. Download tasks can be downloaded according to Priority. The priority can be passed as a parameter in the Cli when downloading the task, or the priority associated with the corresponding application can be set in the Manager Console. For specific functions with different priorities, please refer to the document priority protoc definition [2] .
2. The Scheduler configuration adds a PieceDownloadTimeout field, which is used to set the corresponding Task status to fail when the download of a certain Piece exceeds the Timeout. Prevent abnormal Task metadata from remaining in the Scheduler.
3. The gRPC service adds Health Service and Reflection service.
4. Manager supports Redis sentinel model .
5. Refactor the Dynconfig module to remove the json.Unmarshal operation and improve the runtime efficiency of the Dynconfig module. It also supports filtering abnormal address services through health checks.
6. Fix the exception caused by gRPC not building a hash ring when no service is currently available.
7. In earlier versions, when multiple Pieces of the same Task are downloaded concurrently, the Scheduler will schedule multiple Parents to download to the current Peer, but multiple Pieces will basically be downloaded from a single Parent. The current version changes that multiple Pieces can be downloaded concurrently from different Parents, which can prevent traffic from concentrating on some hotspot Parents, improve download efficiency and increase the average bandwidth utilization.
8. When the Peer calls the Manager to obtain the matching Scheduler Cluster. If no Scheduler Cluster is matched, Manager will return all Scheduler Clusters to Peer. Peer will use the Scheduler Cluster address passed by Health Check for subsequent download scheduling.
9. Support the back-to-source download protocol of the ORAS [3] network client, and expand the ecological support of container mirroring.
10. Add support for UDP Ping packets and gRPC Protoc definition of virtual network topology. In the future, a virtual network topology structure based on network detection will be added to improve the accuracy of the scheduling algorithm.
11. Complete the definition of the V2 version of the P2P protocol [4] network. The Scheduler and Manager correspond to the functions of the V2 version of the P2P protocol. In the future, Dfdaemon will be rewritten based on the V2 version of the P2P protocol and the Rust language to improve client performance while relying on the more standard and more scalable V2 version of the P2P protocol.
12. The OSS client back-to-origin protocol is newly added based on the STS temporary access credentials to access the OSS origin site.
13. Added Host TTLandhostGCInterval configuration in Scheduler, which is mainly used for the release of Host metadata. It can be guaranteed that when the Peer host exits abnormally, the abnormal Host metadata can still be released to prevent dirty data from remaining.
14. The Searcher module of the Manager is added to filter the Scheduler Cluster matched by the current Peer according to the CIDR [5] conditions, providing a more accurate matching calculation method.
15. Refactored the Metric of the V1 version of the P2P protocol, added the Metric of the V2 version of the P2P protocol, and updated the alarm rules corresponding to the PrometheusRule of Helm Charts [6] according to the new Metrics . Reorganize Dragonfly Grafana Dashboards [7] so that users can import Dashboards with one click to observe P2P network traffic and service-related data.
For specific documents, please refer to Monitoring [8] , and Grafana Dashboard is maintained in the project Dragonflyoss/Monitoring [9] . More details about what version updates contain can be found in ChangeLog [10] .
breaking change
The old version Manager uses the default JWT Key to generate JWT Token, which will cause security issues. For details, please refer to security issues [11] . Therefore, the JWT configuration [12] is added to the Manager service configuration file .
When the old version of Manager is upgraded, it is necessary to add JWT Key configuration in the configuration file of the new version of Manager, and the JWT Key needs to be generated by the user. For details on how to configure the JWT Key, please refer to Setting it in the Manager Configuration [13 ] .
Public cloud vendor support
Alibaba Cloud (Alibaba Cloud) - Alibaba Cloud provides installation of Dragonfly 1.x in Container Registry [14] . For documentation, please refer to Use P2P Acceleration in ASK [15] . It is recommended to use the more efficient and stable Dragonfly 2.0 [16] , and for the deployment documentation, refer to Setup Dragonfly in Kubernetes [17] .
Google Cloud Platform (GCP) - GCP provides one-click deployment of Dragonfly in the Marketplace [ 19] of Google Kubernetes Engine (GKE) [18 ] . For specific documents, please refer to Click to Deploy Dragonfly [20] .
Volcano Engine (Volcano Engine) - Volcano Engine integrates the CNCF incubation project Dragonfly in Container Service ( VKE ) and Registry (CR) . Welcome to VKE [21] & CR [22] to learn more.
Baidu AI Cloud (Baidu Smart Cloud) - Baidu AI Cloud provides one-click deployment of P2P acceleration services on Cloud Container Engine (CCE) [24] , and its capability comes from Dragonfly.
｜Related Links｜
Dragonfly community official website :
[1] Dragonfly official website: https://d7y.io/
[2] Priority Protoc Definition：https://github.com/dragonflyoss/api/blob/main/pkg/apis/common/v2/common.proto#L74
[3] ORAS：https://github.com/oras-project/oras
[4] V2 version of the P2P protocol: https://github.com/dragonflyoss/api/tree/main/proto
[5] CIDR: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing
[6] Helm Charts：https://github.com/dragonflyoss/helm-charts
[7] Dragonfly Grafana Dashboards：https://grafana.com/grafana/dashboards/?search=dragonfly
[8] Monitoring：https://d7y.io/docs/concepts/observability/monitoring
[9] Dragonfly Monitor：https://github.com/dragonflyoss/monitoring
[10] ChangeLog：https://github.com/dragonflyoss/Dragonfly2/blob/main/CHANGELOG.md
[11] Security issues：https://github.com/dragonflyoss/Dragonfly2/security/advisories/GHSA-hpc8-7wpm-889w
[12] JWT configuration: https://github.com/dragonflyoss/Dragonfly2/pull/2161
[13] Setting it in the Manager configuration：https://github.com/dragonflyoss/d7y.io/blob/main/docs/reference/configuration/manager.md?plain=1#L56
[14] Alibaba Cloud Mirror Warehouse: https://www.alibabacloud.com/product/container-registry
[15] Use P2P Acceleration in ASK：https://www.alibabacloud.com/help/en/container-registry/latest/use-the-p2p-acceleration-feature-in-ask-and-ack-clusters
[16] Dragonfly repository: https://github.com/dragonflyoss/Dragonfly2
[17] Setup Dragonfly in Kubernetes：https://d7y.io/docs/getting-started/quick-start/kubernetes/
[18] Google Kubernetes Engine（GKE）：https://cloud.google.com/kubernetes-engine
[19] Google Cloud Platform（GCP）Marketplace：https://console.cloud.google.com/marketplace/product/google/dragonfly
[20] Google Cloud Platform（GCP）Dragonfly Marketplace：https://console.cloud.google.com/marketplace/product/google/dragonfly
[21] Volcano Engine Container Service (VKE): https://www.volcengine.com/product/vke
[22] Volcano Engine Mirror Warehouse (CR): https://www.volcengine.com/product/cr
[23] Baidu Smart Cloud Container Service (CCE): https://intl.cloud.baidu.com/product/cce.html
[24]Dragonfly official website: https://d7y.io/
[25] Dragonfly Charts repository: https://github.com/dragonflyoss/helm-charts.
[26] Dragonfly Monitor repository: https://github.com/dragonflyoss/monitoring.
[27] Alibaba Cloud Container Registry: https://www.alibabacloud.com/product/container-registry.
[28] Alibaba Cloud Container Service (ACK) : https://www.alibabacloud.com/product/kubernetes.
[29]Google Kubernetes Engine（GKE）：https://cloud.google.com/kubernetes-engine。
[30]Google Cloud Platform（GCP）Dragonfly Marketplace：https://console.cloud.google.com/marketplace/product/google/dragonfly
[31] Volcano Engine Container Service (VKE) : https://www.volcengine.com/product/vke
[32] Volcano Engine Container Repository (CR) : https://www.volcengine.com/product/cr
[33] Baidu Smart Cloud Container Service (CCE) : https://intl.cloud.baidu.com/product/cce.html
Scan the QR code to join the Dragonfly DingTalk group, come and discuss with us!
Dragonfly Star ✨:
https://github.com/dragonflyoss/Dragonfly2
Recommended reading this week
Dragonfly P2P-based file and image distribution system
P2P transfer protocol optimization in Dragonfly
Occlum Meetup - Let's talk about confidential computing TEE
Booth recruitment｜"SOFA 5th Anniversary" Open Source Bazaar is waiting for you!
