In 2021, UC Berkeley's Ion Stoica and Scott Shenker will publish a paper on "From Cloud Computing to Sky Computing" at a workshop on hot topics in running systems. Different from the Cloud Computing ("Cloud Computing") that we are all familiar with , Sky Computing ("Sky Computing" ) is the future of cloud computing. It refers to how to solve cross-cloud problems and break the gap between different clouds to maximize the use of cross-cloud data.
The mission of Datan Technology is just like the vision of "Sky Computing" described in the paper: to create a next-generation cloud computing platform, so that there is no gap between clouds . To this end, we translated this paper and published it in two installments. The following is the second part of "From Cloud Computing to Sky Computing", for the content of the first half, please click From Cloud Computing to Sky Computing (1)
across clouds
The compatibility layer is your first step to realize sky computing. Although the compatibility layer allows users to run applications on different cloud platforms without modification, users still need to choose the cloud platform by themselves. This is like letting Internet users choose their own routing paths. This task is too cumbersome, so it is not practical. To solve this problem, the Internet uses BGP for AS-level routing selection, and end users are completely unaware of the routing process. Sky Computing should also provide a cross-cloud layer that completely hides the cloud platform provider, and the user should have no idea which cloud platform the application is running on. The cross-cloud layer should now be on top of the compatibility layer.
Cross-cloud tiers should allow users to customize the policies that determine where the application runs - which cloud platform, and these policies should not be so low-level that the user is overburdened with the decision. Instead, these policies should reflect user preferences, such as trade-offs between performance, stability, and cost. Not only that, these strategies should also take into account that some users do not want the application to run on a competitor's cloud platform, or must run in certain countries due to legal restrictions. To give a specific example, a user can determine that his TensorFlow application must run in Germany, and must run within the next two hours, while ensuring that the cost does not exceed a certain threshold.
Cross-cloud tiers also have the potential to improve application reliability and security, as the chances of multiple cloud platforms failing at the same time are very low. And recently there have been some proposals to treat different cloud platforms as different trust domains, so that more efficient security solutions can be provided for more applications.
We believe that there are no technical constraints to achieve cross-cloud tiers, which is not difficult to understand, because the task of migrating between clouds is similar to that of migrating within a data center. Once an application considers multi-cloud scenarios, the remaining problems are completed by the following three functions:
1. Provide agreed naming conventions for OSS services . 2. A search registration service , each cloud provider registers services on it, and each user selects services based on preferences. 3. A cross-cloud billing service .
We now discuss these three functions separately.
The first is the naming convention . In order to be able to accurately find a running instance of a service, we need a globally unique name to name it. There are many ways to achieve this goal, for example, you can use the DNS service to give the name, which will not be discussed here. We also need to associate some metadata with these instances, including how to start these instances, vendor name, location, API version and hardware type, etc. We may also need to attach some dynamically changing information, such as price, load and whether it is currently available and so on.
The second is to retrieve the registration service . Each application needs to find a suitable service instance, which requires such a retrieval registration service. Each cloud provider will publish its own service name and metadata on the service, and the cloud provider should also regularly update dynamic data, such as load and price, etc. Likewise, applications should make requests according to their preferences. Of course, the detailed scheduling algorithm is out of the scope of this paper.
Finally, there is the payment system . In the sky computing scenario, the user's application can run on one cloud platform or multiple cloud platforms, but the payment needs to be calculated separately. If the payment operation is completed independently by each platform, then the user needs to create an account on these platforms, which is obviously inconvenient. Another solution is to hand over the billing service to a third-party service provider, which has accounts for all cloud platforms, so that users only need to pay once, and the service provider will handle the rest.
Based on the above discussion, there are no significant technical barriers to achieving cross-cloud, and of course there are many details that need to be further determined. With this cross-cloud vision, the next question is whether the market will naturally give birth to a product.
Collaboration among cloud providers
The cross-cloud layer is designed to help users run tasks on multiple cloud platforms on demand. If the task involves a large amount of data, then data porting will be inevitable. Today, most cloud providers charge far less for data migration in than for data migration out. For example, importing data into the AWS platform is free, but the cost of migrating data out of AWS is $0.05 to $0.09 per GB, and the equivalent cost can be used to store the data for several months. These fees are much higher than those of the leading CDN vendors, and the quotations of CDNs are generally US$0.009 to US$0.02 per GB.
We refer to this form of pricing strategy as "data gravity" pricing strategy , which will encourage users to prefer to process data in the data center instead of moving it out. Of course, for those tasks that are extremely expensive to calculate, it is very cost-effective to migrate the data out. After all, the cost of data in such tasks accounts for a very small proportion. For example, to process a 150GB data set in the ImageNet training task, it will cost about $13 to migrate this much data out, and it will cost more than $40 to complete the related training task in AWS. If the same task is run on Azure, it only costs About $20. Based on the above data, it is not difficult to find that it is cheaper to move the data out for processing. It is not difficult to find that even with the existence of data gravity, sometimes migrating data is still a more economical choice.
Not only that, for some static data, we can store it in a cloud platform archive service, which is very cheap, and then migrate to the blob storage that needs to be processed when there is a need. Long-term storage in blob storage on a cloud platform is more expensive than the above methods.
As far as we know, the cost of data migration is now consistent, that is, no matter where our data is destined, the cost is the same. A possible choice for the cloud platform is to sign a cooperation agreement with each other, and there is no charge for data transmission between each other, and to build a faster transmission link. In this way, data transmission is free and very fast, and the data gravity between the cloud platforms that sign the agreement is also reduced.
speculation about the future
Based on the above discussion, the technical difficulty of making a compatibility layer is not great, but it will make the services of cloud computing providers become ordinary commodities—substitutable, so it is foreseeable that cloud computing providers will treat the compatibility layer negatively. The good news is that unlike other fields, the standardization of the compatibility layer does not require a global negotiation, and even if the cloud computing provider is not willing, more and more software can run on multiple platforms. Then even if the compatibility layer cannot be officially supported, it will be used by more and more users to achieve the purpose of easier migration.
Although large cloud computing providers do not like the compatibility layer, using this compatibility standard for small cloud service providers will help expand their market share, so they will be more willing to be compatible with this standard. We have already seen this scenario in the market, Google released Anthos - a K8S-based application management platform - which claims to be able to "write once, run anytime, anywhere". Anthos is already running on Google Cloud and AWS, and Azure will soon follow.
When a compatibility layer is widely accepted, then cross-cloud development can begin. Of course, it is just speculation and imagination now, and it is not possible to see that these two layers are fully ready in the short term.
When the above two layers are ready, there are only two options left for cloud computing providers: either insist on using their own private API and lock users to their own platform, or be compatible with the above two layers and provide compatible services. The former tends to be relatively large in size and can provide sufficient resources for private agreements. The cloud computing provider who chooses the latter will become a part of Skycomputing and jointly form Skycomputing to coexist and compete in a compatible system.
Why do we believe so strongly that sky computing will happen? We can simply analyze that in a fully competitive market, providers who insist on private protocols need to compete with other similar providers and the entire sky computing system. In order to remain competitive, they need to spend more resources Come and innovate to ensure the competitiveness of your own private API. But on the contrary, the boundaries of providers in Sky Computing are lower, and they don't need to spend more effort to maintain all fields, but only need to focus on their own relatively narrow fields for innovation. For example, Oracle can provide a cloud that focuses on DB, and companies like EMC can provide a cloud that focuses on storage. Hardware manufacturers can also participate in cloud computing. For example, Samsung may provide the most cost-effective cloud storage, and Nvidia may provide hardware-assisted ML services. In the absence of Sky Computing, the manufacturers mentioned above have only two choices, either to deploy their own hardware or services to the computer room of a certain (several) cloud platform providers, or to build a complete cloud computing infrastructure by themselves. Obviously neither of these is the best option.
Obviously, the convenience mentioned above is based on an assumption that users can find these unique convenient services and get timely information updates. In order to achieve this, the above three layers are needed: compatibility layer, cross-cloud layer and inter-provider collaboration.
Of course, we are not saying that private cloud platforms will disappear, and both models will exist for a long time. The private agreement cloud platform will be suitable for those users who need more assistance, and they don't pay much attention to cost performance. But once it comes to large-scale use, the cost performance of sky computing is reflected, and it becomes the best choice.
in conclusion
In this article, we describe the challenges of achieving sky computing. Some challenges are purely technical and don't look too difficult to achieve. However, in order to make the entire financial system work, Sky Computing needs a group of cloud computing providers to cooperate with each other to allow applications to flow on it.
Past recommendation
From Cloud Computing to Sky Computing (1)
Xline v0.4.0: A distributed KV store for metadata management
DatenLord focuses on the next generation of cloud computing - the infrastructure technology of "Sky Computing", and is committed to broadening the boundaries of cloud computing. DatenLord, a new generation of open source cross-cloud storage platform created by Datan Technology, breaks through cloud-cloud barriers through deep integration of software and hardware, realizes unlimited cross-cloud storage and cross-cloud connectivity, and establishes a unified storage access mechanism for massive remote and heterogeneous data. Cloud applications provide high-performance secure storage support. To meet the needs of customers in different industries for high-performance access to massive data across clouds and data centers.
Public account : Datan Technology DatenLord
DatenLord official website : http://www.datenlord.io
Zhihu Account:
Station B :
https://space.bilibili.com/2017027518
Datenlord Email: [email protected]
If you are interested in joining Datan Technology, or joining Datan Technology Technology Exchange Group, please add a small assistant WeChat: DatenLord_Tech
Published on 2023-06-02 23:16・IP territory Hong Kong, China