In the past ten years, Amazon cloud technology has been committed to helping enterprises and developers achieve digital transformation, including how to use cloud technology to help enterprises improve resource utilization in operations; how to innovate business through cloud infrastructure, containers, and DevOps and agility; in the next ten years, Amazon cloud technology will help developers and enterprises start a new sustainable development transformation . Let developers use the same tools to focus on sustainability work, and through best practices and overall infrastructure, help you meet new challenges in the process of sustainable development transformation.
| The Amazon cloud technology developer community provides developers with global development technology resources. There are technical documents, development cases, technical columns, training videos, activities and competitions, etc. Help Chinese developers connect with the world's most cutting-edge technologies, ideas, and projects, and recommend outstanding Chinese developers or technologies to the global cloud community. If you haven't paid attention/favorite yet, please don't rush over when you see this, click here to make it your technical treasure house! |
While committed to the sustainable development and transformation of the cloud itself, we have summarized our practices and experience, and designed effective tools and services for developers.
As a sustainable development goal on the cloud, we believe that two aspects of work are very important:
The first is to continuously reduce the energy consumption of workloads. The second is to continuously improve the efficiency of work between workload components.
Based on the above two goals, we have summarized six best practices to help developers build a sustainable architecture model on the cloud, including: region selection, correct use of cloud resources, software architecture, data usage, and hardware Select and develop deployment.
Amazon Sustainability Best Practices
Choose the right region to deploy your workload
Regions of Amazon Cloud Technology
Amazon cloud technology provides more than 30 regions and 99 availability zones around the world. Our builders can choose resources in different availability zones based on business needs, including network latency, data compliance requirements, and more. And sustainable development, that is, the goal of energy conservation and emission reduction should also be taken into consideration when selecting an appropriate availability zone.
Amazon cloud technology has achieved a total of 15.7 gigawatts of renewable energy through 310 renewable energy projects in North America, Europe, the Middle East and Africa, and the Asia-Pacific region. Different regions have slightly different energy savings for different renewable energy projects. The 32 Asia-Pacific renewable energy projects include two renewable energy projects in Shandong and Jilin, China. The Shandong project is a 100 megawatt (MW) solar project that is already in operation and will generate 128,000 megawatt hours (MWh) of clean energy per year. The Jilin project is the second renewable energy project supported by Amazon in China and is a 100 megawatt (MW) wind project. The project, which will also soon be operational, will provide more than 300,000 megawatt-hours (MWh) of renewable energy per year, equivalent to powering more than 150,000 ordinary Chinese households.
The best practice we recommend is to choose an area close to these renewable energy projects and use Amazon's existing achievements to achieve the first energy saving and emission reduction.
Proper use of cloud resources
Our second recommendation is to use cloud resources correctly .
this point is very important. Unlike self-built data centers, resources on the cloud are paid on demand. Most enterprises do not average the construction cost of the internal data center to the service cost occupied by the specific business. Therefore, they consume cloud resources in the way of self-built data centers. For a single application, there is no obvious advantage in resource consumption on the cloud. . Therefore, establish new usage habits, standardize the use of cloud resources, especially join in the formulation of clear sustainable development goals, and balance them with SLAs to design application delivery methods, such as: expand on demand, eliminate idle resources, Optimize the usage behavior of resources on the cloud according to the geographical layout of the oil painting workload in the geographical location and according to the sustainable development goals. By regulating the use of resources on the cloud, it can help developers further achieve agility, efficiency, and automation, and at the same time accelerate the transformation of sustainable development.
Let's take a look at a specific case to see what is the correct way to use cloud resources:
We run a specific service in two availability zones, and achieve high availability by implementing real-time failover from one availability zone to another. Obviously, this service needs to reserve idle resources matching the actual workload in each availability zone. Therefore, the energy consumption of actual workloads in each availability zone should be limited to less than 50% of the overall resources.
However, if we achieve high availability of this service through three availability zones. Then we only need to reserve 2/3 of the total resources in each availability zone to satisfy real-time failover. Therefore, we can see that for the high-availability requirements of real-time failover, the choice of three availability zones has higher average utilization of resources, lower costs, and minimal impact of failures.
Quite simply, this is one of the correct ways to use resources on the cloud.
Another interesting case is this one of balancing SLAs and SDGs:
For non-real-time failover services, their important consideration is cost, and the waiting time for failover is exchanged for the greatest possible resource saving.
For the business requirements of non-real-time failover, we can achieve the goal of sustainable development by optimizing the duration of failover. The specific method is to select "cold capacity" as the service resource during failover.
When failover is required, we look for or wait for idle resources in the availability zone, and then perform failover. Although we sacrifice the response time during the switching process, we have greatly reduced the waste and energy consumption caused by idle resources, and achieved the goal of sustainable development in terms of operating costs.
In fact, how to correctly use cloud resources to achieve the goal of sustainable development, the biggest consideration is to balance the SLAs of different businesses and the sustainable development goals of the overall business. Making the right choice when balancing workload deployment, business deployment, and architecture deployment is our best practice.
Optimize software and architecture
We can reduce resource consumption and impact by changing software and architecture. Including: asynchrony and scheduling, using energy-efficient programming languages, removing or refactoring low-usage or even useless load components, low-code, optimizing the impact of developers' end devices, using matching data access and storage patterns software and architecture.
Through performance scheduling, reduce energy consumption due to "peak"
The graph shows the performance curve of a service during its business lifecycle. The three baselines represent:
- Consumption line of average energy consumption (Average)
- Energy consumption at business peaks (Peak)
- Resources required for business operations (Provisioned Capacity)
We found that the actual resources and energy consumption used by services are not the area under the average energy consumption curve, but the area under the configured capacity line. It is worth noting that the size of the configuration capacity depends on the peak energy consumption of the application. Usually, based on high availability considerations, the configuration capacity is twice the peak energy consumption.
If we lower the peak value by sacrificing a certain SLA, or we adjust the way the business operates to average the workload over the entire business cycle, or we implement a queue to smooth the load.
As shown below:
Then we can balance the better ratio of energy consumption from peak to average, we can reduce the peak energy consumption, and we can reduce the configured capacity. So as to achieve less resource occupation and less energy consumption.
for example:
Most enterprises run Crontab with the system default time. This is a generic service. Imagine that all Crontabs of all enterprises on the cloud are triggered at 3 o'clock in the middle of the night. Then the peak energy consumption generated by Crontab at this point in time will push up the total capacity of the resources on the cloud they need. This can be optimized for sustainability goals. In fact, for services like Crontab, non-real-time demand response will not affect the business itself. Therefore, peaks in service demand can be smoothed through event-driven architecture and technology. The specific implementation method is to put service requirements into corresponding buffers through event routers, such as Amazon Event Bridge or Amazon SNS, and then implement asynchronous, parallel processing and Crontab requirements through queue services such as Amazon SQS and Amazon MQ the response to.
Of course, you can also schedule jobs or operating procedures by yourself. Instead of triggering at 3:00 in the morning, you can spread the triggering events to reduce the peak energy consumption of the service. However, it is clear that event-driven architecture and serverless technology are used to achieve a better degree of automation, and the effect of reducing peak energy consumption is more obvious.
The Crontab service business scenario is to reduce the peak energy consumption of the business and reduce the total capacity configuration by means of queues or buffers, so as to obtain guaranteed SLA and reduce the use and impact of overall resources. In addition, we can also use optimization methods to reduce resource contention and improve performance without changing the cost, so as to balance the goals of SLA and sustainable development.
Typical scenarios are: scale up in response to increased work. The expansion process and the expansion itself use resources, if it is minimized. Event-driven, serverless technologies can also be used to elastically scale capacity on demand through queues or buffers, thereby smoothing demand peaks and maintaining high continuous utilization of resources used.
Efficient programming languages help reduce energy consumption
Compiling software in an energy-efficient programming language is also a good sustainable experience for developers. Of course, this is also a very big problem for many developers:
What is an energy-efficient language?
What is its standard?
How to measure it?
…
Many scientists have done research on this for a long time. In the 2017 report:
Scientists designed 10 test scenarios and compared the energy consumption of 27 mainstream languages in different scenarios such as execution time, energy consumption and maximum memory. As a result, C and Rust undisputedly beat the other languages in terms of energy efficiency. It uses 50% less energy than Java, which is commonly used by developers, and 98% less energy than Python.
The founder of Linux has also said that he thinks Rust is a programming language that can solve problems very well. While Rust is as efficient as C, it can also avoid various unsafe risks.
Amazon Cloud Technology recommends the Rust language to our developers, and Amazon itself is also a practitioner of the Rust language. Many of Amazon's important open source projects, such as Firecraker , an open source virtualization technology used to support serverless computing, and Bottlerocket , a Linux-based containerized operating system , have been developed using the Rust language and have achieved great success. At Amazon, Rust has become the key to building large-scale infrastructure, providing key services for Amazon S3, Amazon EC2, Amazon CloudFront, etc. Amazon's trust in Rust is also reflected in its continued contribution to him. Amazon has been a sponsor of the Rust project since 2019, started hiring Rust maintainers and contributors in 2020, and became a founding member of the Rust Foundation in 2021.
hardware
The impact of hardware selection and optimization on sustainable development transformation includes:
- Meet needs with minimal hardware
- Choose an Appropriate Compute Instance
- Use hosted services
- Optimization of GPUs resources
The first is to calculate the impact of the selection on the instance. We recommend that you help reduce your carbon footprint by using an instance type that meets your requirements with the least impact.
Take Graviton, which is well known to developers, as an example:
Graviton 2 is an ARM architecture processor designed and developed by Amazon Cloud Technology. Compared with the latest generation of X86 instances, it is more powerful in a wide range of workloads such as application servers, microservices, video encoding, high-performance computing, electronic design, automation, games, open source databases, memory caching, and CPU-based, machine learning reasoning. Both can provide up to 40% cost performance, are very easy to implement and the cost of migration is not high. Graviton 2 is one of the most power-efficient processors currently available from Amazon Cloud Technology.
Graviton 2 does help developers achieve energy savings throughout the workload. Based on Graviton 2 we also released Graviton 3. Compared with Graviton 2, it offers another 25% price-performance improvement and 60% lower power consumption. Many core applications and services in Amazon cloud technology, including container services, serverless services, etc., are currently compatible with Graviton 2.
It can be seen that just selecting a suitable computing instance from the computing instance can help developers optimize the overall sustainable development goals.
data
Of course there is also data to consider. After completing the workload and data migration, the reasonable management of data and the selection of storage media can also help developers further achieve the goal of sustainable development. Here are some best practices to consider, including:
- Select appropriate data access and storage technologies;
- Develop data classification policies;
- Use lifecycle policies to automatically delete unnecessary data;
- Minimize over-provisioning
- Delete unnecessary or redundant data
- Use a shared file system or object storage to access common data
- Minimize data movement between networks
- data backup
Specifically, choose the appropriate storage medium and technology to store different types of data. Or use intelligent layering to dynamically store data, which can help developers save energy.
For example, in object storage Amazon S3, various types are provided for developers to choose from:
Choose the right storage medium and technology
From hot data, warm data, cold data, to data that needs to be deleted or archived, developers can choose different storage media and technologies. They will provide corresponding technical support for response time, storage cost, and data type. On the one hand, developers can easily realize the intelligent layering of data, and on the other hand, they can also manage the effective life cycle of data.
By automatically archiving infrequently used data, marking and reviewing all data and setting up an alarm mechanism, formulating rules according to data access requirements, using policies to migrate or clear data, etc., can greatly reduce unnecessary storage resources consumption, from a large level to ensure that our cloud computing or business on the cloud is closer to a goal of sustainable development. In fact, next I have a small practice based on Amazon's own R&D engineers, which can optimize our storage resources, which is to optimize log storage.
Data Access and Storage
Optimize log storage
We will find that our various applications generate a large number of log files every day. If we archive a large number of unstructured log files or store them in our Amazon S3 object storage, in fact, we can combine these scattered data in Together, we can save a lot of space. Our own experience and practice is to save one exabyte, and everyone knows that one exabyte actually has such a capacity saving of 1 million terabytes. In addition, our service team has also made a lot of attempts on this different compression algorithm. We will find that from gzip to LZ4 and then to ZSTD, it is obvious that its compression effect is gradually increasing. We also show you its different compression ratios, so that you can make a reasonable choice according to the actual situation.
development and deployment
Reduce widget tool size
The last aspect of practice is about development and deployment, which is a very big topic. First of all, we use an interesting real case as an introduction. You may be familiar with the Amazon SDK for JavaScript, because you almost always use it whenever your browser or node.js code calls Amazon cloud technology services. a kit. With the increase in the frequency of use and development, the copy of the library of this service package will gradually expand, which will cause the SDK package to become larger and larger.
We know that the management of large packages actually consumes a lot of energy and the people you manage. So in the third version of the SDK for Javascript, we introduced a modular approach to optimize it. We extract the tools and codes that everyone shares and make it into a basic package. This basic package is based on it, and then go to superimpose some of your personalized code, or one of your tools. Compared with the additional packages generated by your previous development if you want to implement a specific development project, this basic package saves 75% in capacity. On the basis of this package, when superimposing different applications and tools, we will also go through deletions and a series of optimization methods to further save. We will find that the specific application is on top of the package, and then add your personalized ones, or some tools or services that you especially need, you will save 75% of the capacity, and then save 50% of the capacity . This is actually an experience for everyone. You can optimize your development tools to realize the energy consumption of your resources and further save the energy consumption of resources on the cloud.
All in all, in fact, we will find that sustainable development is not the goal we set in a certain link, it can be achieved. Just like what I shared with you at the beginning of this topic, it is a very big and far-reaching goal. In fact, we can subdivide it into different stages, small ones, and attainable ones. Target. In your entire development and testing process, achieve different sustainable development goals in each process. The accumulation of small things will eventually accelerate the realization of your overall sustainable development goals.
I hope that through the content of this issue, you can set your goals, and find your reasonable tools to understand these best practices and these experiences, and use these best practices to apply in your actual work Get validated, generate new best practices, welcome to share with us.
Author Zheng Yubin
Senior developer evangelist of Amazon Cloud Technology, with 20 years of experience in the ICT industry and digital transformation practice, focusing on the field of cloud native and cloud security technology of Amazon Cloud Technology. With 18 years of experience as an architect, he is committed to providing consulting and technical implementation of data center construction and software-defined data center solutions for finance, education, manufacturing, and Fortune 500 corporate users.
Article source: https://dev.amazoncloud.cn/column/article/63fc5d2af699155a74c64f83?sc_medium=regulartraffic&sc_campaign=crossplatform&sc_channel=CSDN