Authorization instructions: This article authorizes the official Amazon Cloud Technology article to forward and rewrite the rights, including but not limited to Amazon Cloud Technology official channels such as Amazon Cloud Technology Developer Community, Zhihu, self-media platforms, third-party developer media, etc.
Since the launch of the Amazon S3 service, it has been a cloud storage service favored by millions of customers from all walks of life around the world. However, as emerging use cases continue to emerge, traditional storage methods face performance bottlenecks, especially for applications that are extremely sensitive to latency. For the recently held Amazon Cloud Technology re:Invent 2023 conference, the storage-related product Amazon S3 Express One Zone was released, a high-performance storage class that provides customers with the fastest data access speed and excellent performance.
Table of contents
1. Technical challenges: data access speed and performance requirements
2. Technological Breakthrough: Innovation of Amazon S3 Express One Zone
3. Technological breakthrough: integrated solution collocated with computing resources
4. Technological breakthrough: solving complexity and improving flexibility
3. Experience and personal summary
1. Cost reduction and efficiency improvement: Performance advantages and cost reduction of S3 Express One Zone
1. Technical challenges: data access speed and performance requirements
As emerging use cases emerge, some applications need to perform millions of data writes and accesses per minute, placing stringent requirements on response times in the single digits of milliseconds. Especially in scenarios such as artificial intelligence and machine learning model training, interactive analysis, financial model simulation, etc., fast access to data is crucial for computing efficiency. Traditional storage methods struggle to meet these performance needs, and customers often move their most frequently accessed data to custom caching solutions to reduce storage latency.
2. Technological Breakthrough: Innovation of Amazon S3 Express One Zone
Amazon S3 Express One Zone innovatively solves the contradiction between data access speed and performance requirements. By replicating and storing data on specialized hardware and software, it achieves 10 times faster access than Amazon S3 Standard, significantly increasing data processing speeds. To further support millions of requests per minute, a new bucket type, the Amazon S3 directory bucket, is introduced, effectively reducing request costs.
3. Technological breakthrough: integrated solution collocated with computing resources
Amazon S3 Express One Zone provides options for collocation with compute resources, allowing customers to run performance-critical applications faster. By placing data and compute resources in the same Amazon Cloud Availability Zone, you reduce compute costs and meet the needs of the most demanding workloads. Performance-critical applications such as query-intensive analytics achieve single-digit millisecond data access speeds while reducing request costs by up to 50% and compute costs by up to 60% compared to Amazon S3 Standard.
4. Technological breakthrough: solving complexity and improving flexibility
Traditionally, customers have solved performance issues by moving data to a custom caching solution, but this also brings operational burden and modifications to the application. Amazon S3 Express One Zone helps customers escape this dilemma by eliminating the need to manage multiple storage systems for low-latency workloads. By using the same Amazon S3 API to access data, customers no longer need additional storage infrastructure, simplifying the architecture and reducing complexity.
2. Experience testUse
Log in to the Amazon Cloud Technology official website. If you don’t have an account, you can register an account. After logging in, enter the personal console and search for S3 bucket service in the top search bar.
Bucket types are divided into general buckets and directory buckets. Directory buckets are optimized for application scenarios that require stable millisecond-level latency. Here we choose to create a directory bucket and create a bucket by clicking the button on the right in the picture.
When creating a bucket, we select the bucket type as Directory - New, which is used for low-latency application scenarios. Other configuration information is selected based on development needs.
After entering the directory bucket, you can perform some related operations, including uploading and managing resources. After entering the upload interface
We can drag and drop target files and folders, or upload file and folder resources locally.S3 Express One Zone supports batch operations in directory buckets< /span>, including list operations. Users can easily manipulate and manage objects in catalog buckets while retaining the validity and integrity of the inventory
After uploading the resource, just wait a moment to check the upload status. As shown in the picture, the photo has been successfully uploaded. Here I uploaded some photos,S3 Express One Zone is a storage class of Amazon S3 that provides users with default server-side encryption (SSE-S3) and HTTPS in transit Secure access. Through regional and regional API endpoints, users can choose access methods based on actual needs. This storage class supports user-selectable checksum algorithms, including SHA and CRC, for verifying data integrity during upload or download.
Enter the bucket we want to view to get relevant information about the bucket. When the bucket needs to be used in other functions, it can be selected and used based on the relevant name and information.
Once you create a directory bucket, you will immediately experience extremely low read and write latency. This experience can be achieved by establishing a VPC endpoint connection, or by using regions and zone endpoints for API operations that manage object and directory buckets.
Directory bucket S3 Express single zone additionCreate bucket policyExample: As an IAM user, ensure that the user has the s3express:CreateBucket permission. This can be accomplished by adding the following permission to the user's IAM policyFor example:The following example bucket policy allows the AWS account ID to use API operations for the default session For the specified directory bucket. This policy grants access to zone endpoints (object level) for API operations.
例 – 允许使用默认会话进行调用的存储桶策略CreateSessionReadWrite
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadWriteAccess",
"Effect": "Allow",
"Resource": "arn:aws:s3express:us-west-2:111122223333:bucket/DOC-EXAMPLE-BUCKET--usw2-az1--x-s3",
"Principal": {
"AWS": [
"111122223333"
]
},
"Action": [//获得对应权限
"s3:CreateBucket",
"s3:PutBucketPolicy",
"s3:GetBucketPolicy" ]
}
]
}
Amazon S3 Express One Zone storage class supports region-level (bucket or control plane) and object-level (data plane) endpoint API operations,Take creating a bucket as an example , the request syntax is as follows:
PUT / HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-acl: ACL
x-amz-grant-full-control: GrantFullControl
x-amz-grant-read: GrantRead
x-amz-grant-read-acp: GrantReadACP
x-amz-grant-write: GrantWrite
x-amz-grant-write-acp: GrantWriteACP
x-amz-bucket-object-lock-enabled: ObjectLockEnabledForBucket
x-amz-object-ownership: ObjectOwnership
<?xml version="1.0" encoding="UTF-8"?>
<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<LocationConstraint>string</LocationConstraint>
<Location>
<Name>string</Name>
<Type>string</Type>
</Location>
<Bucket>
<DataRedundancy>string</DataRedundancy>
<Type>string</Type>
</Bucket>
</CreateBucketConfiguration>
At the same time, Amazon S3 also provides access point functions, as shown in the figure, all with corresponding applicable scenarios and uses. An access point is a named network endpoint connected to a bucket that can be used to perform operations on S3 objects. Access point aliases provide the same functionality as access point ARNs and can be used anywhere in place of S3 bucket names (typically for data access)
Batch operations are the ability to perform large-scale batch operations on Amazon S3 objects. We can perform batch operations on a series of S3 objects by creating a job
3. Experience and insightsAnd personal summary
After learning andusing Amazon S3 Express One Zone,I deeply felt the power of technology How innovation has dramatically improved the efficiency of data processing. Amazon S3 Express One Zonehasexcellent data access speed, which has brought a lot ofhelp to my daily work< /span>When dealing with artificial intelligence and machine learning model training, interactive analysis and other scenarios, I feel the significant improvement in data access speed, while providing greater flexibility for my work. performance and operability. Compared with traditional storage methods, S3 Express One Zone uses dedicated hardware and software to replicate and store data, making access faster than ever before, making the experience of processing large-scale data efficient and convenient. . .
It is worth mentioning that Amazon S3 Express One Zone introduces a new bucket type, the Amazon S3 directory bucket. This innovative design plays a huge role in reducing request costs. Not only is the performance improved, but it is also more cost-effective in terms of economic benefits. This perfect combination of high performance and affordability makesthedata processing experience also< a i=4>becomeparticularlyexcellent.
The technological innovation of Amazon S3 Express One Zone provides an excellent solution to the contradiction between data access speed and performance requirements. Through the innovative use of specialized hardware and software, faster access to data is achieved, bringing significant improvements to performance-critical applications. The introduction of the new bucket design further improves overall performance, effectively reduces request costs, and provides users with a more economical option. This series of technological innovations not only improves the speed and efficiency of data processing, but also provides users with a more flexible and controllable data management experience. Amazon S3 Express One Zone creates a more intelligent, efficient and affordable data processing environment for users through the forward-looking application of technology.