[ Cloud Computing | AWS ] A Complete Guide to Using Amazon S3 for Bucket and Object Operations in Java Applications

insert image description here

Article Directory

I. Introduction

In this article, we'll explore how Java programming can be used to interact with the Amazon S3 (Simple Storage Service) storage system.

It's important to keep in mind that S3's structure is surprisingly simple: each bucket can hold a large number of objects, which can be accessed through a SOAP interface or a RESTful API.

Next, we'll use the AWS SDK for Java to create, enumerate, and delete S3 buckets. At the same time, we will also learn how to upload, enumerate, download, copy, move, rename and delete the individual objects in these buckets.

2. Required Maven dependencies

Before we start, we need to declare the AWS SDK dependencies in our project:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.20.52</version>
</dependency>

To check the latest version, we can check Maven Central , or other domestic Maven repositories.

3. Information about several prerequisites

To use the AWS SDK we need a few things:

AWS Account: We need an Amazon Web Services account. If we don't have one, we can create an account directly in the AWS console.
AWS Security Credentials: These are our access keys that allow us to call AWS API operations programmatically. We can obtain these credentials in two ways: using the AWS root account credentials in the Access Keys section of the Security Credentials page, or using the IAMIAM user credentials in the console.
Select the AWS Region (Region): We must also select the AWS Region where we want to store the Amazon S3 data. Keep in mind that S3 storage prices vary by region. See the official documentation for more details.

Fourth, create a client connection

First, we need to create a client connection to access the Amazon S3 web service. For this, we will use the Amazon S3 interface:

AWSCredentials credentials = new BasicAWSCredentials(
  "<AWS accesskey>", 
  "<AWS secretkey>"
);

Then we'll configure the client:

AmazonS3 s3client = AmazonS3ClientBuilder
  .standard()
  .withCredentials(new AWSStaticCredentialsProvider(credentials))
  .withRegion(Regions.US_EAST_2)
  .build();

5. Amazon S3 Bucket Operations

5.1. Create Bucket

It is important to note that the bucket namespace is shared by all users of the system. Therefore, our bucket name must be unique among all existing bucket names in Amazon S3 (we'll see how to check this later).

In addition, according to official documents, the Bucket name must meet the following requirements:

Name should not contain underscores
Name length should be between 3 and 63 characters
Name should not end with a dash
Name cannot contain adjacent periods
The name cannot contain a dash after it (for example, "my-.bucket.com" and "my.-bucket" are invalid)
Name cannot contain uppercase characters

Now let's create a bucket:

String bucketName = "baeldung-bucket";

if(s3client.doesBucketExist(bucketName)) {
    
    
    LOG.info("Bucket name is not available."
      + " Try again with a different Bucket name.");
    return;
}

CreateBucketRequest bucketRequest = CreateBucketRequest.builder()
    .bucket(bucketName)
    .build();

s3Client.createBucket(bucketRequest);

Before creating a bucket, we have to doesBucketExist()check if the bucket name is available using method. If the name is available, then we build a CreateBucketRequestand provide the bucket name. The last step is to pass bucketRequestto S3Client .CreateBucketRequest createBucketRequestcreateBucket

5.2. List buckets

Now that we've created some buckets, let's use listBuckets ()the method to print a list of all buckets available in the S3 environment. This method will return a ListBucketsResponse, which contains information about the bucket.

ListBucketsResponse listBucketsResponse = s3Client.listBuckets();

// Display the bucket names
List<Bucket> buckets = listBucketsResponse.buckets();
System.out.println("Buckets:");
for (Bucket bucket : buckets) {
    
    
    System.out.println(bucket.name());
}

This will list all buckets present in the S3 environment:

baeldung-bucket
baeldung-bucket-test2
elasticbeanstalk-us-east-2

5.3. Delete Bucket

**It is very important to make sure the bucket is empty before deleting it. ** Otherwise, an exception will be thrown.

First, we need to construct an DeleBucketRequestinstance and pass it the bucket name. We then call deleteBucketmethods on the s3Client object, passing the request as a parameter.

Also note that only the owner of the bucket can delete it, regardless of their permissions (access control policy):

try {
    
    
    DeleteBucketRequest deleteBucketRequest = DeleteBucketRequest.builder()
        .bucket(bucketName)
        .build();

    s3Client.deleteBucket(deleteBucketRequest);
    System.out.println("Successfully deleted bucket : " + bucketName);
} catch (S3Exception e) {
    
    
    System.err.println(e.getMessage());
    System.exit(1);
}

6. Amazon S3 Object Operations

Files or collections of data within an Amazon S3 bucket are called objects. We can perform various operations on objects such as upload, list, download, copy, move, rename and delete.

6.1. Uploading objects

Uploading objects is a very simple process. First, we'll build an PutObjectRequestinstance, specifying a bucket name and key. We then pass the request and the path to the file containing the data to the s3Client putObjectmethod:

PutObjectRequest request = PutObjectRequest.builder()
    .bucket(bucketName)
    .key(key)
    .build();

return s3Client.putObject(request, Path.of(file.toURI()) );

6.2. List objects

We'll listObjects()list all objects available in the S3 bucket using the method:

ListObjectsV2Request listObjectsV2Request = ListObjectsV2Request.builder()
    .bucket(bucketName)
    .build();
ListObjectsV2Response listObjectsV2Response = s3Client.listObjectsV2(listObjectsV2Request);

List<S3Object> contents = listObjectsV2Response.contents();

System.out.println("Number of objects in the bucket: " + contents.stream().count());
contents.stream().forEach(System.out::println);

To list objects in an AWS S3 bucket, we need to create an ListObjectsV2Requestinstance and specify the bucket name. We then call methods on the s3Client object listObjectsV2, passing the request as a parameter. This method returns a ListObjectsV2Response, which contains information about the objects in the bucket.

6.3. Downloading Objects

To download an object, we first create an GetObjectRequestinstance and pass it the bucket name and key as input parameters. We then feed it to getObjectAsBytes()the method and get the response. Once we get the response we can extract the byte array. The last step is to process the byte array:

GetObjectRequest objectRequest = GetObjectRequest.builder()
    .bucket(bucketName)
    .key(objectKey)
    .build();

ResponseBytes<GetObjectResponse> responseResponseBytes = s3Client.getObjectAsBytes(objectRequest);

byte[] data = responseResponseBytes.asByteArray();

// Write the data to a local file.
java.io.File myFile = new java.io.File("/Users/user/Desktop/hello.txt" );
OutputStream os = new FileOutputStream(myFile);
os.write(data);
System.out.println("Successfully obtained bytes from an S3 object");
os.close();

6.4. Copying, renaming and moving objects

We can copy objects by calling a method on s3client copyObject(), which accepts CopyObjectRequestan instance. Therefore, CopyObjectRequestfour parameters are accepted:

source bucket name
Object key in source bucket
Destination bucket name (can be same as source bucket name)
Object key in target bucket

CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder()
    .sourceBucket(sourceBucketName)
    .sourceKey(sourceKey)
    .destinationBucket(destinationBucketName)
    .destinationKey(destinationKey)
    .build();

return s3Client.copyObject(copyObjectRequest);

Note: We can combine copyObject()methods and deleteObject()to perform move and rename tasks. This would involve first duplicating the object, then deleting it from its old location.

6.5. Deleting objects

To delete an object, we'll call a method on the s3client deleteObject()and pass DeleteObjectRequestthe instance. In order to create DeleteObjectRequestan instance, we need to pass the key and bucket name of the object to delete:

DeleteObjectRequest deleteObjectRequest = DeleteObjectRequest.builder()
    .bucket(bucketName)
    .key(objectKey)
    .build();

s3Client.deleteObject(deleteObjectRequest);

6.6. Deleting multiple objects

To delete multiple objects at once, we first create DeleteObjectsRequestthe objects and pass the buckets. Then we'll pass an ArrayList containing all the object keys to delete.

Once we have this DeleteObjectsRequestobject, we can pass it as a parameter to our s3client's deleteObjects()methods. If successful, it deletes all objects we provide:

ArrayList<ObjectIdentifier> toDelete = new ArrayList<>();
for(String objKey : keys) {
    
    
    toDelete.add(ObjectIdentifier.builder()
        .key(objKey)
        .build());
}

DeleteObjectsRequest deleteObjectRequest = DeleteObjectsRequest.builder()
    .bucket(bucketName)
    .delete(Delete.builder()
        .objects(toDelete).build())
    .build();

s3Client.deleteObjects(deleteObjectRequest);

7. Summary at the end of the article

In this article, we focused on the basics of interacting with the Amazon S3 web service at the bucket level and object level. It focuses on the method of using Amazon S3 (Simple Storage Service) for storage buckets and object operations in Java applications, and explains in detail how to perform storage bucket operations, including creating buckets and listing buckets. In the object manipulation section, we covered the process of uploading, listing, and downloading objects, as well as methods for copying, renaming, moving objects, and deleting objects. Additionally, we provide steps to delete multiple objects. Through this article, you can learn how to effectively interact with Amazon S3 through Java code to implement various operations on storage buckets and objects.