Minio introductory series [5] Detailed explanation of the use of bucket operation API of JAVA integrated Minio

1 Introduction

1.1 Official documents and SDK

Official documentation: https://min.io/docs/minio/kubernetes/upstream/index.html?ref=docs-redirect
SDK: https://github.com/minio/minio-java
Minio provides multiple languages SDK, such as java, go, python, etc. The JAVA development platform can choose JS and java SDK, that is, both the front end and the back end can directly integrate minio.

1.2 Technical solutions

Every OSS user will use the upload service. A common upload method on the Web side is that users upload files to the application server on the browser or App side, and the application server then uploads the files to OSS. The specific process is shown in the figure below.

Compared with directly transferring data to OSS, the above method has three disadvantages:

• Slow upload: User data needs to be uploaded to the application server first, and then uploaded to OSS. The network transmission time is twice as long as direct transmission to OSS. If user data is not transferred through the application server, but directly transmitted to OSS, the speed will be greatly improved. Moreover, OSS uses BGP bandwidth to ensure transmission speeds between operators in various places.

• Poor scalability: If there are more subsequent users, the application server will become a bottleneck.

• High cost: multiple application servers need to be prepared. Since OSS upload traffic is free, if the data is transmitted directly to OSS without passing through the application server, several application servers will be saved.

Currently, there are three technical solutions for uploading files to OSS through Web front-end technology:

• Using the OSS js SDK to upload files to OSS means that the front end is directly connected to OSS, but it is easy to expose authentication information and is not very secure.

• Use the form upload method to upload files to OSS. Use the temporary interface provided by OSS to upload files to OSS using the form upload method. Then request the backend to inform you that the upload is complete and perform subsequent processing.

• Uploading to the application server first, and then requesting OSS to upload, has higher security and can control data and authentication, but has the worst performance.

2 Integrate JAVA SDK

Because general non-Internet projects do not have high performance requirements, it is enough to use JAVA SDK to integrate MInio and then provide an interface for calling on the Web side.

2.1 Environment setup

First, build a Maven basic project and introduce relevant dependencies. The latest version 8.3.1 is introduced here. The latest package of okhttp is also introduced, otherwise some APIs will prompt that the version is too low.

<dependency>
  			<groupId>io.minio</groupId>
  			<artifactId>minio</artifactId>
  			<version>8.3.1</version>
		</dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>4.9.2</version>
        </dependency>

2.2 Initialize the client

You can see that minio now uses the Builder model to construct objects, which is very different from before, so you need to pay attention.

//url为地址,accessKey和secretKey为用户名和密码
MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();

2.3 Basic bucket operations

2.3.1 Whether the bucket exists

Check if the bucket exists.

public boolean bucketExists(BucketExistsArgs args)

Sample code:

    /**
     * 判断桶是否存在
     */
    public static boolean bucketExists(String url, String accessKey, String secretKey, String bucketName)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
    }

2.3.2 Create bucket

Creates a bucket with locking enabled for the given region and object.

public void makeBucket(MakeBucketArgs args)

Sample code:

    /**
     * 添加存储桶
     */
    public static void makeBucket(String url, String accessKey, String secretKey, String bucketName, String region, boolean objectLock)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).region(region).objectLock(objectLock).build());
    }

After creation, you can see these buckets in the console. After uploading files and deleting the last locked bucket, it is found that these objects still appear to exist, and the actual files on the disk are not deleted.

2.3.3 Query bucket information list

List bucket information for all buckets.

public List<Bucket> listBuckets()

Sample code:

    /**
     * 查询存储桶信息列表
     */
    public static List<Bucket> listBuckets(String url, String accessKey, String secretKey) throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        return minioClient.listBuckets();
    }

    public static void main(String[] args) throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        List<Bucket> buckets = listBuckets("url", "accessKey", "secretKey");
        for (Bucket bucket : buckets) {
    
    
            System.out.println(bucket.creationDate() + ", " + bucket.name());
        }
    }

The printed information is as follows. The returned creation time is US time, please note.

2.3.4 Delete bucket

Delete an empty bucket.

public void removeBucket(RemoveBucketArgs args) 

Sample code:

    /**
     * 删除存储桶
     */
    public static void removeBucket(String url, String accessKey, String secretKey, String bucketName)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());
    }

Note: Make sure the bucket exists, otherwise an error will be reported. When deleting, it is best to call the bucketExists() method to determine whether it exists.

2.4 Set up bucket operations

2.4.1 Encryption configuration

Set the encryption configuration of the bucket to allow data to be encrypted using the corresponding encryption configuration when uploading objects in the bucket. The currently supported server-side encryption methods are KMS-managed key server-side encryption (SSE-KMS) and AES256 encryption.
Set the encryption configuration of the bucket:

public void setBucketEncryption(SetBucketEncryptionArgs args)

Get the encryption configuration of the bucket:

public SseConfiguration getBucketEncryption(GetBucketEncryptionArgs args)

2.4.2 Life cycle

Lifecycle management can be applied to the following typical scenarios:

• Log files that are uploaded periodically may only need to be retained for a week or a month. Delete them after expiration.

• Some documents are frequently accessed for a period of time, but may no longer be accessed after a certain period of time. These documents need to be converted to low-frequency access storage, archived storage or deleted after a certain period of time.

Bucket lifecycle configuration:

public void setBucketLifecycle(SetBucketLifecycleArgs args)

Get the life cycle configuration of the bucket:

public LifecycleConfiguration getBucketLifecycle(GetBucketLifecycleArgs args) 

Sample code:

 // 5. 生命周期
        List<LifecycleRule> rules = new LinkedList<>();
        // 配置生命周期规则
        rules.add(
                new LifecycleRule(
                        Status.ENABLED, // 开启状态
                        null,
                        new Expiration((ZonedDateTime) null, 365, null), // 保存365天
                        new RuleFilter("logs/"), // 目录配置
                        "rule2",
                        null,
                        null,
                        null));
        LifecycleConfiguration lifecycleConfiguration = new LifecycleConfiguration(rules);
        // 添加生命周期配置
        minioClient.setBucketLifecycle(
                SetBucketLifecycleArgs.builder().bucket("my-bucketname").config(lifecycleConfiguration).build());
        // 获取配置
        LifecycleConfiguration lifecycleConfiguration1111 =
                minioClient.getBucketLifecycle(
                        GetBucketLifecycleArgs.builder().bucket("my-bucketname").build());
        List<LifecycleRule> rules1 = lifecycleConfiguration1111.rules();
        for (int i = 0; i < rules1.size(); i++) {
    
    
            System.out.println("Lifecycle status is " + rules1.get(i).status());
            System.out.println("Lifecycle prefix is " + rules1.get(i).filter().prefix());
            System.out.println("Lifecycle expiration days is " + rules1.get(i).expiration().days());
        }

The print result is as follows:

2.4.3 Notification configuration

You can use bucket event notifications to monitor events that occur on objects in a bucket.

The various event types supported by the MinIO server are:




Bucket configuration notifications:

public void setBucketPolicy(SetBucketPolicyArgs args)

Get the notification configuration of the bucket:

public NotificationConfiguration getBucketNotification(GetBucketNotificationArgs args)

Code example:

// 6. 通知配置 
        // Add a new SQS configuration.
        NotificationConfiguration notificationConfiguration = new NotificationConfiguration();
        List<QueueConfiguration> queueConfigurationList = notificationConfiguration.queueConfigurationList();
        QueueConfiguration queueConfiguration = new QueueConfiguration();
        queueConfiguration.setQueue("arn:minio:sqs::1:webhook");

        List<EventType> eventList = new LinkedList<>();
        eventList.add(EventType.OBJECT_CREATED_PUT);
        eventList.add(EventType.OBJECT_CREATED_COPY);
        queueConfiguration.setEvents(eventList);
        queueConfiguration.setPrefixRule("images");
        queueConfiguration.setSuffixRule("pg");

        queueConfigurationList.add(queueConfiguration);
        notificationConfiguration.setQueueConfigurationList(queueConfigurationList);

        // Set updated notification configuration.
        minioClient.setBucketNotification(
                SetBucketNotificationArgs.builder().bucket("my-bucketname").config(notificationConfiguration).build());
        System.out.println("Bucket notification is set successfully");

        NotificationConfiguration minioClientBucketNotification =
                minioClient.getBucketNotification(
                        GetBucketNotificationArgs.builder().bucket("my-bucketname").build());
        System.out.println(minioClientBucketNotification);

2.4.4 Policy configuration

Add bucket policy configuration.

public void setBucketPolicy(SetBucketPolicyArgs args) 

Get the bucket policy configuration of the bucket.

public String getBucketPolicy(GetBucketPolicyArgs args)

2.4.5 Copy configuration

Bucket copy aims to copy the selected objects in the bucket to the target bucket. There is more content and will be added later.

Add replication configuration for bucket

public void setBucketReplication(SetBucketReplicationArgs args)

Get the bucket replication configuration for a bucket:

public ReplicationConfiguration getBucketReplication(GetBucketReplicationArgs args)

2.4.6 Bucket Label

When tags are added to a bucket, these tags will be included in the billing bills generated by all requests in the bucket, so that the bill reports can be classified and filtered for more detailed cost analysis. For example: an application will upload data to a bucket when it is running. We can use the application name as a label and set it to the bucket being used. When analyzing the bill, you can analyze the cost of this application through the label of the application name.

setBucketTags can set tags for buckets.

public void setBucketTags(SetBucketTagsArgs args)

getBucketTags gets the tags of the bucket.

public Tags getBucketTags(GetBucketTagsArgs args)

Sample code:

        // 1. 存储桶标签
        Map<String, String> map = new HashMap<>();
        map.put("Project", "Project One");
        map.put("User", "jsmith");
        // 设置标签
        minioClient.setBucketTags(SetBucketTagsArgs.builder().bucket("my-bucketname").tags(map).build());
        // 查询标签
        Tags bucketTags = minioClient.getBucketTags(GetBucketTagsArgs.builder().bucket("my-bucketname").build());
        System.out.println(bucketTags.get().toString());

Return results:

2.4.7 Multi-version settings

If multi-version control is enabled, when uploading objects, OBS automatically creates a unique version number for each object. Uploaded objects with the same name will be saved in OBS with different version numbers at the same time.

If multi-version control is not enabled and an object with the same name is uploaded to the same folder, the newly uploaded object will overwrite the original object.

Certain features, such as versioning, object locking, and bucket replication, require distributed deployment of MinIO using erasure coding. When version control is turned on, multiple versions of the same object are allowed to be kept under the same key.

Set the bucket's versioning configuration.

public void setBucketVersioning(SetBucketVersioningArgs args)

Get the versioning configuration of a bucket.

public VersioningConfiguration getBucketVersioning(GetBucketVersioningArgs args)

Code example:

// 2. 版本配置
        // 'my-bucketname'启用版本控制
        minioClient.setBucketVersioning(
                SetBucketVersioningArgs.builder()
                        .bucket("my-bucketname")
                        .config(new VersioningConfiguration(VersioningConfiguration.Status.ENABLED, null))
                        .build());
        System.out.println("Bucket versioning is enabled successfully");

        //  'my-bucketname'暂停版本控制
        minioClient.setBucketVersioning(
                SetBucketVersioningArgs.builder()
                        .bucket("my-bucketname")
                        .config(new VersioningConfiguration(VersioningConfiguration.Status.SUSPENDED, null))
                        .build());
        System.out.println("Bucket versioning is suspended successfully");

2.4.8 Object locking configuration

After the object lock is set, the object will still exist on the disk after it is deleted.

Set the object locking configuration in the bucket.

public void setObjectLockConfiguration(SetObjectLockConfigurationArgs args) 

Get the object lock configuration in the bucket.

public ObjectLockConfiguration getObjectLockConfiguration(GetObjectLockConfigurationArgs args)

You need to set the bucket to object locking mode first, sample code:

// 3. 将保留模式设置为Compliance，且持续时间为100天
        // 设置锁定对象的保留模式及时限
        ObjectLockConfiguration config =
                new ObjectLockConfiguration(RetentionMode.COMPLIANCE, new RetentionDurationDays(100));
        minioClient.setObjectLockConfiguration(
                SetObjectLockConfigurationArgs.builder()
                        .bucket("my-bucketname-in-eu-with-object-lock")
                        .config(config)
                        .build());
        System.out.println("object-lock configuration is set successfully");
        // 获取锁定配置
        ObjectLockConfiguration objectLockConfiguration =
                minioClient.getObjectLockConfiguration(
                        GetObjectLockConfigurationArgs.builder()
                                .bucket("my-lock-enabled-bucketname")
                                .build());

        System.out.println("Object-lock configuration of bucket");
        System.out.println("Mode: " + objectLockConfiguration.mode());
        System.out.println("Duration: " + objectLockConfiguration.duration());

2.5 Delete configuration

minio provides a series of delete methods for deleting configurations, which are relatively simple and will not be explained with examples.

2.5.1 Delete the encryption configuration of the bucket

public void deleteBucketEncryption(DeleteBucketEncryptionArgs args)

2.5.2 Delete the life cycle configuration of a bucket

public void deleteBucketLifecycle(DeleteBucketLifecycleArgs args)

2.5.3 Delete bucket label

public void deleteBucketTags(DeleteBucketTagsArgs args)

2.5.4 Deleting the bucket policy configuration of a bucket

public void deleteBucketPolicy(DeleteBucketPolicyArgs args)

2.5.5 Deleting the bucket replication configuration of a bucket

public void deleteBucketReplication(DeleteBucketReplicationArgs args)

2.5.6 Deleting bucket notification configuration

public void deleteBucketNotification(DeleteBucketNotificationArgs args)

3 related tools

import io.minio.*;
import io.minio.errors.*;
import io.minio.messages.Bucket;

import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.List;

/**
 * minio工具类
 *
 * @author wuKeFan
 * @date 2023-09-08 14:08:10
 */
public class MinioUtil {
    
    

    /**
     * 判断桶是否存在
     */
    public static boolean bucketExists(String url, String accessKey, String secretKey, String bucketName)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
    }

    /**
     * 添加存储桶
     */
    public static void makeBucket(String url, String accessKey, String secretKey, String bucketName, String region, boolean objectLock)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).region(region).objectLock(objectLock).build());
    }

    /**
     * 指定地区添加存储桶
     */
    public static void makeBucket(String url, String accessKey, String secretKey, String bucketName)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
    }

    /**
     * 指定地区添加存储桶并锁定对象
     */
    public static void makeBucket(String url, String accessKey, String secretKey, String bucketName, String region)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).region(region).build());
    }

    /**
     * 删除存储桶
     */
    public static void removeBucket(String url, String accessKey, String secretKey, String bucketName)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());
    }

    /**
     * 设置桶公有
     */
    public static void setBucketPublicPolicy(String url, String accessKey, String secretKey, String bucketName)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        String sb = "{\"Version\":\"2012-10-17\"," +
                "\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":" +
                "{\"AWS\":[\"*\"]},\"Action\":[\"s3:ListBucket\",\"s3:ListBucketMultipartUploads\"," +
                "\"s3:GetBucketLocation\"],\"Resource\":[\"arn:aws:s3:::" + bucketName +
                "\"]},{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"*\"]},\"Action\":[\"s3:PutObject\",\"s3:AbortMultipartUpload\",\"s3:DeleteObject\",\"s3:GetObject\",\"s3:ListMultipartUploadParts\"],\"Resource\":[\"arn:aws:s3:::" +
                bucketName +
                "/*\"]}]}";
        minioClient.setBucketPolicy(
                SetBucketPolicyArgs.builder()
                        .bucket(bucketName)
                        .config(sb)
                        .build());
    }

    /**
     * 设置桶私有
     */
    public static void setBucketPrivatePolicy(String url, String accessKey, String secretKey, String bucketName)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        minioClient.setBucketPolicy(
                SetBucketPolicyArgs.builder().bucket(bucketName)
                        .config(
                                "{\"Version\":\"2012-10-17\",\"Statement\":[]}"
                        )
                        .build());
    }

    /**
     * 查询存储桶信息列表
     */
    public static List<Bucket> listBuckets(String url, String accessKey, String secretKey) throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
    
    
        MinioClient minioClient = MinioClient.builder().endpoint(url)
                .credentials(accessKey, secretKey).build();
        return minioClient.listBuckets();
    }

}