Use boto3 to call S3 object storage

concept explanation

• S3 object storage

  Object storage service following the Amazon S3 protocol

• boto3

  Based on the s3 protocol, it implements the Python SDK of the client API, which is used to call the object storage service

• Endpoint

  The "Endpoint" we use in the code calling the s3 object storage refers to the access domain name of the external service of the object storage

• Bucket

  Bucket refers to the storage space on object storage, which can be understood as a "container" for storing objects. A user who uses object storage can have multiple buckets. Mainstream cloud service providers will support users to configure access rights to specific Buckets.

• Object

  Each file stored by the user on the object storage is an Object, and each Object has a key name/key value. Objects can be text, pictures, audio, video, or web pages.

  When using the simple object upload method, the object size is limited within 5GB.

  When uploading in parts, the size of each block is limited to 5GB, and the number of parts needs to be less than 10,000, that is, the maximum upload object is about 48.82TB.

There are no folders in S3. It's just that most of the available S3 explorer tools show part of the key name, separated by slashes for folders.

Install boto3

pip install

pip install boto3

Source installation

git clone https://github.com/boto/boto3.git && cd boto3 && sudo python setup.py install

use boto3

Next, let's demonstrate with code how to use boto3

# 初始化 client
access_key = 'AKID10zuoeooooooooooooooooooooo' # ak
secret_key = 'nlbPCvHWkoooooooooooooooooooooo'     # sk
s3_host = 'cos.ap-hangzhou.myqcloud.com'           # Endpoint
s3client = boto3.client('s3',aws_secret_access_key = secret_key,aws_access_key_id = access_key,endpoint_url = s3_host)

# 遍历 bucket
list_buckets = s3client.list_buckets()
print(list_buckets)


# 遍历 bucket下的文件（对象）
bucket_name = 'cxy-1300123456'
file_list = []

response = s3client.list_objects_v2(
    Bucket=bucket_name,
    # MaxKeys=1000  # 返回数量，如果为空则为全部
)
file_desc = response['Contents']
for f in file_desc:
    print('file_name:{},file_size:{}'.format(f['Key'], f['Size']))
    file_list.append(f['Key'])
print(file_list)

bucket_name = "chenxy-1300123456"
local_file = "./tmp/hello.txt"
oss_key = "hello.txt"

# 上传文件
s3client.upload_file(local_file, bucket_name, oss_key)

# 下载文件
s3client.download_file(bucket_name, oss_key, local_file)

For more methods supported by boto3, see: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html?highlight=put_object#client