Amazon Simple Storage Service (Amazon S3)是面向 Internet 的存储服务，具有高扩展性、可靠性、安全性和快速价廉的特点，提供 99.999999999% 的持久性，可存储无限量的数据，每个对象最多包含 5 TB 的数据。S3支持版本控制、对象生命周期管理、加密、静态网站托管、Select SQL查询等。
AWS JAVA SDK
S3 架构设计与编程语言无关，提供 REST 和 SOAP 接口。HTTP 上的 SOAP 支持已弃用，但仍可在 HTTPS 上使用。SOAP 将不支持新 S3 功能，建议使用 REST API。
借助 REST，可以使用标准的 HTTP 请求创建、提取和删除存储桶和对象。直接利用REST API进行代码开发是复杂的，AWS SDK包装了底层REST API，可以简化编程任务。
配置AWS Credentials
为使用AWS SDK，必须提供AWS凭证，在 ~/.aws/credentials (Windows 用户为 C:\Users\USER_NAME.aws\credentials) 中创建：
[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_keyPOM
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bom</artifactId>
<version>1.11.433</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>如要使用全部的SDK，不需使用BOM，简单声明如下：
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.433</version>
</dependency>
</dependencies>S3基本操作
下面代码摘自https://github.com/awslabs/aws-java-sample.git ，演示了createBucket、listBuckets、putObject、getObject、listObjects、deleteObject、deleteBucket等S3基本操作。
import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.*;
import java.io.*;
import java.util.UUID;
/**
* This sample demonstrates how to make basic requests to Amazon S3 using the AWS SDK for Java.
* <p>
* <b>Prerequisites:</b> You must have a valid Amazon Web Services developer account, and be signed up to use Amazon S3.
* For more information on Amazon S3, see http://aws.amazon.com/s3.
* <p>
* <b>Important:</b> Be sure to fill in your AWS access credentials in ~/.aws/credentials (C:\Users\USER_NAME\.aws\credentials for Windows
* users) before you try to run this sample.
*/
public class S3Sample {
public static void main(String[] args) throws IOException {
/*
* Create your credentials file at ~/.aws/credentials (C:\Users\USER_NAME\.aws\credentials for Windows users)
* and save the following lines after replacing the underlined values with your own.
*
* [default]
* aws_access_key_id = YOUR_ACCESS_KEY_ID
* aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
*/
AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();
String bucketName = "my-first-s3-bucket-" + UUID.randomUUID();
String key = "MyObjectKey";
System.out.println("===========================================");
System.out.println("Getting Started with Amazon S3");
System.out.println("===========================================\n");
try {
/*
* Create a new S3 bucket - Amazon S3 bucket names are globally unique, so once a bucket name has been taken by any user, you can't create
* another bucket with that same name.
*
* You can optionally specify a location for your bucket if you want to keep your data closer to your applications or users.
*/
System.out.println("Creating bucket " + bucketName + "\n");
s3.createBucket(bucketName);
/*
* List the buckets in your account
*/
System.out.println("Listing buckets");
for (Bucket bucket : s3.listBuckets()) {
System.out.println(" - " + bucket.getName());
}
System.out.println();
/*
* Upload an object to your bucket - You can easily upload a file to S3, or upload directly an InputStream if you know the length of
* the data in the stream. You can also specify your own metadata when uploading to S3, which allows you set a variety of options
* like content-type and content-encoding, plus additional metadata specific to your applications.
*/
System.out.println("Uploading a new object to S3 from a file\n");
s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile()));
/*
* Download an object - When you download an object, you get all of the object's metadata and a stream from which to read the contents.
* It's important to read the contents of the stream as quickly as possibly since the data is streamed directly from Amazon S3 and your
* network connection will remain open until you read all the data or close the input stream.
*
* GetObjectRequest also supports several other options, including conditional downloading of objects based on modification times,
* ETags, and selectively downloading a range of an object.
*/
System.out.println("Downloading an object");
S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
System.out.println("Content-Type: " + object.getObjectMetadata().getContentType());
displayTextInputStream(object.getObjectContent());
/*
* List objects in your bucket by prefix - There are many options for listing the objects in your bucket. Keep in mind that buckets with
* many objects might truncate their results when listing their objects, so be sure to check if the returned object listing is truncated, and
* use the AmazonS3.listNextBatchOfObjects(...) operation to retrieve additional results.
*/
System.out.println("Listing objects");
ObjectListing objectListing = s3.listObjects(new ListObjectsRequest().withBucketName(bucketName).withPrefix("My"));
for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
System.out.println(" - " + objectSummary.getKey() + " " + "(size = " + objectSummary.getSize() + ")");
}
System.out.println();
/*
* Delete an object - Unless versioning has been turned on for your bucket, there is no way to undelete an object, so use caution when deleting objects.
*/
System.out.println("Deleting an object\n");
s3.deleteObject(bucketName, key);
/*
* Delete a bucket - A bucket must be completely empty before it can be deleted, so remember to delete any objects from your buckets before
* you try to delete them.
*/
System.out.println("Deleting bucket " + bucketName + "\n");
s3.deleteBucket(bucketName);
} catch (AmazonServiceException ase) {
System.out.println("Caught an AmazonServiceException, which means your request made it "
+ "to Amazon S3, but was rejected with an error response for some reason.");
System.out.println("Error Message: " + ase.getMessage());
System.out.println("HTTP Status Code: " + ase.getStatusCode());
System.out.println("AWS Error Code: " + ase.getErrorCode());
System.out.println("Error Type: " + ase.getErrorType());
System.out.println("Request ID: " + ase.getRequestId());
} catch (AmazonClientException ace) {
System.out.println("Caught an AmazonClientException, which means the client encountered "
+ "a serious internal problem while trying to communicate with S3, "
+ "such as not being able to access the network.");
System.out.println("Error Message: " + ace.getMessage());
}
}
/**
* Creates a temporary file with text data to demonstrate uploading a file to Amazon S3
*
* @return A newly created temporary file with text data.
* @throws IOException
*/
private static File createSampleFile() throws IOException {
File file = File.createTempFile("aws-java-sdk-", ".txt");
file.deleteOnExit();
Writer writer = new OutputStreamWriter(new FileOutputStream(file));
writer.write("abcdefghijklmnopqrstuvwxyz\n");
writer.write("01234567890112345678901234\n");
writer.write("!@#$%^&*()-=[]{};':',.<>/?\n");
writer.write("01234567890112345678901234\n");
writer.write("abcdefghijklmnopqrstuvwxyz\n");
writer.close();
return file;
}
/**
* Displays the contents of the specified input stream as text.
*
* @param input The input stream to display as text.
* @throws IOException
*/
private static void displayTextInputStream(InputStream input) throws IOException {
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
while (true) {
String line = reader.readLine();
if (line == null) break;
System.out.println(" " + line);
}
System.out.println();
}
}从对象中选择内容
利用 Amazon S3 Select，可以使用SQL语句筛选 S3 对象的内容，检索所需的部分数据。Amazon S3 Select 适用于以 CSV 或 JSON 格式存储的对象，这些对象可以通过 GZIP 或 BZIP2 压缩和服务器端加密。
Amazon S3 Select 支持一部分 SQL，有关支持的SQL的更多信息，请参阅Amazon S3 Select 和 Amazon Glacier Select 的 SQL 参考。
Amazon S3 Select的要求和限制
要求：
- 必须拥有所查询的对象的 s3:GetObject 权限。
- 如果查询的对象已进行加密，则必须使用 https，并且必须在请求中提供加密密钥。
限制：
- SQL 表达式的最大长度为 256 KB。
- 结果中记录的最大长度为 1 MB。
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.*;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.concurrent.atomic.AtomicBoolean;
import static com.amazonaws.util.IOUtils.copy;
/**
* This example shows how to query data from S3Select and consume the response in the form of an
* InputStream of records and write it to a file.
*/
public class RecordInputStreamExample {
private static final String BUCKET_NAME = "${my-s3-bucket}";
private static final String CSV_OBJECT_KEY = "${my-csv-object-key}";
private static final String S3_SELECT_RESULTS_PATH = "${my-s3-select-results-path}";
private static final String QUERY = "select s._1 from S3Object s";
public static void main(String[] args) throws Exception {
final AmazonS3 s3Client = AmazonS3ClientBuilder.defaultClient();
SelectObjectContentRequest request = generateBaseCSVRequest(BUCKET_NAME, CSV_OBJECT_KEY, QUERY);
final AtomicBoolean isResultComplete = new AtomicBoolean(false);
try (OutputStream fileOutputStream = new FileOutputStream(new File(S3_SELECT_RESULTS_PATH));
SelectObjectContentResult result = s3Client.selectObjectContent(request)) {
InputStream resultInputStream = result.getPayload().getRecordsInputStream(
new SelectObjectContentEventVisitor() {
@Override
public void visit(SelectObjectContentEvent.StatsEvent event) {
System.out.println("Received Stats, Bytes Scanned: " + event.getDetails().getBytesScanned()
+ " Bytes Processed: " + event.getDetails().getBytesProcessed());
}
/*
* An End Event informs that the request has finished successfully.
*/
@Override
public void visit(SelectObjectContentEvent.EndEvent event) {
isResultComplete.set(true);
System.out.println("Received End Event. Result is complete.");
}
}
);
copy(resultInputStream, fileOutputStream);
}
/*
* The End Event indicates all matching records have been transmitted.
* If the End Event is not received, the results may be incomplete.
*/
if (!isResultComplete.get()) {
throw new Exception("S3 Select request was incomplete as End Event was not received.");
}
}
private static SelectObjectContentRequest generateBaseCSVRequest(String bucket, String key, String query) {
SelectObjectContentRequest request = new SelectObjectContentRequest();
request.setBucketName(bucket);
request.setKey(key);
request.setExpression(query);
request.setExpressionType(ExpressionType.SQL);
InputSerialization inputSerialization = new InputSerialization();
inputSerialization.setCsv(new CSVInput());
inputSerialization.setCompressionType(CompressionType.NONE);
request.setInputSerialization(inputSerialization);
OutputSerialization outputSerialization = new OutputSerialization();
outputSerialization.setCsv(new CSVOutput());
request.setOutputSerialization(outputSerialization);
return request;
}
}生成预签名URL
默认，S3对象为私有，只有所有者具有访问权限。但是，对象所有者可以使用自己的安全凭证来创建预签名的 URL，授予有限时间内的对象下载许可，从而与其他用户共享对象，收到预签名 URL 的任何人都可以访问对象。
当创建预签名URL时，必须提供安全凭证、存储桶名称和对象键、指定 HTTP 方法 (指定为 GET 以下载对象) 和过期日期和时间。
import com.amazonaws.AmazonServiceException;
import com.amazonaws.HttpMethod;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import java.io.IOException;
import java.net.URL;
import java.util.Date;
public class GeneratePresignedURL {
public static void main(String[] args) throws IOException {
String bucketName = "*** Bucket name ***";
String objectKey = "*** Object key ***";
try {
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.CN_NORTH_1)
.withCredentials(new ProfileCredentialsProvider())
.build();
// Set the presigned URL to expire after one hour.
Date expiration = new Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
// Generate the presigned URL.
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey)
.withMethod(HttpMethod.GET).withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toString());
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process it, so it returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client couldn't parse the response from Amazon S3.
e.printStackTrace();
}
}
}参考文档
Amazon Simple Storage Service Documentation
Working with Amazon S3 Objects
Using the SDK
Programming Examples
Generate a Pre-signed Object URL using AWS SDK for Java