Large-scale website architecture--resource storage solution [Alibaba OSS storage service] introduction and implementation

In a large-scale website architecture, resource storage is very important. The website will definitely involve resource storage operations such as file uploading and image uploading. As the number of website users continues to grow, the resource storage of the website will also increase server consumption. Hard disk, the general architecture will set up a separate static resource server, but if it is the initial stage of the e-commerce website architecture, the T-level server is also very expensive. Therefore, Ali saw a business opportunity and used its own resource storage oss solution for Taobao for commercial use. Users You can register to purchase oss services, and then store your own static resources on oss by calling Alioss api.

Alioss product introduction:

Alibaba Cloud Object Storage Service (OSS) is a massive, secure, low-cost, and highly reliable cloud storage service provided by Alibaba Cloud. You can upload and download data in any application, at any time, and anywhere by calling the API, or simply manage the data through the web console. OSS is suitable for storing any type of files and is suitable for various websites, development companies and developers.

Comparison between OSS and self-built storage

Contrast

Object Storage OSS

Self-built server storage

reliability	- Service availability is not less than 99.9%. - The scale is automatically expanded without affecting external services. - Data durability is not less than 99.99999999%. - Automatic multiple redundant backup of data.	- Limited by hardware reliability, it is prone to problems. Once the disk is bad, it is prone to irreversible data loss. - Manual data recovery is difficult, time-consuming and labor-intensive.
Safety	- Provide enterprise-level multi-layer security protection. - Multi-user resource isolation mechanism to support remote disaster recovery mechanism. - Provides a variety of authentication and authorization mechanisms, as well as whitelist, anti-leech, master and sub-account functions.	- Requires additional purchase of cleaning and black hole equipment. - Security mechanisms need to be implemented separately.
cost	- The minimum cost is only 0.14 yuan/GB/month, and there is a free quota every month. - Multi-line BGP backbone network, no bandwidth limitation, free upstream traffic. - No operation and maintenance personnel and hosting fees, 0 cost operation and maintenance.	- The storage is limited by the capacity of the hard disk and needs to be manually expanded. - Single-line or dual-line access is slow, has bandwidth limitations, and needs to be manually expanded during peak periods. - Special personnel are required for operation and maintenance, and the cost is high.
data processing capability	- Provides a variety of data value-added services such as image processing, audio and video transcoding, accelerated content distribution, pornographic identification services, and archiving services, which are constantly enriched.	- Requires additional purchase and is deployed separately.

Convenient and fast way to use

Provides standard RESTful API interfaces, rich SDK packages, client tools, and consoles. You can upload, download, retrieve, and manage massive amounts of data for web sites or mobile applications as easily as files.
Unlimited number and size of files. You can expand the storage space indefinitely according to the required storage, which solves the problem of traditional hardware storage expansion.
Streaming writes and reads are supported. It is especially suitable for writing and reading business scenarios of large files such as videos.
Support data lifecycle management. You can customize the batch deletion of expired data or transfer it to a low-cost archiving service.

Powerful and flexible security mechanism

Flexible authentication and authorization mechanism. Provides STS and URL authentication and authorization mechanisms, as well as whitelist, anti-leech, master and sub-account functions.
Provides user-level resource isolation mechanism and multi-cluster synchronization mechanism (optional).

Rich and powerful value-added services

Image processing: supports the conversion of jpg, png, bmp, gif, webp, tiff and other image formats, as well as thumbnail, crop, watermark, zoom and other operations.
Audio and video transcoding: Provide high-quality, high-speed parallel audio and video transcoding capabilities, allowing your audio and video files to easily cope with various terminal devices.
Accelerated distribution of content: OSS, as an origin site, works with CDN for accelerated distribution, featuring stability, no back-to-source bandwidth limitation, high cost performance, and one-click configuration.

Let's talk about uploading and calling OSS:

Simulation requirements:

Information maintenance, information can be modified, including two image processing: 1. Public account QR code image upload. 2. Logo upload.

Front page code:

<%@ page contentType="text/html;charset=UTF-8"%>

<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<%@ taglib prefix="shiro" uri="http://shiro.apache.org/tags" %>
<%@ taglib prefix="tags" tagdir="/WEB-INF/tags" %>

<c:set var="ctx" value="${pageContext.request.contextPath}"/>

<html>
<head>
<title>管理</title>
</head>
<body>

<section class="content-header">
<h1>

<small>
修改
</small>
</h1>
<ol class="breadcrumb">
<li><a href="${ctx}"><i class="fa fa-dashboard"></i> 首页</a></li>
<li class="active">
信息
</li>
</ol>
</section>

<section class="content">
<c:if test="${message != null && message != ''}">
<div class="alert alert-info" style="margin-top:20px;">
<button type="button" class="close" data-dismiss="alert">×</button>
${message}
</div>
</c:if>
<div class="row">
<div class="col-xs-12">
<div class="box">
<div class="box-header"></div>

<form id="inputForm" class="form-horizontal" action="${ctx}/org/hospital/update" method="post" enctype="multipart/form-data">
<input type="hidden" name="id" value="${entity.id }"/>
<input type="hidden" name="token" value="${entity.token }"/>
<input type="hidden" name="status" value="${entity.status }"/>
<input type="hidden" id="logoUrl" name="logoUrl" value="${entity.logoUrl}"/>
<input type="hidden" id="codeUrl" name="codeUrl" value="${entity.codeUrl}"/>

<div class="form-group">
<label class="col-sm-2 control-label">名称</label>
<div class="col-sm-3">
<input id="name" name="name" value="${entity.name }" type="text" class="form-control required" readonly="readonly"/>
</div>
</div>
<div class="form-group">
<label class="col-sm-2 control-label">地址</label>
<div class="col-sm-3">
<input name="address" value="${entity.address }" type="text" class="form-control required" />
</div>
</div>
<div class="form-group">
<label class="col-sm-2 control-label">电话</label>
<div class="col-sm-3">
<input name="telphone" value="${entity.telphone }" type="text" class="form-control required number" />
</div>
</div>
<div class="form-group">
<label class="col-sm-2 control-label">logo图片</label>
<div class="col-sm-5">

<input type="file" name="logoFile" id="logoFile">
<a href="javascript:uploadFile();" class="btn btn-primary">上传</a>

</div>
</div>
<div class="form-group">
<label class="col-sm-2 control-label">logo缩略图</label>
<div class="col-sm-3">
<c:if test="${empty entity.logoUrl}">
<img id="logo_img" alt="" src="${ctx }/static/images/no_pro.jpg" width="40px" height="40px"/>
</c:if>
<c:if test="${!empty entity.logoUrl}">
<img id="logo_img" alt="" src="${entity.logoUrl}" width="40px" height="40px"/>
</c:if>
</div>
</div>
<div class="form-group">
<label class="col-sm-2 control-label">微信二维码</label>
<div class="col-sm-5">
<input type="file" name="codeFile" id="codeFile">
<a href="javascript:uploadCodeFile();" class="btn btn-primary">上传</a>
</div>
</div>
<div class="form-group">
<label class="col-sm-2 control-label">二维码缩略图</label>
<div class="col-sm-3">
<c:if test="${empty entity.codeUrl}">
<img id="code_img" alt="" src="${ctx }/static/images/no_pro.jpg" width="140px" height="140px"/>
</c:if>
<c:if test="${!empty entity.codeUrl}">
<img id="code_img" alt="" src="${entity.codeUrl}" width="140px" height="140px"/>
</c:if>
</div>
</div>

<div class="form-group">
<label class="col-sm-2 control-label">简介</label>
<div class="col-sm-3">
<textarea class="form-control" name="summary">${entity.summary }</textarea>
</div>
</div>

<div class="form-group">
<div class="col-sm-offset-2 col-sm-10">
<shiro:hasPermission name="hospital:update">
<button id="submit" class="btn btn-primary">提交</button>
<a href="${ctx}/" class="btn btn-default">取消</a>
</shiro:hasPermission>
</div>
</div>
</form>
<div class="box-footer clearfix"></div>
</div>
</div>
</div>
</section>

<script type="text/javascript">
$(document).ready(function() {
$("#inputForm").validate();

//$('#fileButton').click(function (){

//})
});
function uploadFile(){
//alert();
//判断是否有选择上传文件
var imgPath = $("#logoFile").val();
if (imgPath == "") {
layer.alert("请选择上传图片！");
return;
}
//判断上传文件的后缀名
var strExtension = imgPath.substr(imgPath.lastIndexOf('.') + 1);
if (strExtension != 'jpg' && strExtension != 'gif'
&& strExtension != 'png' && strExtension != 'bmp') {
layer.alert("请选择图片文件!");
return;
}
$("#inputForm").ajaxSubmit({
type: "POST",
url:"${ctx}/org/hospital/uploadLogo",
dataType: "text",
success: function(data){
//alert(11111);
if(data == ""){
layer.alert("上传失败，请检查网络后重试");
return;
}

$("#logoUrl").val(data);
$("#logo_img").prop("src",data);
layer.alert("上传成功");
//layer.alert(data);

}
});
}

function uploadCodeFile(){
//alert();
//判断是否有选择上传文件
var imgPath = $("#codeFile").val();
if (imgPath == "") {
layer.alert("请选择上传图片！");
return;
}
//判断上传文件的后缀名
var strExtension = imgPath.substr(imgPath.lastIndexOf('.') + 1);
if (strExtension != 'jpg' && strExtension != 'gif'
&& strExtension != 'png' && strExtension != 'bmp') {
layer.alert("请选择图片文件!");
return;
}
$("#inputForm").ajaxSubmit({
type: "POST",
url:"${ctx}/org/hospital/uploadCode",
dataType: "text",
success: function(data){
if(data == ""){
layer.alert("上传失败，请检查网络后重试");
return;
}

$("#codeUrl").val(data);
$("#code_img").prop("src",data);
layer.alert("上传成功");
//layer.alert(data);
}
});
}
</script>
</body>
</html>

springmvc 代码：

/**
* 上传logo
*/
@RequestMapping(value = "uploadLogo", method = RequestMethod.POST)
@ResponseBody
public String uploadLogo( @RequestParam(value = "logoFile", required = false) MultipartFile logoFile,
HttpServletRequest request, HttpServletResponse response) {
if (request instanceof MultipartHttpServletRequest) {
String logoUrl = null;
try {
OSSUtil ossUtil = new OSSUtil();
String name = ossUtil.uploadImg2Oss(logoFile);
logoUrl = ossUtil.getImgUrl(name);
ossUtil.destory();
return logoUrl;
} catch (Exception e) {
//e.printStackTrace();
return "";
}
}
else {
return "";
}
}

阿里云OSS工具类代码OSSUtil

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.util.Date;
import java.util.Random;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.util.StringUtils;
import org.springframework.web.multipart.MultipartFile;

import com.aliyun.oss.OSSClient;
import com.aliyun.oss.model.ObjectMetadata;
import com.aliyun.oss.model.PutObjectResult;

/**
* 阿里云 OSS文件类
*
* @author gshen
*/
public class OSSUtil {

Log log = LogFactory.getLog(OSSUtil.class);
// endpoint以杭州为例，其它region请按实际情况填写

// http://oss-cn-beijing.aliyuncs.com/
private String endpoint = "oss url";
// accessKey请登录https://ak-console.aliyun.com/#/查看
private String accessKeyId = "accessKeyId";
private String accessKeySecret = "accessKeySecret";
//空间
private String bucketName = "gshen_workspace";
//文件存储目录
private String filedir = "data/";

private OSSClient ossClient;

public OSSUtil() {
ossClient = new OSSClient(endpoint, accessKeyId, accessKeySecret);
}

/**
* 初始化
*/
public void init() {
ossClient = new OSSClient(endpoint, accessKeyId, accessKeySecret);
}

/**
* 销毁
*/
public void destory() {
ossClient.shutdown();
}

/**
* 上传图片
*
* @param url
*/
public void uploadImg2Oss(String url) {
File fileOnServer = new File(url);
FileInputStream fin;
try {
fin = new FileInputStream(fileOnServer);
String[] split = url.split("/");
this.uploadFile2OSS(fin, split[split.length - 1]);
} catch (FileNotFoundException e) {
throw new RuntimeException("图片上传失败");
}
}

public String uploadImg2Oss(MultipartFile file) {
if (file.getSize() > 1024 * 1024 * 5) {
throw new RuntimeException("上传图片大小不能超过5M！");
}
String originalFilename = file.getOriginalFilename();
String substring = originalFilename.substring(originalFilename.lastIndexOf(".")).toLowerCase();
Random random = new Random();
String name = random.nextInt(10000) + System.currentTimeMillis() + substring;
try {
InputStream inputStream = file.getInputStream();
this.uploadFile2OSS(inputStream, name);
return name;
} catch (Exception e) {
throw new RuntimeException("图片上传失败");
}
}

/**
* 获得图片路径
*
* @param fileUrl
* @return
*/
public String getImgUrl(String fileUrl) {
if (!StringUtils.isEmpty(fileUrl)) {
String[] split = fileUrl.split("/");
return this.getUrl(this.filedir + split[split.length - 1]);
}
return null;
}

/**
* 上传到OSS服务器如果同名文件会覆盖服务器上的
*
* @param instream 文件流
* @param fileName 文件名称包括后缀名
* @return 出错返回"" ,唯一MD5数字签名
*/
public String uploadFile2OSS(InputStream instream, String fileName) {
String ret = "";
try {
//创建上传Object的Metadata
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(instream.available());
objectMetadata.setCacheControl("no-cache");
objectMetadata.setHeader("Pragma", "no-cache");
objectMetadata.setContentType(getcontentType(fileName.substring(fileName.lastIndexOf("."))));
objectMetadata.setContentDisposition("inline;filename=" + fileName);
//上传文件
PutObjectResult putResult = ossClient.putObject(bucketName, filedir + fileName, instream, objectMetadata);
ret = putResult.getETag();
} catch (IOException e) {
log.error(e.getMessage(), e);
} finally {
try {
if (instream != null) {
instream.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
return ret;
}

/**
* Description: 判断OSS服务文件上传时文件的contentType
*
* @param FilenameExtension 文件后缀
* @return String
*/
public static String getcontentType(String FilenameExtension) {
if (FilenameExtension.equalsIgnoreCase("bmp")) {
return "image/bmp";
}
if (FilenameExtension.equalsIgnoreCase("gif")) {
return "image/gif";
}
if (FilenameExtension.equalsIgnoreCase("jpeg") ||
FilenameExtension.equalsIgnoreCase("jpg") ||
FilenameExtension.equalsIgnoreCase("png")) {
return "image/jpeg";
}
if (FilenameExtension.equalsIgnoreCase("html")) {
return "text/html";
}
if (FilenameExtension.equalsIgnoreCase("txt")) {
return "text/plain";
}
if (FilenameExtension.equalsIgnoreCase("vsd")) {
return "application/vnd.visio";
}
if (FilenameExtension.equalsIgnoreCase("pptx") ||
FilenameExtension.equalsIgnoreCase("ppt")) {
return "application/vnd.ms-powerpoint";
}
if (FilenameExtension.equalsIgnoreCase("docx") ||
FilenameExtension.equalsIgnoreCase("doc")) {
return "application/msword";
}
if (FilenameExtension.equalsIgnoreCase("xml")) {
return "text/xml";
}
return "image/jpeg";
}

/**
* 获得url链接
*
* @param key
* @return
*/
public String getUrl(String key) {
// 设置URL过期时间为10年 3600l* 1000*24*365*10
Date expiration = new Date(new Date().getTime() + 3600l * 1000 * 24 * 365 * 10);
// 生成URL
URL url = ossClient.generatePresignedUrl(bucketName, key, expiration);
if (url != null) {
return url.toString();
}
return null;
}
}

图片上传成功后预览：