Distributed file storage
1. Introduction to FastDFS
FastDFS is an open source and lightweight. It manages files. Its functions include file storage, file synchronization, file access (file upload, file download), etc., which solves the problem of mass storage and load balancing. Especially suitable for online services that use files as the carrier, such as photo album websites, video websites, and so on.
FastDFS is tailor-made for the Internet, fully considering mechanisms such as redundant backup, load balancing, linear expansion, and focusing on high availability, high performance and other indicators. It is easy to build a high-performance file server cluster with FastDFS to provide file upload and download And other services.
The FastDFS architecture includes Tracker server and Storage server. The client requests the Tracker server to upload and download files, and the Storage server completes the file upload and download through the Tracker server scheduling.
The tracker server is used for load balancing and scheduling. The Tracker server can find the Storage server to provide file upload services according to some strategies when uploading files. The tracker can be called a tracking server or a dispatch server. The function of the Storage server is file storage. The files uploaded by the client are finally stored on the Storage server. The Storage server does not implement its own file system but uses the file system of the operating system to manage files. Storage can be called a storage server.
Second, the code
Create the changgou-service-file submodule under the changgou-service module.
1.pom file
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>mychanggou-service</artifactId>
<groupId>com.mychanggou</groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>mycahnggou-service-file</artifactId>
<dependencies>
<!-- FastDFS -->
<dependency>
<groupId>net.oschina.zcx7878</groupId>
<artifactId>fastdfs-client-java</artifactId>
<version>1.27.0.0</version>
</dependency>
<dependency>
<groupId>com.mychanggou</groupId>
<artifactId>mychanggou-common</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
</dependencies>
</project>
2. The main startup class
package com.mychanggou;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
/**
* @author ：LiuShihao
* @date ：Created in 2020/9/17 8:20 下午
* @desc ：
* 启动报错 ： 没有数据库信息配置， 因为不需要用到数据库，所以排除数据库自动加载
*/
@SpringBootApplication(exclude = DataSourceAutoConfiguration.class)
@EnableEurekaClient
public class FileApplication {
public static void main(String[] args) {
SpringApplication.run(FileApplication.class);
}
}
3.FastDFS configuration file information
Create new under the resources folderfdfs_client.conf
connect_timeout = 60
network_timeout = 60
charset = UTF-8
http.tracker_http_port = 80
tracker_server = 116.62.13.104:22122
4. File information package
File uploads generally have file name, file content, file extension, file md5 value, file author and other related attributes. We can create an object to encapsulate these attributes.
package com.mychanggou.file;
import lombok.Data;
/**
* @author ：LiuShihao
* @date ：Created in 2020/9/17 8:24 下午
* @desc ：
* 封装文件上传信息
* 时间：
* Author：
* type：
* size：
* 附加信息：
*/
@Data
public class FastDFSFile {
//文件名字
private String name;
//文件内容
private byte[] content;
//文件扩展名
private String ext;
//文件MD5摘要值
private String md5;
//文件创建作者
private String author;
public FastDFSFile(String name, byte[] content, String ext, String height,String width, String author) {
super();
this.name = name;
this.content = content;
this.ext = ext;
this.author = author;
}
public FastDFSFile(String name, byte[] content, String ext) {
super();
this.name = name;
this.content = content;
this.ext = ext;
}
}
5. Package FastDFS tools
package com.mychanggou.util;
import com.mychanggou.file.FastDFSFile;
import org.csource.fastdfs.*;
import org.springframework.core.io.ClassPathResource;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
/**
* @author ：LiuShihao
* @date ：Created in 2020/9/17 8:29 下午
* @desc ：实现文件管理
* 文件上传
* 文件下载
* 文件删除
* 文件信息获取
* Storage信息获取
* Tracker信息获取
*
*
*/
public class FastDFSUtil {
//加载Tracker连接信息
static {
try {
//String fdfsconfpath = "/Users/LiuShihao/IdeaProjects/mychanggou/mychanggou-service/mycahnggou-service-file/src/main/resources/fdfs_client.conf";
//查找classpath下的文件的路径
String path = new ClassPathResource("fdfs_client.conf").getPath();
ClientGlobal.init(path);
}catch (Exception e){
e.printStackTrace();
}
}
/**
* 文件上传
* @param fastDFSFile 上传的文件信息封装
*/
public static String[] upload(FastDFSFile fastDFSFile) throws Exception {
TrackerServer trackerServer = getTrackerServer();
StorageClient storageClient = getStorageClient(trackerServer);
/**
* 通过StorageClient访问Storage,实现文件上传，并获取文件上传后的存储信息
* 1.上传文件的字节数组
* 2. 上传文件的扩展名
* 3. 上传文件的附加信息
*/
//附加信息： 创建文件附加信息 可以用于文件上传的第三个参数 比如 Author：lsh 拍摄地址：上海 拍摄终端：iPhone 11 Pro Max
// NameValuePair[] meta_list = new NameValuePair[1];
// meta_list[0] = new NameValuePair("name",fastDFSFile.getAuthor());
String[] upload = storageClient.upload_file(fastDFSFile.getContent(), fastDFSFile.getExt(), null);
/**
* upload_file返回参数：upload[]
* 1. upload[0]：文件上传所存储的组名 group1
* 2. upload[1]：文件存储到Storage上的文件名字 M00/02/14/wyt.jpg
* 也可以获得StorageClient1客户端 直接使用upload_file1 方法
*/
return upload;
}
/**
* 获取文件信息
* @param groupName group1
* @param remoteFileName M00/00/00/rB52RF9jaiSAP9oLAAFDcbOOQqw66.jpeg
* @return
* @throws Exception
*/
public static FileInfo getFileInfo(String groupName, String remoteFileName) throws Exception {
TrackerServer trackerServer = getTrackerServer();
StorageClient storageClient = getStorageClient(trackerServer);
return storageClient.get_file_info(groupName,remoteFileName);
}
/**
* 下载文件
* @param groupName
* @param remoteFileName
* @return
* @throws Exception
*/
public static InputStream downloadFile(String groupName, String remoteFileName)throws Exception{
TrackerServer trackerServer = getTrackerServer();
StorageClient storageClient = getStorageClient(trackerServer);
//文件下载
byte[] bytes = storageClient.download_file(groupName, remoteFileName);
return new ByteArrayInputStream(bytes);
}
/**
* 删除文件
* @param groupName
* @param remoteFileName
* @throws Exception
*/
public static void deleteFile(String groupName, String remoteFileName)throws Exception{
TrackerServer trackerServer = getTrackerServer();
StorageClient storageClient = getStorageClient(trackerServer);
//文件删除
storageClient.delete_file(groupName,remoteFileName);
}
public static String getTrackerInfo()throws Exception{
TrackerServer trackerServer = getTrackerServer();
StorageClient storageClient = getStorageClient(trackerServer);
//获取Tracker的端口
String hostString = trackerServer.getInetSocketAddress().getHostString();
int tracker_http_port = ClientGlobal.getG_tracker_http_port();
String url = "http://"+hostString+":"+tracker_http_port;
return url;
}
/**
* 获取TeackerServer
* @return
* @throws IOException
*/
public static TrackerServer getTrackerServer() throws IOException {
//创建一个Tracker访问的客户端对象TrackerClient
TrackerClient trackerClient = new TrackerClient();
//通过TrackerClient访问TrackerServer服务，获取连接信息
TrackerServer trackerServer = trackerClient.getConnection();
return trackerServer;
}
/**
* 获取StorageClient
* @param trackerServer
* @return
*/
public static StorageClient getStorageClient(TrackerServer trackerServer){
//通过TrackerServer的连接信息可以获取Storage的连接信息，创建StorageClient对象存储storage连接信息
StorageClient storageClient = new StorageClient(trackerServer, null);
return storageClient;
}
public static void main(String[] args) throws Exception {
//获取文件信息
// FileInfo file = getFileInfo("group1", "M00/00/00/rB52RF9jaiSAP9oLAAFDcbOOQqw66.jpeg");
// System.out.println("FileSize:"+file.getFileSize());
// System.out.println("IPAddr:"+file.getSourceIpAddr());
// //文件下载
// InputStream is = downloadFile("group1", "M00/00/00/rB52RF9jaiSAP9oLAAFDcbOOQqw66.jpeg");
// FileOutputStream os = new FileOutputStream("/Users/LiuShihao/Desktop/1.jpeg");
// //定义一个缓冲区
// byte[] buffer = new byte[1024];
// while(is.read(buffer)!=-1){
// os.write(buffer);
// }
// os.flush();
// os.close();
// is.close();
//文件删除
// deleteFile("group1", "M00/00/00/rB52RF9jaiSAP9oLAAFDcbOOQqw66.jpeg");
// 获取Tracker信息 IP+Port
// System.out.println(getTrackerInfo());
}
}
6. Control layer
package com.mychanggou.controller;
import com.mychanggou.file.FastDFSFile;
import com.mychanggou.util.FastDFSUtil;
import entity.Result;
import entity.StatusCode;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
/**
* @author ：LiuShihao
* @date ：Created in 2020/9/17 8:51 下午
* @desc ：
*/
@RestController
@RequestMapping("/file")
@CrossOrigin//跨域
public class FileUploadController {
@PostMapping("/upload")
public Result upload(@RequestParam("file")MultipartFile file) throws Exception {
if(file == null){
throw new RuntimeException("文件不存在");
}
//调用工具类 上传FastDFS文件服务器
//使用StringUtils.getFilenameExtension 可以获取文件扩展名
//获取文件名字、文件字节数组、文件扩展名 构造FastDFSFile对象
FastDFSFile fastDFSFile = new FastDFSFile(file.getOriginalFilename(),file.getBytes(),StringUtils.getFilenameExtension(file.getOriginalFilename()));
String[] upload = FastDFSUtil.upload(fastDFSFile);
//拼接访问地址
String url =FastDFSUtil.getTrackerInfo()+"/"+upload[0]+"/"+upload[1];
return new Result(true, StatusCode.OK,"上传成功",url);
}
}
Three, upload test
start up:
Finally, share how to modify the welcome page:
First, give a URL:
http://patorjk.com/software/taag/#p=display&f=Graffiti&t=Type%20Something%20
Enter the file you want in the middle area, You can also choose fonts, etc., and then paste and copy the formed image into a txt, name it banner.txt, and put it in the resource directory. After that, you can start the project. Replace the banner every time you want to change the group. The content in txt is fine.