I. Introduction
The download function should be a relatively common function. Although it may not appear many in a project, basically every project will have it, and some download functions are actually quite complicated. It is not difficult, but troublesome.
If I say that you only need one annotation to download any object for you, don’t you think it is very convenient?
@Download(source = "classpath:/download/README.txt")
@GetMapping("/classpath")
public void classpath() {
}
@Download
@GetMapping("/file")
public File file() {
return new File("/Users/Shared/README.txt");
}
@Download
@GetMapping("/http")
public String http() {
return "http://127.0.0.1:8080/concept-download/image.jpg";
}
Don't you think there's much difference? Then listen to a download request I encountered
We have a platform to manage devices, and each device will have a QR code image and an http address stored in a field.
Now we need to export the compressed package of all device QR code images. The image name needs to be the device name plus .png suffix. It is not difficult in terms of requirements, but it is a bit troublesome.
-
First I need to find out the device list
-
Then use the QR code address to download the image and write it to the local cache file
-
Before downloading, you need to determine whether there is already a cache
-
Concurrent downloads are required to improve performance when downloading
-
Wait until all pictures are downloaded
-
Generate another compressed file
-
Then operate the input and output streams and write them into the response.
Seeing that I have implemented nearly 200 lines of code, it is really smelly and long. How can a download function be so troublesome, so I wondered if there is a simpler way.
My needs at the time were very simple. I thought that I only need to provide the data that needs to be downloaded, such as a file path, a file object, a string text, an http address, or a collection of all the previous types, or even our own An instance of a certain class is defined, and I don’t have to worry about the rest.
Is the file path a file or a directory? String text needs to be written to a text file first?
How to download http resources to local? How to compress multiple files? How to finally write it into the response? I don't want to spend my time dealing with this
For example, for my current needs, I only need to return to the device list, and I don’t need to worry about other things.
@Download(filename = "二维码.zip")
@GetMapping("/download")
public List<Device> download() {
return deviceService.all();
}
public class Device {
//设备名称
private String name;
//设备二维码
//注解表示该http地址是需要下载的数据
@SourceObject
private String qrCodeUrl;
//注解表示文件名称
@SourceName
public String getQrCodeName() {
return name + ".png";
}
//省略其他属性方法
}
Specify the file name and file address by marking certain annotations on the Device field (or implementing an interface)
If it could be implemented in this way, wouldn't it be nice to save time, worry, effort, and spend more time writing 199 lines of code?
If you are interested, the introduction on Github is more detailed, including various advanced usages and the overall architecture.
Second idea
Let’s talk about the main design ideas of this library and the pitfalls encountered in the process. If you are interested, you can continue to read below.
In fact, based on my initial assumptions, I felt that the function was not too complicated, so I decided to open the liver
I just never expected that the implementation would be more complicated than I imagined (this is a story for a later time)
Three basics
First of all, the entire library is based on reactive programming, but it is not reactive in the complete sense. It can only be said to be like Mono. . . Strange combination?
Why is this happening? A big reason is that due to the need to be compatible with webmvc and webflux, I just reconstructed the previously implemented InputStream method into a responsive one, so this combination appeared.
This is also the biggest pitfall I encountered. I had basically adjusted the entire download process based on Servlet, and then I wanted to support webflux.
Everyone knows that in webmvc, we can obtain request and response objects through RequestContextHolder, but this does not work in webflux. Of course, we can inject it in the method parameters.
@Download(source = "classpath:/download/README.txt")
@GetMapping("/classpath")
public void classpath(ServerHttpResponse response) {
}
Combined with Spring's own injection function, we can get the response input parameters through AOP, but I always feel that writing this way is a bit redundant, and I can't bear it because of obsessive-compulsive disorder.
Is there any way to get rid of unused input parameters and get the response object at the same time? I found a way to implement it online.
/**
* 用于设置当前的请求和响应。
*
* @see ReactiveDownloadHolder
*/
public class ReactiveDownloadFilter implements WebFilter {
@Override
public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
ServerHttpRequest request = exchange.getRequest();
ServerHttpResponse response = exchange.getResponse();
return chain.filter(exchange)
//低版本使用subscriberContext
.contextWrite(ctx -> ctx.put(ServerHttpRequest.class, request))
.contextWrite(ctx -> ctx.put(ServerHttpResponse.class, response));
}
}
/**
* 用于获得当前的请求和响应。
*
* @see ReactiveDownloadFilter
*/
public class ReactiveDownloadHolder {
public static Mono<ServerHttpRequest> getRequest() {
//低版本使用subscriberContext
return Mono.deferContextual(contextView -> Mono.just(contextView.get(ServerHttpRequest.class)));
}
public static Mono<ServerHttpResponse> getResponse() {
//低版本使用subscriberContext
return Mono.deferContextual(contextView -> Mono.just(contextView.get(ServerHttpResponse.class)));
}
}
You can get the response object by adding WebFilter, but the return value is Mono
So can we get the corresponding object through Mono.block() blocking? The answer is no. Since webflux is based on Netty's non-blocking thread, if this method is called, an exception will be thrown directly.
So there is no other way but to refactor the previous code based on responsiveness
Four architectures
For a download request, we can divide it into several steps. Take downloading a compressed package of multiple files as an example.
-
First, we usually get the paths of multiple files or the corresponding File objects.
-
Then compress these files to generate a compressed file
-
Finally write the compressed file to the response
But for the needs I described above, it is not a file path or object at the beginning, but an http address. Then one more step is needed before compression, and the image needs to be downloaded first.
Then for various needs we may need to add additional steps anywhere in the current step, so I referred to the implementation of the Spring Cloud Gateway interception chain
/**
* 下载处理器。
*/
public interface DownloadHandler extends OrderProvider {
/**
* 执行处理。
*
* @param context {@link DownloadContext}
* @param chain {@link DownloadHandlerChain}
*/
Mono<Void> handle(DownloadContext context, DownloadHandlerChain chain);
}
/**
* 下载处理链。
*/
public interface DownloadHandlerChain {
/**
* 调度下一个下载处理器。
*
* @param context {@link DownloadContext}
*/
Mono<Void> next(DownloadContext context);
}
In this way, each step can implement a separate DownloadHandler, and any combination of steps can be added.
Five download contexts
On this basis, a context DownloadContext is used throughout the entire process to facilitate sharing and transferring intermediate results between steps.
For context DownloadContext also provides DownloadContextFactory which can be used to customize the context.
DownloadContextInitializer and DownloadContextDestroyer are also provided to extend your own logic during context initialization and destruction.
Six download types supported
The type of data we need to download is not fixed. For example, there are files, http addresses, and instances of custom classes that I hoped for before.
So I abstracted all download objects into Source, representing a download source, so that the file can be implemented as FileSource, the http address can be implemented as HttpSource, and then matched and created through the corresponding SourceFactory
For example, FileSourceFactory can match File and create FileSource, and HttpSourceFactory can match http:// prefix and create HttpSource.
/**
* {@link Source} 工厂。
*/
public interface SourceFactory extends OrderProvider {
/**
* 是否支持需要下载的原始数据对象。
*
* @param source 需要下载的原始数据对象
* @param context {@link DownloadContext}
* @return 如果支持则返回 true
*/
boolean support(Object source, DownloadContext context);
/**
* 创建。
*
* @param source 需要下载的原始数据对象
* @param context {@link DownloadContext}
* @return 创建的 {@link Source}
*/
Source create(Object source, DownloadContext context);
}
So how do we support our custom classes? As mentioned before, we can mark annotations on classes or implement specific interfaces. Let’s briefly talk about them using the annotations I implemented.
In fact, the logic is very simple. As long as you can use reflection skillfully, there will be no problem. Let's take a look at the usage.
@Download(filename = "二维码.zip")
@GetMapping("/download")
public List<Device> download() {
return deviceService.all();
}
public class Device {
//设备名称
private String name;
//设备二维码
//注解表示该http地址是需要下载的数据
@SourceObject
private String qrCodeUrl;
//注解表示文件名称
@SourceName
public String getQrCodeName() {
return name + ".png";
}
//省略其他属性方法
}
First, I defined an annotation @SourceModel annotation on the class to indicate that it needs to be parsed, and then defined a @SourceObject annotation annotation on the field (or method) that needs to be downloaded, so that we can get this field (or method) through reflection. value
The corresponding Source can be created based on the currently supported SourceFactory. Next, use @SourceName to specify the name. You can also obtain the value of this method (or field) through reflection and still set it to the created Source through reflection.
This allows for very flexible support of any object type.
Seven concurrent loading
For network resources like http, we need to first load (multiple files) concurrently into local memory or cache files to improve our processing efficiency.
Of course, I can directly set a thread pool for execution, but each machine, each project, and even each requirement have different concurrency requirements and resource allocation.
So I provide SourceLoader to support custom loading logic. You can even use part of the thread pool, part of it using coroutines, and the remaining part not loaded.
/**
* {@link Source} 加载器。
*
* @see DefaultSourceLoader
* @see SchedulerSourceLoader
*/
public interface SourceLoader {
/**
* 执行加载。
*
* @param source {@link Source}
* @param context {@link DownloadContext}
* @return 加载后的 {@link Source}
*/
Mono<Source> load(Source source, DownloadContext context);
}
eight compression
After we load it, we can perform compression. Similarly, I defined a class Compression as the abstraction of the compression object.
Generally speaking, we will first create a cache file locally, and then write the compressed data to the cache file
However, I hate configuring various paths in the configuration file every time, so memory compression is supported during compression. Of course, if the file is relatively large, a cache file will still be generated honestly.
It also provides a completely customizable SourceCompressor interface for compression formats. You can implement a compression protocol yourself without any problem.
/**
* {@link Source} 压缩器。
*
* @see ZipSourceCompressor
*/
public interface SourceCompressor extends OrderProvider {
/**
* 获得压缩格式。
*
* @return 压缩格式
*/
String getFormat();
/**
* 判断是否支持对应的压缩格式。
*
* @param format 压缩格式
* @param context {@link DownloadContext}
* @return 如果支持则返回 true
*/
default boolean support(String format, DownloadContext context) {
return format.equalsIgnoreCase(getFormat());
}
/**
* 如果支持对应的格式就会调用该方法执行压缩。
*
* @param source {@link Source}
* @param writer {@link DownloadWriter}
* @param context {@link DownloadContext}
* @return {@link Compression}
*/
Compression compress(Source source, DownloadWriter writer, DownloadContext context);
}
Nine response writes
I abstracted the response into DownloadResponse, mainly for compatibility with HttpServletResponse and ServerHttpResponse
But the problem arises again. Here is how webmvc and webflux write the response.
//HttpServletResponse
response.getOutputStream().write(byte b[], int off, int len);
//ServerHttpResponse
response.writeWith(Publisher<? extends DataBuffer> body);
My head hurt because of this compatibility, but I finally got it done.
/**
* 持有 {@link ServerHttpResponse} 的 {@link DownloadResponse}，用于 webflux。
*/
@Getter
public class ReactiveDownloadResponse implements DownloadResponse {
private final ServerHttpResponse response;
private OutputStream os;
private Mono<Void> mono;
public ReactiveDownloadResponse(ServerHttpResponse response) {
this.response = response;
}
@Override
public Mono<Void> write(Consumer<OutputStream> consumer) {
if (os == null) {
mono = response.writeWith(Flux.create(fluxSink -> {
try {
os = new FluxSinkOutputStream(fluxSink, response);
consumer.accept(os);
} catch (Throwable e) {
fluxSink.error(e);
}
}));
} else {
consumer.accept(os);
}
return mono;
}
@SneakyThrows
@Override
public void flush() {
if (os != null) {
os.flush();
}
}
@AllArgsConstructor
public static class FluxSinkOutputStream extends OutputStream {
private FluxSink<DataBuffer> fluxSink;
private ServerHttpResponse response;
@Override
public void write(byte[] b) throws IOException {
writeSink(b);
}
@Override
public void write(byte[] b, int off, int len) throws IOException {
byte[] bytes = new byte[len];
System.arraycopy(b, off, bytes, 0, len);
writeSink(bytes);
}
@Override
public void write(int b) throws IOException {
writeSink((byte) b);
}
@Override
public void flush() {
fluxSink.complete();
}
public void writeSink(byte... bytes) {
DataBuffer buffer = response.bufferFactory().wrap(bytes);
fluxSink.next(buffer);
//在这里可能有问题，但是目前没有没有需要释放的数据
DataBufferUtils.release(buffer);
}
}
}
As long as you write byte[] in the end, you can convert each other, but it may be a little more troublesome and you need to use an interface callback.
Disguise FluxSink as an OutputStream, convert byte[] to DataBuffer when writing and call the next method, and finally call the complete method when flushing, perfect.
Response writing is actually the processing of input and output streams. Under normal circumstances, we will define a byte[] to cache the read data, so I will not fix the size of this cache but provide a DownloadWriter that can be customized. Process input and output streams, including the presence of specified encoding or Range headers
/**
* 具体操作 {@link InputStream} 和 {@link OutputStream} 的写入器。
*/
public interface DownloadWriter extends OrderProvider {
/**
* 该写入器是否支持写入。
*
* @param resource {@link Resource}
* @param range {@link Range}
* @param context {@link DownloadContext}
* @return 如果支持则返回 true
*/
boolean support(Resource resource, Range range, DownloadContext context);
/**
* 执行写入。
*
* @param is {@link InputStream}
* @param os {@link OutputStream}
* @param range {@link Range}
* @param charset {@link Charset}
* @param length 总大小，可能为 null
*/
default void write(InputStream is, OutputStream os, Range range, Charset charset, Long length) {
write(is, os, range, charset, length, null);
}
/**
* 执行写入。
*
* @param is {@link InputStream}
* @param os {@link OutputStream}
* @param range {@link Range}
* @param charset {@link Charset}
* @param length 总大小，可能为 null
* @param callback 回调当前进度和增长的大小
*/
void write(InputStream is, OutputStream os, Range range, Charset charset, Long length, Callback callback);
/**
* 进度回调。
*/
interface Callback {
/**
* 回调进度。
*
* @param current 当前值
* @param increase 增长值
*/
void onWrite(long current, long increase);
}
}
ten events
After I implemented the entire download process, I found that the entire logic was actually a bit complicated, so I had to find a way to monitor the entire download process.
At first, I defined several listeners for callbacks, but they were not easy to use. First of all, our entire architecture is designed to be very flexible and scalable, but the defined listener types are few and difficult to expand.
When we subsequently added other processes and steps, we had to add several new types of listeners or add methods to the original listener classes, which was very troublesome.
So I thought that using events can be expanded more flexibly, and defined DownloadEventPublisher for publishing events and DownloadEventListener for listening events, and supported Spring's event listening method.
Eleven Log
Based on the above event method, I implemented several download logs on this basis
Logs corresponding to each process
Loading progress updates, compression progress updates, response writing progress update logs
Time spent logs
These logs also helped me find a lot of bugs because they printed information about the entire download process in more detail.
Twelve other pits
At first, the initialization and destruction of the context each corresponded to one step at the beginning and the end respectively. However, after I finished writing the response in webflux, I found that the destruction of the context would not be executed.
So I followed the Spring source code and found that the write method returns Mono.empty(), which means that after the response is written, the next method will not be called, so the steps after the response is written will always be will not be called
Finally, the context initialization and destruction were separated, and the destruction method was called during doAfterTerminate.
Thirteen Conclusion
The basic content is like this, but I still don’t know much about responsiveness, and I’m not very good at using some operators, but I still know a lot of advanced usage.