简介
这几天在做通过流下载zip文件以及上传zip文件不解压读取zip文件内容的功能，在读取zip文件内容的时候遇到了size为-1的情况，在此记录下遇到的情况、解决办法、以及未解决的问题。
示例
将上传和下载zip文件的功能做成了一个示例，放到了github上，链接：export-import-zip-use-stream，可以尝试运行下。
遇到的问题
通过流下载zip文件之后，再次导入该zip文件，不解压读取zip文件内容，发现ZipEntry的size()返回-1，如下图所示：
但是尝试使用系统自带的压缩软件压缩了一个zip文件，并上传读取，发现一切正常，size不为-1。使用zipinfo命令查看两个文件作为对比，如下：
可以看到上面文件是通过导出功能生成的，红框里缺少size。而下面的是系统压缩软件压缩的zip文件，红框里面带有size大小。故猜测可能是由于代码里生成ZipEntry的时候没有设置size，compressize，crc32等属性的原因。
读取zip文件时ZipEntry的size为-1解决办法
直接读取当前ZipEntry的流，直到为-1为止，代码如下：
@PostMapping("import")
@ResponseBody
public void importZip(@RequestParam("file")MultipartFile file) throws IOException {
ZipInputStream zipInputStream = new ZipInputStream(file.getInputStream());
ZipEntry zipEntry;
while ((zipEntry = zipInputStream.getNextEntry()) != null) {
if (zipEntry.isDirectory()) {
// do nothing
}else {
String name = zipEntry.getName();
long size = zipEntry.getSize();
// unknown size
// ZipEntry的size可能为-1，表示未知
// 通过上面的几种方式下载，就会产生这种情况
if (size == -1) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
while (true) {
int bytes = zipInputStream.read();
if (bytes == -1) break;
baos.write(bytes);
}
baos.close();
System.out.println(String.format("Name:%s,Content:%s",name,new String(baos.toByteArray())));
} else { // ZipEntry的size正常
byte[] bytes = new byte[(int) zipEntry.getSize()];
zipInputStream.read(bytes, 0, (int) zipEntry.getSize());
System.out.println(String.format("Name:%s,Content:%s",name,new String(bytes)));
}
}
}
zipInputStream.closeEntry();
zipInputStream.close();
}
此时可以正确读取文件内容。
下载文件时设置size的解决办法
需要设置ZipEntry的size，compresSize以及crc32等属性。
例如：
public static void main(String[] args) throws IOException {
File infile = new File("test_file.pdf");
File outfile = new File("test.zip");
try (FileInputStream fis = new FileInputStream(infile);
FileOutputStream fos = new FileOutputStream(outfile);
ZipOutputStream zos = new ZipOutputStream(fos) ) {
byte[] buffer = new byte[1024];
ZipEntry entry = new ZipEntry("data");
precalc(entry, fis.getChannel());
zos.putNextEntry(entry);
for(int bytesRead; (bytesRead = fis.read(buffer)) >= 0; )
zos.write(buffer, 0, bytesRead);
zos.closeEntry();
}
try(FileInputStream fin = new FileInputStream(outfile);
ZipInputStream zis = new ZipInputStream(fin) ) {
ZipEntry entry = zis.getNextEntry();
System.out.println("Entry size: " + entry.getSize());
System.out.println("Compressed size: " + entry.getCompressedSize());
System.out.println("CRC: " + entry.getCrc());
zis.closeEntry();
}
}
private static void precalc(ZipEntry entry, FileChannel fch) throws IOException {
long uncompressed = fch.size();
int method = entry.getMethod();
CRC32 crc = new CRC32();
Deflater def;
byte[] drain;
if(method != ZipEntry.STORED) {
def = new Deflater(Deflater.DEFAULT_COMPRESSION, true);
drain = new byte[1024];
}
else {
def = null;
drain = null;
}
ByteBuffer buf = ByteBuffer.allocate((int)Math.min(uncompressed, 4096));
for(int bytesRead; (bytesRead = fch.read(buf)) != -1; buf.clear()) {
crc.update(buf.array(), buf.arrayOffset(), bytesRead);
if(def!=null) {
def.setInput(buf.array(), buf.arrayOffset(), bytesRead);
while(!def.needsInput()) def.deflate(drain, 0, drain.length);
}
}
entry.setSize(uncompressed);
if(def!=null) {
def.finish();
while(!def.finished()) def.deflate(drain, 0, drain.length);
entry.setCompressedSize(def.getBytesWritten());
}
entry.setCrc(crc.getValue());
fch.position(0);
}另外一种解决方法，通过Enumeration
一、分析下原因
（PS:参考大佬的博客地址：https://blog.csdn.net/zbj18314469395/article/details/84109499）
二、解决办法：
把判断条件由【(ze=zin.getNextEntry())!=null】换成【zipEnum.hasMoreElements ()】
使用了枚举， Enumeration 接口（枚举）。
通常用 Enumeration 中的以下两个方法打印向量中的所有元素：
(1) boolean hasMoreElements(); // 是否还有元素，如果返回 true ，则表示至少含有一个元素
(2) public Object nextElement(); // 如果 Enumeration 枚举对象还含有元素，该方法返回对象中的下一个元素。如果没有，则抛出NoSuchElementException 异常。
附上我的代码，已经修改过的，大家可以参考一下
public void processFile(String path) throws Exception {
ZipFile zf = new ZipFile(path);
//主要是这块
java.util.Enumeration zipEnum = zf.entries();
InputStream in = new BufferedInputStream(new FileInputStream(path));
ZipInputStream zin = new ZipInputStream(in, Charset.forName("gbk"));
ZipEntry ze;
//重点是是这块的判断
while (zipEnum.hasMoreElements()) {
ze = (ZipEntry) zipEnum.nextElement();
if (!ze.getName().contains("__MACOSX") && !ze.getName().contains(".DS_Store")) {
if (!ze.isDirectory()) {
System.out.println("##name##" + ze.getName());
System.out.println("##size##" + ze.getSize());
}
}
}
zin.closeEntry();
}另外一个例子：
https://github.com/yrom/shrinker/blob/master/shrinker/src/main/java/net/yrom/tools/JarProcessor.java
package net.yrom.tools;
import com.google.common.collect.ImmutableList;
import org.apache.commons.io.IOUtils;
import org.apache.commons.io.output.ByteArrayOutputStream;
import java.io.ByteArrayInputStream;
import java.io.EOFException;
import java.io.IOException;
import java.io.OutputStream;
import java.io.UncheckedIOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.function.Function;
import java.util.stream.Collectors;
import java.util.zip.CRC32;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;
/**
* @author yrom
* @version 2017/11/29
*/
class JarProcessor extends ClassesProcessor {
JarProcessor(Function<byte[], byte[]> classTransform, Path src, Path dst) {
super(classTransform, src, dst);
}
@Override
public void proceed() {
try {
List<Pair<String, byte[]>> entryList = readZipEntries(src)
.parallelStream()
.map(this::transformClassBlob)
.collect(Collectors.toList());
if (entryList.isEmpty()) return;
try (OutputStream fileOut = Files.newOutputStream(dst)) {
ByteArrayOutputStream buffer = zipEntries(entryList);
buffer.writeTo(fileOut);
}
} catch (IOException e) {
throw new RuntimeException("Reading jar entries failure", e);
}
}
private ByteArrayOutputStream zipEntries(List<Pair<String, byte[]>> entryList) throws IOException {
ByteArrayOutputStream buffer = new ByteArrayOutputStream(8192);
try (ZipOutputStream jar = new ZipOutputStream(buffer)) {
jar.setMethod(ZipOutputStream.STORED);
final CRC32 crc = new CRC32();
for (Pair<String, byte[]> entry : entryList) {
byte[] bytes = entry.second;
final ZipEntry newEntry = new ZipEntry(entry.first);
newEntry.setMethod(ZipEntry.STORED); // chose STORED method
crc.reset();
crc.update(entry.second);
newEntry.setCrc(crc.getValue());
newEntry.setSize(bytes.length);
writeEntryToJar(newEntry, bytes, jar);
}
jar.flush();
}
return buffer;
}
private Pair<String, byte[]> transformClassBlob(Pair<String, byte[]> entry) {
byte[] bytes = entry.second;
entry.second = classTransform.apply(bytes);
return entry;
}
private List<Pair<String, byte[]>> readZipEntries(Path src) throws IOException {
ImmutableList.Builder<Pair<String, byte[]>> list = ImmutableList.builder();
try (ZipInputStream zip = new ZipInputStream(new ByteArrayInputStream(Files.readAllBytes(src)))) {
for (ZipEntry entry = zip.getNextEntry();
entry != null;
entry = zip.getNextEntry()) {
String name = entry.getName();
if (!name.endsWith(".class")) {
// skip
continue;
}
long entrySize = entry.getSize();
if (entrySize >= Integer.MAX_VALUE) {
throw new OutOfMemoryError("Too large class file " + name + ", size is " + entrySize);
}
byte[] bytes = readByteArray(zip, (int) entrySize);
list.add(Pair.of(name, bytes));
}
}
return list.build();
}
private byte[] readByteArray(ZipInputStream zip, int expected) throws IOException {
if (expected == -1) { // unknown size
return IOUtils.toByteArray(zip);
}
final byte[] bytes = new byte[expected];
int read = 0;
do {
int n = zip.read(bytes, read, expected - read);
if (n <= 0) {
break;
}
read += n;
} while (read < expected);
if (expected != bytes.length) {
throw new EOFException("unexpected EOF");
}
return bytes;
}
private static void writeEntryToJar(ZipEntry entry, byte[] bytes, ZipOutputStream jar) {
try {
jar.putNextEntry(entry);
jar.write(bytes);
jar.closeEntry();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
// mutable pair
static class Pair<F, S> {
F first;
S second;
Pair(F first, S second) {
this.first = first;
this.second = second;
}
static <F, S> Pair<F, S> of(F first, S second) {
return new Pair<>(first, second);
}
}
}zip例子：
/**
* Computes the CRC checksum for the given file.
*
* @param file The file to compute checksum for.
* @return A CRC32 checksum.
* @throws IOException If an I/O error occurs.
*/
private static long computeCrc(File file) throws IOException {
CRC32 crc = new CRC32();
InputStream in = new FileInputStream(file);
try {
byte[] buf = new byte[8192];
int n = in.read(buf);
while (n != -1) {
crc.update(buf, 0, n);
n = in.read(buf);
}
} finally {
in.close();
}
return crc.getValue();
}
private static void copyFileToStream(File file, OutputStream out) throws IOException {
//LOG.info("Downloading '" + FileUtil.getShortPath(file) + "' to " + status.getPlayer());
final int bufferSize = 16 * 1024; // 16 Kbit
InputStream in = new BufferedInputStream(new FileInputStream(file), bufferSize);
try {
byte[] buf = new byte[bufferSize];
long bitrateLimit = 0;
long lastLimitCheck = 0;
while (true) {
long before = System.currentTimeMillis();
int n = in.read(buf);
if (n == -1) {
break;
}
out.write(buf, 0, n);
}
} finally {
out.flush();
in.close();
}
}
private static void zip(ZipOutputStream out, String root, File file) throws IOException {
// Exclude all hidden files starting with a "."
if (file.getName().startsWith(".")) {
return;
}
String zipName = file.getCanonicalPath().substring(root.length() + 1);
if (file.isFile()) {
ZipEntry zipEntry = new ZipEntry(zipName);
zipEntry.setSize(file.length());
zipEntry.setCompressedSize(file.length());
zipEntry.setCrc(computeCrc(file));
out.putNextEntry(zipEntry);
copyFileToStream(file, out);
out.closeEntry();
} else {
ZipEntry zipEntry = new ZipEntry(zipName + '/');
zipEntry.setSize(0);
zipEntry.setCompressedSize(0);
zipEntry.setCrc(0);
out.putNextEntry(zipEntry);
out.closeEntry();
File[] children = file.listFiles(); //FileUtil.listFiles(file);
for (File child : children) {
zip(out, root, child);
}
}
}
public static void zipDir(String zipFileName, String dir) {
try {
File zipFile = new File(zipFileName);
FileOutputStream fos = new FileOutputStream(zipFile);
//ZipOutputStream zos = new ZipOutputStream(new BufferedOutputStream(fos));
CheckedOutputStream checksum = new CheckedOutputStream(fos, new CRC32());
ZipOutputStream zos = new ZipOutputStream(checksum);
File dirFile = new File(dir);
String root = zipFile.getCanonicalPath().substring(0, zipFile.getCanonicalPath().length() - zipFile.getName().length() - 1);
zip(zos, root, dirFile);
} catch (IOException e) {
e.printStackTrace();
}
}但是上面的zip方法，压缩zip后，用Zip4J解压会失败，文件也有问题。这个后面来研究。
转自：
