1. Problem background
An interface is made according to business requirements. The main function is to find data in the database according to query rules, then generate files, verify files, compress files, and then upload these three files to the designated server. However, due to the large amount of data, usually 1kw+ peak value is 4kw+, although the processing performance requirements are not high, but you can not run too slow. It takes about half an hour to run 1500w data after the implementation according to the conventional method, which is still in line with business needs. The amount of data in these two days was huge, and OOM appeared when it reached 2200w.
2. Problem solving
The emergence of OOM has two main points, 1. The memory is really insufficient, and the heap memory needs to be expanded during execution. 2. There is an infinite loop in the program. There are instances holding a large number of new objects that cannot be released. Investigate these two points.
First look at the problematic code block
//生成文件
public boolean loadData(EtlRule etlRule) throws IOException, SQLException {
if (etlRule.getCreate_sql().trim().equals(""))
logger.error("\"error: 配置中包括查询语句为空[create_sq;]\"");
long FILE_UNIT_SIZE = 1024 * 1024 * 1024;
String selSql;
Connection destCon = null;// 源数据库连接对象
Statement stmt = null;
BufferedWriter output = null;
long rowCount = 0L;
int colCounts = 0;
long flen = 0L;
ResultSet res=null;
int closeNum = 5000000; //每五百万条记录关闭一次输入流，目的是清空堆内存。
try {
//初始化数据库连接
EtlDBConnect etlDBConnect = new EtlDBConnect();
destCon = etlDBConnect.initConfig2(destCon);
if (etlRule.getCreate_sql().indexOf("${SRC_TABLE}") > 0) {
selSql = etlRule.getCreate_sql().replace("${SRC_TABLE}", etlRule.table_name);
}else {
selSql = etlRule.getCreate_sql();
}
logger.info("数据查询：sql=[" + selSql + "]");
//生成的文件个数
int fileCount = 1;
//向dataPath中添加路文件径数据
String dataPathArr = etlRule.getLocal_path() + "/" + etlRule.getFile_name().replace("${NUM}",
"001");
etlRule.getDataPath().add(dataPathArr);
File file = new File(etlRule.getDataPath().get(fileCount-1));
if (!file.exists()) {
file.createNewFile();
//System.out.println("ORA-000 数据文件创建失败: " + etlRule.getDataPath().get(0));
//return false;
}
logger.info("[*输出数据文件*]:[" + etlRule.getDataPath().get(0) + "]");
assert destCon != null;
//stmt = destCon.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_READ_ONLY);
stmt = destCon.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
logger.info("[查询主表] sql==========" + selSql);
res = stmt.executeQuery(selSql);
//preFetchRowNum 为配置文件中的值
res.setFetchSize(5000); //设置预处理行数
int fetchSize = res.getFetchSize();
System.out.println("------默认fetchSize预处理行数："+fetchSize);
if (res == null || !res.next()) {
logger.error("数据文件sql无数据！");
return false;
}
res.previous();
output = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file, true),
StandardCharsets.UTF_8),1024);
//StringBuilder line = new StringBuilder();
String str = "";
// 获取字段元信息
ResultSetMetaData rsmd1 = res.getMetaData();
colCounts = rsmd1.getColumnCount();
//Date date = new Date();
while (res.next()) {
// 打印进度
rowCount++;
if (rowCount % printUnit == 0) {
//20w条数据的时候打印一条提示信息
//logger.info(rowCount + " ----rows proceed");
System.out.println(rowCount +" ----rows proceed");
}
//获取所有列对应的数据
for (int i = 1; i <= colCounts; i++) {
//line.append(res.getString(i)).append("\n");
str = res.getString(i)+"\n";
}
//读取一条查询结果，写入一次
output.write(str);
output.flush();
str = "";
//如果文件长度大于2G,建立新文件，最多不会超过9个-------c中40亿的定值超int范围了
if (file.length() >= (FILE_UNIT_SIZE * 2)) {
//如果存在${NUM}，则生成新的文件，不存在就继续向原文件中写入
if (etlRule.getFile_name().contains("${NUM}")){
output.close();
fileCount++;
flen = file.length();
//添加记录数
etlRule.getDataCount().add(rowCount);
etlRule.getDataSize().add(flen);
String newFile = etlRule.getLocal_path() + "/" + etlRule.getFile_name().replace(
"${NUM}", "00" + fileCount);
etlRule.getDataPath().add(newFile);
file = new File(etlRule.getDataPath().get(etlRule.getDataPath().size() - 1));
if (!file.exists()) {
file.createNewFile();
}
output = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file, false), StandardCharsets.UTF_8));
}
}
//logger.info("\t"+ String.valueOf(rowCount) +"\t\t\trows processed");
}//while
//数据写入字段
flen = file.length();
etlRule.getDataCount().add(rowCount);
etlRule.getDataSize().add(flen);
} catch (Exception | Error throwables) {
System.out.println("捕获到错误："+throwables.getMessage());
System.out.println("规则运行标识复位。");
ruleConfigReset(etlRule);
//throwables.printStackTrace();
}finally {
if (output != null) {
try {
//刷新缓存区
output.flush();
output.close();
} catch (IOException e) {
logger.error("关闭输出流异常:{}", e);
}
}
//关闭数据库连接
assert res != null;
res.close();
stmt.close();
destCon.commit();
destCon.close();
}
return true;
}
Through the analysis of the code block, the problem is locked in the while() loop, because only there may be an infinite loop or holding too many new objects. Locked in by further searching
output = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file, true),
StandardCharsets.UTF_8),1024);
//读取一条查询结果，写入一次
output.write(str);
output.flush();
Then continue to track the write() method of BufferedWriter
public void write(String str) throws IOException {
write(str, 0, str.length());
}
Another overloaded write(String str, int off, int len) method is called in writre(String str)
public void write(String str, int off, int len) throws IOException {
synchronized (lock) {
char cbuf[];
if (len <= writeBufferSize) {
if (writeBuffer == null) {
writeBuffer = new char[writeBufferSize];
}
cbuf = writeBuffer;
} else {
// Don't permanently allocate very large buffers.
cbuf = new char[len];
}
str.getChars(off, (off + len), cbuf, 0);
write(cbuf, 0, len);
}
}
By viewing the source code, we first find the definition of a few variables
/**
* The object used to synchronize operations on this stream. For
* efficiency, a character-stream object may use an object other than
* itself to protect critical sections. A subclass should therefore use
* the object in this field rather than <tt>this</tt> or a synchronized
* method.
* 用于同步此流上的操作的对象。为了提高效率，字符流对象可以使用其他对象来保护临界区。
* 因此，子类应该在这个字段中使用对象，而不是<tt>这个</tt>或同步方法。
*/
protected Object lock;
/**
* Temporary buffer used to hold writes of strings and single characters
* 用于保存写字符串和单个字符的临时缓冲区
*/
private char[] writeBuffer;
/**
* Size of writeBuffer, must be >= 1
*/
private final int writeBufferSize = 1024;
The following is the interpretation of the write(String str, int off, int len) method
1.synchronized (lock) I don't make any trouble, because I have never used it like this. I just know that when a thread accesses the synchronized (this) block of an object, other threads trying to access the object will be blocked.
2. When len length is less than writeBufferSize writeBuffer = new char[writeBufferSize];
otherwise cbuf = new char[len];
3. Track str.getChars(off, (off + len), cbuf, 0);
public void getChars(int srcBegin, int srcEnd, char dst[], int dstBegin) {
if (srcBegin < 0) {
throw new StringIndexOutOfBoundsException(srcBegin);
}
if (srcEnd > value.length) {
throw new StringIndexOutOfBoundsException(srcEnd);
}
if (srcBegin > srcEnd) {
throw new StringIndexOutOfBoundsException(srcEnd - srcBegin);
}
System.arraycopy(value, srcBegin, dst, dstBegin, srcEnd - srcBegin);
}
The function of this method is to put a certain piece of byte data in one array into another array. As for taking out a few data from the first array, where they are placed in the second array can be controlled by the parameters of this method.
4. Trace write(cbuf, 0, len); found that it is an abstract method
abstract public void write(char cbuf[], int off, int len) throws IOException;
It can be seen that write(cbuf, 0, len) is used to implement specific functions for subclasses or interfaces, and can be ignored if it is not used.
Three, analysis
After a little bit of the above, we found that in any case, as long as the write() method of BufferedWriter is called, a new char[] will be generated. In this case, if the amount of data is large, it will cause OOM, because the instance holds The object cannot be released in time.
Four, the solution
1. Increase the heap memory.
2. Optimize the code.
The first method can only solve the temporary pain, because the amount of data is uncertain, blindly increasing the heap memory not only wastes resources, but may also affect the operation of other programs.
linux下可以用此命令来查看jvm运行时的默认最大堆内存空间
java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
Note that Java HotSpot is Server or Client because the default allocation of heap memory in the two modes is slightly different. Generally speaking, Server mode is used in the development environment
//win下可以采用Runtime.getRuntime()下的方法来测试
public class jvm {
public static void main(String[] args) {
long maxMemory = Runtime.getRuntime().maxMemory();
long totalMemory = Runtime.getRuntime().totalMemory();
long freeMemory = Runtime.getRuntime().freeMemory();
long usableMemory = maxMemory - totalMemory + freeMemory;
System.out.println("可获得的最大内存是："+maxMemory/(1024*1024));
System.out.println("已经分配到的内存大小是："+(totalMemory/(1024*1024)));
System.out.println("所分配内存的剩余大小是："+(freeMemory/(1024*1024)));
System.out.println("最大可用内存是："+usableMemory/(1024*1024));
}
}
//如果懒得新建工程，可以直接新建个java文件，javac编译后直接运行class文件就行了。
The second method is to optimize the code. This difficulty depends on the specific situation.
According to my own situation, I restart the BufferedWriter every time 500w pieces of data are read in order to release the previously occupied heap memory. I didn't expect other methods for the time being. The modified code has been submitted to the test. I don't know if it can work under a large amount of data. If I don't update it later, it should be fine.
Five, talk to yourself
Here I only provide a troubleshooting idea, OOM problems have to be analyzed according to the specific situation of their own code.
Here is another OOM troubleshooting case for your reference to
record the problem of java.lang.OutOfMemoryError: Java heap space [reproduced]