JobReport is on fire
easy-batch contains a metric in the batch processing stage and a report after the batch is completed
- Information contained
Job start and end time
job status
Read record
Written record
Filtered record
Error count
Custom metrics
We can add our own metrics through the addMetrics of JobMetrics, but in order to access JobMetrics we need to implement the JobListenerinterface
- Reference demo
public class RecordProcessingTimeCalculator implements PipelineListener, JobListener {
private long startTime;
private long nbRecords;
private long recordProcessingTimesSum;
@Override
public Record beforeRecordProcessing(Record record) {
nbRecords++;
startTime = System.currentTimeMillis();
return record;
}
@Override
public void afterRecordProcessing(Record input, Record output) {
recordProcessingTimesSum += System.currentTimeMillis() - startTime;
}
@Override
public void onRecordProcessingException(Record record, Throwable throwable) {
recordProcessingTimesSum += System.currentTimeMillis() - startTime;
}
@Override
public void afterJobEnd(JobReport jobReport) {
jobReport.getMetrics().addMetric(
"Record processing time average (in ms)",
(double)recordProcessingTimesSum / (double)nbRecords);
}
}
Combined job report
Many times we are running a parallel job, but after the results need to get, you can use the combined reporting function, the code that
we can JobReportMergerto deal with
reference to FIG.
- Provided metrics
Startup time (based on minimum and performance report)
End time (based on maximum and parallel report)
Total read records (sum of parallel reports)
Total number of records written (sum of reports written in parallel)
Total number of filtered records (sum of parallel report filters)
Total number of error records (sum of parallel report filters)
The final state (COMPLETED, all completed, one of FAILED failed)
The final name is an aggregation of job names
- Code call
JobReportMerger reportMerger = new DefaultJobReportMerger();
JobReport finalReport = reportMerger.mergeReports(report1, report2);