I have a Stream
processing a few millions of elements. The Map-Reduce algorithm behind it takes a few milliseconds, so task completion takes about twenty minutes.
Stream<MyData> myStream = readData();
MyResult result = myStream
.map(row -> process(row))
.peek(stat -> System.out.println("Hi, I processed another item"))
.reduce(MyStat::aggregate);
I'd like a way to display overall progress, instead of printing a line per element (which results in thousands of lines per second, takes time and doesn't provide any useful information regarding overall progress). I would like to display something similar to:
5% (08s)
10% (14s)
15% (20s)
...
What would be the best (and/or easiest) way to do this?
First of all, Streams are not meant to achieve these kind of tasks (as opposed to a classic data structure). If you know already how many elements your stream will be processing you might go with the following option, which is, I repeat, not the goal of streams.
Stream<MyData> myStream = readData();
final AtomicInteger loader = new AtomicInteger();
int fivePercent = elementsCount / 20;
MyResult result = myStream
.map(row -> process(row))
.peek(stat -> {
if (loader.incrementAndGet() % fivePercent == 0) {
System.out.println(loader.get() + " elements on " + elementsCount + " treated");
System.out.println((5*(loader.get() / fivePercent)) + "%");
}
})
.reduce(MyStat::aggregate);