OutOfMemoryError when joining a list of Strings in Java

Aditya :

So. I've tried:

StringBuilder sb = new StringBuilder();
for(String bufferItem: buffer){
    sb.append(bufferItem);
}

and I've also tried:

String.join("\n", buffer)

I am joining a large files(under 10GB) in memory on a system with more than 100GB. The following is the stack trace. How can I solve this problem?

Exception in thread "main" java.lang.OutOfMemoryError
    at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
    at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
    at java.lang.StringBuilder.append(StringBuilder.java:136)
    at java.lang.StringBuilder.append(StringBuilder.java:76)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:484)
    at java.lang.StringBuilder.append(StringBuilder.java:166)
    at java.util.StringJoiner.add(StringJoiner.java:185)
    at java.lang.String.join(String.java:2504)

Erwin Bolwidt :

You cannot create strings with that many characters. The OutOfMemoryError is not because the heap was full, but because you're trying to build a String larger than the maximum possible size.

The maximum possible size is defined as 2 to the power 31, minus 1, minus 8. That's roughly 2Gb if you only use single-byte characters in a file. See the source of AbstractStringBuilder.

/**
 * The maximum size of array to allocate (unless necessary).
 * Some VMs reserve some header words in an array.
 * Attempts to allocate larger arrays may result in
 * OutOfMemoryError: Requested array size exceeds VM limit
 */
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;

You simply cannot create strings larger than that.

Why do you want to join the files in memory when you can join them while streaming them to disk?