Converting a 3GB file into a byte array

Ashish Pandya :

I am trying to convert 3GB file into byte array but getting OOM(OutOfMemoryError).

We tried

RandomAccessFile randomAccessFile = new RandomAccessFile(sourceLocation.getAbsolutePath(), "r");
MappedByteBuffer mappedByteBuffer = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, randomAccessFile.length()); //while it is large file, it threw 'mmap failed: ENOMEM (Out of memory)' exception.

byte[] data = new byte[1024 * 1024];  // 1MB read at time
while (mappedByteBuffer.hasRemaining()) {
    int remaining = data.length;

    if (mappedByteBuffer.remaining() < remaining)
        remaining = mappedByteBuffer.remaining();
        mappedByteBuffer.get(data, 0, remaining);
    }

    mappedByteBuffer.rewind();
    byte fileContent[] = mappedByteBuffer.array(); //while it is small file, it threw 'java.nio.ReadOnlyBufferException' exception.
    randomAccessFile.close();
}

My Custom Request Body: Custom request body class where my request body prepared

import android.os.Looper;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;

import okhttp3.MediaType;
import okhttp3.RequestBody;
import okio.BufferedSink;

public class ProgressRequestBodyTemp extends RequestBody
{
    private static final int DEFAULT_BUFFER_SIZE = 2048;
    private File mFile;
    private String mPath;
    private String mFileType;
    private int mItemIndex;
    private UploadCallbacks mListener;

    private byte[] encryptedData;
    private ByteArrayInputStream bis;

    public ProgressRequestBodyTemp(final String fileType, final File file, byte[] encryptedData, final int itemIndex, final UploadCallbacks listener)
    {
        this.mFile = file;
        this.mFileType = fileType;
        this.mItemIndex = itemIndex;
        this.mListener = listener;

        this.encryptedData = encryptedData;

        try
        {
            bis = new ByteArrayInputStream(encryptedData); // Convert byte array into input stream for send data to server
        }
        catch (Exception ex)
        {
            ex.printStackTrace();
        }
    }

    @Override
    public MediaType contentType()
    {
        try
        {
            return MediaType.parse(mFileType);
        }
        catch (Exception ex)
        {
            return MediaType.parse("");
        }
    }

    @Override
    public long contentLength() throws IOException
    {
        return encryptedData.length;
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException
    {
        long fileLength = mFile.length();
        byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];

        long uploaded = 0;
        try
        {
            int read;
            android.os.Handler handler = new android.os.Handler(Looper.getMainLooper());
            while ((read = bis.read(buffer)) != -1)
            {

                // update progress on UI thread
                handler.post(new ProgressUpdater(uploaded, encryptedData.length));

                uploaded += read;
                sink.write(buffer, 0, read);
            }
        }
        finally
        {
            bis.close();
        }
    }

    public interface UploadCallbacks
    {
        void onProgressUpdate(int itemIndex, int percentage);
    }

    private class ProgressUpdater implements Runnable
    {
        private long mUploaded;
        private long mTotal;

        public ProgressUpdater(long uploaded, long total)
        {
            mUploaded = uploaded;
            mTotal = total;
        }

        @Override
        public void run()
        {
            if (mListener != null)
            {
                mListener.onProgressUpdate(mItemIndex, (int) (100 * mUploaded / mTotal));
            }
        }
    }
}

My Request: request for upload file

File sourceLocation = new File(".....");
byte fileContent[] = .... // byte array

ProgressRequestBody requestFile  = new ProgressRequestBody(fileType, sourceLocation, fileContent, itemIndex, this);
MultipartBody.Part body = MultipartBody.Part.createFormData("file", sourceLocation.getName(), requestFile); // MultipartBody.Part is used to send also the actual file name
Call<String> mediaCall = getRetrofit().addMedia("SOME STRING DATA", body);

It is giving OutOfMemoryError, Please suggest the best way to convert the large file into byte array.

any help appreciated a lot. thanks in advance.

Daniel B. :

Use InputStream and OutputStream.

They are meant for big amounts of data, for example, when you work with 3GB of data and can't load it into memory.

If you are trying to upload a file, use FileInputStream. Create a File object, pass it to the FileOutputStreamconstructor and start reading bytes from its InputStream to a byte[] buffer and send the bytes with an OutputStream to the server.

This approach will not cause an OutOfMemoryError,because you are reading only enough bytes to fill up the buffer, which should be about 2KB - 8KB in size. After the buffer is full, you write the bytes to the server. After the buffer was written to the server, you read into the buffer again, and the process keeps going on until the whole file is uploaded.

Example using FileInputStream

        File file = new File("yourfile.txt");
        FileInputStream fis = null;
        OutputStream outputStream = null;

        url = new URL(desiredUrl);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();


        try {
            fis = new FileInputStream(file);
            connection.setDoOutput(true);
            outputStream = connection.getOutputStream();

            int actuallyRead;
            byte[] buffer = new byte[2048];
            while ((actuallyRead = fis.read(buffer)) != -1) {
                //do something with bytes, for example, write to the server
            outputStream.write(buffer);

            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fis != null)
                    fis.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
            try {
                if (outputStream != null)
                    outputStream.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }

        }

Note: This approach does not mean you need to reconnect to the server each time a buffer is filled. It will continuously write to the server, until it is done processing the file. This will happen all under the same, single connection.