IOException when writing byte array

Question

I am getting IOException: Map Failed when trying to write a large byte array. I use the method below to write a byte array to a file

private static void write(byte[] data) throws Exception {
        File file = new File("C:/temp/file.json");
        int length = data.length;
        RandomAccessFile raf = new RandomAccessFile(file, "rw");
        FileChannel fc = raf.getChannel();
        MappedByteBuffer buffer = fc.map(FileChannel.MapMode.READ_WRITE, 0, length);
        for (int i = 0; i < length; i++) {
            buffer.put(data[i]);
        }
}

The byte array is about 270mb. Can anyone explain what I am doing wrong? Thanks.

can you add the stack trace? also your jdk version?

vivek_jonam
– vivek_jonam

2012-08-30 11:08:39 +00:00
Commented Aug 30, 2012 at 11:08 — vivek_jonam
– vivek_jonam, Commented Aug 30, 2012 at 11:08

Peter Lawrey · Accepted Answer · 2012-08-30 12:03:46Z

4

I am not sure why the map failed, but I wouldn't do it the way you have done.

FileOutputStream out = new FileOutputStream(filename);
out.write(data);
out.close();

to do it progressively you can use

FileOutputStream out = new FileOutputStream(filename);
for(int i = 0; i < data.length; i += 8192)
    out.write(data, i, Math.min(data.length-i, 8192));
out.close();

The map could fail if you have a 32-bit JVM and you call this method repeatedly. e.g. you run out of virtual memory.

edited Aug 30, 2012 at 12:03

answered Aug 30, 2012 at 11:07

Peter Lawrey

535k83 gold badges770 silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

kidloco Over a year ago

ok, so tried that instead and I get an OutOfMemoryError, but I have set -Xmx1024m

Peter Lawrey Over a year ago

Where do you get the OOME? Is it in this code or somewhere else? I suspect the problem is when you generate the very large byte[] If that is the case I suggest writing to the file as you generate the String rather than making several copies in memory.

kidloco Over a year ago

Exception in thread "main" java.lang.OutOfMemoryError 	at java.io.FileOutputStream.writeBytes(Native Method) 	at java.io.FileOutputStream.write(FileOutputStream.java:269)

Peter Lawrey Over a year ago

Hmmmm, You can write the data in blocks or increase the heap size, but the real solution is to write the data as it is generated and not build the byte[] at all. I assume you are not using a 32-bit JVM?

kidloco Over a year ago

Agreed. I guess we're slightly off topic now from the original question. Thanks for your help!

|

phsym · Accepted Answer · 2012-08-30 11:40:24Z

I think the solution is the same than in C with the mmap() function. If you just created the file, you should seek in it to data.length-1 offset and write a byte at this position in order to have a file of that size before mapping it. When I don't do it in C, I get memory corruption while accessing mapped memory.

Something like that should work :

private static void write(byte[] data) throws Exception {
        File file = new File("C:/temp/file.json");
        int length = data.length;
        RandomAccessFile raf = new RandomAccessFile(file, "rw");

        FileChannel fc = raf.getChannel();
        fc.position(size-1);
        ByteBuffer bf = ByteBuffer.wrap(new byte[]{0x00});

        bf.flip();  // Not sure if flip is needed !!!!!!!

        fc.write(bf);

        MappedByteBuffer buffer = fc.map(FileChannel.MapMode.READ_WRITE, 0, length);
        for (int i = 0; i < length; i++) {
            buffer.put(data[i]);
        }
}

To make simple : You can't map more than the file size, that's why you need to increase the file size before mapping it.

This can explain your problem, but I think it's more appropriate to open a FileOutputStream and directly write your data in it. You can still map it after if needed.

John Watts · Accepted Answer · 2012-08-30 11:09:32Z

0

Is your JVM max heap size at least 270*2 MB? You would set this on the command line used to start Java:

java ... -Xmx 1024m ...

answered Aug 30, 2012 at 11:09

John Watts

8,9151 gold badge35 silver badges36 bronze badges

5 Comments

Marko Topolnik Over a year ago

But MappedByteBuffer uses off-heap memory?

John Watts Over a year ago

I see a lot of "may reside" in the javadoc, but it is definitely a possibility the doc points out. Is there another reference you are consulting?

Marko Topolnik Over a year ago

No, I'm speaking from general knowledge---the point of the memory-mapped buffer is that it transparently uses natively allocated memory, which is the only way to avoid heap-to-native memcpy before actually flushing the buffer to the file. Of course, it may possibly turn out that the native buffer is itself on the heap, but that doesn't seem reasonable to me.

John Watts Over a year ago

I suspect you are right. In either case, I think @PeterLawrey has the more relevant answer. The OP may not even want memory mapped IO.

Marko Topolnik Over a year ago

Yes, a clean, straightforward, three-line solution, and speed is not compromised, either.

Collectives™ on Stack Overflow

IOException when writing byte array

3 Answers 3

6 Comments

Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related