1

Im trying to read a byte array from a Pointer using JNA and I keep getting:

Decompress with insz 11107, and outsize 65536
recieved 1
Decompression complete!
Decompress with insz 22112, and outsize 65536
recieved 1
Decompression complete!
Decompress with insz 22041, and outsize 65536
recieved 1
Decompression complete, final out size of 0!
Exception in thread "main" java.lang.Error: Invalid memory access
    at com.sun.jna.Native.read(Native Method)
    at com.sun.jna.Pointer.read(Pointer.java:149)
    at com.sun.jna.Pointer.getByteArray(Pointer.java:715)
    at me.TTARCHExtract.redo(TTARCHExtract.java:330)
    at me.TTARCHExtract.redo(TTARCHExtract.java:323)
    at me.TTARCHExtract.z_decompress(TTARCHExtract.java:313)

when I execute the code below:

public static final Pointer toPointer(byte[] array, int length){
        Memory ret = new Memory(length);
        ret.write(0, array, 0, length);
        return ret;
    }
//code starts here
private byte[] z_decompress(byte[] in,int insize,byte[] out,int outsize) {
        if(in==null)return null;
        System.out.println("Decompress with insz "+insize+", and outsize "+outsize);
        Pointer inptr = toPointer(in, insize);
        Pointer outptr = toPointer(out, outsize);
        ZStream deflate = new ZStream();
        ZStream z = new ZStream();
        TTARCHHelper.load();
        ZlibLibrary lib = TTARCHHelper.ZLIB_LIBRARY;
        this.initz(lib, z, 15);
        this.initz(lib, deflate, -15);
        return this.redo(insize, outsize, z, lib, inptr, outptr, true, deflate);
    }

    private byte[] redo(int insize,int outsize,ZStream z,ZlibLibrary lib,Pointer inptr,Pointer outptr,boolean first,ZStream deflate) {
        lib.inflateReset(z);
        z.next_in=inptr;
        z.next_out=outptr;
        z.avail_in=insize;
        z.avail_out=outsize;
        int out = lib.inflate(z, ZlibLibrary.Z_FINISH);
        if(!first)System.out.println("recieved "+out);
        if(out != ZlibLibrary.Z_STREAM_END) {
            if(first)return this.redo(insize, outsize, deflate, lib, inptr, outptr, false, null);
            System.out.println("Compressed zlib/deflate input at offset "
                    + ""+dgboff+" ("+insize+" > "+outsize+") is wrong or complete");
            System.exit(-1);
            return null;
        }
        System.out.println("Decompression complete!");
        return z.next_out.getByteArray(0, outsize);
    }

private void initz(ZlibLibrary lib,ZStream z, int w) {
        lib.inflateInit2_(z, w, lib.zlibVersion(), z.size());
    }

The getByteArray is where the error is happening. What could be causing this?

This error happens sometimes and with not all zlib input streams so Is it to do with the out size maybe being wrong?

Code is from C written project ttarchext

Improve this question
7
  • I’ve updated to show the full code which might make it make more sense Commented Aug 27, 2020 at 21:55
  • This is from something written in C yes, here is the function in C which does this: pastebin.com/a1LtSkX5 Commented Aug 27, 2020 at 22:01
  • The stack trace points to the error occurring on the first recursive call of redo(). From there, the only change is the fact that you are using deflate rather than z as the method parameter. The only difference between those is the -15 as the windowsize parameter for inflateInit2_(). The docs I read say it should be between 8 and 15. Should you be calling inflateInit2() without the _? Commented Aug 27, 2020 at 22:44
  • Having it negative means its a raw deflate stream, and I debugged it and the error is happeneing on line 330 which is return z.next_out.getByteArray(0, outsize); Does this error occur because there are no bytes at the pointer? The inflate2_ has an underscore for some reason, I checked in the DLL function exports Commented Aug 27, 2020 at 22:47
  • You missed the point. That's the symptom, not the cause. It's suceeding in the first two cases when you don't call recursively. It's failing when it's called recursively and the difference there is a minus sign. The code you are porting uses a different function: is inflateInit2() the same as inflateInit2_()? Commented Aug 27, 2020 at 22:50

1 Answer 1

Reset to default
1

An Invalid Memory Access error in JNA is a sign that you are attempting to access memory which has not been allocated. In this case, the next_out pointer, with the full length of outsize. To debug this, you need to consult the API to see whether the function expects you to allocate the memory and pass it to the native function, or whether the native function itself will allocate the necessary memory. (In the latter case, the native code usually tells you how to free the memory when you're done with it.) For this API, the allocation is apparently done in the inflateInit2() call, so that's a hint toward the bug's root cause.

The output is instructive in that it shows that it succeeds once with a smaller insz but fails the second time with a larger insz. The difference is also evident in the stack trace for the crash, showing that the recursive call occurred in this second (larger input) case, but likely did not in the first case. (To confirm this in debugging, you should add some more output.)

For the recursive call, the only change is that instead of ZStream z, the third parameter is changed to Zstream deflate (where null is passed as the possible next value on the iteration.) While changing z to deflate seems right, I don't see where in the original code there should be a null. This seems to be intended to recurse as a "next" type of iteration until it's done. (This may not be the cause of the error but is suspicious.)

The only difference in the redo() call with the deflate argument instead of z is that deflate was called with a windowsize of -15. This seems to be contrary to the documentation for inflateInit2_() which you have mapped:

The windowBits parameter shall be a base 2 logarithm of the maximum window size to use, and shall be a value between 8 and 15.

Since the original C code you're porting also used -15, this may be correct, but it's clear the different windowsize has an impact on the output.

I would suggest keeping deflate as the last argument of the recursive call instead of null, and adding more output statements to give you more insight on the values of the parameters as you recurse.

The other variable which could cause the error is the outsize value. This seems to imply the full outsize value is avialable to read, which may not be the case if you have reached the end of the allocation. It is possible outsize is a minimum size (perhaps the windowsize = 15 causes this to be true) the first time, but when recursing (the windowsize = -15 case) that can not be relied upon, and you should read fewer bytes from the output on the final iteration (reviewing the original source suggests z.total_out.)

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

I realised in the code I made a mistake, I should have passed Z_FINISH instead of Z_STREAM_END (see refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/… its well documented there). I updated the code and stack trace. I call the redo recursively because it gets called twice and writing the code twice would not be efficient. Im still getting the same error but its working more (z_decompress is called many times for many streams). The second call is the stream which almost always works, the first call of redo() is just in case it is a valid stream before using -15 instead
outsize is almost always 65536 as the data being read is compressed into uncompressed chunks which are each 65536 bytes in size
I debugged more and looped and kept getting data from the next_out ,which is the buffer which should contain outsize bytes of the uncompressed data, and I saw the last amount of bytes which can be retrieved from the pointer before the invalid memory access is thrown. The last number of bytes that can be retrieved fron the next_out pointer varies. This must be problem with the zlib code?
I tried to explain what you describe in my answer. It reads "full" chunks until the last one which only has the remainder in it (which varies). Did you try z.total_out instead of outsize? I doubt the existing library code is broken, but I don't think you've accurately ported those two parts that I pointed out.
You were right!! I was reading from next out which gets read to in chunks. I read from the out pointer instead! It works now! Thank you so much
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.