4

I am trying to deal with a massive numpy array that I then end up writing into a JPEG image using cv2.imwrite(numpy.array). Unfortunately, what I am working with does not fit into my RAM even though the final JPG image should come up to about 200 MB only.

How can I manage such loads without overloading my RAM?

Are there other ways to write an image without storing the entire array in my RAM at once? It's possible for me to load small bits of the array at a time, but I don't know which module/function to use to write to an image without storing the whole thing on my RAM at once.

As of right now, I have saved the whole image in 4 smaller images (quarters) because that's the best I could do with my limited RAM. But I still want to be able to stitch them together into one complete image. The target image is a 3-channel 26112 x 20480 image.

Improve this question

2 Answers 2

Reset to default
2

If the image is size 26112 x 20480 with three channels, at one byte per channel, the uncompressed data takes 3 x 26112 x 20480 bytes, which works out to about 1.5 gigabytes. The JPEG file can be much smaller than that because it uses lossy compression, but the OpenCV representation of the image does not.

Some operations (such as cropping on block boundaries) are possible on the compressed representation as evidenced by Jpegtran but for most things you'd do in OpenCV you will have to decompress the data. If your algorithm happens to be something that can be written directly in terms of the DCT coefficients of 8x8 blocks, you won't have to decompress, but OpenCV won't help you there.

I don't know if it is feasible with OpenCV, but numpy supports memory-mapped I/O of raw array data. When you read a page of data that is not in memory, the operating system reads it from the mapped file, and if it's running out of memory, it will evict some pages. It will depend on your data access patterns if this is going to be performant. In principle you should be able to simply call mmap on a file into which you have decompressed all the image data in the correct memory layout and wrap the pointer in an OpenCV array, but I haven't tried this myself.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

You may be able to use a sparse matrix for large image manipulation. One library that might work for you is Eigen. Here is a tutorial on using sparse matrices with that library.

OpenCV also supports sparse matrices.

Improve this answer

2 Comments

But the image isn't a sparse matrix. Unless I am wrong, aren't sparse matrices the ones with many zeros?
sparse matrix contains only none zero values and when you convert it back to dense matrix it returns you matrix with your original data and when yo save your image using sparse matrix it will write original data of you image to any extension you want..

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.