Most tutorials on compressing a file in Python involve immediately writing that file to disk with no intervening compressed python object. I want to know how to pickle and then compress a python object in memory without ever writing to or reading from disk.
4
-
You can't compress an object, AFAIK. You can compress the representation of an object (serialized, for instance). The most obvious representation would be a string, so you may want to check this questionloopbackbee– loopbackbee2013-10-21 17:04:30 +00:00Commented Oct 21, 2013 at 17:04
-
Ok, so how do I pickle and then compress the representation in memory?Michael– Michael2013-10-21 17:07:10 +00:00Commented Oct 21, 2013 at 17:07
-
Compression with what compression technique? There are multiple to choose from.Martijn Pieters– Martijn Pieters2013-10-21 17:08:50 +00:00Commented Oct 21, 2013 at 17:08
-
Just pickle to a string then compress the string (see my previous link)loopbackbee– loopbackbee2013-10-21 17:08:53 +00:00Commented Oct 21, 2013 at 17:08
Add a comment
|
3 Answers
I use this to save memory in one place:
import cPickle
import zlib
# Compress:
compressed = zlib.compress(cPickle.dumps(obj))
# Get it back:
obj = cPickle.loads(zlib.decompress(compressed))
If obj has references to a number of small objects, this can reduce the amount of memory used by a lot. A lot of small objects in Python add up because of per-object memory overhead as well as memory fragmentation.
1 Comment
Hammad Hassan
In python 3, we need pickle as there is no cPickle.
Batteries included.
>>> zlib.compress(pickle.dumps([42]))
'x\x9c\xd3\xc8)0\xe0\xf241\xe2J\xd4\x03\x00\x10A\x02\x87'
Comments
bz2.compress(pickle.dumps(some_object))
