1

I have about 700 Python source files (.py) of a few kilobytes of size (average file size is 12 kB, but there are many 1 kB files as well), and I'd like to create a compressed archive containing all of them. My requirements:

  • The archive should be small. (.zip files give me a compression ratio of 3.816, I need something smaller than that. A .rar file created with rar -s -m5 a gives me a compression ratio of 6.177, I'd prefer 7 or more.)
  • The compression must be lossless, it must preserve the original file bit-by-bit. (So minification is out.)
  • There must be a small library written in C which can list the archive and extract individual files.
  • The decompression library must be fast, i.e. not much slower than zlib, preferably faster.
  • If I want to extract a single file, I don't have to uncompress large, unrelated portions of the archive. (So compressed .tar files are out, and solid .rar files are out.)
  • Since all .py files are small (only a few kilobytes in size), I don't need a streaming decompressor or seeking support within a file.
  • If possible, decompression should be initialized from a context dictionary generated from the union of the .py files, to save more space.

Which compression algorithm and C decompression library do you recommend?

I know about the concept of code minification (e.g. removing comments and extra whitespace, renaming local variables to single letter), and I'll consider using this technique for some of my .py files, but in this question I'm not interested in it. (See a Python minifier here.)

I know about the concept of bytecode compilation (.pyc files), but in this question I'm not interested in it. (The reason I don't want to have bytecode in the archive is that bytecode is architecture- and version-dependent, so it's less portable. Also .pyc files tend to be a bit larger than minified .py files.)

See my answers containing plan B and plan C. I'm still looking for plan A, which is smaller than ZIP (but it will be most probably larger than .tar.xz), and it has smaller overhead than .tar.xz.

Improve this question
6
  • 3
    I would have recommended a tar with xz (LZMA2) compression, because as far as I can tell, it generally has the highest compression ratios out there, but you ruled them out both by rejecting the tar format, and the fact that LZMA2 is a lot slower than zlib. Commented Jul 30, 2011 at 10:02
  • You can try all the usual suspects in the world of compression algorithms. If none of those fits your needs, you are probably out of luck. No algorithm will guarantee a compression ratio of 7 or more, as this depends on the input. Commented Jul 30, 2011 at 10:05
  • @Ferdinand Beyer: I'm not only looking for a compression algorithm, but also for a suitable archive format and a decompressor for the archive. Also in this question I'm already looking for something readily available before I commit time to try lots of algorithms and craft something based on them manually. Commented Jul 30, 2011 at 10:12
  • 1
    Are you really sure you need all these "features"? Nowadays, disk space is cheap and networks are fast, why don't you use the plain-old Zip format? Another advantage is: Python can import modules from zip files out of the box. Commented Jul 30, 2011 at 10:18
  • 1
    If you do pick a Python minifier, take a look at my minipy as well as Dan McDougall's. Commented Jul 30, 2011 at 11:11

3 Answers 3

Reset to default
4

I know you've rejected .zip, but it might change your decision if you realise that Python is already capable of importing packages straight from zips, in the form of egg files. No extra code required, except for the setuptools configuration file.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I've rejected ZIP in this question, so ZIP is not a good answer to this question. I know the advantages of ZIP, but in this question I need something smaller.
2

You should consider LZMA (also see the C SDK).

Improve this answer

4 Comments

Thank you for suggesting LZMA. But I can't accept is an answer to my question, because I'm not only looking for a compression algorithm, but also for an archive format possibly with a dictionary, and a C decompression library for the archive. I know that I could write something for myself based on LZMA, but now I'm looking for a library already available.
@pts, you can look at 7z as an archive format (library available along with command line tools).
@pts: Have you considered the 7z archive format as described in Radu's link? 7-zip.org/7z.html
Yes, thank you for suggesting 7z, that's something I'll consider and measure. Preliminary measurement: a non-solid 7z archive (and even a half-solid one with 7z -mx=5 -ms=4000k) is too large -- but a solid 7z archive is equivalent to .tar.xz, see the detailed analysis in the question.
0

FYI Plan B is just to use ZIP files. That's what I'm doing currently. Storing .py files in ZIP archives is very convenient for Python, because Python can load .py files from ZIP archives directly. But I need something smaller than a ZIP file, that's why I asked the question.

FYI Plan C is to use .tar.xz. Here is the analysis. The Linux kernel and Busybox 1.8.5 contain an .xz decompressor, which compiles to 18 kB of x86 code, which fulfills my requirement of a small decompression library. The .tar.xz with xz -6 -C crc32 gives a compression ratio of 6.648 over the .tar file. The overhead of the .xz decompressor of Busybox 1.8.5, compiled for x86 is 17840 bytes in code size (comparing the executable to the .tar.xz file). So this is plan C: when the executable starts, extract the whole archive into memory. (This takes about 0.35 second on my machine, the output is a 9MB memory block.) To read a file from the archive, use its in-memory uncompressed representation. This will be very fast. This backup plan is not a solution to my problem, because it involves a 0.35 second overhead at the beginning of execution, and it needs 9MB of extra memory.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.