169

What I'm trying to do is ship my code to a remote server, that may have different python version installed and/or may not have packages my app requires.

Right now to achieve such portability I have to build relocatable virtualenv with interpreter and code. That approach has some issues (for example, you have to manually copy a bunch of libraries into your virtualenv, since --always-copy doesn't work as expected) and generally slow.

There's (in theory) a way to build python itself statically.

I wonder if I could pack interpreter with my code into one binary and run my application as module. Something like that: ./mypython -m myapp run or ./mypython -m gunicorn -c ./gunicorn.conf myapp.wsgi:application.

8
  • Do you mean something like cx_freeze? Commented Oct 7, 2016 at 9:47
  • @TigerhawkT3, I've never heard of cx_freeze until today. Looked through docs, seems like it's what I want, but it fails to build on Ubuntu 12.04.5 with python3.5.1. I'm looking for alternatives, thanks for kick in the right direction. Commented Oct 7, 2016 at 11:30
  • 1
    Sadly, pyinstaller doesn't work with Django 1.9 and py2app/py2exe don't support linux :( Commented Oct 7, 2016 at 11:39
  • 2
    if cx_Freeze and pyinstaller don't work, there is still hope. You can create package(s) for your target distribution(s) which will resolve all dependencies on installation. Or, maybe use something like docker to run your app. Commented Oct 11, 2016 at 21:29
  • 1
    other options are nuitka and cython Commented Oct 11, 2016 at 22:14

7 Answers 7

213
+300

There are two ways you could go about to solve your problem

  1. Use a static builder, like freeze, or pyinstaller, or py2exe
  2. Compile using cython and link against a statically-compiled version of CPython

This answer explains how you can go about doing it using the second approach, since the first method is not cross platform and version, and has been explained in other answers. Also, using programs like pyinstaller typically results in huge file sizes, while using cython will result in a file that's much smaller

  1. First, install cython.

    sudo -H pip3 install cython
    
  2. Then, you can use cython to generate a C file out of the Python .py file (in reference to https://stackoverflow.com/a/22040484/5714445)

    cython example_file.py --embed
    
  3. Use GCC to compile it after getting your statically-compiled python version (Note: The below assumes you are trying to compile it to Python3)

    gcc -Os $(python3-config --includes) example_file.c -o output_bin_file $(python3-config --ldflags --embed)
    

You will now have a binary file output_bin_file, which is what you are looking for

Other things to note:

  1. Change example_file.py to whatever file you are actually trying to compile. Note: Cython may not approve of filenames containing dashes (-).
  2. Cython is used to use C-Type Variable definitions for static memory allocation to speed up Python programs. In your case however, you will still be using traditional Python definitions.
  3. If you are using additional libraries (like opencv, for example), you might have to provide the directory to them using -L and then specify the name of the library using -l in the GCC Flags. For more information on this, please refer to GCC flags
  4. The above method might not work for anaconda python, as you will likely have to install a version of gcc that is compatible with your conda-python.
  5. If your Python version is not statically-compiled, this will result in a dynamic binary
Sign up to request clarification or add additional context in comments.

8 Comments

Hi, what if my application spans across multiple files. This approach seems to only compile the current script and not all the Python dependencies.
no need to rename your python file (say test.py) into a pyx right now
This was of great help to me. To answer the question about multiple files, you could use cythonize: cythonize -i file_0.py [...] file_n.py for all your python modules, then cython main_file.py --embed for your main script. You'll end up with an executable and .so files (.dll, .dylib depending on your OS) that you can use together as a standalone solution.
I am getting "No modules named encodings". I got to know I require certain files from python installation. However, my target system doesn't have python. Please help.
Warning: Cython requires dynamically loading stuff, and it DOES NOT produce self-contained binaries. Don't waste your time!
|
21

You might wish to investigate Nuitka. It takes python source code and converts it in to C++ API calls. Then it compiles into an executable binary (ELF on Linux). It has been around for a few years now and supports a wide range of Python versions.

You will probably also get a performance improvement if you use it. Recommended.

4 Comments

I couldn't find any information about using Nuitka with Django. Do you, by any chance, have such experience?
@roboslone, Prakhar Agarwal Cython supports external libraries really well, when compared to Nuitka PyPy's static builder. For instance, from the link at the end of this comment, "people use Cython because it helps them solve a problem. Which is either that they want to connect to external non-Python libraries from Python code or that they want to be able to manually optimise their code, or both.", the link being blog.behnel.de/posts/indexp241.html
R. S. Nikhil Krishna, I'm not trying to make my code faster, I'm trying to ship it in one binary with python interpreter inside.
Aah okay. I thought you wanted to bind it with some external library. My bad.
6

You're probably looking for something like Freeze, which is able to compile your Python application with all its libraries into a static binary:

PyPi page of Freeze

Python Wiki page of Freeze

Sourceforge page of Freeze

2 Comments

Actually, all current python packaging tool is doing fake package. They strongly rely on libc. Once you put those fake binary file into ubuntu7 and run them, you'll see many errors.
I simply wish someone could do a full static compilling for python. So that the final binary excutable will get running even without glibc.
3

I have found that using cython is the best, minimal and maintainable way to go forward. Below, I have pasted some code which:

  • Demonstrates using multiple files (dependencies)
  • Calling the main file
  • Using the static files from the storage

Background

I was building a flask application, which was intended to be consumed as an API from another service. I wanted to deploy on the server where the code is not readable, so I had to cover my Flask application into some sort of binary format.

File structure:

The following is the file structure of my project:

app.py
functions.py           (used from app.py)
some_more_functions.py (used from functions.py)
resources/
|    some_file_1.pkl
|    some_file_2.pkl

What we want to do

The method I am going to show you will only create binary of the selected files, without touching or compiling the libraries, or any other dependencies.

When you use some other libraries like pyinstaller, it will try to compile everything, even some heavy libraries like torch and it takes time and is often error prone.

If your purpose is to hide the source code and run the file smoothly, then take this approach.

Step - 1: Create a setup.py file

This file is the main file, which will compile your source code.

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules=cythonize(["app.py", "functions.py", "some_more_functions.py"]),
)

That's it.

Step - 2: Compile the files

python setup.py build_ext --inplace

The above command will create the binary versions of your mentioned files. In linux, they will be .so and in windows they will be .pyd.

We are almost done.

Step - 3: Test your files

After compiling, we can't simply run the file with python app.so we need some "app runner".

So, create a new file which will call your main app.

# new file `main_app.py`

import app  # Import your compiled module

if __name__ == '__main__':
    app.app.run(debug=True)  # Assuming 'app' is the Flask app instance in app.so

Then call it with:

python main_app.py

👉🏻 So how is it better?

  • We didn't need to compile any dependency.
  • We just compiled the source code.
  • No need to reset any resources path which the source code points to, they will get automatically be pointed.
  • Compilation usually takes 1-2 minutes.

I hope this helps!

2 Comments

When trying this with a script of mine, and after some gcc invocations happen, I get copying build/lib.linux-x86_64-cpython-39/my_subdir_name/my_script_name.cpython-39-x86_64-linux-gnu.so -> my_subdir_name.
@einpoklum Should not be a dealbreaker
2

If you are on a Mac you can use py2app to create a .app bundle, which starts your Django app when you double-click on it.

I described how to bundle Django and CherryPy into such a bundle at https://moosystems.com/articles/14-distribute-django-app-as-native-desktop-app-01.html

In the article I use pywebview to display your Django site in a local application window.

2 Comments

My application runs on Linux mostly, so py2app doesn't apply
Though the article uses py2app, it can be done using pyinstaller, too.
2

I have created a docker image that relies on Nuitka and a custom statically linked python3.10 to create a static binary.

Did not test it extensively, if you have the chance please let me know if it works for your use case.

You can check it at: https://github.com/joaompinto/docker-build-python-static-bin

2 Comments

Thanks for sharing, it is working in ubuntu7.
And based on your docker file, I even compiled all python module, so that the whole python3.10.4 is full statically usable in any amd64 machine: drive.google.com/file/d/1nR1owjXuzoxOw9lie5Js5JyCOI1xscBQ/…
1

Freeze options:

However, your target server should have the environment you want -> you should be able to 'create' it. If it doesn't, you should build your software to match the environment.

I found this handy guide on how to install custom version of python to a virtualenv, assuming you have ssh access: https://stackoverflow.com/a/5507373/5616110

In virtualenv, you should be able to pip install anything and you shouldn't need to worry about sudo privileges. Of course, having those and access to package manager like apt makes everything a lot easier.

2 Comments

I'm already using virtualenv, this is not the case. My question is about statically compiled python witn required modules and applications.
Then you have to freeze it, no other way around that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.