creating static library from Python code on Ubuntu

Question

Context: I want to use some of the Python code that I wrote and call it from Matlab (not an ideal set up, but that's the constraint I have to live with for now). The code uses numpy, scipy, and pandas packages. I have tried dynamically linking options and got some success with matpy. However, to use numpy I need to override Matlab's own libraries (libgfortran, liblapack, libblas, libstdc++) and instead point to system's version of the same libraries using LD_PRELOAD. Then if Matlab code downstream tries to use these libs, it invariably segfaults. This is turning out to be less reliable, and even if I fix all the issues with this now, it is prone to unpredictable consequences later on as the downstream Matlab code evolves.

Hence I have decided to compile python code into a static lib (including all dependencies from numpy, scipy and pandas), and then write a Mex that links against this static lib.

In that context, I want your expert opinion on following questions:

Which are the most reliable tools for compiling the python code into a STATIC lib with the goal of then linking it into a Mex? I am for now only looking at Ubuntu 12.04 platform. I looked at Cython, Pyrex, pyInstaller, py2exe, cx_freeze -- great tools, but being a newbie to python and not having compiled python into anything but the automated .pyc, I need your help in making this choice.
What are the pitfalls / things that could go wrong / things I should be careful about in this set up? I am referring to (custom.py + numpy + scipy + pandas) --> static lib ---> Mex -->called from Matlab setup.

Thanks a ton!

GaryBishop · Accepted Answer · 2014-10-11 12:30:59Z

2

I suggest you communicate between the Mex and Python using a socket instead of calling the Python code directly. The OS will do the the socket transfer using a memory-to-memory (or mapping tricks) so it will go fast. Once you've got separate executables, you should be able to get out of this shared library hell.

If you still need Python to be a single executable, consider pyinstaller or cx_freeze.

edited Oct 11, 2014 at 12:30

answered Jun 20, 2014 at 11:14

GaryBishop

3,9223 gold badges25 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Kalpit Over a year ago

Thanks Gary! I need to learn how sockets work, will probably come back with more questions. To ensure I understand the bigger picture, the solution is 1. compile all python code including deps as exec (or static lib? or it doesn't matter?) 2. compile Mex that talks to the exec through sockets 3. call this mex from Matlab. Right?

GaryBishop Over a year ago

That'd work though I think you could avoid making all the python into a single executable if you have access to a full python implementation in the server environment where this will run. It all builds pretty nicely in an arbitrary folder without requiring root access, just set --prefix=/your/path in the configure step on the build of python. Sockets are trival to use from Python and I see examples online of how to do it from Matlab using the ability to call Java inline. So, you could avoid using a Mex file all together.

Collectives™ on Stack Overflow

creating static library from Python code on Ubuntu

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related