Link f2py generated *.so file in a python package using setuptools

Question

I wish to deploy a package to PyPi using setuptools. However, the core part of the package is actually written in Fortran, and I am using f2py to wrap it in python. Basically the project's structure looks like this:

my_project

license.txt
README.md
setup.py
my_project
- init.py
- myfunc.py
- hello.so

The module myfunc.py imports hello.so (import my_project.hello) which can then be used by functions inside myfunc.py. This works perfectly on my machine.

Then I tried standard setuptools installation: sudo python3 setup.py install on my Ubuntu, and it gets installed perfectly. But unfortunately, while importing, it throws ModuleNotFoundError: No module named 'hello'.

Now, from what I understand, on Linux based systems, for python, the shared libraries *.so are stored in /usr/lib/python3/dist-packages/. So I manually copied this hello.so there, and I got a working package! But of course this works only locally. What I would like to do is to tell setuptools to include hello.so inside the python-egg and automatically do the copying etc so that when a user uses pip3 install my_package, they will have access to this shared library automatically. I can see that numpy has somehow achieved that but even after looking at their code, I haven't been able to decode how they did it. Can someone help me with this? Thanks in advance.

Are you happy for this to only function on Linux or does it have to work on Windows and Mac as well? Can it only work on the (most common) x86_64 CPUs or does it have to work on i686 too? — FiddleStix
– FiddleStix, Commented Nov 24, 2020 at 17:41
@FiddleStix If it works on Windows and Mac, it would be great but I don't insist on it. Also, x86_64 is sufficient. — Peaceful
– Peaceful, Commented Nov 25, 2020 at 3:58
Just to be certain, you would like to pre-compile the f2py module so that users do not need a Fortran toolchain themselves (but implying all Fortran related dependencies need to be statically compiled into the .so, (.dll, or .dylib) by you, in advance, for all platforms you intend to support)? Or is it an option to make the f2py compilation part of the install process? — jbdv
– jbdv, Commented Nov 25, 2020 at 8:53
@jbdv Either is fine with me. I am happy to compile and ship, but then those .so should automatically get installed at appropriate places on user's machine. Otherwise, it is also fine that user compiles, generates .so and they get placed at appropriate place from where they could be imported by my package. — Peaceful
– Peaceful, Commented Nov 25, 2020 at 9:08
I use a setup based on a SO answer where you commented. Does this not work for you? For a local install (sudo python -m pip install .), it places the .so file at .../site-packages/my_project_f90.cpython-36m-x86_64-linux-gnu.so at the same level as other project files e.g. .../site-packages/my_project-0.1-py3.6.egg-info. This same structure is reproduced when creating a wheel distribution e.g. sudo python setup.py bdist_wheel in build/bdist.linux-x86_64/wheel. Is this what fails for you? — jbdv
– jbdv, Commented Nov 25, 2020 at 10:22

meTchaikovsky · Accepted Answer · 2020-11-27 02:40:11Z

3

You can achieve this with a setup.py file like this (simplified version, keep only the relevant parts for building external modules)

import os
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext


class f2py_Extension(Extension):

    def __init__(self, name, sourcedirs):
        Extension.__init__(self, name, sources=[])
        self.sourcedirs = [os.path.abspath(sourcedir) for sourcedir in sourcedirs]
        self.dirs = sourcedirs

class f2py_Build(build_ext):

    def run(self):
        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        # compile
        for ind,to_compile in enumerate(ext.sourcedirs):
            module_loc = os.path.split(ext.dirs[ind])[0]
            module_name = os.path.split(to_compile)[1].split('.')[0]
            os.system('cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name))

setup(
    name="foo",
    ext_modules=[f2py_Extension('fortran_external',['foo/one.F90','foo/bar/two.F90'])],
    cmdclass=dict(build_ext=f2py_Build),
)

The essential parts for building an external module are ext_modules and cmdclass in setup(...). ext_modules is just a list of Extension instances, each of which describes a set of extension modules. In the setup.py above, I tell ext_modules I want to create two external modules with two source files foo/test.F90 and foo/bar/two.F90. Based on ext_modules, cmdclass is responsible for compiling the two modules, in our case, the command for compiling the module is

'cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name)

Project structure before installation

├── foo
│   ├── __init__.py
│   ├── bar
│   │   └── two.F90
│   └── one.F90
└── setup.py

Project structure after python setup.py install

├── build
│   └── bdist.linux-x86_64
├── dist
│   └── foo-0.0.0-py3.7-linux-x86_64.egg
├── foo
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-37.pyc
│   ├── bar
│   │   ├── two.F90
│   │   └── two.cpython-37m-x86_64-linux-gnu.so
│   ├── one.F90
│   └── one.cpython-37m-x86_64-linux-gnu.so
├── foo.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
└── setup.py

The two source files one.F90 and two.F90 are very simple

one.F90

module test

  implicit none

  contains

  subroutine add(a)

    implicit none
    integer :: a
    integer :: b
    b = a + 1
    print *, 'one',b

  end subroutine add


end module test

two.F90

module test

  implicit none

  contains

  subroutine add(a)

    implicit none
    integer :: a
    integer :: b
    b = a + 2
    print *, 'two',b

  end subroutine add


end module test

After I installed the package, I can successfully run

>>> from foo.bar.two import test
>>> test.add(5)
 two           7

and

>>> from foo.one import test
>>> test.add(5)
 one           6

edited Nov 27, 2020 at 2:40

answered Nov 26, 2020 at 5:50

meTchaikovsky

7,6963 gold badges18 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Peaceful Over a year ago

Thanks. I am testing this. My packages name contains a hyphen: foo-bar whereas actual directory names are foo_bar (with underscores). In setuptools.find_packages do I use foo-bar or foo_bar?

meTchaikovsky Over a year ago

@Peaceful You're welcome. You need to rename foo-bar to foo_bar in that case (foo_bar works), otherwise although you can build the package, you can't import it because of - in the package name.

Peaceful Over a year ago

This does work for me! The only thing I had to change was to change packages=setuptools.find_packages("foo") to a list containing the modulename and submodule names ['foo', 'foo/bar']. But before I accept this answer, as I said, I would also like to understand what is happening. I will then award the bounty!

meTchaikovsky Over a year ago

@Peaceful Cheers! Check out my updated post, I provided a compact setup.py file.

Peaceful Over a year ago

Will this work even on Windows? Will this use even on Windows? Another change that I made was to use numpy.f2py in place of f2py.

|

jbdv · Accepted Answer · 2020-11-27 05:17:23Z

1

+500

Here is an approach based on F2PY's documentation (the example there covers building multiple F2PY modules, and multiple source files per module), making use of numpy.distutils, that supports Fortran source files.

The structure of a minimal example with multiple F2PY extension modules is based on a src directory layout. It is not necessary/required, but has the advantage that the test routine cannot run unless the package has been installed successfully.

Source layout

my_project
|
+-- src
|   |
|   +-- my_project
|       |
|       +-- __init__.py
|       +-- mod1.py
|       +-- funcs_m.f90
|       +-- two
|           |
|           +-- pluss2.f90
|           +-- times2.f90
|
+-- test_my_project.py
+-- setup.py

setup.py

from setuptools import find_packages

from numpy.distutils.core import setup, Extension

ext1 = Extension(name='my_project.modf90',
                 sources=['src/my_project/funcs_m.f90'],
                 f2py_options=['--quiet'],
                )

ext2 = Extension(name='my_project.oldf90',
                 sources=['src/my_project/two/plus2.f90', 'src/my_project/two/times2.f90'],
                 f2py_options=['--quiet'],
                )

setup(name="my_project",
      version="0.0.1",
      package_dir={"": "src"},
      packages=find_packages(where="src"),
      ext_modules=[ext1, ext2])

__init__.py

The __init__.py file is empty. (Can e.g. import the F2PY modules here if desired)

mod1.py

def add(a, b):
  """ add inputs a and b, and return """
  return a + b

funcs_m.f90

module funcs_m
  implicit none
  contains
    subroutine add(a, b, c)
      integer, intent(in)  :: a
      integer, intent(in)  :: b
      integer, intent(out) :: c
      c = a + b
    end subroutine add
end module funcs_m

plus2.f90

subroutine plus2(x, y)
  integer, intent(in)   :: x
  integer, intent(out)  :: y
  y = x + 2
end subroutine plus2

times2.f90

subroutine times2(x, y)
  integer, intent(in)   :: x
  integer, intent(out)  :: y
  y = x * 2
end subroutine times2

test_my_project.py

import my_project.mod1
import my_project.oldf90
import my_project.modf90

print("mod1.add:            1 + 2 = ", my_project.mod1.add(1, 2))
print("modf90.funcs_m.add:  1 + 2 = ", my_project.modf90.funcs_m.add(1, 2))
x = 1
x = my_project.oldf90.plus2(x)
print("oldf90.plus2:        1 + 2 = ", x)
x = my_project.oldf90.times2(x)
print("oldf90.plus2:        3 * 2 = ", x)

Installing

Now, one can use pip to install the package. There are several advantages to using pip (including ease of upgrading, or uninstalling) as opposed to setup.py install (but this can still be used for building the package for distribution!). From the directory containing setup.py:

> python -m pip install .

Testing

And then, to test the just installed package

> python test_my_project.py
mod1.add:            1 + 2 =  3
modf90.funcs_m.add:  1 + 2 =  3
oldf90.plus2:        1 + 2 =  3
oldf90.plus2:        3 * 2 =  6

This setup has been tested with success on Windows 10 (with ifort), on Ubuntu 18.04 (with gfortran) and on MacOS High Sierra (with gfortran), all with Python 3.6.3.

edited Nov 27, 2020 at 5:17

answered Nov 26, 2020 at 23:59

jbdv

1,2631 gold badge11 silver badges21 bronze badges

12 Comments

Peaceful Over a year ago

This looks great. I will test it. If I have multiple fortran files, I will have to use Extension for each of them separately. Is that right? Also, each of my fortran files contains a separate fortran module, and so it should ideally generate .so files for each separately.

jbdv Over a year ago

As in the f2py docs, you have to create an ext_i = Extension(...) for each separate f2py module/.so you want to create, and then add them to ext_modules=[ext_1,...,ext_i] in setup.py. If a single f2py module consists of multiple Fortran source files, you list them under that particular ext_i's sources argument. I'll update my answer when i have a moment

Peaceful Over a year ago

That works on Linux! I will just test this on Mac and Windows and then will accept the answer.

jbdv Over a year ago

You're welcome, very happy with this setup myself, thanks to hours of experimenting/frustration, and eventually great documentation :-)

Peaceful Over a year ago

On Windows, do users need to have fortran installed for this installation to work?

|

Collectives™ on Stack Overflow

Link f2py generated *.so file in a python package using setuptools

2 Answers 2

11 Comments

Source layout

Installing

Testing

12 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

11 Comments

Source layout

Installing

Testing

12 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related