Python: testing & loading different git commits of same package

Question

I have a python package on github, and I can install different commit versions of it using e.g. pip3 install git+https://github.com/my/package@commithash. I would like to benchmark various different commits against each other, ideally comparing two versions within the same python script, so that I can plot metrics from different versions against each other. To me, the most obvious way to do this would be to install multiple different versions of the same package simultaneously, and access them using a syntax something like

import mypackage_commithash1 as p1
import mypackage_commithash2 as p2

results1 = p1.do_something()
results2 = p2.do_something()

plot_comparison(results1, results2)

But as far as I can see, python doesn't support multiple packages of the same name like this, although https://pypi.org/project/pip3-multiple-versions goes some of the way. Does anyone have any suggestions for ways to go about doing these sorts of comparison within a python script?

sinoroc · Accepted Answer · 2020-03-16 13:15:17Z

1

That's too broad of a question to give a clear answer...

Having two versions of the same project running in the same environment, same interpreter session is difficult, near impossible.

First, maybe have a look at this potentially related question:

Versions management of python packages

1. From reading your question, another solution that comes to mind would be to install the 2 versions of the project in 2 different virtual environments. Then in a 3rd virtual environment I would run code that looks like this (kind of untested pseudo-code, some tweaking will be required):

environments = [
    'path/to/env1',
    'path/to/env2',
]

results = []
for environment in environments:
    output = subprocess.check_output(
        [
            environment + 'bin/python',
            '-c',
            'import package; print(package.do_something())',
        ],
    )
    results.append(parse_output(output))

plot_comparison(results)

2. Another approach, would be to eventually use tox to run the test program in different environments containing each a different version of the project. Then have an extra environment to run the code that would interpret and compare the results (maybe written on the filesystem?).

3. Maybe one could try to hack something together with importlib. Install the 2 versions under 2 different paths (pip install --target ...). Then in the test code, something like that:

modify sys.path to include the path containing version 1
import (maybe importlib can help)
run test 1
modify sys.path to remove the path containing version 1 and include the path containing version 2
import again (maybe importlib.reload is necessary)
run test 2
compare results

edited Mar 16, 2020 at 13:15

answered Mar 15, 2020 at 14:50

sinoroc

23.3k4 gold badges58 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

user2667066 Over a year ago

Could I import the packages from 2 local versions under different names? E.g. I could clone the entire github repo twice, to local folders mylibV1 and mylibV2, checkout versions V1 and V2 in the respective dirs, then simply do import mylibV1.mylib as V1 and import mylibV2.mylib as V2, then use V1.do_something() vs V2.do_something?

user2667066 Over a year ago

(p.s. this isn't using pip, obviously. But it might work for me.)

sinoroc Over a year ago

Try it. It could work, but most likely it won't. One thing coming to mind that could prevent this from working is the way the imports are done internally in this project. Usually when one does import somepackage or from somepackage import something, these are absolute imports and somepackage is a top-level package. Placing the packages under another directory, for example MyProjectV1, makes it so that MyProjectV1 is the top-level package. So unless all imports inside somepackage are relative (for example from . import something), the imports will most likely fail.

user2667066 Over a year ago

Thanks. It does seem to work for me:

>>> import versions.tsinfer_efbafff.tsinfer as V1 >>> import versions.tsinfer_1c17964.tsinfer as V2 >>> V1.__version__ '0.1.4' >>> V2.__version__ '0.1.5.dev140+g1c17964'

sinoroc Over a year ago

Looking at the code of what I assume is the project in question, I doubt this technique could work. This project definitely uses absolute imports to itself, which would be broken because of the extra directory levels. It's very important to make sure you don't have another version of the project already installed in the environment (path/to/pythonX.Y -m pip show tsinfer and path/to/pythonX.Y -m pip uninstall tsinfer), because if it were the case then some code would be imported from this version instead of the version in your directory.

|

Collectives™ on Stack Overflow

Python: testing & loading different git commits of same package

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related