Is there a way for a python script to load and use environment modules? os.system('module load xxx') doesn't work since it executes them in a subshell (at least, I think that's what's happening).
Add a comment
|
5 Answers
I know this question's kind of old but it's still relevant enough that I was looking for the answer, so I'm posting what I found that works as well:
At least in the 3.2.9+ sources, you can include the python "init" file to get a python function version of module:
>>> exec(open('/usr/local/Modules/default/init/python.py').read())
>>> module('list')
No Modulefiles Currently Loaded.
>>> module('load','foo')
>>> module('list')
Currently Loaded Modulefiles:
1) foo/1.0
I've been told earlier versions can do the same without the .py extension, but that's second hand, so ymmv.
Alternative "init" file location (from comment by @lib): /usr/share/Modules/init/python.py
To use with Python 3, version 4.0 or later of Environment Modules is required, as that is the first version to have a bug-free Python3-compliant version of the Python init file.
6 Comments
marshall.ward
This is much closer to what we actually use (since the
popen calls to modulecmd are in the python.py file that you cited), so I'll set this to the answer. Thanks!
lib
In the server I am using it is in /usr/share/Modules/init/python.py . I found the path be running module --version (the correct entry seems MODULES_INIT_DIR)
Dave X
Cool, I found this answer looking for a solution for perl, and my particular setup (VERSION 3.2.7 on RHEL 6.2) has initialization code for several shells in /usr/share/Modules/init/: bash, csh, ksh, perl, python, sh, tcsh, and zsh. Thanks for the pointer.
srand9
execfile() is deprecated in python 3.6.0. How to use exec statement with python 3.6 to use environment modules?
jnewman
Thanks @Jarvis. The answer has been updated for Python 3
|
One of our admins was able to solve the problem for me using os.popen() calls to modulecmd:
cmd = os.popen('/path/to/modulecmd python load my-module')
exec(cmd)
Comments
While the accepted solution works, I found it to be easier to write:
import sys
sys.path.insert(0, '/path/to/environment/modules')
# Environment modules become available by loading python.py (the name choice could be better here)
from python import module
# ...
module('use', '/some/dir')
module('load', 'program/1.2.3')
This looks more pythonic to me and also it allows IDEs to offer auto-completion etc.
Comments
Not directly, but here's one possible workaround, depending on your environment. Assuming you can preface your system command with ENVVAR=value, you can do something along these lines:
import os
os.environ['EDITOR'] = 'vi'
cmd = "EDITOR=%(EDITOR)s $EDITOR" % os.environ
os.system(cmd)
The code assigns vi to your EDITOR environment variable, then passes it on the command line and runs the command, which (in this case) is EDITOR.
Comments
I found the answers from jnewman, andreee and ian quite helpful. However, in my case the $MODULESHOME/init/python.py is written in Python 2 and is incompatible with Python 3 (it uses exec output instead of exec(output)). Here is my modified version that supports Python 3:
import os
import subprocess
def load_module(module_name, *, capture_output=True, check=True, **kw):
module_home = os.environ["MODULESHOME"]
modulecmd = os.path.join(module_home, "bin/modulecmd")
process = subprocess.run(
[modulecmd, "python", "load", module_name],
capture_output=capture_output,
check=check,
**kw,
)
return process
Example use case:
CONDA_MODULE = "Anaconda3/2022.05"
load_module(CONDA_MODULE)
print(subprocess.check_output(["conda", "--help"], stderr=subprocess.STDOUT).decode("utf8"))
