0

I work on a computing cluster. The global Python version on the node is 3.8. I have a virtual environment that I use when I start a job. The cluster uses LSF for scheduling. I usually access it via Open On Demand, start a Jupyter notebook, and switch the kernel to the virtual environment the contains all the dependencies of my project.

Now I want to use a Python library that requires Python version >=3.9. If I was just using bsub to submit jobs I would create a docker image with everything I need in it. But I would prefer to work in a Jupyter notebook. It appears that I cannot create a virtual environment with a Python version higher than the global version.

I have contacted support but let's assume I cannot change the global python version on the node. What is the best way to make my project work if some dependencies require Python 3.9? I have thought about trying to use a Docker image to create a virtual environment on the disk, but I am not sure if that will work.

I would appreciate any help in this matter.

Improve this question
7
  • 3
    You cannot create a Python virtual environment with a specific Python version without installing that version first. So you need to install Python 3.9 somehow. Look at conda, pyenv, asdf. Commented Jun 11 at 21:17
  • You could maybe look into devcontainers (containers.dev) as a solution for the general problem of "I don't control the machine I have to dev on but can run containers on it" Commented Jun 11 at 21:28
  • 1
    Download the Python installer for the version of choice, install. Point your virtualenv to that version’s EXE. Done. Commented Jun 11 at 21:28
  • @s3dev, ...that's making a fair set of assumptions. Locked down / highly-secured systems often flag user-writable partitions noexec, preventing any files stored there from being invoked as executables. Commented Jun 11 at 21:34
  • @WCharles, your local sysadmin team will know the constraints that apply to your specific system better than we do. Stack Overflow is better suited to questions about writing code, whereas here details of how your site does their system administration are pertinent to determine exactly which answers will/won't work. Commented Jun 11 at 21:35

2 Answers 2

Reset to default
2

  1. Install miniconda (a minimal version of conda) following instructions in this link. You can do it as a regular user, no sudo needed.

  2. conda init bash and relogin (or source) to add conda to your PATH.

  3. conda create -n env_name python=3.10

  4. conda activate env_name

Now you can install packages with conda install or pip install, but better to stick to one method.

Conda is really cool piece of software - it combines python version management, virtual environment and package installing.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

uv is a very popular way these days to work with current Python without affecting your global version. Under highlights it lists:

"A single tool to replace pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and more."

https://x.com/kolibril13/status/1885269093445460015 Janaury 2025

"With yesterday's release of uv, it's now possible to start any python version with uvx. For example: uvx [email protected] or uvx [email protected] "

https://bsky.app/profile/simonwillison.net/post/3lhf5upktgs2m January 2025

"Tiny TIL: I just figured out how to run pytest with a different Python version against my pyproject.toml/setup.py projects using 'uv run' uv run --python 3.12 --with '.[test]' pytest https://til.simonwillison.net/pytest/pytest-uv "

Plus, see the 'The Modern Approach: pyproject.toml with uv' section of Why Use uv Projects Instead of requirements.txt?.

Oh wait, despite your post's title, I see "But I would prefer to work in a Jupyter notebook." You'll want to see this post from kolibril13 (Jan-Hendrik Müller).

Also see this discussion (Sorry, it is not easy to format that is here. Please check out the links.):

https://x.com/rahuldave/status/1845847321176678893 https://x.com/kolibril13/status/1845771352747221430 October 2024

"It was always difficult to install @ProjectJupyter locally. I think uv simplifies this process a lot!
1️⃣ install uv with https://docs.astral.sh/uv/getting-started/installation/ 2️⃣ install + start jupyter with uvx jupyter lab The whole process took about 30 seconds.
More about uv+jupyter in this 🧵 1/7" "Best of all, the command for both the initial installation-and-first-run, as well as all future runs, is always the same: uvx jupyter lab.
This is thanks to some smart caching magic happening under the hood with uv."
"It's also possible to use other python versions.
"uvx --python 3.13 jupyter lab" will start with the latest python, and it will even manage the python installation for you."
"here's the same with "uvx --python 3.10 jupyter lab". It should be noted that I did not have any python 3.10 installation before on my system, and all the screen recordings are in real time, not edited."
"convenient way to install packages within the notebook: !uv pip install matplotlib" "packages can also be added by the --with flag.
uvx --python 3.12 --with anywidget jupyter lab.
This is in particular useful for packages like #anywidget by @trevmanz, as some packages (e.g. anywidget) cannot be installed in an already running notebook kernel."
"Finally, it’s worth mentioning that the above examples are great for prototyping ideas or running one-off scripts that don't live within a project. And that’s it! Thanks for reading, and feel free to share your experiences with uv in the comments ✨ 7/7" https://x.com/rahuldave/status/1845847321176678893 >"How does uv know to install jupyterlab? I am having my students do pixi init folder; cd folder; pixi add python numpy matplotlib scipy scikit-learn pandas jupyterlab; pixi run jupyter lab these days but you seem to be suggesting that the incantations can be smaller with uv?"
https://x.com/kolibril13/status/1845850589944528963.
>'> How does uv know to install jupyterlab?
uvx is alisas for uv tool run https://docs.astral.sh/uv/reference/cli/#uv-tool-run and tools are Python packages that provide command-line interfaces. https://docs.astral.sh/uv/concepts/tools/ So uvx jupyter lab is the same as uv tool run jupyter lab Furthermore from their docs:
"Tools can be invoked without installation using uv tool run, in which case their dependencies are installed in a temporary virtual environment isolated from the current project." [...] "When running a tool with uvx, a virtual environment is stored in the uv cache directory and is treated as disposable, i.e., if you run uv cache clean the environment will be deleted."
> but you seem to be suggesting that the incantations can be smaller with uv?
yep, I think the uv interactions can be smaller than with pixi, as you don't have to activate environments.'

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.