Skip to content

Conversation

@daytonb
Copy link

@daytonb daytonb commented Nov 14, 2025

This pull request is an effort to implement a more language-native way to get the "module" function in Python scripts. It is the Python portion of #483. I first proposed this pull request in the mailing list. Below is a description of the plan from that conversation.

For clarity I'll use "Python-module" to refer to a Python script defining functions/classes that you import in Python code and "environment-module" to refer to a TCL environment module that you module load/module unload/etc.

The current documentation for Python initialization (https://modules.readthedocs.io/en/latest/module.html#examples-of-initialization) has

import os
exec(open("/usr/share/Modules/init/python.py").read(), globals())
module("load", "modulefile", "modulefile", "...")

A more Pythonic approach would be to import this module function somehow. Currently, you can do that with as follows

import sys
sys.path.append("/usr/share/Modules/init")
from python import module
module("load", "modulefile", "moduelfile", "...")

The issues with this are that you have to modify the sys.path inside the script and since the Python-module import-ed is called "python". The name makes it look like module is a function from some core Python-module from the Python project.

In this pull request I will

  1. rename the Python initialization file to env_module.py (this is what lmod named theirs by the way) and
  2. to have the shell initialization in tcl/subcmd.tcl.in add the folder of initilization scripts to the PYTHONPATH environment variable.

These changes would change Python initialization to

from env_module import module
module("load", "modulefile", "moduelfile", "...")

which cleans up the way the Python script imports the module function and communicates more clearly that it comes from a 3rd-party Python-module related to environment modules.

@daytonb
Copy link
Author

daytonb commented Nov 14, 2025

At this time there are a couple of issues I know I need to address:

  1. Add tests - I'm guessing we probably want some tests added
  2. Resolve test failures - I ran the quicktests locally and got 42 failures, but I couldn't see how you see which tests failed in order to figure out what I broke
  3. Have someone do a code a documentation review
  4. Change the python.py file to an install-time symlink on Linux:
    • In the mailing list there was discussion that for compatibility the installation process on Linux should symlink the new env_modules.py to python.py, and that no such compatibility in neccessary for Windows because the Windows installs have never had the python.py file.

Copy link
Collaborator

@xdelaruelle xdelaruelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for this pull request. This is already in a very good shape. See review comments for the things to adapt.

Regarding tests:

  • the script/mt utility will guide you to locate where the test failures come from
  • a grep -R setbinpath testsuite/ run at the root of the repository will help you to see where new tests should be added (you may largely inspire from these setbinpath tests)

@daytonb daytonb force-pushed the gh483-python-integration branch from 0639451 to a6b1156 Compare November 20, 2025 05:01
@daytonb
Copy link
Author

daytonb commented Nov 20, 2025

I've addressed the issues noted above except the failing tests. I still need to:

  • find and fix the failing tests
  • add new tests for the pythonpath configure options

Byron Boulton added 3 commits November 20, 2025 00:04
Renaming python.py to env_modules.py. On Linux, symlink python.py to
point to env_modules.py for backward compatibility. Windows doesn't
typically support unix-style symlinks, but we don't need to worry about
that since the Windows installations of Environment Modules never had
python.py.

Signed-off-by: Byron Boulton <[email protected]>
Adding the initdir to PYTHONPATH allows users to "import" the "module"
function in Python scripts.

Signed-off-by: Byron Boulton <[email protected]>
@daytonb daytonb force-pushed the gh483-python-integration branch from a6b1156 to 9ad195b Compare November 20, 2025 05:05
Copy link
Collaborator

@xdelaruelle xdelaruelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comments for the few changes to apply.

@daytonb
Copy link
Author

daytonb commented Nov 22, 2025

I pushed up some fixes. I want to rebase/squash them, but I need to sign off for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants