Launch metview in Jupyter when installed in different environments

[savannahferretti]

I have Jupyter Notebooks installed in my base environment. When creating/using separate environments, I use ipykernel to make them accessible as Jupyter kernels. I recently installed metview into an existing conda environment (entitled "monsoon-pr") following:

conda install 'metview-batch>=5.16.0' -c conda-forge

When launching a new notebook, activating the environment where metview is installed (via the Kernel dropdown), and attempting to import metview following import metview as mv, I get the following error:

Could not run the Metview executable ('metview'); check that the binaries for Metview (version 5 at least) are installed and are in the PATH.
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[1], line 1
----> 1 import metview as mv

File ~/Miniconda/envs/metview/lib/python3.9/site-packages/metview/__init__.py:44
     42 except Exception as exp:
     43     if "METVIEW_PYTHON_ONLY" not in os.environ:
---> 44         raise exp

File ~/Miniconda/envs/metview/lib/python3.9/site-packages/metview/__init__.py:28
     25 if len(sys.argv) != 2 or sys.argv[0] != "-m" or sys.argv[1] != "selfcheck":
     27     try:
---> 28         from . import bindings as _bindings
     30         _bindings.bind_functions(globals(), module_name=__name__)
     32         # Remove "_bindings" from the public API.

File ~/Miniconda/envs/metview/lib/python3.9/site-packages/metview/bindings.py:196
    192         signal.signal(signal.SIGTERM, self.sigterm)
    193         signal.signal(signal.SIGALRM, self.sigalarm)
--> 196 mi = MetviewInvoker()
    198 try:
    199     ffi = cffi.FFI()

File ~/Miniconda/envs/metview/lib/python3.9/site-packages/metview/bindings.py:114, in MetviewInvoker.__init__(self)
    106     if "METVIEW_PYTHON_ONLY" not in os.environ:
    107         print(
    108             "Could not run the Metview executable ('"
    109             + metview_startup_cmd
   (...)
    112             "and are in the PATH."
    113         )
--> 114     raise exp
    116 # wait for Metview to respond...
    117 wait_start = time.time()

File ~/Miniconda/envs/metview/lib/python3.9/site-packages/metview/bindings.py:104, in MetviewInvoker.__init__(self)
    101     print(metview_flags)
    103 try:
--> 104     subprocess.Popen(metview_flags)
    105 except Exception as exp:  # pragma: no cover
    106     if "METVIEW_PYTHON_ONLY" not in os.environ:

File ~/Miniconda/envs/metview/lib/python3.9/subprocess.py:951, in Popen.__init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask)
    947         if self.text_mode:
    948             self.stderr = io.TextIOWrapper(self.stderr,
    949                     encoding=encoding, errors=errors)
--> 951     self._execute_child(args, executable, preexec_fn, close_fds,
    952                         pass_fds, cwd, env,
    953                         startupinfo, creationflags, shell,
    954                         p2cread, p2cwrite,
    955                         c2pread, c2pwrite,
    956                         errread, errwrite,
    957                         restore_signals,
    958                         gid, gids, uid, umask,
    959                         start_new_session)
    960 except:
    961     # Cleanup if the child failed starting.
    962     for f in filter(None, (self.stdin, self.stdout, self.stderr)):

File ~/Miniconda/envs/metview/lib/python3.9/subprocess.py:1821, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session)
   1819     if errno_num != 0:
   1820         err_msg = os.strerror(errno_num)
-> 1821     raise child_exception_type(errno_num, err_msg, err_filename)
   1822 raise child_exception_type(err_msg)

FileNotFoundError: [Errno 2] No such file or directory: 'metview'

I'm assuming this error is because metview is operating out of a different environment than Jupyter. In my Jupyter notebook, os.environ['PATH'] gives my Jupyter path as:

'/jet/home/sferrett/.local/bin:/jet/home/sferrett/bin:/opt/packages/psc.allocations.user/bin:/opt/packages/allocations/bin:/jet/home/sferrett/Miniconda/bin:/jet/home/sferrett/Miniconda/condabin:/jet/home/sferrett/.local/bin:/jet/home/sferrett/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/packages/interact/bin:/opt/puppetlabs/bin'

While in my conda environment with metview (monsoon-pr) in Terminal, my path is:

/jet/home/sferrett/Miniconda/envs/monsoon-pr/bin:/jet/home/sferrett/Miniconda/condabin:/jet/home/sferrett/.local/bin:/jet/home/sferrett/bin:/opt/packages/psc.allocations.user/bin:/opt/packages/allocations/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/packages/interact/bin:/opt/puppetlabs/bin

Is there a way to set the notebook PATH so that it can retrieve metview from the conda environment? I don't want to have to install Jupyter within my conda environment.

[iainrussell]

Hi @savannahferretti,

I'm surprised it can't find Metview, but one thing you can do is, from a terminal where Metview can be found, type which metview. This will give the full path to the Metview startup script. You can then set environment variable METVIEW_PYTHON_START_CMD to this path (including the 'metview' at the end), either on the command line before starting Jupyter, or you can set this env var from the Python code in your Jupyter notebook before you import metview.

Good luck!

Cheers,
Iain

[savannahferretti]

Thanks for the quick response! I am stilling having an issue loading the package. I am unsure if it has to do with the fact that I'm SSH tunneling into Jupyter Notebooks from an HPC. Here is what I did:

In my first Terminal window, I SSHed into the HPC, was allocated a compute node, and launched Jupyter Notebooks. I opened a second Terminal window, mapped the port to my local, then launched Jupyter in my browser. Also from this second window, I activated the environment where metview is installed and executed which metview, which returned:

/jet/home/sferrett/Miniconda/envs/monsoon-pr/bin/metview

I deactivated the environment, and from my base environment (where Jupyter Notebooks is installed), I executed:

export METVIEW_PYTHON_START_CMD=/jet/home/sferrett/Miniconda/envs/monsoon-pr/bin/metview

Executing printenv METVIEW_PYTHON_START_CMD returned the same path.

After launching my notebook and running import metview as mv, I still get an error:

ModuleNotFoundError: No module named 'metview'

I tried relaunching Jupyter, but still the same error.

[iainrussell]

Hi,

OK, this should help: first, to make sure the environment inside your notebook has the right path set, you can set the env var before importing metview, and to also get more debug output:

import os
os.environ["METVIEW_PYTHON_START_CMD "] = "/jet/home/sferrett/Miniconda/envs/monsoon-pr/bin/metview"
os.environ["METVIEW_PYTHON_DEBUG"] = "1"
import metview

[savannahferretti]

Hi,

Thanks for the response. Trying the suggestion above still gives the same error (i.e., it cannot find metview):

import os
os.environ['METVIEW_PYTHON_START_CMD'] = '/jet/home/sferrett/Miniconda/envs/monsoon-pr/bin/metview'
os.environ['METVIEW_PYTHON_DEBUG'] = '1'
import metview

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], line 4
      2 os.environ['METVIEW_PYTHON_START_CMD'] = '/jet/home/sferrett/Miniconda/envs/monsoon-pr/bin/metview'
      3 os.environ['METVIEW_PYTHON_DEBUG'] = '1'
----> 4 import metview

ModuleNotFoundError: No module named 'metview'

[iainrussell]

Hi Savannah,

Looks like this one dropped off the radar...! Did you manage to fix this issue? It looks like you were in an environment where Metview's Python interface was not installed.

Cheers,
Iain