Merge pull request #2362 from Parcels-code/MyST-quickstart

Quickstart Guide and MyST-NB implementation

Author: reint-fischer

.gitignore

@@ -1,6 +1,8 @@
 build/*
 docs/_build/*
 docs/_downloads
+docs/jupyter_execute/*
+docs/.jupyter_cache/*
 output
 
 *.log

docs/conf.py

@@ -36,8 +36,7 @@
     "sphinx.ext.linkcode",
     "sphinx.ext.mathjax",
     "sphinx.ext.napoleon",
-    "myst_parser",
-    "nbsphinx",
+    "myst_nb",
     "numpydoc",
     "sphinxcontrib.mermaid",
     "sphinx_design",
@@ -114,7 +113,13 @@
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = ["_build", "**.ipynb_checkpoints", "user_guide/examples_v3"]
+exclude_patterns = [
+    "_build",
+    "jupyter_execute",
+    "**.ipynb_checkpoints",
+    "user_guide/examples_v3",
+    ".jupyter_cache",
+]
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
@@ -294,7 +299,7 @@ def linkcode_resolve(domain, info):
 
 # Custom sidebar templates, maps document names to template names.
 
-html_sidebars = {"**": ["sidebar-nav-bs"], "documentation/additional_examples": []}
+html_sidebars = {"**": ["sidebar-nav-bs"]}
 
 # Additional templates that should be rendered to pages, maps page names to
 # template names.
@@ -343,17 +348,6 @@ def linkcode_resolve(domain, info):
 # Output file base name for HTML help builder.
 htmlhelp_basename = "parcelsdoc"
 
-nbsphinx_thumbnails = {
-    "examples/tutorial_parcels_structure": "_images/parcels_user_diagram.png",
-    "examples/tutorial_timestamps": "_static/calendar-icon.jpg",
-    "examples/documentation_homepage_animation": "_images/homepage.gif",
-    "examples/tutorial_interaction": "_static/pulled_particles_twoatractors_line.gif",
-    "examples/documentation_LargeRunsOutput": "_static/harddrive.png",
-    "examples/tutorial_unitconverters": "_static/globe-icon.jpg",
-    "examples/documentation_geospatial": "_images/tutorial_geospatial_google_earth.png",
-    "examples/tutorial_kernelloop": "_static/loop-icon.jpeg",
-}
-nbsphinx_execute = "never"
 # -- Options for LaTeX output ---------------------------------------------
 
 BRANCH = (
@@ -528,3 +522,9 @@ def linkcode_resolve(domain, info):
 myst_heading_anchors = 3
 
 myst_enable_extensions = ["substitution"]
+
+# -- Options for MyST-nb --------------------------------------------------
+nb_execution_mode = "cache"
+nb_execution_excludepatterns = ["jupyter_execute", ".jupyter_cache"]
+nb_execution_raise_on_error = True
+nb_execution_timeout = 75

docs/development/docsguide.md

@@ -2,7 +2,9 @@
 
 ## Vision
 
-We believe a clear documentation is important to community building, reproducibility, and transparency in our open-source project. To make it easier to write our documentation in a consistent way, here we outline a brief vision for our documentation based heavily on a few common resources.
+We believe a clear documentation is important to community building, reproducibility, and transparency in our open-source
+project. To make it easier to write our documentation in a consistent way, here we outline a brief vision for our
+documentation based heavily on a few common resources.
 
 ```{note}
 TODO: outline functions of the documentation based on resources
@@ -15,6 +17,28 @@ TODO: outline functions of the documentation based on resources
 - [Write the Docs Guide](https://www.writethedocs.org/guide/)
 - [NumPy Documentation Article](https://labs.quansight.org/blog/2020/03/documentation-as-a-way-to-build-community)
 
+## Notebook execution
+
+We run the notebooks in our documentation using [MyST-NB](https://myst-nb.readthedocs.io/en/latest/index.html). Here is
+a table showing the latest notebook execution:
+
+```{nb-exec-table}
+
+```
+
 ## Style guide
 
-- Write documentation in first person plural ("we"). In our open source code, tutorials and guides can be written by any developer or user, so the documentation teaches all of us how to do something with Parcels.
+- **Prefer `import parcels` over `from parcels import class` in tutorials and how-to guides** so its obvious in later
+  code cells which classes and methods are part of Parcels.
+- [**Avoid too much Repitition In Documentation**](https://www.writethedocs.org/guide/writing/docs-principles/#arid):
+  tutorials and how-to guides notebooks will often have repetition of the general **Parcels** steps, (e.g., imports ) -
+  this is needed so that users have complete examples that they can copy and experiment with.`. We try to limit each page
+  in the documentation to a small number of examples.
+- Introduce links and cross-references to maximize discoverability of documentation. This also reduces the necessity for
+  repetition in notebooks.
+- **Import packages at the top of the section in which they are first used** to show what they are used for.
+- **Write documentation in first person plural ("we").** In our open source code, tutorials and guides can be written
+  by any developer or user, so the documentation teaches all of us how to do something with Parcels. Sometimes it can be
+  more natural to take on the tone of a teacher, writing to a student/learner, in which case it is okay to use "you".
+  Please refrain from using impersonal subjects such as "the user".
+- We recommend hard wrapping prose in markdown so that reading it becomes easier in any editor.

docs/getting_started/index.md

@@ -3,25 +3,18 @@
 Getting started with parcels is easy; here you will find:
 
 ```{toctree}
+:maxdepth: 1
 Installation guide <installation.md>
 Quickstart tutorial <tutorial_quickstart.md>
 Parcels concepts explainer <concepts_overview.md>
-<!-- Simple output tutorial <../examples_v3/tutorial_output.ipynb> -->
+Simple output tutorial <tutorial_output.ipynb>
 
 ```
 
 ```{note}
-TODO: Include one line conda installation for most common use
-```
-
-```{note}
-TODO: Write quickstart tutorial. This should focus on getting users familiar with the different steps to take to run a simulation. Implicitly we should introduce them to the important concepts, but it should not be an in depth explanation. Instead we should reference to the concept overview and the reference API
+TODO: Add links to Reference API in quickstart tutorial
 ```
 
 ```{note}
 TODO: Rewrite parcels concepts overview. This .md file should contain most of what is currently in the tutorial_parcels_structure notebook.
 ```
-
-```{note}
-TODO: Move new output tutorial here
-```

docs/user_guide/examples/tutorial_output.ipynb renamed to docs/getting_started/tutorial_output.ipynb

@@ -15,11 +15,11 @@
    "source": [
     "This tutorial covers the format of the trajectory output exported by Parcels. **Parcels does not include advanced analysis or plotting functionality**, which users are suggested to write themselves to suit their research goals. Here we provide some starting points to explore the parcels output files yourself.\n",
     "\n",
-    "- [**Reading the output file**](#Reading-the-output-file)\n",
-    "- [**Trajectory data structure**](#Trajectory-data-structure)\n",
-    "- [**Analysis**](#Analysis)\n",
-    "- [**Plotting**](#Plotting)\n",
-    "- [**Animations**](#Animations)\n",
+    "- [**Reading the output file**](#reading-the-output-file)\n",
+    "- [**Trajectory data structure**](#trajectory-data-structure)\n",
+    "- [**Analysis**](#analysis)\n",
+    "- [**Plotting**](#plotting)\n",
+    "- [**Animations**](#animations)\n",
     "\n",
     "For more advanced reading and tutorials on the analysis of Lagrangian trajectories, we recommend checking out the [Lagrangian Diagnostics Analysis Cookbook](https://lagrangian-diags.readthedocs.io/en/latest/tutorials.html) and the project in general. The [TrajAn package](https://opendrift.github.io/trajan/index.html) can be used to read and plot datasets of Lagrangian trajectories."
    ]
@@ -56,10 +56,10 @@
     "    \"CopernicusMarine_data_for_Argo_tutorial\"\n",
     ")\n",
     "\n",
-    "ds = xr.open_mfdataset(f\"{example_dataset_folder}/*.nc\", combine=\"by_coords\")\n",
-    "ds.load()  # load the dataset into memory\n",
+    "ds_fields = xr.open_mfdataset(f\"{example_dataset_folder}/*.nc\", combine=\"by_coords\")\n",
+    "ds_fields.load()  # load the dataset into memory\n",
     "\n",
-    "fieldset = parcels.FieldSet.from_copernicusmarine(ds)"
+    "fieldset = parcels.FieldSet.from_copernicusmarine(ds_fields)"
    ]
   },
   {
@@ -72,14 +72,14 @@
     "npart = 10  # number of particles to be released\n",
     "lon = 32 * np.ones(npart)\n",
     "lat = np.linspace(-32.5, -30.5, npart, dtype=np.float32)\n",
-    "time = ds.time.values[0] + np.arange(0, npart) * np.timedelta64(2, \"h\")\n",
-    "z = np.repeat(ds.depth.values[0], npart)\n",
+    "time = ds_fields.time.values[0] + np.arange(0, npart) * np.timedelta64(2, \"h\")\n",
+    "z = np.repeat(ds_fields.depth.values[0], npart)\n",
     "\n",
     "pset = parcels.ParticleSet(\n",
     "    fieldset=fieldset, pclass=parcels.Particle, lon=lon, lat=lat, time=time, z=z\n",
     ")\n",
     "\n",
-    "output_file = parcels.ParticleFile(\"Output.zarr\", outputdt=np.timedelta64(2, \"h\"))"
+    "output_file = parcels.ParticleFile(\"output.zarr\", outputdt=np.timedelta64(2, \"h\"))"
    ]
   },
   {
@@ -109,7 +109,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "pset.execute(\n",
@@ -136,9 +140,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "data_xarray = xr.open_zarr(\"Output.zarr\")\n",
+    "ds_particles = xr.open_zarr(\"output.zarr\")\n",
     "\n",
-    "print(data_xarray)"
+    "print(ds_particles)"
    ]
   },
   {
@@ -171,7 +175,7 @@
    "source": [
     "np.set_printoptions(linewidth=160)\n",
     "one_hour = np.timedelta64(1, \"h\")  # Define timedelta object to help with conversion\n",
-    "time_from_start = data_xarray[\"time\"].values - fieldset.time_interval.left\n",
+    "time_from_start = ds_particles[\"time\"].values - fieldset.time_interval.left\n",
     "\n",
     "print(time_from_start / one_hour)  # timedelta / timedelta -> float number of hours"
    ]
@@ -202,8 +206,8 @@
    "source": [
     "import matplotlib.pyplot as plt\n",
     "\n",
-    "x = data_xarray[\"lon\"].values\n",
-    "y = data_xarray[\"lat\"].values\n",
+    "x = ds_particles[\"lon\"].values\n",
+    "y = ds_particles[\"lat\"].values\n",
     "distance = np.cumsum(\n",
     "    np.sqrt(np.square(np.diff(x)) + np.square(np.diff(y))), axis=1\n",
     ")  # d = (dx^2 + dy^2)^(1/2)\n",
@@ -262,7 +266,7 @@
    "source": [
     "### Conditional selection\n",
     "\n",
-    "In other cases, the processing of the data itself however depends on the absolute time at which the observations are made, e.g. studying seasonal phenomena. In that case the array structure is not as simple: the data must be selected by their `time` value. Here we show how the mean location of the particles evolves through time. This also requires the trajectory data to be aligned in time. The data are selected using `xr.DataArray.where()` which compares the time variable to a specific time. This type of selecting data with a condition (`data_xarray['time']==time`) is a powerful tool to analyze trajectory data.\n"
+    "In other cases, the processing of the data itself however depends on the absolute time at which the observations are made, e.g. studying seasonal phenomena. In that case the array structure is not as simple: the data must be selected by their `time` value. Here we show how the mean location of the particles evolves through time. This also requires the trajectory data to be aligned in time. The data are selected using `xr.DataArray.where()` which compares the time variable to a specific time. This type of selecting data with a condition (`ds_particles['time']==time`) is a powerful tool to analyze trajectory data.\n"
    ]
   },
   {
@@ -276,20 +280,20 @@
     "mean_lat_x = []\n",
     "\n",
     "timerange = np.arange(\n",
-    "    np.nanmin(data_xarray[\"time\"].values),\n",
-    "    np.nanmax(data_xarray[\"time\"].values) + np.timedelta64(timedelta(hours=2)),\n",
+    "    np.nanmin(ds_particles[\"time\"].values),\n",
+    "    np.nanmax(ds_particles[\"time\"].values) + np.timedelta64(timedelta(hours=2)),\n",
     "    timedelta(hours=2),\n",
     ")  # timerange in nanoseconds\n",
     "\n",
     "for time in timerange:\n",
     "    # if all trajectories share an observation at time\n",
-    "    if np.all(np.any(data_xarray[\"time\"] == time, axis=1)):\n",
+    "    if np.all(np.any(ds_particles[\"time\"] == time, axis=1)):\n",
     "        # find the data that share the time\n",
     "        mean_lon_x += [\n",
-    "            np.nanmean(data_xarray[\"lon\"].where(data_xarray[\"time\"] == time).values)\n",
+    "            np.nanmean(ds_particles[\"lon\"].where(ds_particles[\"time\"] == time).values)\n",
     "        ]\n",
     "        mean_lat_x += [\n",
-    "            np.nanmean(data_xarray[\"lat\"].where(data_xarray[\"time\"] == time).values)\n",
+    "            np.nanmean(ds_particles[\"lat\"].where(ds_particles[\"time\"] == time).values)\n",
     "        ]"
    ]
   },
@@ -344,16 +348,16 @@
     "\n",
     "###-Points-###\n",
     "ax1.set_title(\"Points\")\n",
-    "ax1.scatter(data_xarray[\"lon\"].T, data_xarray[\"lat\"].T)\n",
+    "ax1.scatter(ds_particles[\"lon\"].T, ds_particles[\"lat\"].T)\n",
     "###-Lines-###\n",
     "ax2.set_title(\"Lines\")\n",
-    "ax2.plot(data_xarray[\"lon\"].T, data_xarray[\"lat\"].T)\n",
+    "ax2.plot(ds_particles[\"lon\"].T, ds_particles[\"lat\"].T)\n",
     "###-Points + Lines-###\n",
     "ax3.set_title(\"Points + Lines\")\n",
-    "ax3.plot(data_xarray[\"lon\"].T, data_xarray[\"lat\"].T, marker=\"o\")\n",
+    "ax3.plot(ds_particles[\"lon\"].T, ds_particles[\"lat\"].T, marker=\"o\")\n",
     "###-Not Transposed-###\n",
     "ax4.set_title(\"Not transposed\")\n",
-    "ax4.plot(data_xarray[\"lon\"], data_xarray[\"lat\"], marker=\"o\")\n",
+    "ax4.plot(ds_particles[\"lon\"], ds_particles[\"lat\"], marker=\"o\")\n",
     "\n",
     "plt.show()"
    ]
@@ -373,7 +377,7 @@
    "source": [
     "Trajectory plots like the ones above can become very cluttered for large sets of particles. To better see patterns, it's a good idea to create an animation in time and space. To do this, matplotlib offers an [animation package](https://matplotlib.org/stable/api/animation_api.html). Here we show how to use the [**FuncAnimation**](https://matplotlib.org/3.3.2/api/_as_gen/matplotlib.animation.FuncAnimation.html#matplotlib.animation.FuncAnimation) class to animate parcels trajectory data, based on [this visualisation tutorial](https://github.com/Parcels-code/10year-anniversary-session5/blob/eaf7ac35f43c222280fa5577858be81dc346c06b/animations_tutorial.ipynb) from 10-years Parcels. \n",
     "\n",
-    "To correctly reveal the patterns in time we must remember that the `obs` dimension does not necessarily correspond to the `time` variable ([see the section of Trajectory data structure above](#Trajectory-data-structure)). In the animation of the particles, we usually want to draw the points at each consecutive moment in time, not necessarily at each moment since the start of the trajectory. To do this we must [select the correct data](#Conditional-selection) in each rendering.\n"
+    "To correctly reveal the patterns in time we must remember that the `obs` dimension does not necessarily correspond to the `time` variable ([see the section of Trajectory data structure above](#trajectory-data-structure)). In the animation of the particles, we usually want to draw the points at each consecutive moment in time, not necessarily at each moment since the start of the trajectory. To do this we must [select the correct data](#conditional-selection) in each rendering.\n"
    ]
   },
   {
@@ -411,7 +415,7 @@
     "\n",
     "# Set up the colors and associated trajectories:\n",
     "# get release times for each particle (first valide obs for each trajectory)\n",
-    "release_times = data_xarray[\"time\"].min(dim=\"obs\", skipna=True).values\n",
+    "release_times = ds_particles[\"time\"].min(dim=\"obs\", skipna=True).values\n",
     "\n",
     "# get unique release times and assign colors\n",
     "unique_release_times = np.unique(release_times[~np.isnat(release_times)])\n",
@@ -431,9 +435,9 @@
     "print(\"Pre-computing all particle positions...\")\n",
     "all_particles_data = []\n",
     "for i, target_time in enumerate(timerange):\n",
-    "    time_id = np.where(data_xarray[\"time\"] == target_time)\n",
-    "    lons = data_xarray[\"lon\"].values[time_id]\n",
-    "    lats = data_xarray[\"lat\"].values[time_id]\n",
+    "    time_id = np.where(ds_particles[\"time\"] == target_time)\n",
+    "    lons = ds_particles[\"lon\"].values[time_id]\n",
+    "    lats = ds_particles[\"lat\"].values[time_id]\n",
     "    particle_indices = time_id[0]\n",
     "    valid = ~np.isnan(lons) & ~np.isnan(lats)\n",
     "\n",