diff --git a/ImageProcessing/PipelineProcessingAPI.ipynb b/ImageProcessing/PipelineProcessingAPI.ipynb new file mode 100644 index 00000000..419fc893 --- /dev/null +++ b/ImageProcessing/PipelineProcessingAPI.ipynb @@ -0,0 +1,578 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Pipeline Task API\n", + "\n", + "
Owner: **Alex Drlica-Wagner** ([@kadrlica](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@kadrlica))\n", + "
Last Verified to Run: **2018-08-10**\n", + "
Verified Stack Release: **v16.0**" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Learning Objectives:\n", + "\n", + "This notebook seeks to teach users how to unpack a pipeline tasks. As an example, we focus on `processCcd.py`, with the goal of diving into the configuration, interface, and structure of pipeline tasks. This notebook is a digression from Justin Myles script that demonstrates how to run a series of pipeline tasks from the command line to rerun HSC data processing [link].\n", + "\n", + "After working through this tutorial you should be able to:\n", + "\n", + "* Find the source code for a pipeline task\n", + "* Configure (and investigate the configuration) of pipeline tasks\n", + "* Investigate and run those tasks in python\n", + "\n", + "## Logistics\n", + "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# What version of the Stack are we using?\n", + "! echo $HOSTNAME\n", + "! eups list -s | grep lsst_distrib" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Filter some warnings printed by v16.0 of the stack\n", + "import warnings\n", + "warnings.simplefilter(\"ignore\", category=FutureWarning)\n", + "warnings.simplefilter(\"ignore\", category=UserWarning)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "import os\n", + "import pydoc\n", + "import matplotlib.pyplot as plt" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import lsst.afw.display as afwDisplay" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Diving into a Pipeline Task\n", + "\n", + "Our goal is to dive into the inner workings of `ProcessCcd.py`. We pickup from the command line processing described in the getting started tutorials [here](https://pipelines.lsst.io/getting-started/data-setup.html#) and [here](https://pipelines.lsst.io/getting-started/processccd.html), as well as Justin Myles HSC reprocessing notebook [here](). We are specifically interested in digging into the following line:\n", + "\n", + "```\n", + "processCcd.py $DATADIR --rerun processCcdOutputs --id\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We start by tracking down the location of the `processCcd.py` shell script" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!(which processCcd.py)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is the proverbial \"end of the thread\". Our goal is to pull on this thread to unravel the python/C++ functions that are being called under the hood. We start by taking a peak in this script" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!cat $(which processCcd.py)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Ok, this hasn't gotten us very far, but after getting through the stock header, we now have the next link in our chain:\n", + "```\n", + "from lsst.pipe.tasks.processCcd import ProcessCcdTask\n", + "```\n", + "\n", + "There are two ways we can proceed from here. One is to [Google](http://lmgtfy.com/?q=lsst.pipe.tasks.processCcd) `lsst.pipe.tasks.processCcd`, which will take us to this [doxygen page](http://doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/classlsst_1_1pipe_1_1tasks_1_1process_ccd_1_1_process_ccd_task.html) and/or the soure code on [GitHub](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/processCcd.py). \n", + "\n", + "The second approach is to do the import the class oursleves and try to investigate it interactively.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import lsst.pipe.tasks.processCcd\n", + "from lsst.pipe.tasks.processCcd import ProcessCcdTask, ProcessCcdConfig" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can get to the source code for these classes directly using the [`stackclub` toolkit module](https://stackclub.readthedocs.io/), as shown in the [FindingDocs.ipynb](https://github.com/LSSTScienceCollaborations/StackClub/blob/master/GettingStarted/FindingDocs.ipynb)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from stackclub import where_is\n", + "where_is(ProcessCcdConfig)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Diving into a Task Config\n", + "\n", + "Pipeline tasks are controlled and tweaked through there associated `TaskConfig` objects. To investigate the configuration parameters of the `ProcessCcdTask`, we create an instance of the `ProcessCcdConfig` and try calling the `help` method (commented out for brevity). What we are really interested in are the \"Data descriptors\", which we can print directly after capturing the documentation output by `help`." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The first step is to try to get at the documentation through the `help` function (commented out below for brevity). We can find " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `ProcessCcdConfig` object contains of both raw configurables like `doCalibrate` and other configs, like `isr`. To investigate one of these in more detail, we can import it and query it's \"Data descriptors\".\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from lsst.ip.isr.isrTask import IsrTask, IsrTaskConfig\n", + "print('\\n'.join(pydoc.render_doc(IsrTaskConfig).split('\\n')[16:40]) + '...')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "These configurationas are pretty self-explanitory, but say that we really want to understand what `doFringe` is doing. Inorder to get that information we need to go to the source code." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "where_is(IsrTaskConfig)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can then search this file for `doFringe` and we find [several lines](https://github.com/lsst/ip_isr/blob/cc4efb7d763d3663c9e989339505df9654f23fd9/python/lsst/ip/isr/isrTask.py#L597-L598) that look like this:\n", + "\n", + " if self.config.doFringe and not self.config.fringeAfterFlat:\n", + " self.fringe.run(ccdExposure, **fringes.getDict())\n", + " \n", + "If we want to go deeper to see what `fringe.run` does, we can repeat the above process" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "isr_task = IsrTask()\n", + "print(isr_task.fringe)\n", + "import lsst.ip.isr.fringe\n", + "where_is(lsst.ip.isr.fringe.FringeTask)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We finally make our way to the source code for [FringeTask.run](https://github.com/lsst/ip_isr/blob/cc4efb7d763d3663c9e989339505df9654f23fd9/python/lsst/ip/isr/fringe.py#L104), which gives us details on how the fringe correction is performed (i.e. by creating a fringe image and subtracting it from the data image)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Running a Task\n", + "\n", + "Now that we've figured out how to investigate the config for a task, let's try to run the task itself." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# First we setup the directories for the raw image and calibration.\n", + "import shutil\n", + "basedir = '/project/stack-club/validation_data_hsc'\n", + "repodir = os.path.join(basedir,'data')\n", + "# By default the calibration directory lives in ${datadir}/CALIB; \n", + "# however, in the validation_data_hsc it lives in ${basedir}/CALIB\n", + "calibdir = os.path.join(basedir,'CALIB')\n", + "# The directory for our processed output\n", + "outdir = '/home/kadrlica/tmpdir/'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Next, we create a butler to manage the data repositories\n", + "from lsst.daf.persistence import Butler\n", + "# output directory cannot exist (wait for Gen3 Butler...)\n", + "if os.path.exists(outdir): shutil.rmtree(outdir)\n", + "\n", + "# This strange structure for Butler initialization is a 'feature' of the Gen2 butler \n", + "# and should be replaced in Gen3\n", + "butler = Butler(inputs={'root': repodir, 'mapperArgs': {'calibRoot': calibdir}}, \n", + " outputs={'root': outdir})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can use the butler to list the available data sets, and to choose the CCD that we want to process." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# We can list available data sets\n", + "ccds = butler.queryMetadata('raw',['visit','ccd'])\n", + "print(ccds[0:10],'...')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# We choose one of these CCDs as our dataRef\n", + "dataRef = butler.dataRef('raw', dataId={'visit':903332,'ccd':25})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we are prepared to create our `IsrTaskConfig`. At this point, we will turn off many of the processing steps to speed up the task." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "config = IsrTaskConfig()\n", + "config.doBias = True\n", + "config.doDark = True\n", + "config.doFlat = True" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "With the `config` created and configured, we can now create the `IsrTask`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "isr_task = IsrTask(config)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And run it!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "struct = isr_task.runDataRef(dataRef)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The output is a `struct` containing the processed exposure:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(struct)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "afwDisplay.setDefaultBackend('matplotlib') \n", + "# ADW: why can't we set the backend before this cell?\n", + "\n", + "figure = plt.figure(1,figsize=(10,10))\n", + "afw_display = afwDisplay.Display(1)\n", + "afw_display.scale('asinh', 'zscale')\n", + "afw_display.mtv(dataRef.get())\n", + "plt.title(\"Raw\")\n", + "\n", + "figure = plt.figure(2,figsize=(10,10))\n", + "afw_display = afwDisplay.Display(2)\n", + "afw_display.scale('asinh', 'zscale')\n", + "afw_display.mtv(struct.exposure)\n", + "plt.title(\"ISR 'Corrected'\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "for maskName, maskBit in struct.exposure.mask.getMaskPlaneDict().items():\n", + " print('{}: {}'.format(afw_display.getMaskPlaneColor(maskName),maskName))" + ] + }, + { + "cell_type": "raw", + "metadata": {}, + "source": [ + "# This appears to work when called from the shell:\n", + "\n", + "cd /project/stack-club/validation_data_hsc\n", + "export OMP_NUM_THREADS=1\n", + "export SETUP_ASTROMETRY_NET_DATA=\"astrometry_net_data sdss-dr9-fink-v5b\"\n", + "export ASTROMETRY_NET_DATA_DIR=sdss-dr9-fink-v5b\n", + "input=\"data\"\n", + "output=\"/home/kadrlica/tmpdir\"\n", + "rm -rf $output\n", + "processCcd.py $input --calib CALIB --output $output --id ccd=25 visit=903982" + ] + }, + { + "cell_type": "raw", + "metadata": {}, + "source": [ + "repo=/project/stack-club/validation_data_hsc\n", + "output=/home/kadrlica/tmpdir\n", + "processCcd.py $repo/data --calib $repo/CALIB --output $output --id ccd=25 visit=903982" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# https://github.com/ipython/ipykernel/issues/111#issuecomment-237089618\n", + "# Try to get the task outputs into the notebook (and failing)\n", + "import sys,logging\n", + "logger = logging.getLogger()\n", + "logger.setLevel(logging.DEBUG)\n", + "\n", + "# Create STDERR handler\n", + "handler = logging.StreamHandler(sys.stderr)\n", + "# ch.setLevel(logging.DEBUG)\n", + "\n", + "# Create formatter and add it to the handler\n", + "formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')\n", + "handler.setFormatter(formatter)\n", + "\n", + "# Set STDERR handler as the only handler \n", + "logger.handlers = [handler]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Run processCcdTask" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First, we run the task as we would from the command line. All that we are doing here is parsing the `cmdline` string as if it is arguments passed to `processCcdTask.py` on the command line. Unfortunately, the task does not print it's output to a notebook cell, so we just need to wait for ~1 minute for this to run. We turn off several optional subtask steps so that things run faster..." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Create the task\n", + "task = ProcessCcdTask()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%timeit\n", + "# output directory cannot exist (wait for Gen3 Butler...)\n", + "if os.path.exists(outdir): shutil.rmtree(outdir)\n", + "cmdline = '{0} --calib {1} --output {2} --id ccd=25 visit=903982'.format(repodir,calibdir,outdir)\n", + "cmdline += ' --config doCalibrate=False isr.doBias=False isr.doDark=False isr.doFlat=False'\n", + "struct = task.parseAndRun(cmdline.split())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "config = ProcessCcdConfig()\n", + "#help(config)\n", + "helplist = pydoc.render_doc(config).split('\\n')\n", + "print('\\n'.join(helplist[18:47]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The next level in complexity is to try to call `task.run`. In order to do this we need to provide a `dataRef`. To do this, we create a butler." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Since we don't want to do calibration processing we turn off these configs\n", + "config = ProcessCcdConfig()\n", + "config.isr.doBias = False\n", + "config.isr.doDark = False\n", + "config.isr.doFlat = False\n", + "config.doCalibrate = False" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "task = ProcessCcdTask(butler=butler, config=config)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Setup the astrometry reference catalogs following:\n", + "# https://github.com/lsst/validation_data_hsc\n", + "#!export SETUP_ASTROMETRY_NET_DATA=\"astrometry_net_data sdss-dr9-fink-v5b\"\n", + "#!export ASTROMETRY_NET_DATA_DIR={repodir + '/sdss-dr9-fink-v5b'}\n", + "\n", + "# TODO: Create the astrometry reference object to pass to the task" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "LSST", + "language": "python", + "name": "lsst" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.2" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/ImageProcessing/ProcessEimage.ipynb b/ImageProcessing/ProcessEimage.ipynb deleted file mode 100644 index 137777cb..00000000 --- a/ImageProcessing/ProcessEimage.ipynb +++ /dev/null @@ -1,197 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Single Exposure Processing\n", - "\n", - "This is intended to walk you through the processing pipeline on jupyterlab. It builds on the first two hands-on tutorials in the LSST [\"Getting started\" tutorial series](https://pipelines.lsst.io/getting-started/index.html#getting-started-tutorial). It is intended for anyone getting started with using the LSST Science Pipelines for data processing. \n", - "\n", - "The goal of this tutorial is to setup a Butler for a simulated LSST data set and to run the `processCCD.py` pipeline task to produced reduced images." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Setting up the data repository\n", - "\n", - "Sample data for this tutorial comes from the `twinkles` LSST simulation and is available in a shared directory on `jupyterlab`. We will make a copy of the input data in our current directory:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!if [ ! -d DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 DATA; fi" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Inside the data directory you'll see a directory structure that looks like this" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!ls -lh DATA/" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cat DATA/_mapper" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "All of the relavent images and calibrations have already been ingested into the Butler for this data set." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Reviewing what data will be processed\n", - "\n", - "We'll now process individual raw LSST simulated images in the Butler `DATA` repository into calibrated exposures. We’ll use the `processCcd.py` command-line task to remove instrumental signatures with dark, bias and flat field calibration images. `processCcd.py` will also use the reference catalog to establish a preliminary WCS and photometric zeropoint solution.\n", - "\n", - "First we'll examine the set of exposures available in the Twinkles data set using the Butler" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we'll do a similar thing using the `processEimageTask` from the LSST pipeline. **There is a bit of ugliness here because the `processEimage.py` command line script is only python2 compatible so we need to parse the arguments through the API. This has the nasty habit of trying to exit after the args.**" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from lsst.obs.lsstSim.processEimage import ProcessEimageTask" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "args = 'DATA --rerun process-eimage --id filter=r --show data'\n", - "ProcessEimageTask.parseAndRun(args=args.split())\n", - "\n", - "# BUG: the command above exits early, due to a namespace problem:\n", - "# /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/15.0/python/lsst/pipe/base/argumentParser.py in parse_args(self, config, args, log, override)\n", - "# 628 \n", - "# 629 if namespace.show and \"run\" not in namespace.show:\n", - "# --> 630 sys.exit(0)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The important arguments here are `--id` and `--show data`.\n", - "\n", - "The `--id` argument allows you to select datasets to process by their data IDs. Data IDs describe individual datasets in the Butler repository. Datasets also have types, and each command-line task will only process data of certain types. In this case, `processEimage.py` will processes raw simulated e-images **(need more description of e-images)**.\n", - "\n", - "In the above command, the `--id filter=r` argument selects data from the r filter. Specifying `--id` without any arguments acts as a wildcard that selects all raw-type data in the repository.\n", - "\n", - "The `--show data` argument puts `processEimage.py` into a dry-run mode that prints a list of data IDs to standard output that would be processed according to the `--id` argument rather than actually processing the data. \n", - "\n", - "Notice the keys that describe each data ID, such as the visit (exposure identifier), raft (identifies a specific LSST camera raft), sensor (identifies an individual ccd on a raft) and filter, among others. With these keys you can select exactly what data you want to process." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next we perform the same task directly with the Butler:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import lsst.daf.persistence as dafPersist" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "butler = dafPersist.Butler(inputs='DATA')\n", - "butler.queryMetadata('eimage', ['visit', 'raft', 'sensor','filter'], dataId={'filter': 'r'})" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Processing data\n", - "\n", - "Now we'll move on to actually process some of the Twinkles data. To do this, we'll remove the `--show data` argument." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "args = 'DATA --rerun process-eimage --id filter=r --show data'\n", - "# The command below also exits early - see the error message above.\n", - "ProcessEimageTask.parseAndRun(args=args.split())" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "LSST_Stack (Python 3)", - "language": "python", - "name": "lsst_stack" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.2" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/ImageProcessing/README.rst b/ImageProcessing/README.rst index 3f4b3c73..a1d41ddf 100644 --- a/ImageProcessing/README.rst +++ b/ImageProcessing/README.rst @@ -1,6 +1,3 @@ -Image Processing -================ - This folder contains a set of tutorial notebooks exploring the image processing routines in the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs. @@ -13,7 +10,6 @@ This folder contains a set of tutorial notebooks exploring the image processing - Links - Owner - * - **Re-run HSC** - End-to-end processing of the ``ci_hsc`` test dataset using the DM Stack. - `ipynb `_, @@ -35,3 +31,12 @@ This folder contains a set of tutorial notebooks exploring the image processing - `Andrew Bradshaw `_ + * - **PipelineProcessingApi.ipynb** + - Take a deep dive into the configuration, interface, and structure of pipeline tasks. + - `ipynb `_, + `rendered `_ + + .. image:: https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/ImageProcessing/log/PipelineTasks.svg + :target: https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/ImageProcessing/log/PipelineTasks.log + + - `Alex Drlica-Wagner `_