Speedup cosmodc2 #195

jkrick · 2025-12-13T01:53:16Z

This will close #144.
@bsipocz please also let me know if this covers #132

One problem in those issues was a backend problem. Yesterday Judith did something to it, and now it is lightning fast, and not as error prone. This PR is therefore not a significant change to the original code, but since I was working on that notebook, and spending time waiting for queries to finish, I cleaned up/added a bunch of clarifying text. I also added a section on further query examples that we don't run, but for which we show users the syntax. This now includes the previous commented out section which did a count on the table.

The notebook now runs in 10s on Fornax small.

Here is what Judith did, for documentation purposes:

"I’m starting to suspect that the slurmctld daemon might be in a funny state, though systemctl status slurmctld shows it as “active”. There are archived slurmctld logs under /var/log/slurm but no active log."

"journalctl -u slurmctld -f shows nothing. I checked the path.

Okay, will restart it.

Yes, we have a slurmctld.log file now."

bsipocz

It would be nice to get the rendering going on for this one for a better review. So I go ahead and add the minor fix and will return with a better review.

bsipocz · 2025-12-16T00:25:56Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

-  display_name: Python 3 (ipykernel)
-  language: python
  name: python3
+  display_name: python3
+  language: python


would be great of not changing the order of these as it break CI as it currently is (I do use the name: python3 field in some matching with the assumption that it's the last one.

sorry if that was me, and not jupyterlab!

tutorials/cosmodc2/cosmoDC2_TAP_access.md

bsipocz · 2025-12-16T00:27:05Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+
+```{code-cell} ipython3
+
+```


Suggested change

```{code-cell} ipython3

```

bsipocz · 2025-12-16T00:28:01Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+    plt.show()
+```
+
+++ {"jp-MarkdownHeadingCollapsed": true}


Not sure if this was intended?

I don't even know what it is. could be I was clicking around and it did this without me realizing. It doesn't show up in the notebook view, so I never saw it.

This was just meant to be a comment-review, not an approval.

tutorials/cosmodc2/cosmoDC2_TAP_access.md

bsipocz · 2025-12-16T01:00:53Z

Well, we don't have a preview as ADS is down and that break the linkchecker 🤦‍♀️

troyraen

Thanks for taking this on. A few simple comments after reading through it.

troyraen · 2025-12-16T18:41:30Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+This tutorial demonstrates how to access and query the **CosmoDC2 Mock v1** catalogs using IRSA’s Table Access Protocol (TAP) service. Background information on the catalogs is available on the [IRSA CosmoDC2 page](https://irsa.ipac.caltech.edu/Missions/cosmodc2.html).

-# Querying CosmoDC2 Mock v1 catalogs
+The catalogs are served through IRSA’s Virtual Observatory–standard **TAP** interface (see the [IVOA TAP specification](https://www.ivoa.net/documents/TAP/)), which you can access programmatically in Python via the **PyVO** library. TAP queries are written in the **Astronomical Data Query Language (ADQL)** — a SQL-like language designed for astronomical catalogs (see the [ADQL specification](https://www.ivoa.net/documents/latest/ADQL.html)).


How about linking to IRSA's TAP page instead of the IVOA specs? https://irsa.ipac.caltech.edu/docs/program_interface/TAP.html. IMO, our page gives more practical and easily applicable info. It also includes both of these links to the IVOA specs.

troyraen · 2025-12-16T18:45:16Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+```{code-cell} ipython3
+import time
+starttime = time.time()
+```


This is helpful for us, but from the user perspective I feel this is clutter so would suggest not having it here. If leaving in, there is no endtime, which seems like an oversight.

good catch, I just forgot to remove it

troyraen · 2025-12-16T18:47:10Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+import numpy as np
+import matplotlib.mlab as mlab
+import matplotlib.pyplot as plt
+import time


Suggested change

import time

troyraen · 2025-12-16T18:54:20Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+# Now that we have a list of galaxy redshifts in that region, we can
+# create a histogram of the redshifts to see what redshifts this survey includes.
+
+if cone_results:


I don't think the if statement is doing anything.

good point, I have removed it here and below.

troyraen · 2025-12-16T18:58:33Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

-# we will construct a 2D histogram rather than a scatter plot.
-plt.hist2d(results['mag_true_r_sdss_z0'], results['mag_true_g_sdss_z0']-results['mag_true_r_sdss_z0'],
-           bins=200, cmap='plasma', cmax=500)
+if redshift_results:


I don't think this if statement is doing anything either. Seems unlikely for the TAP query to return successfully but only return a blank table.

troyraen · 2025-12-16T18:59:18Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+    # Show the plot.
+    plt.show()


Unnecessary in a notebook.

I never understood that distinction, thanks. I removed it.

jkrick

thanks for the reviews.

@bsipocz did you want to add more review comments? If not, this can be merged.

jkrick · 2025-12-16T01:32:44Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

-  display_name: Python 3 (ipykernel)
-  language: python
  name: python3
+  display_name: python3
+  language: python


sorry if that was me, and not jupyterlab!

jkrick · 2025-12-16T01:36:16Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+    plt.show()
+```
+
+++ {"jp-MarkdownHeadingCollapsed": true}


I don't even know what it is. could be I was clicking around and it did this without me realizing. It doesn't show up in the notebook view, so I never saw it.

jkrick · 2025-12-16T20:52:00Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+This tutorial demonstrates how to access and query the **CosmoDC2 Mock v1** catalogs using IRSA’s Table Access Protocol (TAP) service. Background information on the catalogs is available on the [IRSA CosmoDC2 page](https://irsa.ipac.caltech.edu/Missions/cosmodc2.html).

-# Querying CosmoDC2 Mock v1 catalogs
+The catalogs are served through IRSA’s Virtual Observatory–standard **TAP** interface (see the [IVOA TAP specification](https://www.ivoa.net/documents/TAP/)), which you can access programmatically in Python via the **PyVO** library. TAP queries are written in the **Astronomical Data Query Language (ADQL)** — a SQL-like language designed for astronomical catalogs (see the [ADQL specification](https://www.ivoa.net/documents/latest/ADQL.html)).


jkrick · 2025-12-16T20:52:35Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+```{code-cell} ipython3
+import time
+starttime = time.time()
+```


good catch, I just forgot to remove it

jkrick · 2025-12-16T20:53:23Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+import numpy as np
+import matplotlib.mlab as mlab
+import matplotlib.pyplot as plt
+import time


jkrick · 2025-12-16T21:01:43Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+# Now that we have a list of galaxy redshifts in that region, we can
+# create a histogram of the redshifts to see what redshifts this survey includes.
+
+if cone_results:


good point, I have removed it here and below.

jkrick · 2025-12-16T21:19:32Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+    # Show the plot.
+    plt.show()


I never understood that distinction, thanks. I removed it.

jkrick · 2025-12-16T21:21:58Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+
+```{code-cell} ipython3
+
+```


bsipocz · 2025-12-16T21:56:17Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

-  display_name: Python 3 (ipykernel)
-  language: python
  name: python3
-execution:
-  timeout: 2600
+  display_name: python3
+  language: python


This is back again, I suspect jupytext is doing it, and it is breaking the CI.

One workaround for now could be to not include this patch in the commit, e.g. adding changes with git add --patch. This will be a problem for all notebooks, not just this one.

bsipocz

Some more comments based on the rendering I get locally.

bsipocz · 2025-12-16T22:14:41Z

tutorials/cosmodc2/cosmoDC2_TAP_access.md

+### Count the total number of redshifts in the chosen table
+answer for this table= 597,488,849 redshifts
+```
+adql = f"SELECT count(redshift) FROM {tablename}"  
 ```
-***
+
+### Count galaxies in a sky region (cone search)
+Useful for: estimating source density, validating spatial footprint, testing spatial completeness
+```


I think we could unify the comment's/subtext's format a little bit. Have it for all the examples; have it as a complete sentence, either keep the numerical values in all (where applicable) or none.

And have the code block be sql at all or at none (though I don't see we have syntax colouring for sql)

tutorials/cosmodc2/cosmoDC2_TAP_access.md

bsipocz

OK, so skipping for #201 is working again, so while I hijacked this PR as a testing bed, this is now good to go in.

Speedup cosmodc2 30cf6cc

jkrick self-assigned this Dec 13, 2025

jkrick added the content Content related issues/PRs. label Dec 13, 2025

jkrick requested a review from bsipocz December 13, 2025 01:53

bsipocz previously approved these changes Dec 16, 2025

View reviewed changes

bsipocz reviewed Dec 16, 2025

View reviewed changes

tutorials/cosmodc2/cosmoDC2_TAP_access.md Outdated Show resolved Hide resolved

troyraen reviewed Dec 16, 2025

View reviewed changes

jkrick commented Dec 16, 2025

View reviewed changes

bsipocz reviewed Dec 16, 2025

View reviewed changes

bsipocz mentioned this pull request Dec 16, 2025

BUG: buildhtml should not run on PRs by default #201

Closed

bsipocz force-pushed the speedup_cosmodc2 branch from 433d0a2 to 0eeccc2 Compare December 16, 2025 23:14

jkrick and others added 5 commits December 16, 2025 15:53

totally floundering with TAP

5fcfc2d

still TAP errors but text updated

dac5916

fully functional code, improved outline and text

a8839a3

code review comments

0e21427

Some whitespace and styling fixes

c859996

bsipocz force-pushed the speedup_cosmodc2 branch from 0eeccc2 to c859996 Compare December 16, 2025 23:53

bsipocz approved these changes Dec 17, 2025

View reviewed changes

bsipocz merged commit 30cf6cc into Caltech-IPAC:main Dec 17, 2025
6 of 8 checks passed

bsipocz mentioned this pull request Dec 17, 2025

CI: cosmodc2 is consistently failing in cron #132

Closed

github-actions bot pushed a commit that referenced this pull request Dec 17, 2025

Merge pull request #195 from jkrick/speedup_cosmodc2

e733d35

Speedup cosmodc2 30cf6cc

Speedup cosmodc2 #195

Speedup cosmodc2 #195

Uh oh!

Conversation

jkrick commented Dec 13, 2025

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bsipocz commented Dec 16, 2025

Uh oh!

troyraen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkrick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment