feat(http): Switch ACMS authentication to cookie-based sessions

[ERosendo]

Background

ACMS recently refactored its site to use HTMX, which broke CL's existing authentication flow.

Two key behaviors changed:

• The bearer token is no longer present in the SAML response (/Saml2/Acs). It now appears only alongside docket data returned by the cookie-authenticated IndexContent endpoint.
• Docket data is now served from the ...-showdoc.../Home/IndexContent?caseIdentifier=... endpoint, which authenticates using cookies alone. The ...-showdocservices.../api/... endpoints continue to require a bearer token.

As a result, ACMS authentication stopped working in production.

What this PR does

Reworks PacerSession so ACMS requests authenticate with cookies instead of a pre-fetched bearer token:

• SAML handshake on an isolated session seeded with the PACER cookies (the IdP needs NextGenCSO), persisting only the azurewebsites.us cookies into a per-court jar (self.acms_cookies).
• Separated from the PACER jar so a PACER (re-)login can't wipe ACMS auth, and per-court jars prevent broadly-scoped cookies from bleeding across courts.
• Per-request cookie handling: the court jar is sent on every ACMS request and refreshed from each response; a bearer token would be attached only once one has been stored.
• Clears the Secure flag on ACMS cookies so they survive webhook-sentry's HTTPS-to-HTTP proxy hop.

This PR restores the foundation required for ACMS access in CourtListener.

With cookie-based authentication working again, CourtListener can successfully establish ACMS sessions and access the IndexContent endpoint. A follow-up PR will implement the docket retrieval flow.

That work is intentionally deferred so this PR can focus exclusively on the authentication and session-management layer, keeping the change smaller and easier to review.

[ERosendo]

I put together a small script to test the updated auth flow. It logs into PACER, fetches a few sample CA2/CA9 showdoc pages, extracts the window.showDocViewModel payload, and prints the first caseDetails entry for each case.

import json
from typing import Any, Dict, Optional, List

from juriscraper.pacer import PacerSession

# -----------------------------
# CONFIG — UPDATE CREDENTIALS
# -----------------------------
USERNAME = ""
PASSWORD = ""

CASES = [
    "https://ca2-showdoc.azurewebsites.us/Home/IndexContent?caseIdentifier=26-1518",
    "https://ca9-showdoc.azurewebsites.us/Home/IndexContent?caseIdentifier=26-3513",
    "https://ca9-showdoc.azurewebsites.us/Home/IndexContent?caseIdentifier=26-3961",
]


# -----------------------------
# PARSING LOGIC
# -----------------------------
def extract_show_doc_view_model(html_text: str) -> Dict[str, Any]:
    """
    Extracts `window.showDocViewModel = {...}` from the showdoc HTML page.
    """
    marker = "window.showDocViewModel = "
    start = html_text.index(marker) + len(marker)

    obj, _ = json.JSONDecoder().raw_decode(html_text, start)
    return obj


def get_case_details(view_model: Dict[str, Any]) -> Optional[Dict[str, Any]]:
    """
    Safely extract first case details entry.
    """
    case_details = view_model.get("caseDetails")
    if case_details and len(case_details) > 0:
        return case_details[0]
    return None


# -----------------------------
# PACER HELPERS
# -----------------------------
def fetch_case(session: PacerSession, url: str) -> Optional[Dict[str, Any]]:
    """
    Fetch a case page and return parsed case details.
    """
    response = session.get(url)
    view_model = extract_show_doc_view_model(response.text)
    return get_case_details(view_model)


session = PacerSession(username=USERNAME, password=PASSWORD)
session.login()
for url in CASES:
    print(f"\nFetching: {url}\n")
    try:
        case_details = fetch_case(session, url)
        print(json.dumps(case_details, indent=2, default=str))
    except Exception as e:
        print(f"Error processing {url}: {e}")
    print("\n" + "*" * 40)

[claude]

🔴 _update_acms_cookies merges response.cookies into the per-court jar via .update() but never re-runs _desecure_acms_cookies, so any server-refreshed cookie carrying Secure=True replaces the desecured copy. Once that happens (Azure App Service routinely re-issues ARRAffinity/ARRAffinitySameSite with Secure on every response), the requests library will refuse to send the cookie over webhook-sentry's plain-http leg — defeating the very workaround this PR adds. One-line fix: call self._desecure_acms_cookies(self.acms_cookies[court_id]) after the .update() at http.py:211.

Extended reasoning...

What the bug is

The PR adds _desecure_acms_cookies (http.py:331-336) and calls it once at the end of establish_acms_session (http.py:683) so the bootstrap ACMS jar's cookies all have secure=False. The PR comment is explicit about why: "Clear Secure on ACMS cookies so webhook-sentry's https->http downgrade doesn't strip them."

But _update_acms_cookies (http.py:202-211), which is called after every ACMS get()/post() (http.py:248 and http.py:297), only does:

if court_id:
    self.acms_cookies[court_id].update(response.cookies)

RequestsCookieJar.update() is a straight merge keyed on (domain, path, name) — it preserves each incoming cookie's secure attribute verbatim. There is no symmetric desecure call on the refresh path, so any cookie the server re-issues with Set-Cookie: …; Secure lands in the jar with secure=True and replaces the desecured bootstrap copy.

Why this is realistic for the target deployment

The PR exists specifically to support CourtListener behind webhook-sentry (see freelawproject/courtlistener#5921 in the comment at http.py:332-335). webhook-sentry terminates TLS and forwards over plain HTTP, so even though juriscraper calls an https:// URL, the actual transport is HTTP and cookielib's policy refuses to attach cookies with secure=True.

The ACMS hosts are *.azurewebsites.us (Azure App Service). Azure App Service emits ARRAffinity / ARRAffinitySameSite sticky-session cookies with the Secure flag, and ARR re-issues affinity cookies on responses — particularly when the LB rebalances or the inbound cookie state changes. ASP.NET session cookies (.AspNet.* / .ASPXAUTH) are similarly refreshed mid-session and emitted Secure. Either of these landing in response.cookies re-secures a cookie that the bootstrap pass had cleared.

Step-by-step proof

1. login() → establish_acms_session('ca9') runs the SAML handshake, builds the per-court jar, and calls _desecure_acms_cookies(acms_jar) (http.py:683). State: jar has ARRAffinity with secure=False.
2. juriscraper calls session.get('https://ca9-showdoc.azurewebsites.us/Home/IndexContent?…'). _prepare_acms_request attaches the jar (all secure=False). webhook-sentry downgrades to http; requests sends Cookie: ARRAffinity=…. Request succeeds.
3. The response carries Set-Cookie: ARRAffinitySameSite=NEW_VALUE; Secure; HttpOnly (Azure rotated the affinity cookie). _update_acms_cookies(court_id, r) runs self.acms_cookies['ca9'].update(r.cookies). Confirmed by runtime test: RequestsCookieJar.update(new_jar) where new_jar has secure=True for an existing name replaces the entry — final jar.get_dict() for that cookie is the new one with secure=True.
4. Next juriscraper call: session.get('https://ca9-showdoc.azurewebsites.us/Home/IndexContent?…'). webhook-sentry downgrades to http again. Confirmed by runtime test: requests.PreparedRequest omits Secure cookies on http:// URLs. The freshly re-secured ARRAffinitySameSite is not sent.
5. Without the sticky-session affinity cookie, the ASP.NET app behind ARR routes the request to a backend that has no session for this user, and ACMS auth fails — silently, mid-session.

Why existing code doesn't prevent it

_desecure_acms_cookies is only called once, at the very end of establish_acms_session. Once the cookie jar is in steady state, every subsequent server response can re-secure cookies and there is no symmetric pass. The update() call does not filter by attribute and response.cookies carries the Set-Cookie attributes verbatim.

Impact

This defeats the PR's stated webhook-sentry workaround as soon as the ACMS server refreshes any Secure cookie — typical within the first few requests of an Azure App Service session. The desecure protection only covers the lifetime up to the first cookie rotation; after that, auth quietly drops for the exact deployment scenario this PR exists to support.

Fix

One line inside _update_acms_cookies:

def _update_acms_cookies(self, court_id, response):
    if court_id:
        self.acms_cookies[court_id].update(response.cookies)
        self._desecure_acms_cookies(self.acms_cookies[court_id])

Alternatively, iterate response.cookies and clear secure before .update(). Either way restores the symmetry the bootstrap path already has.

_{🔬 also observed by verify-runtime}

[ERosendo]

Fixed in 2ecf6e9

[claude]

🔴 The new get_acms_auth_object = establish_acms_session alias (http.py:687-688) is described as backwards-compatible "for existing callers (e.g. acms_attachment_page)", but it is not behaviorally equivalent for that exact caller. ACMSAttachmentPage.query() (acms_attachment_page.py:122-130) calls get_acms_auth_object specifically to populate session.acms_user_data (CsoId/ContactType), which the next line — get_docket_entries — then indexes in network_utils.py:226-227. The old implementation set acms_user_data from model_json["PacerUser"]; establish_acms_session only persists cookies, so a fresh ACMSAttachmentPage().query(...) after login() now raises KeyError: '\''CsoId'\''. The new store_acms_token path that would populate this is wired only to the deferred IndexContent flow, so the attachment-page caller is broken in the interim — either restore user-data extraction in establish_acms_session (the SAML response still embeds PacerUser), or drop the misleading "backwards-compatible" framing and document that this caller breaks until the follow-up PR lands.

Extended reasoning...

What the bug is

ACMSAttachmentPage.query() (juriscraper/pacer/acms_attachment_page.py:122-130) — code unchanged by this PR — has an explicit gatekeeper whose stated purpose is to populate acms_user_data before issuing the attachment request:

# Ensure ACMS user data is available before fetching attachments.
# Attachment requests include user-specific data in the request body,
# so the session must have the necessary authentication data
# initialized beforehand.
if not self.session.acms_user_data:
    self.session.get_acms_auth_object(self.court_id)

# Fetch Docket Entry Details
docket_info = self.api_client.get_docket_entries(case_id)

The very next call, get_docket_entries (juriscraper/lib/network_utils.py:226-227), unconditionally indexes acms_user_data for the request body:

json={
    "csoId": self.session.acms_user_data["CsoId"],
    "contactType": self.session.acms_user_data["ContactType"],
}

Why this PR breaks it

Before this PR, get_acms_auth_object parsed the SAML response, populated both acms_tokens[court_id] (from model_json["AuthToken"]) and acms_user_data (from model_json["PacerUser"] — CsoId/ContactType).

After this PR, http.py:687-688 reduces get_acms_auth_object to a one-liner alias:

# Backwards-compatible alias for existing callers (e.g. acms_attachment_page).
get_acms_auth_object = establish_acms_session

establish_acms_session (http.py:632-685) performs the SAML POST but only persists cookies to self.acms_cookies[court_id] — it never parses var model = ..., never reads PacerUser, and never assigns acms_user_data or acms_tokens. The only writer of acms_user_data post-PR is store_acms_token (http.py:213-225), which has no caller in this PR (the PR description acknowledges the IndexContent flow that would call it is deferred to a follow-up).

Step-by-step proof

1. Caller does session = PacerSession(...); session.login() — acms_user_data defaults to {}.
2. Caller does ACMSAttachmentPage(court_id="ca9", pacer_session=session).query(case_id, entry_id).
3. acms_attachment_page.py:126 evaluates if not self.session.acms_user_data: → True (empty dict).
4. acms_attachment_page.py:127 calls self.session.get_acms_auth_object("ca9"), which is now establish_acms_session("ca9"). It posts to /Saml2/Acs, copies azurewebsites.us cookies into self.acms_cookies["ca9"], and returns. acms_user_data is still {}.
5. acms_attachment_page.py:130 calls self.api_client.get_docket_entries(case_id).
6. network_utils.py:226 does self.session.acms_user_data["CsoId"] → KeyError: '\''CsoId'\''.

Why the alias comment is actively misleading

The comment at http.py:687 — "Backwards-compatible alias for existing callers (e.g. acms_attachment_page)." — explicitly names the caller it claims to preserve, but the behavior preserved is only the function name, not the side effect the caller depends on. The previous failure mode for this caller was a clean PacerLoginException (if SAML parsing failed); the new failure mode is a confusing KeyError on an empty dict, which is strictly worse.

Impact

ACMSAttachmentPage is an existing, shipped feature (released in 2.6.83 / 2025-08-05 per CHANGES.md). Any caller in CourtListener or downstream that uses it on a freshly-logged-in PacerSession is broken by this PR until the deferred follow-up lands.

How to fix

Three reasonable options, in rough order of preference:

1. Restore user-data extraction inside establish_acms_session — the SAML /Saml2/Acs response still embeds the var model = ... JSON containing PacerUser, so the parsing block removed from the old method can be re-added (or hoisted into a helper) and called from both establish_acms_session and store_acms_token. This actually delivers the backwards compatibility the comment claims.
2. Route ACMSAttachmentPage.query through whatever the follow-up PR introduces (likely an IndexContent call that ends in store_acms_token), making the attachment-page flow share the new bearer-token-population path.
3. At minimum, drop the misleading "Backwards-compatible alias" framing and raise a clear NotImplementedError from get_acms_auth_object (or document the broken state), so callers fail with a meaningful message until the follow-up lands.

Option 1 keeps the PR self-contained and honors the comment`s promise; option 2 ties the fix to the follow-up; option 3 is the bare minimum if the breakage is intentional.