nvidia repos failing on regular sync - Error: Task failed: ''NoneType' object has no attribute 'artifact_set_sha256''

[hbrown-uiowa]

Version
pulpcore 3.89.1
pulp_deb 3.7.0
Installed via container: quay.io/pulp/pulp:3.89.1

Describe the bug
Repeated syncs of the nvidia cuda repos fail with 'NoneType' object has no attribute 'artifact_set_sha256''

To Reproduce

Create the repo:

NAME="nvidia-cuda-24.04-amd64"
pulp deb remote create --name=$NAME --url="https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/" --distribution /
pulp deb repository create --name=$NAME --remote=$NAME
pulp deb repository sync --name=$NAME
PUB_HREF=$(pulp deb publication -t verbatim create --repository=$NAME | jq -r '.pulp_href')
pulp deb distribution create --name=${NAME}-bimonthly --base-path=${NAME}-bimonthly --publication $PUB_HREF

Note that the /pulp/content/$NAME/Packages file exists

Re-run a sync:

pulp deb repository sync --name=$NAME 
PUB_HREF=$(pulp deb publication -t verbatim create --repository=$NAME | jq -r '.pulp_href')
pulp deb distribution update --name=${NAME}-bimonthly --base-path=${NAME}-bimonthly --publication $PUB_HREF

Note that /pulp/content/$NAME/Packages has disappeared.

Re-run the sync pulp deb repository sync --name=$NAME and get the sync fail with 'NoneType' object has no attribute 'artifact_set_sha256''

Expected behavior
A new sync/publication/distribution update would not delete the Packages file and would complete successfully.

Additional context
Re-doing the sync as

pulp deb repository sync --name=$NAME --no-optimize

makes the repository usable again until the next sync happens that does not use --no-optimize

[quba42]

Interesting. I don't think we changed much with pulp_deb (other than maybe ACS support, but I don't think that would cause this). Maybe something changed in the repo? It would be interesting to re-test using an older pulp_deb version to see if that is also affected.

[hbrown-uiowa]

The Packages file disappearing occurs on pulpcore 3.82.1 and pulp_deb 3.5.2, but subsequent syncs don't fail. Ultimately, with no Packages file, the repo isn't usable.

[hbrown-uiowa]

I'm also seeing the same behavior when I try it with pulpcore 3.65.0 and pulp_deb 3.4.0. I'm wondering if there's a sync with no new debs to download that it somehow short circuits and removes the Packages file and then never regenerates it.

[quba42]

Since I am unsure when I will get around to experimenting myself, I will start by giving some high level thoughts and pointers:

First, a quick summary of what --optimize does: It checks if the sha256 checksum of a release file or Packages file has not changed since the last sync, and if it has not, it skips processing of that Packages file during the sync. It is weird that not processing something would lead to it being removed from the repo.

My best guess is that there is some unexpected interaction with independent features like mirror=True sync option. mirror sync is supposed to have the effect to remove things from the repo that are no longer part of the upstream repo. Are you using mirror=True for your syncs? If yes, try switching to mirror=False to check if that "fixes" the issue. Still would not explain why only this one repository is suddenly affected.

The other thing I can say is that I remember nvidia repos in particular being a particularly extreme example of a cobbled together flat repo format that just barely meets any APT repo specifications. I would not be at all surprised if they found another creative way to add a new idiosyncrasy to their repo metadata.

[hbrown-uiowa]

As far as I can tell, mirror has always been False and my debugging has confirmed that it was False.

Thus far, initial sync (or a --no-optimize sync) is clean and then the second sync with --optimize as the default causes the Packages file to disappear and then when I check the repository version:

# pulp deb repository version show --repository=$NAME | jq -r '.content_summary.removed'
{
  "deb.package_index": {
    "count": 1,
    "href": "/pulp/api/v3/content/deb/package_indices/?repository_version_removed=/pulp/api/v3/repositories/deb/apt/019b0f3f-edca-7661-83e1-7fe334fb1cf8/versions/14/"
  }
}

The Releases file just references the Packages file.

I will continue to try and figure out where things are breaking down.