Pulp offers outdated content

[dmgeurts-mm]

Version
Katello v4.15.0

Describe the bug
A recent host build using Foreman failed the FreeIPA enrollment, digging into why, I found that an apt package listed for update had been removed from the upstream Ubuntu repo. the host I was trying to update could no longer download this package, resulting in the 404 error I was seeing.

The Katello Ubuntu product repo is configured as ‘on demand’. Looking at the logs it's clear that the host tries to download a package no longer available upstream. I tried to sync the repo, but could not get it to update and present the current package version to the host.

To Reproduce
N/A, this was working before.

Expected behavior
Pulp repo to remain in sync with the upstream repo even if using on demand.

Additional context
https://community.theforeman.org/t/katello-deb-repo-out-of-sync-with-upstream/42370

Pulpcore-content logs:

Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb')
Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: pulp [None]: pulpcore.content.handler:WARNING: Could not download remote artifact at 'http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb': 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb'
Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb')

[quba42]

As described here, this is not a bug. On demand works by only downloading packages once a host requests those packages. If, at that point in time, those packages no longer exist in the remote repo, there is nothing on demand can do.

=> On demand should only be used for repositories that do not discard old package versions.

A simple re-sync (using content mirroring) should initially fix the issue until such time as the upstream repo starts dropping packages once more.

There is a feature proposal to overcome this limitation for Debian by fetching packages from the Debian archive: #382
I do not know if there is a similar service for Ubuntu and in any case this feature proposal currently has no one who is willing to work on it.

[dmgeurts-mm]

A simple re-sync (using content mirroring) should initially fix the issue until such time as the upstream repo starts dropping packages once more.

I'm sorry to report that a resync did not fix the issue. I was expecting the problem to be stale data in pulp, which would be fixed by a resync, but alas, no. Apologies, I should have made this clearer in the original message.