Pulp import export support by aidenfine · Pull Request #455 · theforeman/foremanctl

aidenfine · 2026-04-17T15:28:11Z

Why are you introducing these changes? (Problem description, related links)

Add pulp content import/export arguments.

What are the changes introduced in this pull request?

You can pass in multiple args of either --content-import-path or --content-export-path as many times as you want, since these will be appended as a list. These folders will be automatically created.

How to test this pull request

pytest tests/pulp_test.py

Can also manually test these steps.

# create env
./foremanctl deploy --foreman-initial-admin-password=changeme --tuning development --content-import-path /mycontentpath/import --content-export-path /mycontentpath/export --content-export-path /mycontentpath/export2

# ssh 
vagrant ssh quadlet

# verify imports (should see a list with exported/imported paths)
sudo podman exec pulp-api pulpcore-manager shell -c "from django.conf import settings; print(settings.ALLOWED_IMPORT_PATHS)"

# exports
sudo podman exec pulp-api pulpcore-manager shell -c "from django.conf import settings; print(settings.ALLOWED_EXPORT_PATHS)"

sjha4 · 2026-04-17T15:56:56Z

+  PULP_ALLOWED_IMPORT_PATHS: >-
+    {{ pulp_default_import_path + pulp_extra_import_paths }}
+  PULP_ALLOWED_EXPORT_PATHS: >-
+    {{ pulp_default_export_path + pulp_extra_export_paths }}


Locally I have this:
PULP_ALLOWED_IMPORT_PATHS: >-
['/var/lib/pulp/sync_imports', '/var/lib/pulp/imports']
PULP_ALLOWED_EXPORT_PATHS: >-
['/var/lib/pulp/exports']

These would be the default paths..

ianballou · 2026-04-17T16:23:46Z

Can we make these import/export arguments also available via forge deploy-dev ?

ehelms · 2026-04-17T16:28:13Z

Can we make these import/export arguments also available via forge deploy-dev ?

Consider this method for defining them -- https://github.com/theforeman/foremanctl/blob/master/docs/developer/playbooks-and-roles.md#shared-metadata-fragments

ianballou · 2026-04-17T17:36:08Z

I just tried this out, seeing the following in my pulp-api service definition:

Environment='PULP_ALLOWED_IMPORT_PATHS=['"'"'/var/lib/pulp/imports'"'"', '"'"'/var/lib/pulp/my_imports'"'"', '"'"'/var/lib/pulp/my_exports'"'"', '"'"'/var/lib/pulp/exports'"'"']'
Environment='PULP_ALLOWED_EXPORT_PATHS=['"'"'/var/lib/pulp/exports'"'"', '"'"'/var/lib/pulp/my_exports'"'"']'

So far so good, assuming the escaping is correct. And yes, I like to be able to import from my export directory.

ianballou · 2026-04-17T17:39:05Z

Export and import worked! I was able to generate /var/lib/pulp/exports/Default_Organization/Export-Zoo-1/1.0/2026-04-17T17-37-04-00-00/ and import it back into another repository. Perfect.

aidenfine · 2026-04-17T18:31:04Z

@ehelms I am not too sure on how to make sure the created import and export directories are mounted into the container. Can you point me into the right direction. Maybe somewhere in the code we handle a similar workflow

ehelms · 2026-04-17T20:01:11Z

@ehelms I am not too sure on how to make sure the created import and export directories are mounted into the container. Can you point me into the right direction. Maybe somewhere in the code we handle a similar workflow

You'll need to do something like this -- https://github.com/theforeman/foremanctl/blob/master/src/roles/pulp/tasks/main.yaml#L137

ehelms · 2026-04-24T19:25:42Z

+pulp_volumes: >-
+  {{
+    ['/var/lib/pulp:/var/lib/pulp'] +
+    (pulp_import_paths | map('regex_replace', '^(.+)$', '\1:\1') | list) +


What are these regexs doing?

lets take one item from pulp_import_paths and call it string1, so string1="/foo/bar"
this regex turns string1 into the same format as the line above. ('/var/lib/pulp:/var/lib/pulp').

So after string1 runs through regex, we get "/foo/bar:/foo:bar"

As /var/lib/pulp is already mounted as a volume, do we want / need to mount the default paths? I realize this can make the logic a bit more complicated but I worry about performance of mounting unnecessary volumes, especially those that are sub-directories of an existing mount.

While it shouldn't have a perf-impact, I'd avoid mounting anything under /var/lib/pulp extra, just to make the resulting container less confusing to look at.

But, if you look closely, this uses pulp_import_paths which is the user-supplied value, not the *_default_* values we have, so it "should do the right thing" already

ehelms · 2026-04-24T19:26:59Z

  foreman_development_github_username:
    help: GitHub username to add as additional remote for git checkouts
    action: store
+  pulp_import_paths:


You could define these in a file like https://github.com/theforeman/foremanctl/blob/master/src/playbooks/_database_connection/metadata.obsah.yaml and then include that file in both deploy and deploy-dev.

ehelms · 2026-04-29T21:29:54Z

+pulp_volumes: >-
+  {{
+    ['/var/lib/pulp:/var/lib/pulp'] +
+    ((pulp_default_import_paths + pulp_import_paths) | map('regex_replace', '^(.+)$', '\1:\1') | list) +


Suggested change

((pulp_default_import_paths + pulp_import_paths) | map('regex_replace', '^(.+)$', '\1:\1') | list) +

((pulp_import_paths) | map('regex_replace', '^(.+)$', '\1:\1') | list) +

I think this would be the way to prevent known double-mounting? Same for line below.

Okay I will make this change today

Okay pushed that change with some test changes as well

ehelms · 2026-04-30T19:49:17Z

+        assert server.file(path).is_directory
+
+@pytest.mark.parametrize("container", ["pulp-api", "pulp-content", "pulp-worker-1"])
+def test_pulp_import_export_volume_mounts(server, container, pulp_import_export_paths):


My read on this is that it will check only the defaults right now, since neither the tests or the test setup is specifying any additional paths. Have you thought about that and adding specification of additional paths to the test setup?

It will also read custom paths as well the fixture will pull items from the environment and verify that any path in the PULP_ALLOWED_IMPORT_PATHS or PULP_ALLOWED_EXPORT_PATHS is present AND a volume mount.

My point is that this isn't testing the user input anywhere and therefore not checking whether the custom parts are working. Consider adding these to CI to be able to assert that they work.

I think I fixed this in my latest commit going to watch the CI run

ehelms · 2026-05-05T19:30:59Z

@aidenfine this will need a rebase now

ehelms · 2026-05-05T19:34:57Z

    action: append_unique
    type: FQDN
    parameter: --certificate-cname
+  pulp_import_paths:


Use the fragment you created instead of repeating these.

ehelms · 2026-05-05T19:46:40Z

Consider also:

Major: docs/user/parameters.md — Add --content-import-path and --content-export-path to the Mapped section with their foreman-installer equivalents. Remove
--foreman-proxy-content-pulpcore-additional-import-paths from the Undetermined list on line 158.
Minor: Consider adding a section to docs/user/ (either a new file or a section in an existing doc) explaining the content import/export path configuration — what the paths are used for,
when users need to set them, and example usage with --content-import-path.

ianballou · 2026-05-06T21:13:04Z

Just wanted to note that this is the upstream issue for it: #445

evgeni · 2026-05-15T06:44:24Z

+def pulp_import_export_paths(server):
+    result = server.run("podman inspect pulp-api --format '{{json .Config.Env}}'")
+    assert result.succeeded
+    env = {v.split('=', 1)[0]: v.split('=', 1)[1] for v in json.loads(result.stdout)}
+    import_paths = json.loads(env['PULP_ALLOWED_IMPORT_PATHS'].replace("'", '"'))
+    export_paths = json.loads(env['PULP_ALLOWED_EXPORT_PATHS'].replace("'", '"'))
+    return import_paths, export_paths


This implies setting of the env vars was correct, which is part of the flow we're supposed to test.

We had a similar issue in #484 and I think the correct path here would to load parameters.yaml and taking the user-input from there assert that the values have been properly propagated to the various consumers.

evgeni · 2026-05-15T06:49:21Z

+pulp_volumes: >-
+  {{
+    ['/var/lib/pulp:/var/lib/pulp'] +
+    (pulp_import_paths | map('regex_replace', '^(.+)$', '\1:\1') | list) +


As an alternative, you could write: pulp_import_paths | zip(pulp_import_paths) | map('join', ':') | list

I am not sure it's actually nicer than the regex, but wanted to still mention it :)

evgeni

the (newly added) ruff check is complaining, and I think this needs some polishing on the tests side, but conceptually this looks solid!

aidenfine · 2026-05-15T15:55:45Z

@evgeni I updated the PR to use the parameters.yaml file in the import/export tests and fixed ruff errors. Tests are passing

evgeni · 2026-05-18T06:25:20Z

+
+
+@pytest.mark.parametrize("container", ["pulp-api", "pulp-content", "pulp-worker-1"])
+def test_pulp_import_export_env_vars(server, container, pulp_import_export_paths):


instead of testing the env vars, you could test whether the setting is actually passed on to pulp as you write in your PR description via pulpcore-manager, seems cleaner to me?

aidenfine · 2026-05-18T14:03:23Z

Will fix these failing tests today.

On another note @evgeni is this test change what you expected? I had Claude help create this test for me.

ehelms · 2026-05-19T03:00:42Z

Testing out a skill that analyzes failures, so bear with me a bit here :)

Universal Failure

  tests/pulp_test.py::test_pulp_import_export_settings

  Error: SyntaxError: invalid syntax inside pulpcore-manager shell -c

  Root cause: Shell quoting collision. The test constructs a Python one-liner with single-quoted dict keys and wraps it in single quotes for the -c argument:

  # tests/pulp_test.py:114
  py = "from django.conf import settings; import json; print(json.dumps({'import': list(settings.ALLOWED_IMPORT_PATHS), 'export': list(settings.ALLOWED_EXPORT_PATHS)}))"
  result = server.run(f"podman exec pulp-api pulpcore-manager shell -c '{py}'")

  The f-string wraps py in single quotes. The shell then sees '...{' as the first token and import as a bare (unquoted) word — which Python parses as the import keyword, producing the
  SyntaxError. The actual command that reaches pulpcore-manager is:

  ...shell -c 'from django.conf import settings; import json; print(json.dumps({import: ..., export: ...}))'

  Suggested fix — use double quotes for the dict keys so they survive the single-quote shell wrapping (tests/pulp_test.py:114):

  py = 'from django.conf import settings; import json; print(json.dumps({"import": list(settings.ALLOWED_IMPORT_PATHS), "export": list(settings.ALLOWED_EXPORT_PATHS)}))'
  result = server.run(f"podman exec pulp-api pulpcore-manager shell -c '{py}'")

aidenfine · 2026-05-19T12:54:08Z

Testing out a skill that analyzes failures, so bear with me a bit here :)

Universal Failure

  tests/pulp_test.py::test_pulp_import_export_settings

  Error: SyntaxError: invalid syntax inside pulpcore-manager shell -c

  Root cause: Shell quoting collision. The test constructs a Python one-liner with single-quoted dict keys and wraps it in single quotes for the -c argument:

  # tests/pulp_test.py:114
  py = "from django.conf import settings; import json; print(json.dumps({'import': list(settings.ALLOWED_IMPORT_PATHS), 'export': list(settings.ALLOWED_EXPORT_PATHS)}))"
  result = server.run(f"podman exec pulp-api pulpcore-manager shell -c '{py}'")

  The f-string wraps py in single quotes. The shell then sees '...{' as the first token and import as a bare (unquoted) word — which Python parses as the import keyword, producing the
  SyntaxError. The actual command that reaches pulpcore-manager is:

  ...shell -c 'from django.conf import settings; import json; print(json.dumps({import: ..., export: ...}))'

  Suggested fix — use double quotes for the dict keys so they survive the single-quote shell wrapping (tests/pulp_test.py:114):

  py = 'from django.conf import settings; import json; print(json.dumps({"import": list(settings.ALLOWED_IMPORT_PATHS), "export": list(settings.ALLOWED_EXPORT_PATHS)}))'
  result = server.run(f"podman exec pulp-api pulpcore-manager shell -c '{py}'")

Lets see if it passes 🤞

evgeni · 2026-05-19T13:14:01Z

Yes, the test looks like what I was thinking of, thanks!

While the existing quoting seems to be fixed, for future reference: shlex.quote can be useful in such cases, similar to Ruby's Shellwords.

aidenfine · 2026-05-19T13:18:25Z

Yes, the test looks like what I was thinking of, thanks!

While the existing quoting seems to be fixed, for future reference: shlex.quote can be useful in such cases, similar to Ruby's Shellwords.

Great, thank you I wasn't aware of shlex. Are we good to merge once all tests pass?

pr-processor Bot added the Not yet reviewed label Apr 17, 2026

aidenfine changed the title ~~Sat-44059 pulp import export support~~ Pulp import export support Apr 17, 2026