Provide an upgrade path to V0.10 while preserving existing directory layouts

[woutervdstoel]

The behavior of pkg_tars include_runfiles attribute seems to have changed quite dramatically in the v0.10 release.

When combined with the strip_prefix = "/" attribute, the directory layout of the runfiles tree before it was just nested at the root of the tarball, i.e. a runfile at $package_dir/path/to/file.yaml would be found exactly in that place.

Since v0.10, the runfiles somehow end up at $package_dir/$myservice.runfiles/$package_dir/path/to/file.yaml.

This completely breaks our existing workflows and compatibility. The PR notes for this change (#754) in my opinion do not really capture the magnitude of this breaking change. Could you describe a way to preserve the old behavior? I think many users upgrading to v0.10 will run into this issue.

[mortenmj]

cc: @aiuto

[aiuto]

So, don't upgrade? Is there a compelling need for you to?

…

On Wed, Jan 17, 2024, 5:05 AM Morten Mjelva ***@***.***> wrote: cc: @aiuto <https://github.com/aiuto> — Reply to this email directly, view it on GitHub <#813 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHHYZSBPGUGRRMESCP3YO6O6VAVCNFSM6AAAAABB6H5BFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJVGQ4DCOBYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[woutervdstoel]

Not at the moment, but I would think that packaging an executable target with runfiles in a way that one can extract it and reference those runfiles using the same relative paths as those in bazel run immediately after extracting would be a common use case that no longer seems to be supported.

[Wyverald]

So, don't upgrade? Is there a compelling need for you to?

Just a note that "don't upgrade" isn't always feasible -- any module in the dependency graph could choose to upgrade rules_pkg to 0.10.0 and potentially break everyone else who's using the older API.

[jwbee]

This response is ridiculous since the person telling us to not upgrade is also the person who ripped out the (completely adequate, long-working) pkg_tar from bazel itself, with the comment that he was forcing us to upgrade. Then the same person turns around and tells us if we don't like ot we don't have to upgrade. Oh also because the version number starts with a zero it's unsupported and all of its features can break silently at any time.

[psigen]

@woutervdstoel I was able to use the remap_paths argument to pkg_tar re-root the runfiles into another location:

remap_paths = { "$package_dir/$myservice.runfiles": "$package_dir" }

Maybe that would work for you?

[blackliner]

I have the same issue here. I am using pkg_tar to package a py_binary with all it's runfiles, including data dependencies. I am using oci_image to deploy that py_binary. In my case the /app fails because it cannot find the data files, and they can only be found if the PWD is /job_list.runfiles/_main, and not /, when running /app (which contains all the runfiles magic)

filegroup(
    name = "params",
    srcs = ["params/params.json"],
)

py_binary(
    name = "app",
    srcs = ["app.py"],
    deps = [
        ":some_libs",
    ],
    data = [
        ":params",
    ],
)

pkg_tar(
    name = "app_tar",
    srcs = [":app"],
    include_runfiles = True,
)

oci_image(
    name = "app_image",
    base = "@python_base",
    cmd = ["/app"],
    workdir = "/app.runfiles/_main", # workaround for https://github.com/bazelbuild/rules_pkg/issues/813
    tars = [":app_tar"],
)

Since I do not know how remap_paths would help me, I set the workdir of the oci_image to /job_list.runfiles/_main.