Error when using packages of kind `localSourceControl`

[nmcc24]

While adopting local packages that mirror remote ones using swift package manager's set-mirror (https://docs.swift.org/swiftpm/documentation/packagemanagerdocs/packageconfigsetmirror) I have faced the following error as part of the build process when SwiftPackageListPlugin executes

Error: The file “AEXML” couldn’t be opened because there is no such file. Program ended with exit code: 1

Below follows the contents of mirrors.json and Package.resolved files:

Mirrors.json
{ "object": [ { "mirror": "<my_directory>/TestLocalMirrors/LocalMirrors/AEXML", "original": "https://github.com/tadija/AEXML" }, { "mirror": "/<my_directory>/TestLocalMirrors/LocalMirrors/analytics-ios", "original": "https://github.com/segmentio/analytics-ios" }, { "mirror": "<my_directory>/TestLocalMirrors/LocalMirrors/swift-argument-parser", "original": "https://github.com/apple/swift-argument-parser" }, { "mirror": "<my_directory>eTstLocalMirrors/LocalMirrors/swift-package-list", "original": "https://github.com/FelixHerrmann/swift-package-list" } ], "version": 1 }

Package.resolved
{ "originHash" : "7e1b589942a81651aa427ee5fbf71b7b9bc62038ebe88abd5c1ae1f2ce397b0d", "pins" : [ { "identity" : "aexml", "kind" : "localSourceControl", "location" : "https://github.com/tadija/AEXML", "state" : { "revision" : "db806756c989760b35108146381535aec231092b", "version" : "4.7.0" } }, { "identity" : "analytics-ios", "kind" : "localSourceControl", "location" : "https://github.com/segmentio/analytics-ios", "state" : { "revision" : "78423aaae2e48e32a07246850f37c76f3d34cb84", "version" : "4.1.8" } }, { "identity" : "swift-argument-parser", "kind" : "localSourceControl", "location" : "https://github.com/apple/swift-argument-parser", "state" : { "revision" : "cdd0ef3755280949551dc26dee5de9ddeda89f54", "version" : "1.6.2" } }, { "identity" : "swift-package-list", "kind" : "localSourceControl", "location" : "https://github.com/FelixHerrmann/swift-package-list", "state" : { "revision" : "22ef2a2b1b51b453be02ce2bd780753718e8893a", "version" : "4.8.0" } } ], "version" : 3 }

After debugging your source code, I found the issue lies on the following lines of code PackageResolved.swift:

Currently, it is being assumed that packages of kind localSourceControl have a local file URL, which is not the case. pin.location will contain the value of location contained in Package.resolved. In other words, AEXML's pin.location would be "https://github.com/tadija/AEXML" and not a local path. In turn, URL(fileURLWithPath: will create an invalid URL and the code will later on throw.

As far as I am aware, packages are of kind localSourceControl only when using this concept of local mirroring relying of git repos, not for packages within the codebase itself. Furthermore, in this later case, the dependencies won't be part of Package.resolved.

How to solve this issue?
By modifying the code snipet above to treat both remoteSourceControl and localSourceControl similarly and allowing them both to be derived from packageSource = checkouts.packageSource(location: pin.location) instead, I managed to resolve the issue and get the correct output as expected.

@FelixHerrmann could you please share your views? Please let me know if I am missing something.

Thank you.

[FelixHerrmann]

Hey @nmcc24, I think I never could test it in a real-world scenario and it is just an assumption from for example here:

https://github.com/swiftlang/swift-package-manager/blob/36c3549fd26dd418e27c80ed5a9968e1ab5e7578/Sources/PackageGraph/ResolvedPackagesStore.swift#L448-L453

I'm happy to get that change in if you can confirm that the local ones also are located in the checkouts directory (if I read that correct you already tested that with your setup).

[FelixHerrmann]

Ok, just opened a PR but also played around with mirrors myself and noticed one problem with that; when you give your mirror a different name than the original repository it will no longer find it in the checkouts directory (it looks for the mirror but checkouts contains the original one). We might actually need to respect the mirror.json; will give you an update when I have some progress.

[nmcc24]

Hey @nmcc24, I think I never could test it in a real-world scenario and it is just an assumption from for example here:

https://github.com/swiftlang/swift-package-manager/blob/36c3549fd26dd418e27c80ed5a9968e1ab5e7578/Sources/PackageGraph/ResolvedPackagesStore.swift#L448-L453

It doesn’t surprise me @FelixHerrmann - this is a very niche use case that unfortunately I’m having to explore due to limitation within my organisation.

I'm happy to get that change in if you can confirm that the local ones also are located in the checkouts directory (if I read that correct you already tested that with your setup).

Yes, that’s correct. The checkouts directly will have all the necessary code and info, including the correct naming, for both local and remote source control dependencies.

Nonetheless, I’ll push my testing repo shortly for you to take a look if you are interested.

[nmcc24]

Ok, just opened a PR but also played around with mirrors myself and noticed one problem with that; when you give your mirror a different name than the original repository it will no longer find it in the checkouts directory (it looks for the mirror but checkouts contains the original one). We might actually need to respect the mirror.json; will give you an update when I have some progress.

Cool - expect an MR to be raised during the course of today.

In regards to the naming point you raised - that would be an invalid use case and swiftpm would not work. One of the iterations I tested during this journey was trying to point to a local mirror with name <target_name>-version and that failed. In hindsight, that didn’t make sense because, as the name states, it is supposed to be a 1-2-1 mapping of the remote git. Having naming differences (or not being a git repo with versioning info) breaks that principle and SPM will complain during package resolution.

[nmcc24]

@FelixHerrmann here is my public testing repo https://github.com/nmcc24/TestLocalMirrors

Note: On the mirror.json file, you must manually replace with the correct value pointing to the local repo folder

[nmcc24]

Below you'll find the screenshots taken from DerivedData showing that:

1. SPM has labeled these dependencies with kind local source control

2. All dependencies are available in the checkout directory

[FelixHerrmann]

Hey @nmcc24, after some investigation I came to the conclusion that adding support for SPM mirrors is the only proper solution and I will close #163 again. I have opened a new PR #164 which adds support for mirrors; please give it a try and let me know if the output looks as expected, for my test setup it works now as expected.

[FelixHerrmann]

Just tested with your example project and this is working as well:

[nmcc24]

Thanks @FelixHerrmann for looking into this in depth for testing it on my sample project. It looks good to me and I’ll be looking forward to integrating this change into our project.

[nmcc24]

I have tested your branch in my professional project and I confirm all is working great. Thanks a lot

[FelixHerrmann]

Great @nmcc24! I will merge the PR now but plan to add at least one other feature for the next release; in the meantime you can integrate the dependency pointing to the master branch.