Unable to Push to Standard LFS Remote and S3 Remote Together

[AlJohri]

I'm trying to use a git repository with multiple remotes, one to an internal GitFarm repository and another to an S3 bucket based repository. The repository has several large Git LFS artifacts that I want to publish to both GitFarm and S3. Essentially, I want the S3 bucket to act as a replica.

When I set up the new S3 remote, the LFS settings are not scoped to the S3 remote so I am not longer able to push to the GitFarm LFS.

Before

❯ git remote -v
origin	ssh://git.amazon.com/pkg/TK (fetch)
origin	ssh://git.amazon.com/pkg/TK (push)

❯ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

❯ git pull
Already up to date.

❯ git push
Everything up-to-date

After adding S3 Remote unable to push to GitFarm

git remote add s3 s3://TK/TK
git config --add lfs.customtransfer.git-lfs-s3.path git-lfs-s3
git config --add lfs.standalonetransferagent git-lfs-s3

❯ git remote -v
origin	ssh://git.amazon.com/pkg/TK (fetch)
origin	ssh://git.amazon.com/pkg/TK (push)
s3	s3://TK/TK (fetch)
s3	s3://TK/TK (push)

❯ git push s3 main
Everything up-to-date

❯ git pull s3 main
From s3://TK/TK
 * branch            main       -> FETCH_HEAD
Already up to date.

❯ cp output.parquet output2.parquet
❯ git add output2.parquet
❯ git commit -m "test add parquet file"

❯ git push origin main
error initializing custom adapter "git-lfs-s3" worker 0: [32] s3 uri ssh://git.amazon.com/pkg/TK is invalid
error: failed to push some refs to 'ssh://git.amazon.com/pkg/TK'

As you can see, I am unable to push to GitFarm now because it is trying to use the git-lfs-s3. I scoured the documentation and I could not find a way to configure git-lfs-s3 only for the s3 remote.

Is it possible to make git-lfs-s3 act as a pass through if it is trying to push to a non-s3 uri?

Git Config

Here is my full git config for reference:

❯ cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = ssh://git.amazon.com/pkg/TK
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
    remote = origin
    merge = refs/heads/main
[lfs]
    repositoryformatversion = 0
    standalonetransferagent = git-lfs-s3
[remote "s3"]
    url = s3://TK/TK
    fetch = +refs/heads/*:refs/remotes/s3/*
[lfs "customtransfer.git-lfs-s3"]
    path = git-lfs-s3

[AlJohri]

Based on these docs:

lfs.<url>.standalonetransferagent can be used to configure a custom transfer agent for individual remotes. lfs.standalonetransferagent unconditionally configures a custom transfer agent for all remotes.

I tried the following git config:

❯ cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = ssh://git.amazon.com/pkg/TK
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
    remote = origin
    merge = refs/heads/main
[lfs]
    repositoryformatversion = 0
[lfs "s3://TK/TK"]
    standalonetransferagent = git-lfs-s3
[remote "s3"]
    url = s3://TK/TK
    fetch = +refs/heads/*:refs/remotes/s3/*
[lfs "customtransfer.git-lfs-s3"]
    path = git-lfs-s3
[branch "aljohri"]
    remote = s3
    merge = refs/heads/aljohri

I verified that the git config was correct by running these commands:

❯ git config --get-urlmatch lfs.standalonetransferagent s3://TK/TK
git-lfs-s3

❯ git config --get-urlmatch lfs.standalonetransferagent ssh://git.amazon.com/pkg/TK

As you can see, we get git-lfs-s3 for the first and no output for the second.

However, when I try to push I get this error:

❯ git push origin --verbose
Pushing to ssh://git.amazon.com/pkg/TK
batch response: 1 validation error detected: Value '[ssh, lfs-standalone-file, git-lfs-s3, basic]' at 'transfers' failed to satisfy constraint: Member must satisfy constraint: [Member must satisfy enum value set: [ssh, basic, lfs-standalone-file]]
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
error: failed to push some refs to 'ssh://git.amazon.com/pkg/TK'

[AlJohri]

Here are the versions of git and git-lfs that I am using:

❯ git --version
git version 2.47.1

❯ git-lfs --version
git-lfs/3.1.4 (GitHub; linux amd64; go 1.19.3)

[AlJohri]

I was able to reduce the config down to a more minimal example that reproduces the error:

❯ cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = ssh://git.amazon.com/pkg/TK
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
    remote = origin
    merge = refs/heads/main
[lfs "customtransfer.git-lfs-s3"]
    path = git-lfs-s3

As you can see, we don't even have the s3 remote setup here and we are not using the git-lfs-s3 custom transfer agent for any remote.

However, with the above config, I get this error:

❯ git push origin main
batch response: 1 validation error detected: Value '[ssh, lfs-standalone-file, git-lfs-s3, basic]' at 'transfers' failed to satisfy constraint: Member must satisfy constraint: [Member must satisfy enum value set: [ssh, basic, lfs-standalone-file]]
error: failed to push some refs to 'ssh://git.amazon.com/pkg/TK'

[AlJohri]

If I run this command:

GIT_CURL_VERBOSE=1 git push origin main

I can see the payload sent to the server looks like:

{
    "operation":"upload",
    "objects":
    [
        {
            "oid": "ae03b6e74e4cfdaf72183cc19e1ac8f31223bf2d4a0d36f3cb3cc52cff210bf8",
            "size":186284
        }
    ],
    "transfers": ["lfs-standalone-file","git-lfs-s3","basic","ssh"],
    "ref": {"name":"refs/heads/main"},
    "hash_algo": "sha256"
}

I believe this is what the server is rejecting because the transfers array has git-lfs-s3 in there.

Here are the docs on the API request format: https://github.com/git-lfs/git-lfs/blob/41999a064abfc4c30ff6864da6bab8fad32d3536/docs/api/batch.md?plain=1#L28-L48

transfers - An optional Array of String identifiers for transfer adapters
that the client has configured. If omitted, the basic transfer adapter MUST
be assumed by the server.

[AlJohri]

Some success, I found that if I add this setting:

[lfs]
    basictransfersonly = true

Full config:

❯ cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = ssh://git.amazon.com/pkg/TK
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
    remote = origin
    merge = refs/heads/main
[lfs]
    basictransfersonly = true
[lfs "customtransfer.git-lfs-s3"]
    path = git-lfs-s3

Then I am able to push successfully to the GitFarm remote. Let's try to add the S3 remote back in now and try again.

[AlJohri]

While adding basictransfersonly allowed GitFarm to start working in the example above, I was unable to get the scoped lfs config to work with the s3 remote. Despite adding this config:

[lfs "s3://TK/TK"]
    standalonetransferagent = git-lfs-s3

I see that GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push s3 will use the wrong command to try to upload.

It will use this:

11:35:29.323717 trace git-lfs: tq: sending batch of size 1
11:35:29.323854 trace git-lfs: run_command: ssh s3 git-lfs-authenticate //TK/TK upload

when it should be this:

11:30:12.072889 trace git-lfs: tq: sending batch of size 1
11:30:12.072933 trace git-lfs: tq: starting transfer adapter "git-lfs-s3"
11:30:12.072945 trace git-lfs: xfer: adapter "git-lfs-s3" Begin() with 8 workers
11:30:12.072955 trace git-lfs: xfer: starting up custom transfer process "git-lfs-s3" for worker 0
11:30:12.073085 trace git-lfs: exec: sh '-c' 'git-lfs-s3 '

The only way to fix is to run git-lfs-s3 install which adds the lfs configs unscoped:

[lfs]
    standalonetransferagent = git-lfs-s3

[AlJohri]

The only way I can figure out how to work together with the two remotes at the moment is a shell script which will add the configs, run the git operation, and then remove the configs:

#!/usr/bin/env bash

export AWS_PROFILE=TK
export AWS_DEFAULT_PROFILE=TK
export AWS_REGION=us-east-1
export AWS_DEFAULT_REGION=us-east-1

git remote add s3 s3://TK/TK
git config --add lfs.customtransfer.git-lfs-s3.path git-lfs-s3
git config --add lfs.standalonetransferagent git-lfs-s3

# run command like pull or push
git pull s3 main

git config --unset lfs.standalonetransferagent
git config --unset lfs.customtransfer.git-lfs-s3.path
git remote remove s3

Looking forward to hear any suggestions on how to allow GitFarm and S3 remotes to properly co-exist.

[thevinchi]

Have a fix coming for this. The root cause is one layer below where this thread landed. Sharing for the record, will include more detail in the PR.

Why the scoped config didn't take effect

@AlJohri investigation got to:

[lfs "s3://TK/TK"] standalonetransferagent = git-lfs-s3

…with git config --get-urlmatch returning the right value, but git push s3 still running ssh s3 git-lfs-authenticate //TK/TK upload. Tracing through git-lfs:

• lfsapi/endpoint_finder.go:234-274: NewEndpoint parses the remote URL. For s3://..., the scheme isn't http/https/ssh/git/file, so it falls through to the default branch and is treated as a bare SSH URL. That's where the ssh s3 git-lfs-authenticate ... trace comes from.
• tq/manifest.go:270-284: findStandaloneTransfer looks up lfs.<ep.Url>.standalonetransferagent against the resolved endpoint URL, not the remote URL. Once s3:// has been mis-parsed as SSH, the URL git-lfs queries against scoped config no longer matches what git-lfs-s3 install wrote, so the scoped agent is never selected.

So scoping by remote URL alone can't work for s3:// — git-lfs's URL parser eats the scheme before the scoped lookup runs.

Fix

Set remote.<name>.lfsurl to a synthetic, deterministic HTTPS URL — RemoteEndpoint (endpoint_finder.go:117-134) checks lfsurl before falling back to remote.<name>.url, so this short-circuits the SSH-discovery path. The standalone agent is then scoped to that synthetic URL. The URL is never contacted (IsStandaloneTransfer() skips the LFS server entirely — see commands/lockverifier.go:164), it's purely a match key.

Concretely, git-lfs-s3 install --remote <name> will write:

[remote "<name>"]
    url = s3://<bucket>/<prefix>
    lfsurl = https://lfs-alias.git-remote-s3.test/<bucket>/<prefix>
[lfs "https://lfs-alias.git-remote-s3.test/<bucket>/<prefix>"]
    standalonetransferagent = git-lfs-s3
[lfs "customtransfer.git-lfs-s3"]
    path = git-lfs-s3

.test is RFC 6761-reserved (synthetic, non-resolvable) so the alias can't accidentally collide with a real host. Bare git-lfs-s3 install continues to work for single-remote setups. I would recommend --remote in multi-remote setups and likely warn from bare install when more than one remote is configured.

PR to follow.