Add a mechanism to allow files to be uploaded in a specific order

[cowboy]

So I was using FTP-Deploy-Action and ran into an issue, but looking at the code, it seemed more relevant to file the issue in this repo - LMK if you want me to move it. Let me explain what I'm doing.

The website I'm deploying has:

• an assets directory containing fingerprinted css and js files
• an index.html file that loads fingerprinted assets and runs the app
• a build.json file that is generated at build time with the commit sha, which the app polls for to see if it has changed, at which time it reloads the app

eg.

build/client/
|-- assets
|   |-- api-CxujkEL4.js
|   |-- build-info-uX_ocp1e.js
|   |-- chunk-KNED5TY2-B2LzAy-9.js
|   |-- (etc)
|   |-- test-nav-Df4njhOQ.css
|   |-- test-nav-QvsV6Ysp.js
|   `-- with-props-ColFkw8L.js
|-- build.json
`-- index.html

The problem Im running into is that I have no way to specify the order in which these files are uploaded. However, the files must be uploaded in a very specific order:

1. All new fingerprinted asset files must first be uploaded. If a user loads the app, or the upload fails during this step, the app will continue to work as before, because nothing has changed with any existing files.
2. The index.html file may now be uploaded. Once this file is uploaded, any user loading the app will see the new version.
3. The build.json file may now be uploaded. Once this file is uploaded, the app will see the change and reload itself.
4. No-longer-used files may now be deleted from the server.

If step 2 happens before the asset files are uploaded, any user loading the app would run into problems, because necessary asset files don't exist on the server yet.

If step 3 happens before index.html is uploaded, the app would reload itself prematurely and see no changes, because index.html wouldn't be on the server yet, so it would reload the old version. Once index.html was uploaded, the app would have no way of knowing it needed to re-reload itself.

Right now, I'm working around this limitation like so, but as you can imagine, this isn't ideal, as it adds complexity and makes the entire deploy process take significantly longer.

name: Deploy Web Dev

on:
  push:
    branches: [main]

jobs:
  deploy-dev:
    runs-on: ubuntu-latest
    concurrency:
      group: deploy-dev
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version-file: 'package.json'
      - run: npm ci
      - run: npm run build-ci

      # step 1: Upload fingerprinted assets, use a dummy state file so it doesn't delete anything.
      #         Unfortunately, this will re-upload unchanged files.

      - name: ftp deploy - step 1 - fingerprinted assets
        uses: SamKirkland/FTP-Deploy-Action@v4.3.5
        with:
          local-dir: ./build/client/
          username: ci
          password: ${{ secrets.FTP_PASSWORD }}
          server: ${{ secrets.FTP_SERVER }}
          state-name: ftp-deploy-sync-state-dummy-1.json
          exclude: |
            index.html
            build.json

      # step 2: Upload index.html which effectively makes the new version go live, still don't delete anything.

      - run: rm ./build/client/ftp-deploy-sync-state-dummy-1.json
      - name: ftp deploy - step 2 - index.html
        uses: SamKirkland/FTP-Deploy-Action@v4.3.5
        with:
          local-dir: ./build/client/
          username: ci
          password: ${{ secrets.FTP_PASSWORD }}
          server: ${{ secrets.FTP_SERVER }}
          state-name: ftp-deploy-sync-state-dummy-2.json
          exclude: |
            assets/**
            build.json

      # step 3: Upload build.json so that app can auto-update, overwrite dummy state files, delete as usual.
      #         Also, force-upload new dummy state files so no files will be deleted in steps 1+2 next time.

      - run: echo "cachebust-${{ github.run_id }}" > ./build/client/ftp-deploy-sync-state-dummy-1.json
      - run: echo "cachebust-${{ github.run_id }}" > ./build/client/ftp-deploy-sync-state-dummy-2.json
      - name: ftp deploy - step 3 - build.json
        uses: SamKirkland/FTP-Deploy-Action@v4.3.5
        with:
          local-dir: ./build/client/
          username: ci
          password: ${{ secrets.FTP_PASSWORD }}
          server: ${{ secrets.FTP_SERVER }}

So, here's the ask. Can an option be added to this library and to FTP-Deploy-Action that lets the user specify the order in which files should be uploaded?

It could possibly just be an array of glob patterns (like how exclude works). The glob patterns would be iterated over in order, and for each one, all not-already-added matching files would be added to a sorted output array.

The underlying code would be something along these lines:

function sortFiles(files, patterns) {
  const map = {};
  const sortedFiles = [];

  patterns.forEach((pattern) => {
    const files = multimatch(unsortedFiles, pattern);
    for (const file of files) {
      if (!map[file]) {
        map[file] = true;
        sortedFiles.push(file);
      }
    }
  });

  return sortedFiles;
}

Eg.

const unsortedFiles = [
  "build.json",
  "index.html",
  "assets/home-C6JigMfh.js",
  "assets/chunk-KNED5TY2-B2LzAy-9.js",
  "assets/build-info-uX_ocp1e.js",
  "assets/overlay-B98S4KzY.css",
  "assets/test-Dfnwt5EG.css",
  "assets/entry.client-Jga_eV3f.js",
];

const sortPatterns = ["assets/**", "index.html", "build.json"];
const sortedFiles = sortFiles(unsortedFiles, sortPatterns);

console.log(sortedFiles);
/*
[
  "assets/home-C6JigMfh.js",
  "assets/chunk-KNED5TY2-B2LzAy-9.js",
  "assets/build-info-uX_ocp1e.js",
  "assets/overlay-B98S4KzY.css",
  "assets/test-Dfnwt5EG.css",
  "assets/entry.client-Jga_eV3f.js",
  "index.html",
  "build.json",
]
*/

Which would allow me (or anyone else deploying a static generated site with fingerprinted assets) to do this:

      - name: ftp deploy
        uses: SamKirkland/FTP-Deploy-Action@v4.3.5
        with:
          local-dir: ./build/client/
          username: ci
          password: ${{ secrets.FTP_PASSWORD }}
          server: ${{ secrets.FTP_SERVER }}
          order: |
            assets/**
            index.html
            build.json

Thanks!

[cowboy]

FWIW, I worked on it a bit this evening and created a patch against @samkirkland/ftp-deploy@1.2.4 to use with patch-package, so I can now just put something like this in an npm deploy script:

ftp-deploy --local-dir ./build/client/ --username ci \
  --server $FTP_SERVER --password $FTP_PASSWORD \
  --sort '!**/{index.html,build.json}' index.html build.json

And clean up my workflow like so:

      - run: npm run deploy
        env:
          FTP_PASSWORD: ${{ secrets.FTP_PASSWORD }}
          FTP_SERVER: ${{ secrets.FTP_SERVER }}

Most use cases would probably be to add everything except the file(s) which need to be deployed last, followed by those file(s). In my case, --sort '!**/{index.html,build.json}' index.html build.json does:

1. First, add all files (recursively) that are not index.html or build.json
2. Then add index.html
3. Then add build.json

Here's the patch. It's certainly not directly usable given how you build your source into dist but it should save you time if you choose to implement this.

For anyone else in a similar situation, in the meantime, you can:

• save the following file as patches/@samkirkland+ftp-deploy+1.2.4.patch
• add the patch-package and @samkirkland/ftp-deploy@1.2.4 npm dependiencies
• add the postinstall script per the patch-package README

diff --git a/node_modules/@samkirkland/ftp-deploy/dist/cli.js b/node_modules/@samkirkland/ftp-deploy/dist/cli.js
index dc9d075..b8aeecd 100755
--- a/node_modules/@samkirkland/ftp-deploy/dist/cli.js
+++ b/node_modules/@samkirkland/ftp-deploy/dist/cli.js
@@ -26,6 +26,7 @@ const argv = yargs_1.default.options({
     "dry-run": { type: "boolean", default: false, description: "Prints which modifications will be made with current config options, but doesn't actually make any changes" },
     "dangerous-clean-slate": { type: "boolean", default: false, description: "Deletes ALL contents of server-dir, even items in excluded with 'exclude' argument" },
     "exclude": { type: "array", default: module_1.excludeDefaults, description: "An array of glob patterns, these files will not be included in the publish/delete process" },
+    "sort": { type: "array", default: [], description: "An array of glob patterns used to sort files to be uploaded. See this github issue for more info: https://github.com/SamKirkland/ftp-deploy/issues/44" },
     "log-level": { choices: ["minimal", "standard", "verbose"], default: "standard", description: "How much information should print. minimal=only important info, standard=important info and basic file changes, verbose=print everything the script is doing" },
     "security": { choices: ["strict", "loose"], default: "loose", description: "" }
 })
diff --git a/node_modules/@samkirkland/ftp-deploy/dist/deploy.js b/node_modules/@samkirkland/ftp-deploy/dist/deploy.js
index 8825951..2e57ee2 100644
--- a/node_modules/@samkirkland/ftp-deploy/dist/deploy.js
+++ b/node_modules/@samkirkland/ftp-deploy/dist/deploy.js
@@ -195,7 +195,7 @@ function deploy(args, logger, timings) {
             totalBytesUploaded = diffs.sizeUpload + diffs.sizeReplace;
             timings.start("upload");
             try {
-                const syncProvider = new syncProvider_1.FTPSyncProvider(client, logger, timings, args["local-dir"], args["server-dir"], args["state-name"], args["dry-run"]);
+                const syncProvider = new syncProvider_1.FTPSyncProvider(client, logger, timings, args["local-dir"], args["server-dir"], args["state-name"], args["dry-run"], args.sort);
                 yield syncProvider.syncLocalToServer(diffs);
             }
             finally {
diff --git a/node_modules/@samkirkland/ftp-deploy/dist/syncProvider.js b/node_modules/@samkirkland/ftp-deploy/dist/syncProvider.js
index 243d1e4..69eb71e 100644
--- a/node_modules/@samkirkland/ftp-deploy/dist/syncProvider.js
+++ b/node_modules/@samkirkland/ftp-deploy/dist/syncProvider.js
@@ -27,7 +27,7 @@ function ensureDir(client, logger, timings, folder) {
 }
 exports.ensureDir = ensureDir;
 class FTPSyncProvider {
-    constructor(client, logger, timings, localPath, serverPath, stateName, dryRun) {
+    constructor(client, logger, timings, localPath, serverPath, stateName, dryRun, sort) {
         this.client = client;
         this.logger = logger;
         this.timings = timings;
@@ -35,6 +35,7 @@ class FTPSyncProvider {
         this.serverPath = serverPath;
         this.stateName = stateName;
         this.dryRun = dryRun;
+        this.sort = sort;
     }
     /**
      * Converts a file path (ex: "folder/otherfolder/file.txt") to an array of folder and a file path
@@ -133,19 +134,54 @@ class FTPSyncProvider {
             this.logger.all(`----------------------------------------------------------------`);
             this.logger.all(`Making changes to ${totalCount} ${(0, utilities_1.pluralize)(totalCount, "file/folder", "files/folders")} to sync server state`);
             this.logger.all(`Uploading: ${(0, pretty_bytes_1.default)(diffs.sizeUpload)} -- Deleting: ${(0, pretty_bytes_1.default)(diffs.sizeDelete)} -- Replacing: ${(0, pretty_bytes_1.default)(diffs.sizeReplace)}`);
+            if (this.sort.length > 0) {
+                this.logger.all(`Sorting uploads in this order: ${this.sort.join(', ')}`);
+            }
             this.logger.all(`----------------------------------------------------------------`);
             // create new folders
             for (const file of diffs.upload.filter(item => item.type === "folder")) {
                 yield this.createFolder(file.name);
             }
-            // upload new files
-            for (const file of diffs.upload.filter(item => item.type === "file").filter(item => item.name !== this.stateName)) {
-                yield this.uploadFile(file.name, "upload");
-            }
-            // replace new files
-            for (const file of diffs.replace.filter(item => item.type === "file").filter(item => item.name !== this.stateName)) {
-                // note: FTP will replace old files with new files. We run replacements after uploads to limit downtime
-                yield this.uploadFile(file.name, "replace");
+            if (this.sort.length > 0) {
+                function sortRecords(records, patterns) {
+                    const map = {};
+                    const sortedFiles = Array.from(patterns, () => []);
+                  
+                    patterns.forEach((pattern, i) => {
+                        for (const record of records) {
+                            const { name } = record;
+                            if (pattern[0] === '!') {
+                                pattern = ['**', pattern];
+                            }
+                            const isMatch = require("multimatch")(name, pattern).length > 0;
+                            if (isMatch && !map[name]) {
+                                map[name] = true;
+                                sortedFiles[i].push(record);
+                            }
+                        }
+                    });
+                  
+                    return sortedFiles.reduce((acc, arr) => [...acc, ...arr], []);
+                }
+                let records = [
+                    ...diffs.upload.map(obj => ({ ...obj, mode: 'upload' })),
+                    ...diffs.replace.map(obj => ({ ...obj, mode: 'replace' })),
+                ];
+                records = records.filter(item => item.type === "file").filter(item => item.name !== this.stateName);
+                records = sortRecords(records, this.sort);
+                for (const file of records) {
+                    yield this.uploadFile(file.name, file.mode);
+                }
+            } else {
+                // upload new files
+                for (const file of diffs.upload.filter(item => item.type === "file").filter(item => item.name !== this.stateName)) {
+                    yield this.uploadFile(file.name, "upload");
+                }
+                // replace new files
+                for (const file of diffs.replace.filter(item => item.type === "file").filter(item => item.name !== this.stateName)) {
+                    // note: FTP will replace old files with new files. We run replacements after uploads to limit downtime
+                    yield this.uploadFile(file.name, "replace");
+                }
             }
             // delete old files
             for (const file of diffs.delete.filter(item => item.type === "file")) {
diff --git a/node_modules/@samkirkland/ftp-deploy/dist/utilities.js b/node_modules/@samkirkland/ftp-deploy/dist/utilities.js
index 5980d49..c33caf3 100644
--- a/node_modules/@samkirkland/ftp-deploy/dist/utilities.js
+++ b/node_modules/@samkirkland/ftp-deploy/dist/utilities.js
@@ -160,6 +160,7 @@ function getDefaultSettings(withoutDefaults) {
         "dry-run": (_f = withoutDefaults["dry-run"]) !== null && _f !== void 0 ? _f : false,
         "dangerous-clean-slate": (_g = withoutDefaults["dangerous-clean-slate"]) !== null && _g !== void 0 ? _g : false,
         "exclude": (_h = withoutDefaults.exclude) !== null && _h !== void 0 ? _h : module_1.excludeDefaults,
+        "sort": (_h = withoutDefaults.sort) !== null && _h !== void 0 ? _h : [],
         "log-level": (_j = withoutDefaults["log-level"]) !== null && _j !== void 0 ? _j : "standard",
         "security": (_k = withoutDefaults.security) !== null && _k !== void 0 ? _k : "loose",
         "timeout": (_l = withoutDefaults.timeout) !== null && _l !== void 0 ? _l : 30000,

[SamKirkland]

Very good idea, nice patch file too!

[cowboy]

Note that I just updated the patch to support negations so you can do something like --sort '!**/{index.html,build.json}' index.html build.json (which I also explained in the previous post).