Implement Scratch.external - extensions part

[GarboMuffin]

5 new APIs:

• Scratch.external.importModule() is a wrapper around import() that gets inlined at build time
• Scratch.external.fetch() is a wrapper around fetch() that gets inlined at build time
• Scratch.external.dataURL() is like external.fetch() but it gives you a base64 data: URL
• Scratch.external.blob() is like external.fetch() but it gives you a Blob
• Scratch.external.evalAndReturn() is for executing scripts inside of a function and then returning an expression of your choice

In development, all of these call directly into VM implementation which does actually go and fetch each time the extension runs. In production, all of these are inlined in various ways so that everything is inside the JS file. All references to Scratch.external go away, so we're not breaking compatibility with anything. A new concept called "build snippets" are added into built JS as needed to implement things like base85 and zstd decompression.

URLs should point to jsdelivr, unpkg, raw.githubusercontent.com, etc. In development mode, new URLs will be found and fetched so their SHA-256 can be stored. In production builds, these resources get fetched and the hashes are checked to ensure no tampering. Must use an immutable URL pinned to an npm tag or commit hash.

Face sensing has been updated to use these as a first example.

Ideally this API would be called Scratch.import, but using "import" verbatim causes various issues in TypeScript and elsewhere so we're avoiding that. Hence, "Scratch.external" and "importModule"

[GarboMuffin]

Using import() is not sufficient for things like https://raw.githubusercontent.com/PsychoGoldfishNG/NewgroundsIO-JS/refs/heads/main/dist/NewgroundsIO.min.js that aren't ES modules

So, we could have several different ways to import things, and make them available as Scratch.import(), Scratch.import.asPlainScript(), Scratch.import.executeInIIFEAndExport(), etc.

[FurryR]

Using import() is not sufficient for things like https://raw.githubusercontent.com/PsychoGoldfishNG/NewgroundsIO-JS/refs/heads/main/dist/NewgroundsIO.min.js that aren't ES modules

So, we could have several different ways to import things, and make them available as Scratch.import(), Scratch.import.asPlainScript(), Scratch.import.executeInIIFEAndExport(), etc.

I prefer letting extension developers decide whether to use import(). Bundling is also okay, but the production should not be minified. (Webpack development build may work, but since its obsolete i recommend tsup)

[GarboMuffin]

no, it uses URLs to websites

see #2264's imports for general kind of syntax that might work, though actually that PR and this one don't integrate quite right yet, and there's still some missing mechanisms that need to be added

there's a lot in the air - might restrict it to certain websites only (trusted, unlikely to vanish, operated by neutral parties that won't do anything weird with traffic logs) or make it skip the CDNs and download from npm instead. lots of things like that still being decided

there's some nice properties of this general approach:

• avoid bloating the repository with every version of every dependency
• no opportunity for anyone to tamper a dependency before copying it into the repository - you would have to tamper with upstream or the CDN which is presumably more difficult and you would have much higher priorities to attack
• you can copy an extension's unprocessed source code out of the repo, paste it into the editor, and it will always work even before merge. if libraries got fetched from e.tw.o, extension would need to be merged first for those to work

[GarboMuffin]

If your school blocks major CDNs then you have much larger problems, but also the point of this PR is that all the dependencies will get inlined during a production build which solves that problem

[GarboMuffin]

The URLs in the source code get used in development (easier to debug) while in production they get transparently converted into another format

[GarboMuffin]

Re: bundlers were mentioned earlier

I don't intend to bring in bundlers right now. The goal is to use the files provided by upstream without modification. That's needed for non-production usage where the URLs are left unchanged, is a lot simpler, and I think most of the dependencies that people want to integrate tend to offer scripts that work without changes. Biggest hurdle is that few dependencies offer ES modules. Hence the idea of allowing an import-as-script-tag or something that fetches the source code and evals it in an IIFE

Letting people configure bundlers inevitably leads to dependencies requiring random versions of rollup or parcel or webpack or whatever with random configuration and plugins. I don't want to be maintaining that if I don't need to.

If upstream doesn't provide workable files then we can figure something out on a case-by-case basis, possibly by forking a version and publishing under @turbowarp/*

[Brackets-Coder]

If your school blocks major CDNs then you have much larger problems, but also the point of this PR is that all the dependencies will get inlined during a production build which solves that problem

I'm not worried about me and I'm not on a school network. I'm concerned for our users who might have a whitelisted school network, but I suppose I misunderstood the fact that the inlining happens when the extension is built, not every time it's loaded, which entirely resolves that

Now that I really understand what's happening here I'm realizing how much better it actually is than I thought

Sorry for the confusion

[GarboMuffin]

Initially, face sensing became 16MB larger with this. Cut that about in half by not supporting devices without WASM SIMD. Cut in half again by putting the WASM inside a zip and inflating it at runtime along with using base85 instead of base64. Now it's down to 3.1MB. Still a bit excessive -- wouldn't have merged this if Scratch didn't add it first -- but tolerable.

[GarboMuffin]

It'll come down to 2.7MB by switching to fzstd instead of relying on jszip. It'll also load faster that way

[GarboMuffin]

Down to 2.2MB by compressing the tensorflow scripts. Might end up reverting that later since it does make things take a fair bit longer to start up