feat(sdk): TinyGo WASM canary builds and spike plan#3063
Draft
pflynn-virtru wants to merge 13 commits intomainfrom
Draft
feat(sdk): TinyGo WASM canary builds and spike plan#3063pflynn-virtru wants to merge 13 commits intomainfrom
pflynn-virtru wants to merge 13 commits intomainfrom
Conversation
… and zipwrite modules
ADR documenting the TinyGo hybrid WASM architecture spike (SDK-WASM-1), including host crypto ABI, go/no-go criteria, and task breakdown. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
github-actions
bot
added
comp:ci
Contributor
Summary of ChangesHello @pflynn-virtru, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request lays the groundwork for a significant architectural shift by exploring the feasibility of a TinyGo-compiled WebAssembly (WASM) core engine for TDF operations. It introduces a set of targeted canary programs to assess TinyGo's compatibility with crucial Go standard library components and establishes a detailed spike plan. The primary aim is to validate a hybrid approach where cryptographic primitives are delegated to the host environment, while core TDF logic resides within a compact and portable WASM module, paving the way for enhanced flexibility and deployment options. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Ignored Files
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Go code, now WASM bound, TinyGo makes it small and fast, New frontiers are found. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive spike plan (ADR) for a TinyGo-based WASM core engine and adds several canary programs to test the compatibility of necessary Go standard library packages with TinyGo. The ADR is well-structured and detailed, covering architecture, ABI, risks, and a task breakdown. The canary programs are good initial steps to de-risk the effort.
My review focuses on improving the clarity of the ADR and addressing a potential memory safety issue in the wasmimport canary. I've suggested a clarification in the ADR's objective to align it better with the documented scope. I've also recommended adding a crucial comment to the wasmMalloc implementation to highlight its dependency on the -gc=leaking build flag for memory safety, which is critical for future maintenance.
Comment on lines
44
to
45
| buf := make([]byte, size) | ||
| return uint32(uintptr(unsafe.Pointer(&buf[0]))) |
Contributor
There was a problem hiding this comment.
The current implementation of wasmMalloc has a potential memory safety issue. The buf slice is allocated within the function, and returning a pointer to its underlying data can lead to a dangling pointer if the garbage collector reclaims buf after the function returns. While this is mitigated by compiling with the -gc=leaking flag as mentioned in the ADR, this critical dependency is not apparent from the code itself. For better maintainability and to prevent accidental misuse, I recommend adding a comment explaining why this is safe under the specific build conditions and highlighting the intentional memory leak.
buf := make([]byte, size)
// NOTE: This leaks memory. This is only safe because the module is compiled
// with the `-gc=leaking` flag, which prevents the garbage collector from
// reclaiming the memory.
return uint32(uintptr(unsafe.Pointer(&buf[0])))| ## Objective | ||
|
|
||
| Validate that a TinyGo-compiled WASM module can perform TDF3 single-segment | ||
| encrypt/decrypt with all crypto delegated to host functions, producing output |
Contributor
There was a problem hiding this comment.
The objective states encrypt/decrypt, but the scope of the spike seems to be focused on implementing encryption within the WASM module and then validating the output using the existing Go SDK for decryption. The 'Explicitly Out of Scope' section also mentions 'Decrypt inside WASM' is for a future milestone (M2). To better reflect the spike's goal, consider clarifying that only encryption will be performed by the WASM module by removing /decrypt.
Suggested change
| encrypt/decrypt with all crypto delegated to host functions, producing output | |
| encrypt with all crypto delegated to host functions, producing output |
Contributor
X-Test Failure Report |
…e logic - Removed `wasmimport` module and redundant crypto directives. - Consolidated `calculateSignature` logic into `writer.go` and removed it from `manifest.go`. - Updated TinyGo canary workflow to reflect WASM module restructuring.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| - name: "Report result" | ||
| if: always() | ||
| run: | | ||
| BUILD="${{ steps.build.outcome }}" |
Check notice
Code scanning / zizmor
code injection via template expansion Note
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
Adds self-contained decrypt benchmarks that construct valid TDFs programmatically and inject payload keys directly, bypassing KAS. Covers 1MB–2GB WriteTo path and 100MB streaming Read() path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
Benchmarks for the experimental streaming TDF Writer covering: - End-to-end encrypt (NewWriter + WriteSegment + Finalize) - Single segment encrypt throughput (WriteSegment only) - Full TDF assembly (segments + finalize bytes) Sizes: 1MB, 100MB, 1GB (short-mode skippable). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
…ibility
Replace reflection-based encoding/json with tinyjson (CosmWasm fork) codegen
for manifest, assertion, and policy structs in the TDF write path. This is a
prerequisite for the WASM core engine spike (SDK-WASM-1) since encoding/json
panics at runtime under TinyGo.
Changes:
- Add tinyjson codegen for manifest.go (13 types) and assertion_types.go (3 types)
- Change KeyAccess.PolicyBinding from interface{} to concrete PolicyBinding type
- Replace json.Marshal with .MarshalJSON() in writer.go, key_access.go, assertion.go
- Move Assertion/Statement/Binding structs to assertion_types.go for codegen
- Drop polymorphic Statement.Value UnmarshalJSON (reader concern, out of WASM scope)
- Add tinyjson TinyGo canary with manifest/policy/assertion round-trip validation
- Add wasm/Makefile with toolcheck, build, run, generate targets
- Add wasm/README.md with TinyGo/tinyjson/wasmtime install instructions
Canary results: tinyjson module compiles to 62KB raw / 29KB gzipped WASM,
all round-trip tests pass under wasmtime.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
…r WASM Copy production zipstream writer code (5 files from sdk/internal/zipstream/) into a standalone canary module and verify it compiles and runs correctly under TinyGo WASM. This completes Phase 1 (Foundation) of the WASM core engine spike (SDK-WASM-1). The canary exercises: - Single-segment TDF ZIP creation (header + manifest + central directory) - Multi-segment out-of-order writing (3 segments in order 2, 0, 1) - ZIP64 mode (ZIP64 EOCD + locator signatures) - CRC32 combine (multi-part checksum matches direct computation) Key finding: time.Time and time.Now() work correctly under TinyGo — the only identified risk for zipstream compatibility. Binary size: 113KB raw / 59KB gzipped (well under 300KB budget). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
X-Test Failure Report |
Use grouped redirects ({ cmd1; cmd2; } >> file) instead of individual
redirects to satisfy shellcheck/actionlint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| if: always() | ||
| run: | | ||
| BUILD="${{ steps.build.outcome }}" | ||
| RUN="${{ steps.run.outcome }}" |
Check notice
Code scanning / zizmor
code injection via template expansion Note
Contributor
X-Test Failure Report |
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
Document the three I/O models evaluated (WASM-drives, host-drives, hybrid) and recommend hybrid streaming I/O for M2 with read_input and write_output host imports. Update ABI evolution to 13 functions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
Move raw go:wasmimport declarations out of wasm/main.go into a dedicated hostcrypto package with idiomatic Go APIs ([]byte/string params, error returns). Wraps all 8 crypto host functions plus 2 streaming I/O hooks (read_input, write_output) for the M2 hybrid I/O architecture. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement the host side of the WASM crypto/IO ABI using Wazero. The host package registers "crypto" and "io" modules that fulfill the go:wasmimport calls from the hostcrypto guest package, delegating all crypto operations to lib/ocrypto. Host functions: random_bytes, aes_gcm_encrypt/decrypt, hmac_sha256, rsa_oaep_sha1_encrypt/decrypt, rsa_generate_keypair, get_last_error, read_input, write_output. Includes 20 tests covering ABI conformance (programmatic WASM module with matching imports), happy-path round-trips, error paths (bad keys, corrupted ciphertext, wrong-key decrypt, OOB writes, nil I/O), error state lifecycle, and truncation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
Replace the tdf_encrypt stub with a complete single-segment TDF3 encrypt path running inside the WASM sandbox. All crypto is delegated to the host via hostcrypto; manifest construction, policy binding, HS256 integrity, and ZIP assembly run inside WASM using tinyjson types and zipstream. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 11 integration tests that compile the WASM module, load it in wazero, and exercise tdf_encrypt end-to-end: round-trip decrypt, manifest field validation, policy binding, segment/root integrity, UUID format, attributes, empty plaintext, deterministic sizes, and error paths. Fix Go 1.25 wasip1 proc_exit(0) closing the module by using wazero's FunctionExporter to provide real WASI with a custom proc_exit that panics (non-sys.ExitError) instead of closing. Fix GC-safety in malloc by keeping allocations reachable, and copy data in ptrToString/ptrToBytes instead of using raw unsafe references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sdk/experimental/tdf/wasm/that exercise the stdlib and third-party packages needed for a hybrid WASM TDF core enginesdk/experimental/tdf/manifest, assertion, and key-access structs fromencoding/jsonto tinyjson codegen for TinyGo WASM compatibility — includes generated marshal/unmarshal code and updated teststinyjsonandzipstreamcanaries that validate these migrations compile and round-trip correctly under TinyGo.github/workflows/tinygo-wasm-canary.yaml) that compiles each canary with TinyGo targetingwasip1— expected to have failures until spike work is completedocs/adr/spike-wasm-core-tinygo-hybrid.md) documenting the TinyGo hybrid architecture, host crypto ABI (8 functions), go/no-go criteria, and task breakdownsdk/benchmark_test.go) measuring pure decrypt throughput at 1MB–2GB using direct key injection (no KAS dependency)sdk/experimental/tdf/benchmark_test.go) measuring streaming encrypt throughput, single-segment performance, and full TDF assemblyhostcryptopackage (sdk/experimental/tdf/wasm/hostcrypto/) with typed Go wrappers for allgo:wasmimporthost functions — the WASM-side ABI layer for crypto and I/Osdk/experimental/tdf/wasm/host/) implementing the host side of the crypto/IO ABI — registers"crypto"and"io"modules with Wazero, delegating all crypto tolib/ocrypto. Includes 20 tests covering ABI conformance, round-trip correctness, error paths, and OOB handling.hostcrypto; manifest construction, policy binding (HS256), integrity computation, and ZIP assembly run inside WASM using tinyjson types and zipstream.tdf_encrypt: round-trip decrypt, manifest field validation, policy binding, segment/root integrity, UUID format, attributes, empty plaintext, deterministic sizes, and error paths. Includes wazero proc_exit fix for Go 1.25 wasip1 and GC-safety fixes for WASM malloc.Canary programs
base64hexencoding/base64,encoding/hexzipwriteencoding/binary,hash/crc32,bytes,sort,synctinyjsonzipstreamiocontextio,context,strings,strconv,fmt,errorsstdjsonencoding/jsonwith TDF manifest structs (superseded bytinyjson)wasmgo:wasmimporthost ABI +tdfpackageHost crypto ABI
random_bytesocrypto.RandomBytes(n)aes_gcm_encryptocrypto.NewAESGcm(key).Encrypt(pt)aes_gcm_decryptocrypto.NewAESGcm(key).Decrypt(ct)hmac_sha256ocrypto.CalculateSHA256Hmac(key, data)rsa_oaep_sha1_encryptocrypto.NewAsymEncryption(pem).Encrypt(pt)rsa_oaep_sha1_decryptocrypto.NewAsymDecryption(pem).Decrypt(ct)rsa_generate_keypairocrypto.NewRSAKeyPair(bits)get_last_errorread_inputcfg.Input.Read(buf)write_outputcfg.Output.Write(buf)WASM TDF encrypt (Task 3.1)
Single-segment TDF3 encrypt running entirely inside the WASM sandbox:
hostcrypto.RandomBytes(32)hostcrypto.RsaOaepSha1Encrypt(kasPub, dek)MarshalJSON()HMAC-SHA256(dek, base64Policy)→ hex → base64hostcrypto.AesGcmEncrypt(dek, plaintext)HMAC-SHA256(dek, cipher)→ base64HMAC-SHA256(dek, segmentSig)→ base64WriteSegment+FinalizeEncrypt integration tests (11 tests)
TestTDFEncryptRoundTripTestTDFEncryptManifestFieldsTestTDFEncryptPolicyBindingTestTDFEncryptSegmentIntegrityTestTDFEncryptPolicyUUIDTestTDFEncryptWithAttributesTestTDFEncryptEmptyPlaintextTestTDFEncryptDeterministicSizesTestTDFEncryptErrorInvalidKeyTestTDFEncryptErrorBufferTooSmallTestTDFEncryptGetErrorClearsAfterReadget_errorretrievalBenchmarks
Measured on a single dev machine (Apple Silicon); numbers are indicative, not normative.
BenchmarkDecrypt/1MB–2GBsdk/benchmark_test.goBenchmarkStreamDecrypt(100MB, 32KB reads)sdk/benchmark_test.goBenchmarkWriterEncrypt/1MB–1GBsdk/experimental/tdf/benchmark_test.goBenchmarkWriterWriteSegment(2MB)sdk/experimental/tdf/benchmark_test.goBenchmarkWriterAssemble/1MB–100MBsdk/experimental/tdf/benchmark_test.goCI workflow
ciaggregation job, does not block PRsfail-fast: false— all canaries run independentlysdk/experimental/tdf/**,sdk/internal/zipstream/**,sdk/manifest.go,lib/ocrypto/**Test plan
base64hex,zipwrite,tinyjson, andzipstreamcanaries pass TinyGo compilation and executiongo build ./sdk/experimental/tdf/wasm/...)ciaggregation jobneedslistcd sdk && go test -bench=BenchmarkDecrypt -short -run=^$ .cd sdk/experimental/tdf && go test -bench=Benchmark -short -run=^$ .go test -v ./sdk/experimental/tdf/wasm/host/(20 host ABI tests)go test -v ./sdk/experimental/tdf/wasm/host/ -run TestTDFEncrypt(11 end-to-end tests)go build ./sdk/...GOOS=wasip1 GOARCH=wasm go build ./sdk/experimental/tdf/wasm/Related: SDK-WASM-1 (Jira)
🤖 Generated with Claude Code