Add pluggable page-level AEAD encryption support by HTHou · Pull Request #866 · apache/tsfile

HTHou · 2026-07-15T10:22:24Z

Summary

add an optional file encryption header carrying provider, profile, key, wrapped data key, and file identity metadata
introduce a provider-neutral AEAD SPI and registry without embedding a concrete cryptographic implementation
encrypt compressed page bodies for non-aligned, aligned, and table-model write paths, and decrypt them on the corresponding read paths
preserve existing unencrypted TsFile behavior and propagate encryption metadata through append, recovery, lazy loading, and sketch tooling
add compatibility, round-trip, tamper-detection, append, and recovery tests

Motivation

TsFile currently has legacy encryption interfaces but no self-describing file-level context for pluggable page-level authenticated encryption. This change establishes the format and I/O integration points while leaving algorithm and key-management implementations to external providers.

Compatibility

Existing unencrypted TsFiles keep their current layout and remain readable.
Encrypted files require a registered provider matching the identifiers stored in the encryption header.
Page metadata remains readable while compressed page bodies are protected with AEAD.
The encrypted-file format and public API are still under discussion and are not intended as a compatibility commitment in this draft.

Validation

./mvnw spotless:check -P with-java -pl java/tsfile -am
./mvnw test -P with-java -pl java/tsfile -am -Dtest='PageCryptoContextTest,FileEncryptionHeaderTest,TDEPageAeadTsFileTest,UnClosedTsFileReaderTest,ForceAppendTsFileWriterTest,RestorableTsFileIOWriterTest,TimePageWriterTest' -Dsurefire.failIfNoSpecifiedTests=false
30 targeted tests passed with no failures or errors

Draft discussion points

finalize the encryption-header schema and format-version semantics
bind page authentication data to an unambiguous chunk/page identity
refine page-index ownership and the separation between legacy encryption parameters and file AEAD context
confirm append/recovery behavior and backward-compatibility expectations before stabilizing the format

codecov-commenter · 2026-07-15T10:44:40Z

Codecov Report

❌ Patch coverage is 80.37825% with 166 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.29%. Comparing base (d33e640) to head (517b06b).
⚠️ Report is 7 commits behind head on develop.

Files with missing lines	Patch %	Lines
...pache/tsfile/file/header/FileEncryptionHeader.java	73.56%	23 Missing ⚠️
...sfile/write/v4/AbstractTableModelTsFileWriter.java	41.37%	17 Missing ⚠️
...a/org/apache/tsfile/encrypt/PageCryptoContext.java	74.19%	16 Missing ⚠️
...a/org/apache/tsfile/read/TsFileSequenceReader.java	82.75%	15 Missing ⚠️
...g/apache/tsfile/read/reader/chunk/ChunkReader.java	62.16%	14 Missing ⚠️
...ain/java/org/apache/tsfile/write/TsFileWriter.java	59.37%	13 Missing ⚠️
...ache/tsfile/read/reader/page/LazyLoadPageData.java	65.51%	10 Missing ⚠️
...va/org/apache/tsfile/encrypt/EncryptParameter.java	92.24%	9 Missing ⚠️
...ava/org/apache/tsfile/file/header/ChunkHeader.java	76.47%	8 Missing ⚠️
...che/tsfile/encrypt/EncryptionProviderRegistry.java	78.57%	6 Missing ⚠️
... and 12 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #866      +/-   ##
===========================================
+ Coverage    64.53%   65.29%   +0.76%     
===========================================
  Files          760      766       +6     
  Lines        53314    54198     +884     
  Branches      8453     8620     +167     
===========================================
+ Hits         34404    35388     +984     
+ Misses       17258    17154     -104     
- Partials      1652     1656       +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…x/tsfile-tde-page-aead # Conflicts: # java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java # java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java

…age-aead

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings

InputStream#readNBytes(...) is only available starting in Java 9. If this project targets Java 8… · New
getAssociatedData() always allocates a new byte[] copy. Since this is called on every page… · New
For page-AEAD encrypted files, decrypting requires both the correct pageIndex and a nonnegative… · New
registerProvider(...) trims provider ids before storing them, but unregisterProvider(...) does not.… · New

What changed in this PR

This PR introduces pluggable page-level AEAD encryption for TsFile by adding a self-describing file encryption header, a provider-neutral encryption SPI/registry, and integrating page body encrypt/decrypt across write and read paths while preserving unencrypted TsFile compatibility.

Changes:

Added FileEncryptionHeader (written after the TsFile version byte) and propagated encryption context through writers/readers, append, and recovery flows.
Introduced provider-neutral AEAD SPI (IEncryptProvider, EncryptionProviderRegistry) and page-associated-data binding (PageCryptoContext) with per-chunk ordinals and per-page indices.
Added/updated tests to validate round-trips, tamper detection, ordinal continuity across append/recovery, and reader behavior.

File	Description
java/tsfile/src/test/java/org/apache/tsfile/write/writer/TimePageWriterTest.java	Adds an AEAD encrypt/decrypt page round-trip test using the test provider.
java/tsfile/src/test/java/org/apache/tsfile/write/writer/RestorableTsFileIOWriterTest.java	Adds recovery/ownership tests for encrypted headers and chunk ordinal continuation.
java/tsfile/src/test/java/org/apache/tsfile/write/writer/ForceAppendTsFileWriterTest.java	Adds force-append coverage for encrypted files and chunk ordinal uniqueness.
java/tsfile/src/test/java/org/apache/tsfile/write/TDEPageAeadTsFileTest.java	New integration tests for encrypted read/write paths, tamper detection, and chunk reuse rejection.
java/tsfile/src/test/java/org/apache/tsfile/read/UnClosedTsFileReaderTest.java	Adds reading encrypted pages from file header in unclosed reader scenario.
java/tsfile/src/test/java/org/apache/tsfile/file/header/FileEncryptionHeaderTest.java	New round-trip and validation tests for the file encryption header format.
java/tsfile/src/test/java/org/apache/tsfile/encrypt/TestAeadEncryptionProvider.java	Adds a JDK-only AES-GCM test provider implementing the new SPI.
java/tsfile/src/test/java/org/apache/tsfile/encrypt/PageCryptoContextTest.java	Adds tests for AAD binding/tamper detection and ordinal allocation behavior.
java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java	Writes encryption header before data, threads encryption context through chunk headers, and blocks encrypted chunk copy.
java/tsfile/src/main/java/org/apache/tsfile/write/writer/RestorableTsFileIOWriter.java	Loads encryption header on recovery, resumes ordinal allocation, and closes owned AEAD parameters.
java/tsfile/src/main/java/org/apache/tsfile/write/writer/ForceAppendTsFileWriter.java	Loads encryption header for append, resumes ordinal allocation, and closes owned AEAD parameters.
java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java	Propagates file-level AEAD context and ensures owned AEAD parameters are closed.
java/tsfile/src/main/java/org/apache/tsfile/write/page/ValuePageWriter.java	Uses centralized `PageBodyEncoder` and adds pageIndex/chunkOrdinal support.
java/tsfile/src/main/java/org/apache/tsfile/write/page/TimePageWriter.java	Uses centralized `PageBodyEncoder` and adds pageIndex/chunkOrdinal support.
java/tsfile/src/main/java/org/apache/tsfile/write/page/PageWriter.java	Uses centralized `PageBodyEncoder` and adds pageIndex/chunkOrdinal support.
java/tsfile/src/main/java/org/apache/tsfile/write/page/PageBodyEncoder.java	New helper to compress + encrypt page bodies (including AEAD AAD binding).
java/tsfile/src/main/java/org/apache/tsfile/write/page/EncodedPageBody.java	New lightweight holder for encoded page body bytes (data/offset/size).
java/tsfile/src/main/java/org/apache/tsfile/write/chunk/ValueChunkWriter.java	Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers.
java/tsfile/src/main/java/org/apache/tsfile/write/chunk/TimeChunkWriter.java	Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers.
java/tsfile/src/main/java/org/apache/tsfile/write/chunk/ChunkWriterImpl.java	Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers.
java/tsfile/src/main/java/org/apache/tsfile/write/TsFileWriter.java	Propagates file-level AEAD context and closes owned AEAD parameters on close.
java/tsfile/src/main/java/org/apache/tsfile/utils/TsFileSketchTool.java	Uses `reader.getDataStartOffset()` to handle optional encryption header.
java/tsfile/src/main/java/org/apache/tsfile/read/reader/page/LazyLoadPageData.java	Adds AEAD decrypt path using pageIndex/chunkOrdinal and validates plaintext sizes.
java/tsfile/src/main/java/org/apache/tsfile/read/reader/chunk/ChunkReader.java	Threads pageIndex/chunkOrdinal into page readers and adds AEAD page decrypt/uncompress overloads.
java/tsfile/src/main/java/org/apache/tsfile/read/reader/chunk/AbstractAlignedChunkReader.java	Threads pageIndex/chunkOrdinal through aligned chunk page reads.
java/tsfile/src/main/java/org/apache/tsfile/read/common/Chunk.java	Rejects merging AEAD-encrypted chunks and ensures ordinal allocation during rewrite.
java/tsfile/src/main/java/org/apache/tsfile/read/UnClosedTsFileReader.java	Uses file-level encrypt param directly for unclosed reader.
java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java	Parses optional encryption header, tracks data start offset, records chunk ordinals, and supports AEAD page decryption with AAD.
java/tsfile/src/main/java/org/apache/tsfile/file/header/FileEncryptionHeader.java	New persisted encryption header format (versioned, size-bounded, wrapped key only).
java/tsfile/src/main/java/org/apache/tsfile/file/header/ChunkHeader.java	Adds optional chunk ordinal field for AEAD binding and updates (de)serialization accordingly.
java/tsfile/src/main/java/org/apache/tsfile/file/MetaMarker.java	Adds `ENCRYPTION_HEADER` marker for file-level encryption header.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/PageCryptoContext.java	New canonical AEAD page context with stable associated data construction.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncryptor.java	Adds encryptor selection for AEAD params and `encryptPage` default method.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncryptProvider.java	New SPI interface for provider implementations.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncrypt.java	Extends `AutoCloseable` and adds `getPageBodyOverhead` contract for AEAD profiles.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/IDecryptor.java	Adds decryptor selection for AEAD params and `decryptPage` default method.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptionProviderRegistry.java	New registry (and service loader integration) for stable provider ids.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptUtils.java	Adds AEAD-aware `getEncryptParameter` and `getEncrypt` overload.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptParameter.java	Extends parameters for AEAD (provider/profile ids, wrapped key, file id, chunk ordinal allocation, destroy/close).
java/common/src/main/resources/org/apache/tsfile/i18n/messages_zh.properties	Adds i18n strings for new encryption header/provider/AAD errors.
java/common/src/main/resources/org/apache/tsfile/i18n/messages.properties	Adds i18n strings for new encryption header/provider/AAD errors.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

Unresolved context-validation, fail-open SPI, and missing last-reader decryption paths can produce insecure or unreadable files.

4 open findings

4 resolved since last review

InputStream#readNBytes(...) is only available starting in Java 9. If this project targets Java 8…
For page-AEAD encrypted files, decrypting requires both the correct pageIndex and a nonnegative…
getAssociatedData() always allocates a new byte[] copy. Since this is called on every page…
registerProvider(...) trims provider ids before storing them, but unregisterProvider(...) does not.…

Previously missed (1)

In code that hasn't changed since last review

Decrypt aligned BLOB/OBJECT pages before last-point decoding

java/tsfile/src/main/java/org/apache/tsfile/write/page/PageBodyEncoder.java:69

Encrypting every page body leaves TsFileLastReader.readAlignedLastPoint incompatible with encrypted aligned BLOB/OBJECT columns: that path slices the stored body and passes it directly to IUnCompressor/ValuePageReader without decryption (TsFileLastReader.java:179-216). It will feed ciphertext to the decoder (or decompressor), so last-point reads fail for these supported types. Route that path through the AEAD-aware page deserialization while tracking the selected page index and chunk ordinal.

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Headerless recovery can create unreadable mixed-format files, and encryption-context validation and lifetime issues remain unresolved.

2 open findings

4 resolved since last review

Previously missed (2)

In code that hasn't changed since last review

Normalize provider IDs during lookup

java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptionProviderRegistry.java:58

Provider IDs are trimmed when registering and unregistering, but lookup uses the untrimmed persisted value. A provider whose declared ID contains surrounding whitespace is therefore registered successfully under the trimmed key yet can never be resolved from a parameter/header carrying that declared ID. Apply the same normalization during lookup (or reject non-canonical IDs consistently).

Preserve encryption context for materialized chunks

java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java:2417

The same EncryptParameter instance is attached to every Chunk returned by readMemChunk() and to its lazy page readers. Destroying it here means an already materialized in-memory chunk becomes unusable as soon as the sequence reader is closed; subsequent page loading fails with parameter_destroyed. Give returned chunks an independently owned context or otherwise coordinate the context lifetime instead of invalidating it with the file handle.

🧠 Review effort: Balanced

+          EncryptParameter recoveredParameter = reader.getEncryptParam();
+          if (recoveredParameter != null && recoveredParameter.isTdePageAead()) {
+            if (this.param != null && this.param.isTdePageAead()) {
+              this.param.close();
+            }
+            this.param = recoveredParameter.copy();
+            setEncryptParam(this.param);
+          }
+          markExistingFileStarted(recoveredParameter != null && recoveredParameter.isTdePageAead());


+  private void validateChunkEncryptionContext(Chunk chunk) throws IOException {
+    EncryptParameter sourceParameter = chunk.getEncryptParam();
+    boolean targetUsesPageAead = encryptParameter != null && encryptParameter.isTdePageAead();
+    boolean sourceUsesPageAead = sourceParameter != null && sourceParameter.isTdePageAead();
+    if (!targetUsesPageAead && !sourceUsesPageAead) {
+      return;
+    }
+    throw new IOException(Messages.get("error.write.encrypted_chunk_copy_unsupported"));
+  }


Add pluggable page-level AEAD encryption support

76754e0

JackieTien97 and others added 5 commits August 6, 2026 16:37

Bump version to 2.4.1-260806-SNAPSHOT

7d25839

Merge remote-tracking branch 'origin/snapshot/2.4.1-260806' into code…

dd20f4d

…x/tsfile-tde-page-aead # Conflicts: # java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java # java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java

Merge develop into TsFile TDE branch

f7b5789

Merge remote-tracking branch 'origin/develop' into codex/tsfile-tde-p…

b940382

…age-aead

Bind AEAD pages to file-unique chunk ordinals

dbd0609

HTHou marked this pull request as ready for review October 9, 2026 09:35

HTHou requested a balanced review from Copilot October 9, 2026 09:35

Copilot AI reviewed Oct 9, 2026

View reviewed changes

Copilot stopped reviewing on behalf of HTHou due to an error October 9, 2026 10:22
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

Address page AEAD review feedback

f2f58a7

HTHou requested a balanced review from Copilot October 10, 2026 02:39

Copilot started reviewing on behalf of HTHou October 10, 2026 02:40 View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Fail closed for page AEAD and validate TsFile encryption context

0eb25b0

HTHou requested a balanced review from Copilot October 10, 2026 03:07

Copilot started reviewing on behalf of HTHou October 10, 2026 03:08 View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Fix page AEAD append and reader lifecycle edge cases

517b06b

Navigation Menu

Repository navigation

Add pluggable page-level AEAD encryption support - #866

Add pluggable page-level AEAD encryption support#866
HTHou wants to merge 9 commits into
developfrom
codex/tsfile-tde-page-aead

HTHou commented Jul 15, 2026

Uh oh!

codecov-commenter commented Jul 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

HTHou commented Jul 15, 2026

Summary

Motivation

Compatibility

Validation

Draft discussion points

Uh oh!

codecov-commenter commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Jul 15, 2026 •

edited

Loading