diff --git a/CHANGELOG.md b/CHANGELOG.md index 1006a93844c..a6b3c572e6b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,6 +88,16 @@ changes. observed a block (so a stalled chain backend can be detected even while drift is frozen). [#2749](https://github.com/cardano-scaling/hydra/issues/2749) +- **BREAKING**: Event payloads in the `hydra.db` persistence database are now + CBOR-encoded instead of JSON, reducing disk usage and speeding up event + append and replay. Existing databases are migrated automatically on first + start (one-shot re-encode of all rows followed by `VACUUM`); a corrupt row + aborts startup and leaves the database untouched. After migration, older + hydra-node versions refuse to open the database — there is no downgrade + path. + +## [2.3.0] - 2026.07.15 + - Add **selective partial fanout**: distribute a chosen subset of a closed head's UTxO instead of draining it all at once. Introduces the `PartialFanout` client input, the `HeadPartiallyFannedOut` server output (with a `fanoutMode` diff --git a/docs/docs/dev/architecture/event-sourcing.md b/docs/docs/dev/architecture/event-sourcing.md index 1e9aab59d08..9e72d5b4bdf 100644 --- a/docs/docs/dev/architecture/event-sourcing.md +++ b/docs/docs/dev/architecture/event-sourcing.md @@ -21,7 +21,18 @@ The database contains a single `events` table: | Column | Type | Description | |-------------|---------|------------------------------------------| | `event_id` | INTEGER | Primary key, matches the in-memory event id | -| `event_data` | BLOB | JSON-encoded event payload | +| `event_data` | BLOB | CBOR-encoded event payload (`ToCBOR`/`FromCBOR`) | + +Event payloads use a constructor-name-tagged CBOR format: the constructor name +as a CBOR text string, followed by the constructor fields in declaration order. +Most instances are derived generically via `genericToCBOR` / `genericFromCBOR` +from `Hydra.CBOR.Generic` (hydra-prelude), which makes the data type +declaration itself the on-disk format: changing the order or types of an +existing constructor's fields is a breaking change that requires a schema +migration, while adding, removing or reordering constructors keeps existing +data decodable thanks to the name tags. The golden tests in `Hydra.CBORSpec` +lock the concrete bytes per constructor and fail on any accidental format +change. The following connection pragmas are set on every open: @@ -39,16 +50,18 @@ The last-seen event id is updated atomically at enqueue time (not write time), s #### Schema versioning -The database schema is versioned using SQLite's built-in `PRAGMA user_version`. On startup, `initSchema` reads the current version and applies any pending migration steps incrementally up to `nextVersion`. Each step is defined as a case in `migrateStep`: +The database schema is versioned using SQLite's built-in `PRAGMA user_version`. On startup, `initSchema` reads the current version and applies any pending migration steps incrementally up to `nextVersion`. Each step runs together with its version bump in a single transaction, so a crash or failure mid-migration rolls back to a well-defined version. The steps are defined as cases in `migrateStep`: ```haskell -migrateStep conn = \case +migrateStep conn reencodeRow = \case 0 -> -- create the events table - 1 -> -- (future) e.g. add an index or new column + 1 -> -- re-encode all JSON event payloads to CBOR ``` A fresh database starts at version 0 (SQLite default). After all migrations run, `user_version` is set to `nextVersion`. If the database has a version higher than `nextVersion` (e.g. from a newer release), the node refuses to start to prevent silent data corruption on downgrade. +Version 1 databases (written by earlier hydra-node releases) store event payloads as JSON. Opening one automatically re-encodes every row to CBOR and runs `VACUUM` afterwards to reclaim the freed space. A row that fails to decode aborts the migration — and thereby node startup — leaving the database untouched at version 1, still usable by the previous hydra-node version. Note that after a successful migration there is no downgrade path: older releases refuse to open a version 2 database. The legacy `state` file migration also inserts CBOR (the file itself remains JSON lines). + To add a new migration: 1. Bump `nextVersion` in `Hydra.Events.SQLiteBased` diff --git a/head-state-viewer/head-state-viewer.cabal b/head-state-viewer/head-state-viewer.cabal index 50932ce589d..4cfb62cec94 100644 --- a/head-state-viewer/head-state-viewer.cabal +++ b/head-state-viewer/head-state-viewer.cabal @@ -57,6 +57,7 @@ library , aeson , base , bytestring + , cardano-binary , containers , hydra-node , hydra-prelude diff --git a/head-state-viewer/src/HydraVis/History.hs b/head-state-viewer/src/HydraVis/History.hs index 07bebc4ae5e..51e0a1813a4 100644 --- a/head-state-viewer/src/HydraVis/History.hs +++ b/head-state-viewer/src/HydraVis/History.hs @@ -6,8 +6,10 @@ -- The Hydra node persists events using the schema defined by -- "Hydra.Events.SQLiteBased": a single @events@ table with -- @event_id INTEGER PRIMARY KEY@ and a @event_data BLOB@ column that holds a --- JSON-encoded @StateEvent tx@. We open the database read-only (so a running --- node can keep writing) and replay the events in event-id order. +-- CBOR-encoded @StateEvent tx@ (JSON in version 1 databases; we decode CBOR +-- first and fall back to JSON so pre-migration databases and archives still +-- open). We open the database read-only (so a running node can keep writing) +-- and replay the events in event-id order. module HydraVis.History ( HistoryStep (..), loadHistoryFor, @@ -18,6 +20,7 @@ module HydraVis.History ( import Hydra.Prelude +import Cardano.Binary (decodeFull') import Data.Aeson qualified as Aeson import Data.Text.Encoding qualified as TE import Database.SQLite.Simple (Only (..), SQLData (..), open, query, query_) @@ -50,7 +53,7 @@ deriving stock instance (IsChainState tx, Show (NodeState tx)) => Show (HistoryS -- recover. If the file does not exist the underlying 'open' call throws. loadHistoryFor :: forall tx. - (FromJSON (StateEvent tx), IsChainState tx) => + (FromCBOR (StateEvent tx), FromJSON (StateEvent tx), IsChainState tx) => NodeState tx -> FilePath -> IO [HistoryStep tx] @@ -62,13 +65,29 @@ loadHistoryFor initial path = do "SELECT event_data FROM events ORDER BY event_id ASC" :: IO [Only EventBlob] SQL.close conn - events <- forM rows $ \(Only (EventBlob blob)) -> - case Aeson.eitherDecodeStrict' blob of + events <- forM rows $ \(Only (EventBlob blob)) -> decodeEventBlob path blob + pure (buildHistory initial events) + +-- | Decode an event payload: CBOR (the current production format) first, +-- falling back to JSON for version 1 databases and pre-rotation archives. +decodeEventBlob :: + (FromCBOR (StateEvent tx), FromJSON (StateEvent tx)) => + FilePath -> + ByteString -> + IO (StateEvent tx) +decodeEventBlob path blob = + case decodeFull' blob of + Right e -> pure e + Left cborErr -> case Aeson.eitherDecodeStrict' blob of Right e -> pure e - Left err -> + Left jsonErr -> fail $ - "HydraVis.History: failed to decode event in " <> path <> ": " <> err - pure (buildHistory initial events) + "HydraVis.History: failed to decode event in " + <> path + <> ": CBOR: " + <> show cborErr + <> "; JSON: " + <> jsonErr -- | Wrapper that accepts both BLOB (the production format) and TEXT (what -- a quick @sqlite3 CLI@ insert produces) columns when reading event payloads. @@ -98,7 +117,7 @@ buildHistory initial events = -- Used by the follow loop to fetch only newly persisted rows on each poll. loadEventsAfter :: forall tx. - FromJSON (StateEvent tx) => + (FromCBOR (StateEvent tx), FromJSON (StateEvent tx)) => FilePath -> Maybe EventId -> IO [StateEvent tx] @@ -117,12 +136,7 @@ loadEventsAfter path lastSeen = do (Only eid) :: IO [Only EventBlob] SQL.close conn - forM rows $ \(Only (EventBlob blob)) -> - case Aeson.eitherDecodeStrict' blob of - Right e -> pure e - Left err -> - fail $ - "HydraVis.History: failed to decode event in " <> path <> ": " <> err + forM rows $ \(Only (EventBlob blob)) -> decodeEventBlob path blob -- | Extend an existing history with newly observed events, folding them -- through 'aggregateNodeState' starting from the final state of the existing diff --git a/head-state-viewer/src/HydraVis/SampleDb.hs b/head-state-viewer/src/HydraVis/SampleDb.hs index 77daed4cee3..fdbbe809486 100644 --- a/head-state-viewer/src/HydraVis/SampleDb.hs +++ b/head-state-viewer/src/HydraVis/SampleDb.hs @@ -3,14 +3,14 @@ -- -- Real Hydra nodes write to @/hydra.db@ via -- "Hydra.Events.SQLiteBased"; this module writes a file with the same shape --- (the @events (event_id, event_data BLOB)@ schema, JSON-encoded --- @StateEvent tx@ per row) using purely synthesised events so the --- visualizer can be exercised without a running node. +-- (the @events (event_id, event_data BLOB)@ schema, CBOR-encoded +-- @StateEvent tx@ per row, schema version 2) using purely synthesised events +-- so the visualizer can be exercised without a running node. module HydraVis.SampleDb (writeSampleDb, sampleStateEvents) where import Hydra.Prelude -import Data.Aeson qualified as Aeson +import Cardano.Binary (serialize') import Database.SQLite.Simple (close, execute, execute_, open) import Hydra.Events (EventId) import Hydra.HeadLogic (aggregateState, update) @@ -62,7 +62,8 @@ sampleStateEvents = in (n, scs) : collect (n + 1) s' rest -- | Create (or recreate) a SQLite database at @path@ holding --- 'sampleStateEvents'. Overwrites any existing rows. +-- 'sampleStateEvents'. Overwrites any existing rows. Rows are CBOR-encoded +-- and the schema version set to 2, matching what a current hydra-node writes. writeSampleDb :: FilePath -> IO () writeSampleDb path = do conn <- open path @@ -70,10 +71,11 @@ writeSampleDb path = do conn "CREATE TABLE IF NOT EXISTS events \ \(event_id INTEGER NOT NULL PRIMARY KEY, event_data BLOB NOT NULL)" + execute_ conn "PRAGMA user_version = 2" execute_ conn "DELETE FROM events" forM_ sampleStateEvents $ \e -> execute conn "INSERT INTO events (event_id, event_data) VALUES (?, ?)" - (eventId e, toStrict (Aeson.encode e)) + (eventId e, serialize' e) close conn diff --git a/head-state-viewer/src/HydraVis/UI/Update.hs b/head-state-viewer/src/HydraVis/UI/Update.hs index b2a4af6a09a..ad6433d415a 100644 --- a/head-state-viewer/src/HydraVis/UI/Update.hs +++ b/head-state-viewer/src/HydraVis/UI/Update.hs @@ -242,7 +242,7 @@ applyAction a m = case a of -- | Background poll loop. Reads any new rows from the SQLite file at a -- fixed cadence and dispatches them as 'AppendEvents'. followSub :: - FromJSON (StateEvent tx) => + (FromCBOR (StateEvent tx), FromJSON (StateEvent tx)) => FilePath -> Maybe EventId -> Sub (Action tx) diff --git a/hydra-cardano-api/src/Hydra/Cardano/Api/AddressInEra.hs b/hydra-cardano-api/src/Hydra/Cardano/Api/AddressInEra.hs index b73ce27756f..1b853e76783 100644 --- a/hydra-cardano-api/src/Hydra/Cardano/Api/AddressInEra.hs +++ b/hydra-cardano-api/src/Hydra/Cardano/Api/AddressInEra.hs @@ -1,7 +1,10 @@ +{-# OPTIONS_GHC -Wno-orphans #-} + module Hydra.Cardano.Api.AddressInEra where import Hydra.Cardano.Api.Prelude +import Cardano.Api qualified as Api import Cardano.Ledger.Address qualified as Ledger import Cardano.Ledger.BaseTypes qualified as Ledger import Cardano.Ledger.Credential qualified as Ledger @@ -14,6 +17,32 @@ import PlutusLedgerApi.V3 ( ) import PlutusLedgerApi.V3 qualified as Plutus +-- * Orphans + +-- missing CBOR instances + +-- NOTE: Encoded as the bech32/base58 address text, consistent with the JSON +-- representation. +instance (IsShelleyBasedEra era, Typeable era) => ToCBOR (AddressInEra era) where + toCBOR = toCBOR . serialiseAddress + +instance (IsShelleyBasedEra era, Typeable era) => FromCBOR (AddressInEra era) where + fromCBOR = do + t <- fromCBOR + case deserialiseAddress (proxyToAsType $ Proxy @(AddressInEra era)) t of + Nothing -> fail $ "failed to deserialise AddressInEra from " <> show (t :: Text) + Just addr -> pure addr + +instance ToCBOR (Api.Address ByronAddr) where + toCBOR = toCBOR . serialiseToRawBytes + +instance FromCBOR (Api.Address ByronAddr) where + fromCBOR = do + bs <- fromCBOR + case deserialiseFromRawBytes (proxyToAsType $ Proxy @(Api.Address ByronAddr)) bs of + Left err -> fail (show err) + Right v -> pure v + -- * Extras -- | Construct a Shelley-style address from a verification key. This address has diff --git a/hydra-cardano-api/src/Hydra/Cardano/Api/ChainPoint.hs b/hydra-cardano-api/src/Hydra/Cardano/Api/ChainPoint.hs index b35cb736ad7..79bf74d6853 100644 --- a/hydra-cardano-api/src/Hydra/Cardano/Api/ChainPoint.hs +++ b/hydra-cardano-api/src/Hydra/Cardano/Api/ChainPoint.hs @@ -10,3 +10,26 @@ getChainPoint header = ChainPoint slotNo headerHash where (BlockHeader slotNo headerHash _) = header + +-- * Orphans + +-- missing CBOR instances + +instance ToCBOR ChainPoint where + toCBOR = \case + ChainPointAtGenesis -> + toCBOR ("ChainPointAtGenesis" :: Text) + ChainPoint slotNo headerHash -> + toCBOR ("ChainPoint" :: Text) <> toCBOR slotNo <> toCBOR (serialiseToRawBytes headerHash) + +instance FromCBOR ChainPoint where + fromCBOR = + fromCBOR >>= \case + ("ChainPointAtGenesis" :: Text) -> pure ChainPointAtGenesis + "ChainPoint" -> do + slotNo <- fromCBOR + bytes <- fromCBOR + case deserialiseFromRawBytes (proxyToAsType $ Proxy @(Hash BlockHeader)) bytes of + Left err -> fail (show err) + Right headerHash -> pure $ ChainPoint slotNo headerHash + tag -> fail $ show (tag :: Text) <> " is not a proper CBOR-encoded ChainPoint" diff --git a/hydra-cardano-api/src/Hydra/Cardano/Api/NetworkId.hs b/hydra-cardano-api/src/Hydra/Cardano/Api/NetworkId.hs index b9ebafd3d87..9596cf23a27 100644 --- a/hydra-cardano-api/src/Hydra/Cardano/Api/NetworkId.hs +++ b/hydra-cardano-api/src/Hydra/Cardano/Api/NetworkId.hs @@ -25,3 +25,17 @@ instance FromJSON NetworkId where "Mainnet" -> pure Mainnet "Testnet" -> Testnet <$> o .: "magic" _ -> fail "Expected tag to be Mainnet | Testnet" + +-- missing CBOR instances + +instance ToCBOR NetworkId where + toCBOR = \case + Mainnet -> toCBOR ("Mainnet" :: Text) + Testnet (NetworkMagic magic) -> toCBOR ("Testnet" :: Text) <> toCBOR magic + +instance FromCBOR NetworkId where + fromCBOR = + fromCBOR >>= \case + ("Mainnet" :: Text) -> pure Mainnet + "Testnet" -> Testnet . NetworkMagic <$> fromCBOR + tag -> fail $ show tag <> " is not a proper CBOR-encoded NetworkId" diff --git a/hydra-cardano-api/src/Hydra/Cardano/Api/PolicyAssets.hs b/hydra-cardano-api/src/Hydra/Cardano/Api/PolicyAssets.hs index 2fae022e449..69d406c21db 100644 --- a/hydra-cardano-api/src/Hydra/Cardano/Api/PolicyAssets.hs +++ b/hydra-cardano-api/src/Hydra/Cardano/Api/PolicyAssets.hs @@ -11,3 +11,27 @@ instance ToJSON PolicyAssets where instance FromJSON PolicyAssets where parseJSON v = PolicyAssets <$> parseJSON v + +-- missing CBOR instances + +instance ToCBOR AssetName where + toCBOR = toCBOR . serialiseToRawBytes + +instance FromCBOR AssetName where + fromCBOR = do + bs <- fromCBOR + case deserialiseFromRawBytes AsAssetName bs of + Left err -> fail (show err) + Right v -> pure v + +instance ToCBOR Quantity where + toCBOR (Quantity q) = toCBOR q + +instance FromCBOR Quantity where + fromCBOR = Quantity <$> fromCBOR + +instance ToCBOR PolicyAssets where + toCBOR (PolicyAssets assets) = toCBOR assets + +instance FromCBOR PolicyAssets where + fromCBOR = PolicyAssets <$> fromCBOR diff --git a/hydra-cardano-api/src/Hydra/Cardano/Api/PolicyId.hs b/hydra-cardano-api/src/Hydra/Cardano/Api/PolicyId.hs index 1a56d40df3d..6501cf33b49 100644 --- a/hydra-cardano-api/src/Hydra/Cardano/Api/PolicyId.hs +++ b/hydra-cardano-api/src/Hydra/Cardano/Api/PolicyId.hs @@ -9,6 +9,20 @@ import Cardano.Ledger.Mary.Value qualified as Ledger import Hydra.Cardano.Api.ScriptHash () import PlutusLedgerApi.V3 (CurrencySymbol, fromBuiltin, unCurrencySymbol) +-- * Orphans + +-- missing CBOR instances + +instance ToCBOR PolicyId where + toCBOR = toCBOR . serialiseToRawBytes + +instance FromCBOR PolicyId where + fromCBOR = do + bs <- fromCBOR + case deserialiseFromRawBytes AsPolicyId bs of + Left err -> fail (show err) + Right v -> pure v + -- * Type conversions -- | Convert Cardano api 'PolicyId' to Cardano ledger `PolicyID`. diff --git a/hydra-node/golden/ChainEvent.cbor b/hydra-node/golden/ChainEvent.cbor new file mode 100644 index 00000000000..a3050a3b69c Binary files /dev/null and b/hydra-node/golden/ChainEvent.cbor differ diff --git a/hydra-node/golden/ClientInput.cbor b/hydra-node/golden/ClientInput.cbor new file mode 100644 index 00000000000..960b41767f6 Binary files /dev/null and b/hydra-node/golden/ClientInput.cbor differ diff --git a/hydra-node/golden/ClientMessage.cbor b/hydra-node/golden/ClientMessage.cbor new file mode 100644 index 00000000000..1ceb12b48e2 Binary files /dev/null and b/hydra-node/golden/ClientMessage.cbor differ diff --git a/hydra-node/golden/ConfirmedSnapshot.cbor b/hydra-node/golden/ConfirmedSnapshot.cbor new file mode 100644 index 00000000000..8dfc55ee4a5 Binary files /dev/null and b/hydra-node/golden/ConfirmedSnapshot.cbor differ diff --git a/hydra-node/golden/Connectivity.cbor b/hydra-node/golden/Connectivity.cbor new file mode 100644 index 00000000000..fbc2995fde5 Binary files /dev/null and b/hydra-node/golden/Connectivity.cbor differ diff --git a/hydra-node/golden/DecommitInvalidReason.cbor b/hydra-node/golden/DecommitInvalidReason.cbor new file mode 100644 index 00000000000..048a8994d10 Binary files /dev/null and b/hydra-node/golden/DecommitInvalidReason.cbor differ diff --git a/hydra-node/golden/Deposit.cbor b/hydra-node/golden/Deposit.cbor new file mode 100644 index 00000000000..ce24bf85011 Binary files /dev/null and b/hydra-node/golden/Deposit.cbor differ diff --git a/hydra-node/golden/DepositStatus.cbor b/hydra-node/golden/DepositStatus.cbor new file mode 100644 index 00000000000..b064bce4df0 --- /dev/null +++ b/hydra-node/golden/DepositStatus.cbor @@ -0,0 +1 @@ +ŸhInactivefActivegExpiredÿ \ No newline at end of file diff --git a/hydra-node/golden/FanoutMode.cbor b/hydra-node/golden/FanoutMode.cbor new file mode 100644 index 00000000000..da3787a34c6 Binary files /dev/null and b/hydra-node/golden/FanoutMode.cbor differ diff --git a/hydra-node/golden/FanoutProgressMode.cbor b/hydra-node/golden/FanoutProgressMode.cbor new file mode 100644 index 00000000000..436b8036e44 --- /dev/null +++ b/hydra-node/golden/FanoutProgressMode.cbor @@ -0,0 +1 @@ +ŸnAutoFanningOutwAwaitingFanoutSelectionÿ \ No newline at end of file diff --git a/hydra-node/golden/Greetings.cbor b/hydra-node/golden/Greetings.cbor new file mode 100644 index 00000000000..ad3c86ba85c Binary files /dev/null and b/hydra-node/golden/Greetings.cbor differ diff --git a/hydra-node/golden/HeadState.cbor b/hydra-node/golden/HeadState.cbor new file mode 100644 index 00000000000..304c859e8d7 Binary files /dev/null and b/hydra-node/golden/HeadState.cbor differ diff --git a/hydra-node/golden/HeadStatus.cbor b/hydra-node/golden/HeadStatus.cbor new file mode 100644 index 00000000000..d85e9a5b3f9 --- /dev/null +++ b/hydra-node/golden/HeadStatus.cbor @@ -0,0 +1 @@ +ŸdIdledOpenfClosednFanoutPossiblejFanningOutÿ \ No newline at end of file diff --git a/hydra-node/golden/InvalidInput.cbor b/hydra-node/golden/InvalidInput.cbor new file mode 100644 index 00000000000..e6d5f8e17f2 --- /dev/null +++ b/hydra-node/golden/InvalidInput.cbor @@ -0,0 +1 @@ +ŸlInvalidInputdKnWokQô€¸Œð¤£Ÿ=vÿ \ No newline at end of file diff --git a/hydra-node/golden/Message.cbor b/hydra-node/golden/Message.cbor new file mode 100644 index 00000000000..00ee606b45a Binary files /dev/null and b/hydra-node/golden/Message.cbor differ diff --git a/hydra-node/golden/NetworkInfo.cbor b/hydra-node/golden/NetworkInfo.cbor new file mode 100644 index 00000000000..ee533d06421 Binary files /dev/null and b/hydra-node/golden/NetworkInfo.cbor differ diff --git a/hydra-node/golden/NodeState.cbor b/hydra-node/golden/NodeState.cbor new file mode 100644 index 00000000000..d74f3dbcdc2 Binary files /dev/null and b/hydra-node/golden/NodeState.cbor differ diff --git a/hydra-node/golden/OnChainTx.cbor b/hydra-node/golden/OnChainTx.cbor new file mode 100644 index 00000000000..09e078c9211 Binary files /dev/null and b/hydra-node/golden/OnChainTx.cbor differ diff --git a/hydra-node/golden/PostChainTx.cbor b/hydra-node/golden/PostChainTx.cbor new file mode 100644 index 00000000000..9cbbc2c3cd8 Binary files /dev/null and b/hydra-node/golden/PostChainTx.cbor differ diff --git a/hydra-node/golden/PostTxError.cbor b/hydra-node/golden/PostTxError.cbor new file mode 100644 index 00000000000..57611f614ff Binary files /dev/null and b/hydra-node/golden/PostTxError.cbor differ diff --git a/hydra-node/golden/RequirementFailure.cbor b/hydra-node/golden/RequirementFailure.cbor new file mode 100644 index 00000000000..5f7ba2c2ec9 Binary files /dev/null and b/hydra-node/golden/RequirementFailure.cbor differ diff --git a/hydra-node/golden/SeenSnapshot.cbor b/hydra-node/golden/SeenSnapshot.cbor new file mode 100644 index 00000000000..229af10458e Binary files /dev/null and b/hydra-node/golden/SeenSnapshot.cbor differ diff --git a/hydra-node/golden/ServerOutput.cbor b/hydra-node/golden/ServerOutput.cbor new file mode 100644 index 00000000000..4147b90af89 Binary files /dev/null and b/hydra-node/golden/ServerOutput.cbor differ diff --git a/hydra-node/golden/SideLoadRequirementFailure.cbor b/hydra-node/golden/SideLoadRequirementFailure.cbor new file mode 100644 index 00000000000..9e008ae6945 Binary files /dev/null and b/hydra-node/golden/SideLoadRequirementFailure.cbor differ diff --git a/hydra-node/golden/StateEvent.cbor b/hydra-node/golden/StateEvent.cbor new file mode 100644 index 00000000000..0f75f8d6a0a Binary files /dev/null and b/hydra-node/golden/StateEvent.cbor differ diff --git a/hydra-node/golden/SyncedStatus.cbor b/hydra-node/golden/SyncedStatus.cbor new file mode 100644 index 00000000000..06503aa89d1 --- /dev/null +++ b/hydra-node/golden/SyncedStatus.cbor @@ -0,0 +1 @@ +ŸfInSyncjCatchingUpÿ \ No newline at end of file diff --git a/hydra-node/golden/TimedServerOutput.cbor b/hydra-node/golden/TimedServerOutput.cbor new file mode 100644 index 00000000000..f73529d45f0 Binary files /dev/null and b/hydra-node/golden/TimedServerOutput.cbor differ diff --git a/hydra-node/golden/WhichEtcd.cbor b/hydra-node/golden/WhichEtcd.cbor new file mode 100644 index 00000000000..bbc4b20f6e4 --- /dev/null +++ b/hydra-node/golden/WhichEtcd.cbor @@ -0,0 +1 @@ +ŸlEmbeddedEtcdjSystemEtcdÿ \ No newline at end of file diff --git a/hydra-node/hydra-node.cabal b/hydra-node/hydra-node.cabal index 7aed86aeecf..7f66fb4e4dc 100644 --- a/hydra-node/hydra-node.cabal +++ b/hydra-node/hydra-node.cabal @@ -56,6 +56,7 @@ library Hydra.API.ServerOutput Hydra.API.ServerOutputFilter Hydra.API.WSServer + Hydra.CBOR.Orphans Hydra.Chain Hydra.Chain.Backend Hydra.Chain.Blockfrost @@ -127,6 +128,7 @@ library , cardano-slotting , cardano-strict-containers , cborg + , cborg-json , conduit , containers , contra-tracer @@ -190,6 +192,7 @@ library testlib Test.Hydra.API.ClientInput Test.Hydra.API.HTTPServer Test.Hydra.API.ServerOutput + Test.Hydra.CBOR Test.Hydra.Chain Test.Hydra.Chain.Direct.State Test.Hydra.Chain.Direct.TimeHandle @@ -217,12 +220,14 @@ library testlib , base , bytestring , cardano-api + , cardano-binary , cardano-crypto-class , cardano-ledger-babbage:testlib , cardano-ledger-conway:testlib , cardano-ledger-core , cardano-ledger-core:testlib , cardano-slotting + , cborg , containers , directory , filepath @@ -380,6 +385,7 @@ test-suite tests Hydra.API.ServerOutputSpec Hydra.API.ServerSpec Hydra.BehaviorSpec + Hydra.CBORSpec Hydra.Chain.BlockfrostSpec Hydra.Chain.Direct.HandlersSpec Hydra.Chain.Direct.ScriptRegistrySpec @@ -480,6 +486,7 @@ test-suite tests , plutus-ledger-api >=1.1.1.0 , plutus-tx , QuickCheck + , quickcheck-arbitrary-adt , quickcheck-dynamic >=3.4 && <3.5 , quickcheck-instances , regex-tdfa diff --git a/hydra-node/src/Hydra/API/ClientInput.hs b/hydra-node/src/Hydra/API/ClientInput.hs index e8e8886895e..843428ef8f3 100644 --- a/hydra-node/src/Hydra/API/ClientInput.hs +++ b/hydra-node/src/Hydra/API/ClientInput.hs @@ -27,3 +27,9 @@ deriving stock instance IsTx tx => Eq (ClientInput tx) deriving stock instance IsTx tx => Show (ClientInput tx) deriving anyclass instance IsTx tx => ToJSON (ClientInput tx) deriving anyclass instance IsTx tx => FromJSON (ClientInput tx) + +instance IsTx tx => ToCBOR (ClientInput tx) where + toCBOR = genericToCBOR + +instance IsTx tx => FromCBOR (ClientInput tx) where + fromCBOR = genericFromCBOR diff --git a/hydra-node/src/Hydra/API/ServerOutput.hs b/hydra-node/src/Hydra/API/ServerOutput.hs index 1b3fcd16dc8..6aa424a1b01 100644 --- a/hydra-node/src/Hydra/API/ServerOutput.hs +++ b/hydra-node/src/Hydra/API/ServerOutput.hs @@ -4,6 +4,7 @@ module Hydra.API.ServerOutput where +import Cardano.Binary (Decoder) import Control.Lens ((.~)) import Data.Aeson (Value (..), defaultOptions, encode, genericParseJSON, genericToJSON, omitNothingFields, tagSingleConstructors, withObject, (.:)) import Data.Aeson.KeyMap qualified as KeyMap @@ -13,7 +14,7 @@ import Hydra.API.ClientInput (ClientInput) import Hydra.Chain (PostChainTx, PostTxError) import Hydra.Chain.ChainState (ChainSlot, IsChainState) import Hydra.HeadLogic.Error (SideLoadRequirementFailure) -import Hydra.HeadLogic.State (ClosedState (..), FanoutMode (..), HeadState (..), OpenState (..), PartialFanoutState (..), SeenSnapshot (..)) +import Hydra.HeadLogic.State (ClosedState (..), FanoutMode (..), HeadState, OpenState (..), PartialFanoutState (..), SeenSnapshot (..)) import Hydra.HeadLogic.State qualified as HeadState import Hydra.Ledger (ValidationError) import Hydra.Network (Host, ProtocolVersion) @@ -48,6 +49,29 @@ instance IsChainState tx => FromJSON (TimedServerOutput tx) where parseJSON v = flip (withObject "TimedServerOutput") v $ \o -> TimedServerOutput <$> parseJSON v <*> o .: "seq" <*> o .: "timestamp" +-- NOTE: Unlike the JSON instance, which merges 'seq' and 'timestamp' into the +-- inner 'ServerOutput' object, the CBOR encoding is a plain tagged envelope. +-- The tag makes any server-sent message start with a unique text token, so +-- decoders can dispatch on it. +instance IsChainState tx => ToCBOR (TimedServerOutput tx) where + toCBOR TimedServerOutput{output, seq, time} = + toCBOR ("TimedServerOutput" :: Text) <> toCBOR seq <> toCBOR time <> toCBOR output + +instance IsChainState tx => FromCBOR (TimedServerOutput tx) where + fromCBOR = + fromCBOR >>= \case + ("TimedServerOutput" :: Text) -> decodeTimedServerOutputBody + tag -> fail $ show tag <> " is not a proper CBOR-encoded TimedServerOutput" + +-- | Decode a 'TimedServerOutput' after its @TimedServerOutput@ tag has already +-- been consumed (used for tag-based dispatch). +decodeTimedServerOutputBody :: IsChainState tx => Decoder s (TimedServerOutput tx) +decodeTimedServerOutputBody = do + seq <- fromCBOR + time <- fromCBOR + output <- fromCBOR + pure TimedServerOutput{output, seq, time} + data DecommitInvalidReason tx = DecommitTxInvalid {localUTxO :: UTxOType tx, validationError :: ValidationError} | DecommitAlreadyInFlight {otherDecommitTxId :: TxIdType tx} @@ -62,6 +86,12 @@ instance (ToJSON (TxIdType tx), ToJSON (UTxOType tx)) => ToJSON (DecommitInvalid instance (FromJSON (TxIdType tx), FromJSON (UTxOType tx)) => FromJSON (DecommitInvalidReason tx) where parseJSON = genericParseJSON defaultOptions +instance IsTx tx => ToCBOR (DecommitInvalidReason tx) where + toCBOR = genericToCBOR + +instance IsTx tx => FromCBOR (DecommitInvalidReason tx) where + fromCBOR = genericFromCBOR + -- | Individual messages as produced by the 'Hydra.HeadLogic' in -- the 'ClientEffect'. data ClientMessage tx @@ -85,6 +115,12 @@ instance IsChainState tx => FromJSON (ClientMessage tx) where { omitNothingFields = True } +instance IsChainState tx => ToCBOR (ClientMessage tx) where + toCBOR = genericToCBOR + +instance IsChainState tx => FromCBOR (ClientMessage tx) where + fromCBOR = genericFromCBOR + -- | A friendly welcome message which tells a client something about the -- node. Currently used for knowing what signing key the server uses (it -- only knows one), 'HeadStatus' and optionally (if 'HeadIsOpen' or @@ -121,6 +157,40 @@ instance IsChainState tx => FromJSON (Greetings tx) where , tagSingleConstructors = True } +instance IsChainState tx => ToCBOR (Greetings tx) where + toCBOR Greetings{me, headStatus, hydraHeadId, snapshotUtxo, hydraNodeVersion, env, networkInfo, chainSyncedStatus, currentSlot} = + toCBOR ("Greetings" :: Text) + <> toCBOR me + <> toCBOR headStatus + <> toCBOR hydraHeadId + <> toCBOR snapshotUtxo + <> toCBOR (toText hydraNodeVersion) + <> toCBOR env + <> toCBOR networkInfo + <> toCBOR chainSyncedStatus + <> toCBOR currentSlot + +instance IsChainState tx => FromCBOR (Greetings tx) where + fromCBOR = + fromCBOR >>= \case + ("Greetings" :: Text) -> decodeGreetingsBody + tag -> fail $ show tag <> " is not a proper CBOR-encoded Greetings" + +-- | Decode a 'Greetings' after its @Greetings@ tag has already been consumed +-- (used for tag-based dispatch). +decodeGreetingsBody :: IsChainState tx => Decoder s (Greetings tx) +decodeGreetingsBody = do + me <- fromCBOR + headStatus <- fromCBOR + hydraHeadId <- fromCBOR + snapshotUtxo <- fromCBOR + hydraNodeVersion <- toString <$> fromCBOR @Text + env <- fromCBOR + networkInfo <- fromCBOR + chainSyncedStatus <- fromCBOR + currentSlot <- fromCBOR + pure Greetings{me, headStatus, hydraHeadId, snapshotUtxo, hydraNodeVersion, env, networkInfo, chainSyncedStatus, currentSlot} + data InvalidInput = InvalidInput { reason :: String , input :: Text @@ -130,6 +200,24 @@ data InvalidInput = InvalidInput deriving anyclass instance ToJSON InvalidInput deriving anyclass instance FromJSON InvalidInput +instance ToCBOR InvalidInput where + toCBOR InvalidInput{reason, input} = + toCBOR ("InvalidInput" :: Text) <> toCBOR (toText reason) <> toCBOR input + +instance FromCBOR InvalidInput where + fromCBOR = + fromCBOR >>= \case + ("InvalidInput" :: Text) -> decodeInvalidInputBody + tag -> fail $ show tag <> " is not a proper CBOR-encoded InvalidInput" + +-- | Decode an 'InvalidInput' after its @InvalidInput@ tag has already been +-- consumed (used for tag-based dispatch). +decodeInvalidInputBody :: Decoder s InvalidInput +decodeInvalidInputBody = do + reason <- toString <$> fromCBOR @Text + input <- fromCBOR + pure InvalidInput{reason, input} + data ServerOutput tx = NetworkConnected | NetworkDisconnected @@ -219,6 +307,12 @@ deriving stock instance IsChainState tx => Show (ServerOutput tx) deriving anyclass instance IsChainState tx => FromJSON (ServerOutput tx) deriving anyclass instance IsChainState tx => ToJSON (ServerOutput tx) +instance IsChainState tx => ToCBOR (ServerOutput tx) where + toCBOR = genericToCBOR + +instance IsChainState tx => FromCBOR (ServerOutput tx) where + fromCBOR = genericFromCBOR + -- | Whether or not to include full UTxO in server outputs. data WithUTxO = WithUTxO | WithoutUTxO deriving stock (Eq, Show) @@ -303,6 +397,12 @@ data HeadStatus deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) +instance ToCBOR HeadStatus where + toCBOR = genericToCBOR + +instance FromCBOR HeadStatus where + fromCBOR = genericFromCBOR + -- | Client-facing projection of the node's fanout 'FanoutMode': whether a -- fanning-out head will continue draining on its own or is waiting for the -- client to choose the next 'PartialFanout'. Surfacing this lets clients render @@ -326,6 +426,12 @@ fanoutProgressMode = \case DistributingSelection{} -> AutoFanningOut AwaitingSelection -> AwaitingFanoutSelection +instance ToCBOR FanoutProgressMode where + toCBOR = genericToCBOR + +instance FromCBOR FanoutProgressMode where + fromCBOR = genericFromCBOR + -- | All information needed to distinguish behavior of the commit endpoint. data CommitInfo = CannotCommit @@ -339,6 +445,12 @@ data NetworkInfo = NetworkInfo deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) +instance ToCBOR NetworkInfo where + toCBOR = genericToCBOR + +instance FromCBOR NetworkInfo where + fromCBOR = genericFromCBOR + -- | Get latest confirmed snapshot UTxO from 'HeadState'. getSnapshotUtxo :: IsTx tx => HeadState tx -> Maybe (UTxOType tx) getSnapshotUtxo = \case diff --git a/hydra-node/src/Hydra/CBOR/Orphans.hs b/hydra-node/src/Hydra/CBOR/Orphans.hs new file mode 100644 index 00000000000..c4ff496b504 --- /dev/null +++ b/hydra-node/src/Hydra/CBOR/Orphans.hs @@ -0,0 +1,49 @@ +{-# OPTIONS_GHC -Wno-orphans #-} + +-- | Orphan 'ToCBOR' / 'FromCBOR' instances for third-party types that appear +-- in CBOR-encoded hydra-node data (logs and API messages). +module Hydra.CBOR.Orphans () where + +import Hydra.Prelude + +import Codec.CBOR.JSON (decodeValue, encodeValue) +import Data.Aeson qualified as Aeson +import Data.IP (IP) +import Hydra.Cardano.Api (File (..)) +import Network.Socket (PortNumber) + +-- | 'Aeson.Value' is CBOR-encoded using the standard JSON-in-CBOR mapping +-- from cborg-json. This is used for the few log types which carry raw JSON +-- payloads (e.g. 'Hydra.API.APIServerLog.APIServerLog'). +-- +-- NOTE: Non-integral JSON numbers round-trip through 'Double', which may lose +-- precision; integers are exact. +instance ToCBOR Aeson.Value where + toCBOR = encodeValue + +instance FromCBOR Aeson.Value where + fromCBOR = decodeValue False + +-- | Encoded as its textual representation (e.g. @127.0.0.1@). +instance ToCBOR IP where + toCBOR = toCBOR . (show :: IP -> Text) + +instance FromCBOR IP where + fromCBOR = do + t :: Text <- fromCBOR + case readMaybe (toString t) of + Nothing -> fail $ "failed to parse IP address from " <> show t + Just ip -> pure ip + +instance ToCBOR PortNumber where + toCBOR = toCBOR . (fromIntegral :: PortNumber -> Word16) + +instance FromCBOR PortNumber where + fromCBOR = (fromIntegral :: Word16 -> PortNumber) <$> fromCBOR + +-- | Encoded as the file path text (e.g. for 'Hydra.Cardano.Api.SocketPath'). +instance (Typeable content, Typeable direction) => ToCBOR (File content direction) where + toCBOR (File fp) = toCBOR (toText fp) + +instance (Typeable content, Typeable direction) => FromCBOR (File content direction) where + fromCBOR = File . toString <$> fromCBOR @Text diff --git a/hydra-node/src/Hydra/Chain.hs b/hydra-node/src/Hydra/Chain.hs index fe84cc4012d..12e107421c6 100644 --- a/hydra-node/src/Hydra/Chain.hs +++ b/hydra-node/src/Hydra/Chain.hs @@ -113,6 +113,12 @@ deriving stock instance IsTx tx => Show (PostChainTx tx) deriving anyclass instance IsTx tx => ToJSON (PostChainTx tx) deriving anyclass instance IsTx tx => FromJSON (PostChainTx tx) +instance IsTx tx => ToCBOR (PostChainTx tx) where + toCBOR = genericToCBOR + +instance IsTx tx => FromCBOR (PostChainTx tx) where + fromCBOR = genericFromCBOR + -- | Describes transactions as seen on chain. Holds as minimal information as -- possible to simplify observing the chain. data OnChainTx tx @@ -163,6 +169,12 @@ deriving stock instance IsTx tx => Show (OnChainTx tx) deriving anyclass instance IsTx tx => ToJSON (OnChainTx tx) deriving anyclass instance IsTx tx => FromJSON (OnChainTx tx) +instance IsTx tx => ToCBOR (OnChainTx tx) where + toCBOR = genericToCBOR + +instance IsTx tx => FromCBOR (OnChainTx tx) where + fromCBOR = genericFromCBOR + -- | Exceptions thrown by 'postTx'. data PostTxError tx = NoSeedInput @@ -214,6 +226,12 @@ deriving anyclass instance IsChainState tx => FromJSON (PostTxError tx) instance IsChainState tx => Exception (PostTxError tx) +instance IsChainState tx => ToCBOR (PostTxError tx) where + toCBOR = genericToCBOR + +instance IsChainState tx => FromCBOR (PostTxError tx) where + fromCBOR = genericFromCBOR + -- | A non empty sequence of chain states that can be rolled back. -- This is expected to be constructed by using the smart constructor -- 'initHistory'. @@ -355,6 +373,12 @@ deriving stock instance (IsTx tx, IsChainState tx) => Show (ChainEvent tx) deriving anyclass instance (IsTx tx, IsChainState tx) => ToJSON (ChainEvent tx) deriving anyclass instance (IsTx tx, IsChainState tx) => FromJSON (ChainEvent tx) +instance IsChainState tx => ToCBOR (ChainEvent tx) where + toCBOR = genericToCBOR + +instance IsChainState tx => FromCBOR (ChainEvent tx) where + fromCBOR = genericFromCBOR + -- | A callback indicating a 'ChainEvent tx' happened. Most importantly the -- 'Observation' of a relevant Hydra transaction. type ChainCallback tx m = ChainEvent tx -> m () diff --git a/hydra-node/src/Hydra/Chain/Direct/State.hs b/hydra-node/src/Hydra/Chain/Direct/State.hs index f3da475e93c..f30494a35a0 100644 --- a/hydra-node/src/Hydra/Chain/Direct/State.hs +++ b/hydra-node/src/Hydra/Chain/Direct/State.hs @@ -114,6 +114,13 @@ data ChainStateAt = ChainStateAt deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) +instance ToCBOR ChainStateAt where + toCBOR ChainStateAt{spendableUTxO, recordedAt} = + toCBOR spendableUTxO <> toCBOR recordedAt + +instance FromCBOR ChainStateAt where + fromCBOR = ChainStateAt <$> fromCBOR <*> fromCBOR + instance IsChainState Tx where type ChainPointType Tx = ChainPoint diff --git a/hydra-node/src/Hydra/Events/SQLiteBased.hs b/hydra-node/src/Hydra/Events/SQLiteBased.hs index b04d2a912b6..e7acec2d63b 100644 --- a/hydra-node/src/Hydra/Events/SQLiteBased.hs +++ b/hydra-node/src/Hydra/Events/SQLiteBased.hs @@ -7,9 +7,21 @@ -- == Architecture -- -- Events are stored in a single @events@ table with an integer primary key --- (@event_id@) and a BLOB column (@event_data@) containing JSON-encoded event --- data. The database uses WAL journal mode with @synchronous=NORMAL@ to avoid --- per-write fsyncs while still syncing at WAL checkpoints. +-- (@event_id@) and a BLOB column (@event_data@) containing CBOR-encoded event +-- data (via 'ToCBOR' / 'FromCBOR'). The database uses WAL journal mode with +-- @synchronous=NORMAL@ to avoid per-write fsyncs while still syncing at WAL +-- checkpoints. +-- +-- == Schema migrations +-- +-- The schema version is tracked in @PRAGMA user_version@ and migrated on open +-- (see 'applyMigrations'). Version 1 stored event data as JSON; opening a +-- version 1 database re-encodes every row to CBOR in one transaction and runs +-- @VACUUM@ afterwards to reclaim the freed space. A row that fails to decode +-- aborts the migration (and thereby node startup) with +-- 'EventDecodingException', rolling back to an intact version 1 database. +-- The legacy file-based store (JSON lines) is migrated by decoding each line +-- as JSON and inserting CBOR. -- -- == Async write-behind -- @@ -58,6 +70,7 @@ module Hydra.Events.SQLiteBased where import Hydra.Prelude +import Cardano.Binary (decodeFull', serialize') import Conduit (ConduitT, ResourceT, bracketP, runConduitRes, sourceFile, yield, (.|)) import Control.Concurrent.Class.MonadSTM (flushTBQueue, newEmptyTMVarIO, newTBQueueIO, putTMVar, readTBQueue, takeTMVar, writeTBQueue, writeTVar) import Control.Monad.Class.MonadAsync (async, cancel, link) @@ -65,7 +78,7 @@ import Data.Aeson qualified as Aeson import Data.ByteString qualified as BS import Data.Conduit.Combinators (linesUnboundedAscii) import Data.Conduit.Combinators qualified as C -import Database.SQLite.Simple (Connection, Only (..), Statement, close, closeStatement, execute, executeMany, execute_, nextRow, open, openStatement, query_, withTransaction) +import Database.SQLite.Simple (Connection, Only (..), Statement, close, closeStatement, execute, executeMany, execute_, nextRow, open, openStatement, query, query_, withTransaction) import Hydra.Events (EventSink (..), EventSource (..), HasEventId (..)) import Hydra.Events.Rotation (EventStore (..)) import Hydra.Logging (Tracer, traceWith) @@ -91,7 +104,7 @@ data SQLiteLog -- | Items in the write-behind queue: either an event to insert or a flush -- marker that the writer thread signals after processing all preceding items. -- --- Events are queued unencoded and JSON-encoded on the writer thread: the +-- Events are queued unencoded and CBOR-encoded on the writer thread: the -- encoding of e.g. a SnapshotRequested event carrying a large UTxO otherwise -- sits on the node loop between processing a ReqSn and broadcasting the -- AckSn. The bounded queue briefly pins event values instead of compact @@ -112,7 +125,7 @@ type WriteItem e = Either (TMVar IO ()) (Word64, e) -- event id TVar, and this bracket flushes on exit. withSQLiteEventStore :: forall e a. - (FromJSON e, ToJSON e, HasEventId e) => + (ToCBOR e, FromCBOR e, FromJSON e, HasEventId e) => Tracer IO SQLiteLog -> FilePath -> FilePath -> @@ -131,13 +144,20 @@ withSQLiteEventStore tracer dbFile legacyStateFile callback = do -- automatically. mkSQLiteEventStore :: forall e. - (ToJSON e, FromJSON e, HasEventId e) => + (ToCBOR e, FromCBOR e, FromJSON e, HasEventId e) => FilePath -> IO (Connection, EventStore e IO, IO (), IO (), IO ()) mkSQLiteEventStore dbFile = do createDirectoryIfMissing True (takeDirectory dbFile) conn <- open dbFile - initSchema conn + -- Rows of a version 1 database are JSON-encoded and get re-encoded to CBOR + -- by the schema migration; a row that fails to decode aborts startup. + let reencodeRow :: Word64 -> ByteString -> IO ByteString + reencodeRow eid bytes = + case Aeson.eitherDecodeStrict' @e bytes of + Right evt -> pure $ serialize' evt + Left err -> throwIO EventDecodingException{eventId = eid, decodeError = err} + initSchema conn reencodeRow -- Dedicated connection for 'sourceEvents' streams, so concurrent client -- history replay cannot hold statements open on the connection rotation -- runs VACUUM INTO on (see module header). @@ -161,11 +181,11 @@ mkSQLiteEventStore dbFile = do decodeRow :: (Word64, ByteString) -> IO e decodeRow (eid, evData) = - case Aeson.eitherDecodeStrict' evData of + case decodeFull' evData of Right evt -> pure evt -- NOTE: This will prevent the node from starting, which is intentional — -- starting with missing events would silently corrupt the head state. - Left err -> throwIO EventDecodingException{eventId = eid, decodeError = err} + Left err -> throwIO EventDecodingException{eventId = eid, decodeError = show err} sourceEvents :: ConduitT () e (ResourceT IO) () sourceEvents = do @@ -215,7 +235,7 @@ mkSQLiteEventStore dbFile = do -- Archive the current database before removing events, so the -- pre-rotation log is retained (mirrors the old file-based backup). backupDatabase conn dbFile logId - let evData = toStrict $ Aeson.encode checkpointEvent + let evData = serialize' checkpointEvent withTransaction conn $ do deleteAllEvents conn insertEvent conn (getEventId checkpointEvent, evData) @@ -241,17 +261,17 @@ mkSQLiteEventStore dbFile = do -- | Background writer that drains the queue and batch-inserts into SQLite. -- Each iteration blocks for at least one item, then flushes everything --- available. Events are JSON-encoded here, off the caller's thread, then +-- available. Events are CBOR-encoded here, off the caller's thread, then -- batch-inserted in a single transaction, and any flush markers in the batch -- are signalled. Encode errors surface as writer thread crashes, which are -- 'link'ed to the node. -writerLoop :: ToJSON e => Connection -> TBQueue IO (WriteItem e) -> IO () +writerLoop :: ToCBOR e => Connection -> TBQueue IO (WriteItem e) -> IO () writerLoop conn queue = forever $ do first' <- atomically $ readTBQueue queue rest <- atomically $ flushTBQueue queue let allItems = first' : rest (flushSignals, events) = partitionEithers allItems - eventRows = map (second (toStrict . Aeson.encode)) events + eventRows = map (second serialize') events unless (null eventRows) $ withTransaction conn $ insertEvents conn eventRows @@ -278,7 +298,7 @@ flushWriteQueue queue = do -- subsequent node restarts skip the migration step automatically. migrateFromFileBased :: forall e. - (FromJSON e, HasEventId e) => + (FromJSON e, ToCBOR e, HasEventId e) => Proxy e -> Tracer IO SQLiteLog -> FilePath -> @@ -297,11 +317,12 @@ migrateFromFileBased _proxy tracer legacyFile conn reinitLastSeen = do .| linesUnboundedAscii .| C.filter (not . BS.null) .| C.sinkList - -- Decode each line to extract the event id, then store the original raw - -- bytes. Invalid JSON is caught here so corrupt files fail at migration. + -- Decode each JSON line (legacy files are always JSON) and store the + -- event re-encoded as CBOR. Invalid JSON is caught here so corrupt + -- files fail at migration. rowParams <- forM (zip [1 ..] rawLines) $ \(lineNo :: Int, line) -> case Aeson.eitherDecodeStrict' @e line of - Right evt -> pure (getEventId evt, line) + Right evt -> pure (getEventId evt, serialize' evt) Left err -> throwIO EventDecodingException{eventId = fromIntegral lineNo, decodeError = err} unless (null rowParams) $ withTransaction conn $ @@ -315,17 +336,26 @@ migrateFromFileBased _proxy tracer legacyFile conn reinitLastSeen = do -- Internal -- | Current schema version. Bump this and add a migration step to --- 'applyMigrations' whenever the schema changes. +-- 'migrateStep' whenever the schema changes. nextVersion :: Int -nextVersion = 1 +nextVersion = 2 + +-- | Re-encode a single event row given its event id and stored bytes, used by +-- the version 1 (JSON) to version 2 (CBOR) migration. Must throw when the row +-- cannot be decoded. +type ReencodeRow = Word64 -> ByteString -> IO ByteString -- | Initialise connection pragmas, then create or migrate the schema to -- 'nextVersion' using SQLite's built-in @user_version@ pragma. -initSchema :: Connection -> IO () -initSchema conn = do +initSchema :: Connection -> ReencodeRow -> IO () +initSchema conn reencodeRow = do configurePragmas conn v <- getSchemaVersion conn - applyMigrations conn v + applyMigrations conn reencodeRow v + -- Reclaim the space freed by the JSON -> CBOR re-encode. VACUUM cannot run + -- inside a transaction and is a space optimization only: a crash between + -- the migration commit and here costs disk space, not correctness. + when (v == 1) $ execute_ conn "VACUUM" configurePragmas :: Connection -> IO () configurePragmas conn = @@ -353,25 +383,50 @@ setSchemaVersion conn v = execute_ conn $ fromString $ "PRAGMA user_version = " <> show v -- | Apply all pending migrations from version @v@ up to 'nextVersion'. --- Each step runs in its own transaction so that a crash mid-migration leaves --- the database at a well-defined version. -applyMigrations :: Connection -> Int -> IO () -applyMigrations conn v +-- Each step runs together with its version bump in one transaction +-- (@PRAGMA user_version@ is transactional), so a crash or decoding failure +-- mid-migration rolls back to a well-defined version. +applyMigrations :: Connection -> ReencodeRow -> Int -> IO () +applyMigrations conn reencodeRow v | v > nextVersion = error $ "Database schema version " <> show v <> " is newer than supported " <> show nextVersion <> ", cannot downgrade" | v == nextVersion = pure () | otherwise = do - migrateStep conn v - setSchemaVersion conn (v + 1) - applyMigrations conn (v + 1) + withTransaction conn $ do + migrateStep conn reencodeRow v + setSchemaVersion conn (v + 1) + applyMigrations conn reencodeRow (v + 1) -- | Individual migration steps. Pattern-match on the /source/ version. -migrateStep :: Connection -> Int -> IO () -migrateStep conn = \case +migrateStep :: Connection -> ReencodeRow -> Int -> IO () +migrateStep conn reencodeRow = \case 0 -> createEventsTable conn + 1 -> reencodeAllEvents conn reencodeRow unknown -> error $ "Unknown schema version " <> show unknown <> ", cannot migrate" +-- | Re-encode all event rows using the given 'ReencodeRow' function (the +-- version 1 JSON to version 2 CBOR migration). Rows are processed in batches +-- of ascending event id so memory stays bounded for large databases. Runs +-- inside the caller's transaction. +reencodeAllEvents :: Connection -> ReencodeRow -> IO () +reencodeAllEvents conn reencodeRow = go 0 + where + batchSize = 1000 :: Int + + go :: Word64 -> IO () + go startId = do + rows :: [(Word64, ByteString)] <- + query conn "SELECT event_id, event_data FROM events WHERE event_id >= ? ORDER BY event_id LIMIT ?" (startId, batchSize) + case nonEmpty rows of + Nothing -> pure () + Just neRows -> do + updates <- forM rows $ \(eid, evData) -> do + encoded <- reencodeRow eid evData + pure (encoded, eid) + executeMany conn "UPDATE events SET event_data = ? WHERE event_id = ?" updates + go (fst (last neRows) + 1) + -- SQL queries createEventsTable :: Connection -> IO () diff --git a/hydra-node/src/Hydra/HeadLogic/Error.hs b/hydra-node/src/Hydra/HeadLogic/Error.hs index cf13c7d06a9..9ca10d6b3d8 100644 --- a/hydra-node/src/Hydra/HeadLogic/Error.hs +++ b/hydra-node/src/Hydra/HeadLogic/Error.hs @@ -5,6 +5,7 @@ module Hydra.HeadLogic.Error where import Hydra.Prelude +import Hydra.Chain.ChainState (IsChainState) import Hydra.HeadLogic.Input (Input) import Hydra.HeadLogic.State (HeadState) import Hydra.Ledger (ValidationError) @@ -49,6 +50,12 @@ deriving anyclass instance ) => ToJSON (LogicError tx) +instance (IsChainState tx, ToCBOR (Input tx)) => ToCBOR (LogicError tx) where + toCBOR = genericToCBOR + +instance (IsChainState tx, FromCBOR (Input tx)) => FromCBOR (LogicError tx) where + fromCBOR = genericFromCBOR + data RequirementFailure tx = ReqSnNumberInvalid {requestedSn :: SnapshotNumber, lastSeenSn :: SnapshotNumber} | ReqSvNumberInvalid {requestedSv :: SnapshotVersion, lastSeenSv :: SnapshotVersion} @@ -69,6 +76,12 @@ deriving stock instance Eq (TxIdType tx) => Eq (RequirementFailure tx) deriving stock instance Show (TxIdType tx) => Show (RequirementFailure tx) deriving anyclass instance ToJSON (TxIdType tx) => ToJSON (RequirementFailure tx) +instance IsTx tx => ToCBOR (RequirementFailure tx) where + toCBOR = genericToCBOR + +instance IsTx tx => FromCBOR (RequirementFailure tx) where + fromCBOR = genericFromCBOR + data SideLoadRequirementFailure tx = SideLoadInitialSnapshotMismatch | SideLoadSnNumberInvalid {requestedSn :: SnapshotNumber, lastSeenSn :: SnapshotNumber} @@ -82,3 +95,9 @@ deriving stock instance Eq (UTxOType tx) => Eq (SideLoadRequirementFailure tx) deriving stock instance Show (UTxOType tx) => Show (SideLoadRequirementFailure tx) deriving anyclass instance ToJSON (UTxOType tx) => ToJSON (SideLoadRequirementFailure tx) deriving anyclass instance FromJSON (UTxOType tx) => FromJSON (SideLoadRequirementFailure tx) + +instance IsTx tx => ToCBOR (SideLoadRequirementFailure tx) where + toCBOR = genericToCBOR + +instance IsTx tx => FromCBOR (SideLoadRequirementFailure tx) where + fromCBOR = genericFromCBOR diff --git a/hydra-node/src/Hydra/HeadLogic/Outcome.hs b/hydra-node/src/Hydra/HeadLogic/Outcome.hs index 6bb9277f629..bffcba398fc 100644 --- a/hydra-node/src/Hydra/HeadLogic/Outcome.hs +++ b/hydra-node/src/Hydra/HeadLogic/Outcome.hs @@ -195,6 +195,15 @@ instance forall tx. (IsChainState tx, IsTx tx, FromJSON (NodeState tx), FromJSON Object (KeyMap.insert "mode" (toJSON (AwaitingSelection :: FanoutMode tx)) o) v -> v +-- NOTE: This codec defines the event format persisted in the hydra.db events +-- table (see "Hydra.Events.SQLiteBased"). Changing an encoding here breaks +-- decoding of existing databases and requires a schema migration. +instance IsChainState tx => ToCBOR (StateChanged tx) where + toCBOR = genericToCBOR + +instance IsChainState tx => FromCBOR (StateChanged tx) where + fromCBOR = genericFromCBOR + data Outcome tx = -- | Continue with the given state updates and side effects. Continue {stateChanges :: [StateChanged tx], effects :: [Effect tx]} diff --git a/hydra-node/src/Hydra/HeadLogic/State.hs b/hydra-node/src/Hydra/HeadLogic/State.hs index d7e2ddbee53..8bcc192843b 100644 --- a/hydra-node/src/Hydra/HeadLogic/State.hs +++ b/hydra-node/src/Hydra/HeadLogic/State.hs @@ -52,6 +52,12 @@ deriving stock instance (IsTx tx, Show (ChainStateType tx)) => Show (HeadState t deriving anyclass instance (IsTx tx, ToJSON (ChainStateType tx)) => ToJSON (HeadState tx) deriving anyclass instance (IsTx tx, FromJSON (ChainStateType tx)) => FromJSON (HeadState tx) +instance IsChainState tx => ToCBOR (HeadState tx) where + toCBOR = genericToCBOR + +instance IsChainState tx => FromCBOR (HeadState tx) where + fromCBOR = genericFromCBOR + -- | Update the chain state in any 'HeadState'. setChainState :: ChainStateType tx -> HeadState tx -> HeadState tx setChainState chainState = \case @@ -95,6 +101,12 @@ deriving stock instance Show (ChainStateType tx) => Show (IdleState tx) deriving anyclass instance ToJSON (ChainStateType tx) => ToJSON (IdleState tx) deriving anyclass instance FromJSON (ChainStateType tx) => FromJSON (IdleState tx) +instance IsChainState tx => ToCBOR (IdleState tx) where + toCBOR IdleState{chainState} = toCBOR chainState + +instance IsChainState tx => FromCBOR (IdleState tx) where + fromCBOR = IdleState <$> fromCBOR + -- ** Open -- | An 'Open' head with a 'CoordinatedHeadState' tracking off-chain @@ -113,6 +125,17 @@ deriving stock instance (IsTx tx, Show (ChainStateType tx)) => Show (OpenState t deriving anyclass instance (IsTx tx, ToJSON (ChainStateType tx)) => ToJSON (OpenState tx) deriving anyclass instance (IsTx tx, FromJSON (ChainStateType tx)) => FromJSON (OpenState tx) +instance IsChainState tx => ToCBOR (OpenState tx) where + toCBOR OpenState{parameters, coordinatedHeadState, chainState, headId, headSeed} = + toCBOR parameters + <> toCBOR coordinatedHeadState + <> toCBOR chainState + <> toCBOR headId + <> toCBOR headSeed + +instance IsChainState tx => FromCBOR (OpenState tx) where + fromCBOR = OpenState <$> fromCBOR <*> fromCBOR <*> fromCBOR <*> fromCBOR <*> fromCBOR + -- | Off-chain state of the Coordinated Head protocol. data CoordinatedHeadState tx = CoordinatedHeadState { localUTxO :: UTxOType tx @@ -144,6 +167,29 @@ deriving stock instance IsTx tx => Show (CoordinatedHeadState tx) deriving anyclass instance IsTx tx => ToJSON (CoordinatedHeadState tx) deriving anyclass instance IsTx tx => FromJSON (CoordinatedHeadState tx) +instance IsTx tx => ToCBOR (CoordinatedHeadState tx) where + toCBOR CoordinatedHeadState{localUTxO, localTxs, allTxs, confirmedSnapshot, seenSnapshot, currentDepositTxId, decommitTx, version} = + toCBOR localUTxO + <> toCBOR localTxs + <> toCBOR allTxs + <> toCBOR confirmedSnapshot + <> toCBOR seenSnapshot + <> toCBOR currentDepositTxId + <> toCBOR decommitTx + <> toCBOR version + +instance IsTx tx => FromCBOR (CoordinatedHeadState tx) where + fromCBOR = + CoordinatedHeadState + <$> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + -- | Data structure to help in tracking whether we have seen or requested a -- ReqSn already and if seen, the signatures we collected already. data SeenSnapshot tx @@ -211,6 +257,28 @@ instance IsTx tx => FromJSON (SeenSnapshot tx) where pure $ mkSeenSnapshot snapshot signatories other -> fail $ "unknown SeenSnapshot tag: " <> toString other +-- Manual instances that exclude 'signableBytes' from CBOR (it is derived from +-- 'snapshot' and recomputed on deserialisation), like the JSON instances above. +instance IsTx tx => ToCBOR (SeenSnapshot tx) where + toCBOR = \case + NoSeenSnapshot -> + toCBOR ("NoSeenSnapshot" :: Text) + LastSeenSnapshot{lastSeen} -> + toCBOR ("LastSeenSnapshot" :: Text) <> toCBOR lastSeen + RequestedSnapshot{lastSeen, requested} -> + toCBOR ("RequestedSnapshot" :: Text) <> toCBOR lastSeen <> toCBOR requested + SeenSnapshot{snapshot, signatories} -> + toCBOR ("SeenSnapshot" :: Text) <> toCBOR snapshot <> toCBOR signatories + +instance IsTx tx => FromCBOR (SeenSnapshot tx) where + fromCBOR = + fromCBOR >>= \case + ("NoSeenSnapshot" :: Text) -> pure NoSeenSnapshot + "LastSeenSnapshot" -> LastSeenSnapshot <$> fromCBOR + "RequestedSnapshot" -> RequestedSnapshot <$> fromCBOR <*> fromCBOR + "SeenSnapshot" -> mkSeenSnapshot <$> fromCBOR <*> fromCBOR + tag -> fail $ show tag <> " is not a proper CBOR-encoded SeenSnapshot" + -- | Smart constructor for 'SeenSnapshot' that computes and caches -- 'signableBytes' from 'snapshot', enforcing the invariant that they stay in sync. mkSeenSnapshot :: @@ -269,6 +337,29 @@ deriving stock instance (IsTx tx, Show (ChainStateType tx)) => Show (ClosedState deriving anyclass instance (IsTx tx, ToJSON (ChainStateType tx)) => ToJSON (ClosedState tx) deriving anyclass instance (IsTx tx, FromJSON (ChainStateType tx)) => FromJSON (ClosedState tx) +instance IsChainState tx => ToCBOR (ClosedState tx) where + toCBOR ClosedState{parameters, confirmedSnapshot, contestationDeadline, readyToFanoutSent, chainState, headId, headSeed, version} = + toCBOR parameters + <> toCBOR confirmedSnapshot + <> toCBOR contestationDeadline + <> toCBOR readyToFanoutSent + <> toCBOR chainState + <> toCBOR headId + <> toCBOR headSeed + <> toCBOR version + +instance IsChainState tx => FromCBOR (ClosedState tx) where + fromCBOR = + ClosedState + <$> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + -- ** PartialFanout -- Terminology note: the selective-fanout feature spans several near-synonymous @@ -300,6 +391,12 @@ deriving stock instance IsTx tx => Show (FanoutMode tx) deriving anyclass instance IsTx tx => ToJSON (FanoutMode tx) deriving anyclass instance IsTx tx => FromJSON (FanoutMode tx) +instance IsTx tx => ToCBOR (FanoutMode tx) where + toCBOR = genericToCBOR + +instance IsTx tx => FromCBOR (FanoutMode tx) where + fromCBOR = genericFromCBOR + -- | A closed head whose UTxO is being distributed across multiple fanout -- transactions (on-chain @FanoutProgress@). Holds the partial-fanout bookkeeping -- that used to live in 'ClosedState'. @@ -325,3 +422,30 @@ deriving stock instance (IsTx tx, Eq (ChainStateType tx)) => Eq (PartialFanoutSt deriving stock instance (IsTx tx, Show (ChainStateType tx)) => Show (PartialFanoutState tx) deriving anyclass instance (IsTx tx, ToJSON (ChainStateType tx)) => ToJSON (PartialFanoutState tx) deriving anyclass instance (IsTx tx, FromJSON (ChainStateType tx)) => FromJSON (PartialFanoutState tx) + +instance IsChainState tx => ToCBOR (PartialFanoutState tx) where + toCBOR PartialFanoutState{parameters, confirmedSnapshot, contestationDeadline, chainState, headId, headSeed, version, remainingOutputs, distributedOutputs, mode} = + toCBOR parameters + <> toCBOR confirmedSnapshot + <> toCBOR contestationDeadline + <> toCBOR chainState + <> toCBOR headId + <> toCBOR headSeed + <> toCBOR version + <> toCBOR remainingOutputs + <> toCBOR distributedOutputs + <> toCBOR mode + +instance IsChainState tx => FromCBOR (PartialFanoutState tx) where + fromCBOR = + PartialFanoutState + <$> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR + <*> fromCBOR diff --git a/hydra-node/src/Hydra/HeadLogic/StateEvent.hs b/hydra-node/src/Hydra/HeadLogic/StateEvent.hs index 6d4b7339c45..afc3d2d109b 100644 --- a/hydra-node/src/Hydra/HeadLogic/StateEvent.hs +++ b/hydra-node/src/Hydra/HeadLogic/StateEvent.hs @@ -28,6 +28,16 @@ deriving stock instance IsChainState tx => Eq (StateEvent tx) deriving anyclass instance IsChainState tx => ToJSON (StateEvent tx) deriving anyclass instance IsChainState tx => FromJSON (StateEvent tx) +-- NOTE: This codec defines the row format persisted in the hydra.db events +-- table (see "Hydra.Events.SQLiteBased"). Changing it breaks decoding of +-- existing databases and requires a schema migration. +instance IsChainState tx => ToCBOR (StateEvent tx) where + toCBOR StateEvent{eventId, stateChanged, time} = + toCBOR eventId <> toCBOR stateChanged <> toCBOR time + +instance IsChainState tx => FromCBOR (StateEvent tx) where + fromCBOR = StateEvent <$> fromCBOR <*> fromCBOR <*> fromCBOR + mkCheckpoint :: NodeState tx -> EventId -> UTCTime -> StateEvent tx mkCheckpoint nodeState eventId time = StateEvent diff --git a/hydra-node/src/Hydra/Ledger.hs b/hydra-node/src/Hydra/Ledger.hs index f58a41b9e8e..588e7e85607 100644 --- a/hydra-node/src/Hydra/Ledger.hs +++ b/hydra-node/src/Hydra/Ledger.hs @@ -62,3 +62,4 @@ data ValidationResult newtype ValidationError = ValidationError {reason :: Text} deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) + deriving newtype (ToCBOR, FromCBOR) diff --git a/hydra-node/src/Hydra/Ledger/Simple.hs b/hydra-node/src/Hydra/Ledger/Simple.hs index 996842c78fc..1a3c9b48d2e 100644 --- a/hydra-node/src/Hydra/Ledger/Simple.hs +++ b/hydra-node/src/Hydra/Ledger/Simple.hs @@ -108,7 +108,7 @@ instance IsTx SimpleTx where newtype SimpleChainState = SimpleChainState {slot :: ChainSlot} deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) - deriving newtype (Num) + deriving newtype (Num, ToCBOR, FromCBOR) instance IsChainState SimpleTx where type ChainPointType SimpleTx = ChainSlot diff --git a/hydra-node/src/Hydra/Network.hs b/hydra-node/src/Hydra/Network.hs index 1869c994d40..0f82771dda5 100644 --- a/hydra-node/src/Hydra/Network.hs +++ b/hydra-node/src/Hydra/Network.hs @@ -61,6 +61,12 @@ data WhichEtcd = EmbeddedEtcd | SystemEtcd deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) +instance ToCBOR WhichEtcd where + toCBOR = genericToCBOR + +instance FromCBOR WhichEtcd where + fromCBOR = genericFromCBOR + -- | Configuration for a `Node` network layer. data NetworkConfiguration = NetworkConfiguration { persistenceDir :: FilePath @@ -116,7 +122,7 @@ instance FromJSON PortNumber where -- ** NodeId newtype NodeId = NodeId {nodeId :: Text} - deriving newtype (Eq, Show, IsString, Read, Ord, ToJSON, FromJSON) + deriving newtype (Eq, Show, IsString, Read, Ord, ToJSON, FromJSON, ToCBOR, FromCBOR) -- ** Host @@ -183,6 +189,12 @@ data Connectivity deriving stock (Generic, Eq, Show) deriving anyclass (ToJSON, FromJSON) +instance ToCBOR Connectivity where + toCBOR = genericToCBOR + +instance FromCBOR Connectivity where + fromCBOR = genericFromCBOR + newtype ProtocolVersion = ProtocolVersion Natural deriving stock (Eq, Show, Generic, Ord) deriving newtype (ToCBOR, FromCBOR) diff --git a/hydra-node/src/Hydra/Network/Message.hs b/hydra-node/src/Hydra/Network/Message.hs index 0ddf8ee1707..adff23244c6 100644 --- a/hydra-node/src/Hydra/Network/Message.hs +++ b/hydra-node/src/Hydra/Network/Message.hs @@ -45,20 +45,10 @@ deriving anyclass instance IsTx tx => ToJSON (Message tx) deriving anyclass instance IsTx tx => FromJSON (Message tx) instance (ToCBOR tx, ToCBOR (UTxOType tx), ToCBOR (TxIdType tx)) => ToCBOR (Message tx) where - toCBOR = \case - ReqTx tx -> toCBOR ("ReqTx" :: Text) <> toCBOR tx - ReqSn sv sn txs decommitTx incrementUTxO -> toCBOR ("ReqSn" :: Text) <> toCBOR sv <> toCBOR sn <> toCBOR txs <> toCBOR decommitTx <> toCBOR incrementUTxO - AckSn sig sn -> toCBOR ("AckSn" :: Text) <> toCBOR sig <> toCBOR sn - ReqDec utxo -> toCBOR ("ReqDec" :: Text) <> toCBOR utxo + toCBOR = genericToCBOR instance (FromCBOR tx, FromCBOR (UTxOType tx), FromCBOR (TxIdType tx)) => FromCBOR (Message tx) where - fromCBOR = - fromCBOR >>= \case - ("ReqTx" :: Text) -> ReqTx <$> fromCBOR - "ReqSn" -> ReqSn <$> fromCBOR <*> fromCBOR <*> fromCBOR <*> fromCBOR <*> fromCBOR - "AckSn" -> AckSn <$> fromCBOR <*> fromCBOR - "ReqDec" -> ReqDec <$> fromCBOR - msg -> fail $ show msg <> " is not a proper CBOR-encoded Message" + fromCBOR = genericFromCBOR instance IsTx tx => SignableRepresentation (Message tx) where getSignableRepresentation = serialize' diff --git a/hydra-node/src/Hydra/Node/ApiTransactionTimeout.hs b/hydra-node/src/Hydra/Node/ApiTransactionTimeout.hs index 8cb39230b7d..6b70e82a842 100644 --- a/hydra-node/src/Hydra/Node/ApiTransactionTimeout.hs +++ b/hydra-node/src/Hydra/Node/ApiTransactionTimeout.hs @@ -7,7 +7,7 @@ newtype ApiTransactionTimeout = ApiTransactionTimeout { apiTransactionTimeoutNominalDiffTime :: NominalDiffTime } deriving stock (Eq, Ord) - deriving newtype (Show, Read, Num, Enum, Real, ToJSON, FromJSON) + deriving newtype (Show, Read, Num, Enum, Real, ToJSON, FromJSON, ToCBOR, FromCBOR) -- | Truncates to whole seconds. instance Integral ApiTransactionTimeout where diff --git a/hydra-node/src/Hydra/Node/Environment.hs b/hydra-node/src/Hydra/Node/Environment.hs index 948a1d35a16..cf0867c8c5a 100644 --- a/hydra-node/src/Hydra/Node/Environment.hs +++ b/hydra-node/src/Hydra/Node/Environment.hs @@ -93,6 +93,44 @@ instance FromJSON Environment where <*> o .: "unsyncedPeriod" <*> o .: "configuredPeers" +-- | Like the JSON instance above, 'ToCBOR' deliberately omits 'signingKey' +-- (CBOR-encoding a 'Secret' is a compile-time error by design). +instance ToCBOR Environment where + toCBOR Environment{party, otherParties, participants, contestationPeriod, depositPeriod, depositActivation, unsyncedPeriod, configuredPeers} = + toCBOR party + <> toCBOR otherParties + <> toCBOR participants + <> toCBOR contestationPeriod + <> toCBOR depositPeriod + <> toCBOR depositActivation + <> toCBOR unsyncedPeriod + <> toCBOR configuredPeers + +-- | Like the JSON instance above, the decoded signing key is +-- 'placeholderSigningKey', NOT the real key. +instance FromCBOR Environment where + fromCBOR = do + party <- fromCBOR + otherParties <- fromCBOR + participants <- fromCBOR + contestationPeriod <- fromCBOR + depositPeriod <- fromCBOR + depositActivation <- fromCBOR + unsyncedPeriod <- fromCBOR + configuredPeers <- fromCBOR + pure + Environment + { party + , signingKey = placeholderSigningKey + , otherParties + , participants + , contestationPeriod + , depositPeriod + , depositActivation + , unsyncedPeriod + , configuredPeers + } + -- | Sentinel signing key used when an 'Environment' has to be -- reconstructed without access to the real one (e.g. JSON roundtrip -- tests). Exported so 'Arbitrary' generators can use the same value, diff --git a/hydra-node/src/Hydra/Node/State.hs b/hydra-node/src/Hydra/Node/State.hs index 5355d47c563..c8f584e1e8a 100644 --- a/hydra-node/src/Hydra/Node/State.hs +++ b/hydra-node/src/Hydra/Node/State.hs @@ -26,6 +26,13 @@ data ChainPointTime = ChainPointTime deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) +instance ToCBOR ChainPointTime where + toCBOR ChainPointTime{currentSlot, currentChainTime, drift} = + toCBOR currentSlot <> toCBOR currentChainTime <> toCBOR drift + +instance FromCBOR ChainPointTime where + fromCBOR = ChainPointTime <$> fromCBOR <*> fromCBOR <*> fromCBOR + data NodeState tx = -- | Normal operation of the node where it is connected and has a recent -- view of the chain. @@ -54,6 +61,12 @@ deriving stock instance (IsTx tx, Show (ChainStateType tx)) => Show (NodeState t deriving anyclass instance (IsTx tx, ToJSON (ChainStateType tx)) => ToJSON (NodeState tx) deriving anyclass instance (IsTx tx, FromJSON (ChainStateType tx)) => FromJSON (NodeState tx) +instance IsChainState tx => ToCBOR (NodeState tx) where + toCBOR = genericToCBOR + +instance IsChainState tx => FromCBOR (NodeState tx) where + fromCBOR = genericFromCBOR + initNodeState :: IsChainState tx => ChainStateType tx -> NodeState tx initNodeState chainState = NodeCatchingUp @@ -77,6 +90,12 @@ data SyncedStatus = InSync | CatchingUp deriving stock (Generic, Eq, Show) deriving anyclass (ToJSON, FromJSON) +instance ToCBOR SyncedStatus where + toCBOR = genericToCBOR + +instance FromCBOR SyncedStatus where + fromCBOR = genericFromCBOR + syncedStatus :: NodeState tx -> SyncedStatus syncedStatus NodeInSync{} = InSync syncedStatus NodeCatchingUp{} = CatchingUp @@ -97,10 +116,27 @@ deriving stock instance IsTx tx => Show (Deposit tx) deriving anyclass instance IsTx tx => ToJSON (Deposit tx) deriving anyclass instance IsTx tx => FromJSON (Deposit tx) +instance IsTx tx => ToCBOR (Deposit tx) where + toCBOR Deposit{headId, deposited, created, deadline, status} = + toCBOR headId + <> toCBOR deposited + <> toCBOR created + <> toCBOR deadline + <> toCBOR status + +instance IsTx tx => FromCBOR (Deposit tx) where + fromCBOR = Deposit <$> fromCBOR <*> fromCBOR <*> fromCBOR <*> fromCBOR <*> fromCBOR + data DepositStatus = Inactive | Active | Expired deriving stock (Generic, Eq, Show) deriving anyclass (ToJSON, FromJSON) +instance ToCBOR DepositStatus where + toCBOR = genericToCBOR + +instance FromCBOR DepositStatus where + fromCBOR = genericFromCBOR + depositsForHead :: HeadId -> PendingDeposits tx -> PendingDeposits tx depositsForHead targetHeadId = Map.filter (\Deposit{headId} -> headId == targetHeadId) diff --git a/hydra-node/src/Hydra/Node/UnsyncedPeriod.hs b/hydra-node/src/Hydra/Node/UnsyncedPeriod.hs index 024cbd4ab3d..6a206f1dae5 100644 --- a/hydra-node/src/Hydra/Node/UnsyncedPeriod.hs +++ b/hydra-node/src/Hydra/Node/UnsyncedPeriod.hs @@ -8,7 +8,7 @@ import Hydra.Tx.ContestationPeriod (ContestationPeriod, toNominalDiffTime) -- Beyond this period the node will refuse to process new transactions and signing snapshots. newtype UnsyncedPeriod = UnsyncedPeriod {unsyncedPeriodToNominalDiffTime :: NominalDiffTime} deriving stock (Eq, Ord) - deriving newtype (Show, Read, Num, Real, ToJSON, FromJSON) + deriving newtype (Show, Read, Num, Real, ToJSON, FromJSON, ToCBOR, FromCBOR) -- | Compute a default 'UnsyncedPeriod' based on the 'ContestationPeriod'. -- This is the legacy behavior: half of the contestation period. diff --git a/hydra-node/test/Hydra/CBORSpec.hs b/hydra-node/test/Hydra/CBORSpec.hs new file mode 100644 index 00000000000..ee4bcd29894 --- /dev/null +++ b/hydra-node/test/Hydra/CBORSpec.hs @@ -0,0 +1,225 @@ +{-# OPTIONS_GHC -Wno-orphans #-} + +-- | Tests for the 'ToCBOR' / 'FromCBOR' codecs of hydra-node types. +-- +-- Three layers of protection: +-- +-- * Unit tests for 'genericToCBOR' / 'genericFromCBOR' pinning down the +-- constructor-name-tagged format they produce (including that newtypes +-- and single-constructor records carry the tag). +-- +-- * Roundtrip properties keeping encoder/decoder pairs in sync: adding a +-- constructor without a matching codec (or with fields decoded in the +-- wrong order) fails here. +-- +-- * Golden tests locking the concrete byte-level formats, one sample per +-- constructor. These catch changes that roundtrip properties cannot see, +-- e.g. reordering fields in a data declaration of a generically derived +-- codec, or symmetric encoder+decoder drift. If one fails, the change +-- breaks decoding of persisted data (hydra.db) or the API wire format; +-- only delete and regenerate a golden file as a deliberate, documented +-- format change. +module Hydra.CBORSpec where + +import Hydra.Prelude +import Test.Hydra.Prelude + +import Cardano.Binary (decodeFull', serialize') +import Codec.CBOR.Write (toStrictByteString) +import Hydra.API.ClientInput (ClientInput) +import Hydra.API.ServerOutput ( + ClientMessage, + DecommitInvalidReason, + FanoutProgressMode, + Greetings, + HeadStatus, + InvalidInput (..), + NetworkInfo, + ServerOutput, + TimedServerOutput, + ) +import Hydra.Chain (ChainEvent, OnChainTx, PostChainTx, PostTxError) +import Hydra.Chain.Direct.State (ChainStateAt) +import Hydra.HeadLogic.Error (RequirementFailure, SideLoadRequirementFailure) +import Hydra.HeadLogic.Outcome (StateChanged) +import Hydra.HeadLogic.State (FanoutMode, HeadState, SeenSnapshot) +import Hydra.HeadLogic.StateEvent (StateEvent (..)) +import Hydra.Ledger.Cardano (Tx) +import Hydra.Network (Connectivity, WhichEtcd) +import Hydra.Network.Message (Message) +import Hydra.Node.Environment (Environment) +import Hydra.Node.State (Deposit, DepositStatus, NodeState, SyncedStatus) +import Hydra.Tx (ConfirmedSnapshot, Snapshot) +import Test.Hydra.API.ClientInput () +import Test.Hydra.API.ServerOutput () +import Test.Hydra.CBOR (genGoldenSamples, goldenCBOR, roundtripCBOR) +import Test.Hydra.Chain.Direct.State () +import Test.Hydra.HeadLogic.Outcome () +import Test.Hydra.HeadLogic.StateEvent () +import Test.Hydra.Network.Message () +import Test.Hydra.Node.Environment () +import Test.QuickCheck (resize) +import Test.QuickCheck.Arbitrary.ADT (ADTArbitrary (..), ConstructorArbitraryPair (..), ToADTArbitrary, toADTArbitrary) + +instance Arbitrary InvalidInput where + arbitrary = InvalidInput <$> arbitrary <*> arbitrary + +-- * ToADTArbitrary instances for per-constructor golden samples + +instance ToADTArbitrary (ClientMessage Tx) +instance ToADTArbitrary (TimedServerOutput Tx) +instance ToADTArbitrary InvalidInput +instance ToADTArbitrary HeadStatus +instance ToADTArbitrary FanoutProgressMode +instance ToADTArbitrary NetworkInfo +instance ToADTArbitrary (DecommitInvalidReason Tx) +instance ToADTArbitrary (ConfirmedSnapshot Tx) +instance ToADTArbitrary (PostChainTx Tx) +instance ToADTArbitrary (OnChainTx Tx) +instance ToADTArbitrary (PostTxError Tx) +instance ToADTArbitrary (ChainEvent Tx) +instance ToADTArbitrary (HeadState Tx) +instance ToADTArbitrary (SeenSnapshot Tx) +instance ToADTArbitrary (FanoutMode Tx) +instance ToADTArbitrary (NodeState Tx) +instance ToADTArbitrary SyncedStatus +instance ToADTArbitrary DepositStatus +instance ToADTArbitrary (Deposit Tx) +instance ToADTArbitrary Connectivity +instance ToADTArbitrary WhichEtcd +instance ToADTArbitrary (RequirementFailure Tx) +instance ToADTArbitrary (SideLoadRequirementFailure Tx) + +-- * Test types for the generic codec + +data GenericCBORSum + = GenNullary + | GenPositional Int Text + | GenRecord {genA :: Word64, genB :: [Int], genC :: Maybe Text} + deriving stock (Eq, Show, Generic) + +instance ToCBOR GenericCBORSum where + toCBOR = genericToCBOR + +instance FromCBOR GenericCBORSum where + fromCBOR = genericFromCBOR + +newtype GenericCBORNewtype = GenericCBORNewtype Int + deriving stock (Eq, Show, Generic) + +instance ToCBOR GenericCBORNewtype where + toCBOR = genericToCBOR + +instance FromCBOR GenericCBORNewtype where + fromCBOR = genericFromCBOR + +spec :: Spec +spec = parallel $ do + describe "genericToCBOR / genericFromCBOR" $ do + it "encodes a nullary constructor as just its name tag" $ + serialize' GenNullary `shouldBe` serialize' ("GenNullary" :: Text) + + it "encodes the name tag followed by fields in declaration order" $ + serialize' (GenPositional 42 "hi") + `shouldBe` toStrictByteString + (toCBOR ("GenPositional" :: Text) <> toCBOR (42 :: Int) <> toCBOR ("hi" :: Text)) + + it "encodes record fields in declaration order" $ + serialize' GenRecord{genA = 7, genB = [1, 2], genC = Just "x"} + `shouldBe` toStrictByteString + ( toCBOR ("GenRecord" :: Text) + <> toCBOR (7 :: Word64) + <> toCBOR ([1, 2] :: [Int]) + <> toCBOR (Just ("x" :: Text)) + ) + + it "tags newtypes with their constructor name too" $ + serialize' (GenericCBORNewtype 7) + `shouldBe` toStrictByteString (toCBOR ("GenericCBORNewtype" :: Text) <> toCBOR (7 :: Int)) + + it "roundtrips all constructor shapes" $ do + decodeFull' (serialize' GenNullary) `shouldBe` Right GenNullary + decodeFull' (serialize' (GenPositional 42 "hi")) `shouldBe` Right (GenPositional 42 "hi") + decodeFull' (serialize' GenRecord{genA = 7, genB = [1, 2], genC = Nothing}) + `shouldBe` Right GenRecord{genA = 7, genB = [1, 2], genC = Nothing} + decodeFull' (serialize' (GenericCBORNewtype 7)) `shouldBe` Right (GenericCBORNewtype 7) + + it "fails decoding an unknown tag naming the type" $ + case decodeFull' @GenericCBORSum (serialize' ("Bogus" :: Text)) of + Left err -> show err `shouldContain` "is not a proper CBOR-encoded GenericCBORSum" + Right v -> expectationFailure $ "unexpectedly decoded: " <> show v + + describe "API types" $ do + roundtripCBOR $ Proxy @(ClientInput Tx) + roundtripCBOR $ Proxy @(ServerOutput Tx) + roundtripCBOR $ Proxy @(TimedServerOutput Tx) + roundtripCBOR $ Proxy @(ClientMessage Tx) + roundtripCBOR $ Proxy @(Greetings Tx) + roundtripCBOR $ Proxy @InvalidInput + roundtripCBOR $ Proxy @HeadStatus + roundtripCBOR $ Proxy @NetworkInfo + + describe "protocol types" $ do + roundtripCBOR $ Proxy @(Snapshot Tx) + roundtripCBOR $ Proxy @(ConfirmedSnapshot Tx) + roundtripCBOR $ Proxy @(PostChainTx Tx) + roundtripCBOR $ Proxy @(OnChainTx Tx) + roundtripCBOR $ Proxy @(PostTxError Tx) + roundtripCBOR $ Proxy @(ChainEvent Tx) + roundtripCBOR $ Proxy @ChainStateAt + roundtripCBOR $ Proxy @(HeadState Tx) + roundtripCBOR $ Proxy @(SeenSnapshot Tx) + roundtripCBOR $ Proxy @(NodeState Tx) + roundtripCBOR $ Proxy @(Deposit Tx) + roundtripCBOR $ Proxy @Environment + roundtripCBOR $ Proxy @Connectivity + + describe "persisted types" $ do + roundtripCBOR $ Proxy @(StateChanged Tx) + roundtripCBOR $ Proxy @(StateEvent Tx) + -- Locks the on-disk format of hydra.db events: one sample per + -- 'StateChanged' constructor, stored as raw CBOR. + goldenCBOR "StateEvent Tx" "golden/StateEvent.cbor" genGoldenStateEvents + + -- One golden per CBOR-encoded type, one sample per constructor. Locks the + -- byte-level formats of the hydra.db event payloads and the API/network + -- wire messages. + describe "golden formats" $ do + goldenCBOR "ClientInput Tx" "golden/ClientInput.cbor" (genGoldenSamples @(ClientInput Tx)) + goldenCBOR "ServerOutput Tx" "golden/ServerOutput.cbor" (genGoldenSamples @(ServerOutput Tx)) + goldenCBOR "TimedServerOutput Tx" "golden/TimedServerOutput.cbor" (genGoldenSamples @(TimedServerOutput Tx)) + goldenCBOR "ClientMessage Tx" "golden/ClientMessage.cbor" (genGoldenSamples @(ClientMessage Tx)) + goldenCBOR "Greetings Tx" "golden/Greetings.cbor" (genGoldenSamples @(Greetings Tx)) + goldenCBOR "InvalidInput" "golden/InvalidInput.cbor" (genGoldenSamples @InvalidInput) + goldenCBOR "HeadStatus" "golden/HeadStatus.cbor" (genGoldenSamples @HeadStatus) + goldenCBOR "FanoutProgressMode" "golden/FanoutProgressMode.cbor" (genGoldenSamples @FanoutProgressMode) + goldenCBOR "NetworkInfo" "golden/NetworkInfo.cbor" (genGoldenSamples @NetworkInfo) + goldenCBOR "DecommitInvalidReason Tx" "golden/DecommitInvalidReason.cbor" (genGoldenSamples @(DecommitInvalidReason Tx)) + goldenCBOR "Message Tx" "golden/Message.cbor" (genGoldenSamples @(Message Tx)) + goldenCBOR "ConfirmedSnapshot Tx" "golden/ConfirmedSnapshot.cbor" (genGoldenSamples @(ConfirmedSnapshot Tx)) + goldenCBOR "PostChainTx Tx" "golden/PostChainTx.cbor" (genGoldenSamples @(PostChainTx Tx)) + goldenCBOR "OnChainTx Tx" "golden/OnChainTx.cbor" (genGoldenSamples @(OnChainTx Tx)) + goldenCBOR "PostTxError Tx" "golden/PostTxError.cbor" (genGoldenSamples @(PostTxError Tx)) + goldenCBOR "ChainEvent Tx" "golden/ChainEvent.cbor" (genGoldenSamples @(ChainEvent Tx)) + goldenCBOR "HeadState Tx" "golden/HeadState.cbor" (genGoldenSamples @(HeadState Tx)) + goldenCBOR "SeenSnapshot Tx" "golden/SeenSnapshot.cbor" (genGoldenSamples @(SeenSnapshot Tx)) + goldenCBOR "FanoutMode Tx" "golden/FanoutMode.cbor" (genGoldenSamples @(FanoutMode Tx)) + goldenCBOR "NodeState Tx" "golden/NodeState.cbor" (genGoldenSamples @(NodeState Tx)) + goldenCBOR "SyncedStatus" "golden/SyncedStatus.cbor" (genGoldenSamples @SyncedStatus) + goldenCBOR "DepositStatus" "golden/DepositStatus.cbor" (genGoldenSamples @DepositStatus) + goldenCBOR "Deposit Tx" "golden/Deposit.cbor" (genGoldenSamples @(Deposit Tx)) + goldenCBOR "Connectivity" "golden/Connectivity.cbor" (genGoldenSamples @Connectivity) + goldenCBOR "WhichEtcd" "golden/WhichEtcd.cbor" (genGoldenSamples @WhichEtcd) + goldenCBOR "RequirementFailure Tx" "golden/RequirementFailure.cbor" (genGoldenSamples @(RequirementFailure Tx)) + goldenCBOR "SideLoadRequirementFailure Tx" "golden/SideLoadRequirementFailure.cbor" (genGoldenSamples @(SideLoadRequirementFailure Tx)) + +-- | One 'StateEvent' per 'StateChanged' constructor, in declaration order: +-- 'ToADTArbitrary' enumerates the constructors generically, so coverage of +-- every constructor holds by construction and new constructors are included +-- automatically. Samples are generated small (resized): the golden file +-- locks tags and field order, which small values exercise just as well. +genGoldenStateEvents :: Gen [StateEvent Tx] +genGoldenStateEvents = do + ADTArbitrary{adtCAPs} <- resize 5 $ toADTArbitrary (Proxy @(StateChanged Tx)) + forM (zip [0 ..] adtCAPs) $ \(i, ConstructorArbitraryPair{capArbitrary}) -> + StateEvent i capArbitrary <$> arbitrary diff --git a/hydra-node/test/Hydra/Events/SQLiteBasedSpec.hs b/hydra-node/test/Hydra/Events/SQLiteBasedSpec.hs index 449d2085e04..3e57e891c28 100644 --- a/hydra-node/test/Hydra/Events/SQLiteBasedSpec.hs +++ b/hydra-node/test/Hydra/Events/SQLiteBasedSpec.hs @@ -4,22 +4,24 @@ module Hydra.Events.SQLiteBasedSpec where import Hydra.Prelude hiding (label) import Test.Hydra.Prelude +import Cardano.Binary (decodeFull') import Data.Aeson qualified as Aeson import Data.ByteString qualified as BS import Data.List (zipWith3) import Data.List qualified as List -import Database.SQLite.Simple (close, execute, execute_, open) +import Database.SQLite.Simple (Only (..), close, execute, execute_, open, query) import Hydra.Events (EventSink (..), EventSource (..), getEvents, putEvent) import Hydra.Events.Rotation (EventStore (..)) import Hydra.Events.SQLiteBased (EventDecodingException, SQLiteLog (..), getSchemaVersion, nextVersion, withSQLiteEventStore) +import Hydra.HeadLogic.Outcome (StateChanged) import Hydra.HeadLogic.StateEvent (StateEvent (..)) import Hydra.Ledger.Simple (SimpleTx) import Hydra.Logging (Envelope (..), nullTracer) -import System.Directory (doesFileExist) +import System.Directory (doesFileExist, getFileSize) import Test.Hydra.Chain.Direct.State () import Test.Hydra.HeadLogic.StateEvent () import Test.Hydra.Ledger.Simple () -import Test.QuickCheck (forAllShrink, generate, ioProperty, sublistOf, suchThat, (===)) +import Test.QuickCheck (forAllShrink, generate, ioProperty, sublistOf, suchThat, vectorOf, (===)) import Test.QuickCheck.Gen (listOf) import Test.Util (captureTracer) @@ -71,9 +73,9 @@ spec = do let dbFile = tmpDir <> "/hydra.db" stateFile = tmpDir <> "/state" withSQLiteEventStore @(StateEvent SimpleTx) nullTracer dbFile stateFile $ \store -> do - -- Insert a row with invalid JSON directly via a separate connection + -- Insert a row with undecodable data directly via a separate connection bracket (open dbFile) close $ \conn -> - execute conn "INSERT INTO events (event_id, event_data) VALUES (?, ?)" (1 :: Word64, "not valid json" :: ByteString) + execute conn "INSERT INTO events (event_id, event_data) VALUES (?, ?)" (1 :: Word64, "not valid cbor" :: ByteString) getEvents (eventSource store) `shouldThrow` \(_ :: EventDecodingException) -> True @@ -112,6 +114,64 @@ spec = do withSQLiteEventStore @(StateEvent SimpleTx) nullTracer dbFile stateFile (\_ -> pure ()) `shouldThrow` anyErrorCall + it "migrates a v1 JSON database to v2 CBOR" $ do + withTempDir "hydra-sqlite-persistence" $ \tmpDir -> do + let dbFile = tmpDir <> "/hydra.db" + stateFile = tmpDir <> "/state" + events <- generate $ mkContinuousEvents <$> vectorOf 300 arbitrary <*> vectorOf 300 arbitrary + -- Hand-create a version 1 database with JSON-encoded rows, as written + -- by hydra-node versions before the CBOR switch. + bracket (open dbFile) close $ \conn -> do + execute_ conn "CREATE TABLE events (event_id INTEGER NOT NULL PRIMARY KEY, event_data BLOB NOT NULL)" + execute_ conn "PRAGMA user_version = 1" + forM_ events $ \e -> + execute conn "INSERT INTO events (event_id, event_data) VALUES (?, ?)" (eventId e, toStrict (Aeson.encode e)) + sizeBefore <- getFileSize dbFile + withSQLiteEventStore @(StateEvent SimpleTx) nullTracer dbFile stateFile $ \store -> do + loadedEvents <- getEvents (eventSource store) + loadedEvents `shouldBe` events + v <- bracket (open dbFile) close getSchemaVersion + v `shouldBe` nextVersion + -- Random events from the original JSON database must be found in the + -- migrated database under the same event_id, with the row blob + -- decoding (as CBOR) to exactly the event that was inserted. + picked <- generate $ sublistOf events `suchThat` (not . null) + bracket (open dbFile) close $ \conn -> + forM_ picked $ \e -> do + rows :: [Only ByteString] <- + query conn "SELECT event_data FROM events WHERE event_id = ?" (Only (eventId e)) + case rows of + [Only bytes] -> + case decodeFull' bytes of + Left err -> + expectationFailure $ + "failed to decode migrated row " <> show (eventId e) <> " as CBOR: " <> show err + Right (decoded :: StateEvent SimpleTx) -> decoded `shouldBe` e + _ -> + expectationFailure $ + "expected exactly one row for event_id " <> show (eventId e) <> ", got " <> show (length rows) + -- Re-encoding + VACUUM must not grow the database; with a few hundred + -- events CBOR is strictly smaller, but we only assert non-growth to + -- keep this stable for edge cases. + sizeAfter <- getFileSize dbFile + sizeAfter `shouldSatisfy` (<= sizeBefore) + + it "aborts migration and keeps v1 intact on a corrupt row" $ do + withTempDir "hydra-sqlite-persistence" $ \tmpDir -> do + let dbFile = tmpDir <> "/hydra.db" + stateFile = tmpDir <> "/state" + goodEvent :: StateEvent SimpleTx <- generate arbitrary + bracket (open dbFile) close $ \conn -> do + execute_ conn "CREATE TABLE events (event_id INTEGER NOT NULL PRIMARY KEY, event_data BLOB NOT NULL)" + execute_ conn "PRAGMA user_version = 1" + execute conn "INSERT INTO events (event_id, event_data) VALUES (?, ?)" (1 :: Word64, toStrict (Aeson.encode goodEvent)) + execute conn "INSERT INTO events (event_id, event_data) VALUES (?, ?)" (2 :: Word64, "not valid json" :: ByteString) + withSQLiteEventStore @(StateEvent SimpleTx) nullTracer dbFile stateFile (\_ -> pure ()) + `shouldThrow` \(_ :: EventDecodingException) -> True + -- The failed migration must roll back: still version 1, rows untouched. + v <- bracket (open dbFile) close getSchemaVersion + v `shouldBe` 1 + prop "can migrate from file-based store" $ forAllShrink genContinuousEvents shrink $ \events -> ioProperty $ do @@ -157,7 +217,10 @@ spec = do genContinuousEvents :: Gen [StateEvent SimpleTx] genContinuousEvents = - zipWith3 StateEvent [0 ..] <$> listOf arbitrary <*> listOf arbitrary + mkContinuousEvents <$> listOf arbitrary <*> listOf arbitrary + +mkContinuousEvents :: [StateChanged SimpleTx] -> [UTCTime] -> [StateEvent SimpleTx] +mkContinuousEvents = zipWith3 StateEvent [0 ..] withEventSourceAndSink :: (EventSource (StateEvent SimpleTx) IO -> EventSink (StateEvent SimpleTx) IO -> IO b) -> IO b withEventSourceAndSink action = diff --git a/hydra-node/test/Main.hs b/hydra-node/test/Main.hs index 6940db3ef84..e81530f185f 100644 --- a/hydra-node/test/Main.hs +++ b/hydra-node/test/Main.hs @@ -7,6 +7,7 @@ import Hydra.API.HTTPServerSpec qualified import Hydra.API.ServerOutputSpec qualified import Hydra.API.ServerSpec qualified import Hydra.BehaviorSpec qualified +import Hydra.CBORSpec qualified import Hydra.Chain.BlockfrostSpec qualified import Hydra.Chain.Direct.HandlersSpec qualified import Hydra.Chain.Direct.ScriptRegistrySpec qualified @@ -56,6 +57,7 @@ main = , testSpec "API.ServerOutput" Hydra.API.ServerOutputSpec.spec , testSpec "API.Server" Hydra.API.ServerSpec.spec , testSpec "Behavior" Hydra.BehaviorSpec.spec + , testSpec "CBOR" Hydra.CBORSpec.spec , testSpec "Chain.Blockfrost" Hydra.Chain.BlockfrostSpec.spec , testSpec "Chain.Direct.Handlers" Hydra.Chain.Direct.HandlersSpec.spec , testSpec "Chain.Direct.ScriptRegistry" Hydra.Chain.Direct.ScriptRegistrySpec.spec diff --git a/hydra-node/testlib/Test/Hydra/CBOR.hs b/hydra-node/testlib/Test/Hydra/CBOR.hs new file mode 100644 index 00000000000..9d61d0b5f06 --- /dev/null +++ b/hydra-node/testlib/Test/Hydra/CBOR.hs @@ -0,0 +1,74 @@ +-- | Helpers to test 'ToCBOR' / 'FromCBOR' instances. +module Test.Hydra.CBOR where + +import Hydra.Prelude +import Test.Hydra.Prelude + +import Cardano.Binary (decodeFull', serialize') +import Codec.CBOR.Read (deserialiseFromBytes) +import Codec.CBOR.Write (toLazyByteString) +import Data.Typeable (typeRep) +import System.Directory (createDirectoryIfMissing, doesFileExist) +import System.FilePath (takeDirectory) +import Test.QuickCheck (Property, resize, (===)) +import Test.QuickCheck.Arbitrary.ADT (ADTArbitrary (..), ConstructorArbitraryPair (..), ToADTArbitrary, toADTArbitrary) + +-- | Test that a value can be roundtripped through its CBOR encoding. +prop_canRoundtripCBOREncoding :: + (ToCBOR a, FromCBOR a, Eq a, Show a) => a -> Property +prop_canRoundtripCBOREncoding a = + let encoded = toLazyByteString $ toCBOR a + in (snd <$> deserialiseFromBytes fromCBOR encoded) === Right a + +-- | A property spec asserting CBOR roundtrips for arbitrary values of @a@. +-- This is the CI guard that keeps the hand-written encoder/decoder pairs in +-- sync: a new constructor without a (correct) codec fails here. +roundtripCBOR :: + forall a. + (Arbitrary a, ToCBOR a, FromCBOR a, Eq a, Show a) => + Proxy a -> + Spec +roundtripCBOR p = + prop ("roundtrips CBOR encoding: " <> show (typeRep p)) $ + prop_canRoundtripCBOREncoding @a + +-- | Golden test locking a persisted CBOR format. The golden file holds the +-- raw CBOR of a list of samples. When the file is missing it is created from +-- the given generator with a fixed seed — commit the result. On every other +-- run the stored bytes must decode successfully and re-encode to the exact +-- same bytes. +-- +-- This catches codec changes that would break decoding of already persisted +-- data (e.g. hydra.db events) — including symmetric encoder+decoder drift +-- (say, reordering the fields on both sides) that roundtrip properties +-- cannot see. If this fails, the change breaks existing databases and needs +-- a schema migration; only delete and regenerate the golden file alongside +-- one. +-- | One sample per constructor of @a@, in declaration order, enumerated +-- generically by 'ToADTArbitrary': coverage of every constructor holds by +-- construction and new constructors are included automatically. Samples are +-- generated small (resized): golden files lock tags and field order, which +-- small values exercise just as well. +genGoldenSamples :: forall a. ToADTArbitrary a => Gen [a] +genGoldenSamples = do + ADTArbitrary{adtCAPs} <- resize 5 $ toADTArbitrary (Proxy @a) + pure $ capArbitrary <$> adtCAPs + +goldenCBOR :: + forall a. + (ToCBOR a, FromCBOR a) => + String -> + FilePath -> + Gen [a] -> + Spec +goldenCBOR name path gen = + it ("golden CBOR: " <> name) $ do + unlessM (doesFileExist path) $ do + createDirectoryIfMissing True (takeDirectory path) + writeFileBS path . serialize' $ generateWith gen 42 + bytes <- readFileBS path + case decodeFull' @[a] bytes of + Left err -> + expectationFailure $ + "failed to decode golden file " <> path <> ": " <> show err + Right samples -> serialize' samples `shouldBe` bytes diff --git a/hydra-node/testlib/Test/Hydra/HeadLogic/Outcome.hs b/hydra-node/testlib/Test/Hydra/HeadLogic/Outcome.hs index 93ba301ee9f..143d04a137e 100644 --- a/hydra-node/testlib/Test/Hydra/HeadLogic/Outcome.hs +++ b/hydra-node/testlib/Test/Hydra/HeadLogic/Outcome.hs @@ -6,12 +6,14 @@ module Test.Hydra.HeadLogic.Outcome where import Hydra.Prelude import Test.Hydra.Prelude -import Hydra.Chain.ChainState (ChainStateType (..), IsChainState) +import Hydra.Chain.ChainState (ChainPointType (..), ChainStateType (..), IsChainState) import Hydra.HeadLogic.Outcome (StateChanged (..)) import Hydra.Node.Environment (Environment (..), mkHeadParameters) import Test.Hydra.API.ServerOutput () import Test.Hydra.Chain () import Test.Hydra.HeadLogic.State () +import Test.Hydra.Network () +import Test.Hydra.Node.State () import Test.Hydra.Tx.Gen (ArbitraryIsTx) import Test.QuickCheck (oneof) import Test.QuickCheck.Arbitrary.ADT (ToADTArbitrary) diff --git a/hydra-prelude/hydra-prelude.cabal b/hydra-prelude/hydra-prelude.cabal index 44e49e4a76f..9ab4eedf6ad 100644 --- a/hydra-prelude/hydra-prelude.cabal +++ b/hydra-prelude/hydra-prelude.cabal @@ -17,6 +17,7 @@ library hs-source-dirs: src c-sources: cbits/revision.c exposed-modules: + Hydra.CBOR.Generic Hydra.Prelude Hydra.Version @@ -31,6 +32,7 @@ library , io-classes:si-timers , pretty-simple , relude + , time , transformers default-language: GHC2021 diff --git a/hydra-prelude/src/Hydra/CBOR/Generic.hs b/hydra-prelude/src/Hydra/CBOR/Generic.hs new file mode 100644 index 00000000000..c26e2163a70 --- /dev/null +++ b/hydra-prelude/src/Hydra/CBOR/Generic.hs @@ -0,0 +1,133 @@ +{-# LANGUAGE DataKinds #-} + +-- | Generic derivation of 'ToCBOR' / 'FromCBOR' instances in the +-- constructor-name-tagged format used for hydra-node persistence and API +-- messages. +-- +-- The encoding of a value is the constructor name as a CBOR text string, +-- followed by the encodings of the constructor fields in declaration order, +-- concatenated without any list framing: +-- +-- @ +-- toCBOR ("ConstructorName" :: Text) <> toCBOR field1 <> toCBOR field2 <> ... +-- @ +-- +-- Every constructor is tagged, including the single constructor of records +-- and newtypes. Tagging by name (instead of by declaration index like the +-- @serialise@ package) means adding or reordering constructors does not +-- change the encoding of existing data; removing or renaming a constructor, +-- or changing the order or type of its fields, does and requires a migration +-- of persisted data. +-- +-- Since the field encodings come from the data type declaration, the +-- declaration itself becomes the wire format: reordering record fields is a +-- format change that the type checker will not flag. Golden tests (see +-- 'Hydra.CBORSpec' in hydra-node) are the guard against that. +-- +-- Decoding fails with @"\" is not a proper CBOR-encoded \@ +-- when the decoded tag matches no constructor. +module Hydra.CBOR.Generic ( + genericToCBOR, + genericFromCBOR, +) where + +import Relude + +import Cardano.Binary (Decoder, Encoding, FromCBOR (..), ToCBOR (..)) +import GHC.Generics +import GHC.TypeLits (KnownSymbol, symbolVal) + +-- | Encode a value in the constructor-name-tagged CBOR format: +-- +-- @ +-- instance ToCBOR MyType where +-- toCBOR = genericToCBOR +-- @ +genericToCBOR :: (Generic a, GToCBOR (Rep a)) => a -> Encoding +genericToCBOR = gToCBOR . from + +-- | Decode a value from the constructor-name-tagged CBOR format produced by +-- 'genericToCBOR': +-- +-- @ +-- instance FromCBOR MyType where +-- fromCBOR = genericFromCBOR +-- @ +genericFromCBOR :: (Generic a, GFromCBOR (Rep a)) => Decoder s a +genericFromCBOR = to <$> gFromCBOR + +-- * Encoding + +class GToCBOR f where + gToCBOR :: f p -> Encoding + +instance GToCBOR f => GToCBOR (D1 meta f) where + gToCBOR (M1 x) = gToCBOR x + +instance (GToCBOR f, GToCBOR g) => GToCBOR (f :+: g) where + gToCBOR = \case + L1 x -> gToCBOR x + R1 x -> gToCBOR x + +instance (KnownSymbol name, GFieldsToCBOR f) => GToCBOR (C1 ('MetaCons name fixity hasSelectors) f) where + gToCBOR (M1 x) = toCBOR (conNameText (Proxy @name)) <> gFieldsToCBOR x + +class GFieldsToCBOR f where + gFieldsToCBOR :: f p -> Encoding + +instance GFieldsToCBOR U1 where + gFieldsToCBOR U1 = mempty + +instance (GFieldsToCBOR f, GFieldsToCBOR g) => GFieldsToCBOR (f :*: g) where + gFieldsToCBOR (x :*: y) = gFieldsToCBOR x <> gFieldsToCBOR y + +instance GFieldsToCBOR f => GFieldsToCBOR (S1 meta f) where + gFieldsToCBOR (M1 x) = gFieldsToCBOR x + +instance ToCBOR a => GFieldsToCBOR (K1 i a) where + gFieldsToCBOR (K1 a) = toCBOR a + +-- * Decoding + +class GFromCBOR f where + gFromCBOR :: Decoder s (f p) + +instance (KnownSymbol typeName, GConsFromCBOR f) => GFromCBOR (D1 ('MetaData typeName moduleName package isNewtype) f) where + gFromCBOR = do + tag <- fromCBOR + case gConsFromCBOR tag of + Just decodeFields -> M1 <$> decodeFields + Nothing -> + fail $ show tag <> " is not a proper CBOR-encoded " <> symbolVal (Proxy @typeName) + +class GConsFromCBOR f where + -- | The field decoder of the constructor matching the given tag, if any. + gConsFromCBOR :: Text -> Maybe (Decoder s (f p)) + +instance (GConsFromCBOR f, GConsFromCBOR g) => GConsFromCBOR (f :+: g) where + gConsFromCBOR tag = + (fmap L1 <$> gConsFromCBOR tag) <|> (fmap R1 <$> gConsFromCBOR tag) + +instance (KnownSymbol name, GFieldsFromCBOR f) => GConsFromCBOR (C1 ('MetaCons name fixity hasSelectors) f) where + gConsFromCBOR tag + | tag == conNameText (Proxy @name) = Just (M1 <$> gFieldsFromCBOR) + | otherwise = Nothing + +class GFieldsFromCBOR f where + gFieldsFromCBOR :: Decoder s (f p) + +instance GFieldsFromCBOR U1 where + gFieldsFromCBOR = pure U1 + +instance (GFieldsFromCBOR f, GFieldsFromCBOR g) => GFieldsFromCBOR (f :*: g) where + gFieldsFromCBOR = (:*:) <$> gFieldsFromCBOR <*> gFieldsFromCBOR + +instance GFieldsFromCBOR f => GFieldsFromCBOR (S1 meta f) where + gFieldsFromCBOR = M1 <$> gFieldsFromCBOR + +instance FromCBOR a => GFieldsFromCBOR (K1 i a) where + gFieldsFromCBOR = K1 <$> fromCBOR + +-- | The constructor name from generic metadata, as 'Text'. +conNameText :: KnownSymbol name => Proxy name -> Text +conNameText = toText . symbolVal diff --git a/hydra-prelude/src/Hydra/Prelude.hs b/hydra-prelude/src/Hydra/Prelude.hs index bef4ee3abf0..03e6c4805ca 100644 --- a/hydra-prelude/src/Hydra/Prelude.hs +++ b/hydra-prelude/src/Hydra/Prelude.hs @@ -1,5 +1,7 @@ -- NOTE: Usage of 'trace' in 'spy' is accepted here. {-# OPTIONS_GHC -Wno-deprecations #-} +-- NOTE: For the 'NominalDiffTime' CBOR instances below. +{-# OPTIONS_GHC -Wno-orphans #-} module Hydra.Prelude ( module Relude, @@ -28,6 +30,8 @@ module Hydra.Prelude ( elems, FromCBOR (..), ToCBOR (..), + genericToCBOR, + genericFromCBOR, FromJSON (..), ToJSON (..), encodePretty, @@ -108,7 +112,10 @@ import Data.Aeson.Encode.Pretty ( encodePretty, ) import Data.ByteString.Base16 qualified as Base16 +import Data.Fixed (Fixed (..)) import Data.Text qualified as T +import Data.Time.Clock (nominalDiffTimeToSeconds, secondsToNominalDiffTime) +import Hydra.CBOR.Generic (genericFromCBOR, genericToCBOR) import Relude hiding ( MVar, Nat, @@ -158,6 +165,15 @@ import Relude.Extra.Map ( import System.IO qualified import Text.Pretty.Simple (pShow) +-- Orphan CBOR instances: cardano-binary provides 'UTCTime' instances, but +-- lacks 'NominalDiffTime'. Encoded as integer picoseconds, which is lossless +-- since 'NominalDiffTime' has fixed picosecond resolution. +instance ToCBOR NominalDiffTime where + toCBOR ndt = case nominalDiffTimeToSeconds ndt of MkFixed i -> toCBOR i + +instance FromCBOR NominalDiffTime where + fromCBOR = secondsToNominalDiffTime . MkFixed <$> fromCBOR + -- | Pad a text-string to right with the given character until it reaches the given -- length. -- diff --git a/hydra-tx/src/Hydra/Chain/ChainState.hs b/hydra-tx/src/Hydra/Chain/ChainState.hs index 83f06743026..8b39e1e6345 100644 --- a/hydra-tx/src/Hydra/Chain/ChainState.hs +++ b/hydra-tx/src/Hydra/Chain/ChainState.hs @@ -9,7 +9,7 @@ import Hydra.Tx (IsTx (..)) -- | A generic description for a chain slot all implementations need to use. newtype ChainSlot = ChainSlot Natural deriving stock (Ord, Eq, Show, Generic) - deriving newtype (Num, ToJSON, FromJSON) + deriving newtype (Num, ToJSON, FromJSON, ToCBOR, FromCBOR) -- | Types that can be used on-chain by the Hydra protocol. This includes the -- information that needs to be retained about the L1 state to interact with @@ -23,10 +23,14 @@ class , Show (ChainPointType tx) , FromJSON (ChainPointType tx) , ToJSON (ChainPointType tx) + , FromCBOR (ChainPointType tx) + , ToCBOR (ChainPointType tx) , Eq (ChainStateType tx) , Show (ChainStateType tx) , FromJSON (ChainStateType tx) , ToJSON (ChainStateType tx) + , FromCBOR (ChainStateType tx) + , ToCBOR (ChainStateType tx) ) => IsChainState tx where diff --git a/hydra-tx/src/Hydra/Tx/ContestationPeriod.hs b/hydra-tx/src/Hydra/Tx/ContestationPeriod.hs index f36e7458723..d69a98d2003 100644 --- a/hydra-tx/src/Hydra/Tx/ContestationPeriod.hs +++ b/hydra-tx/src/Hydra/Tx/ContestationPeriod.hs @@ -13,7 +13,7 @@ import Text.Show (Show (..)) -- values of unknown sign. newtype ContestationPeriod = UnsafeContestationPeriod Natural deriving stock (Eq, Ord) - deriving newtype (Real, Integral, ToJSON, FromJSON) + deriving newtype (Real, Integral, ToJSON, FromJSON, ToCBOR, FromCBOR) instance Show ContestationPeriod where show (UnsafeContestationPeriod s) = show s <> "s" diff --git a/hydra-tx/src/Hydra/Tx/Crypto.hs b/hydra-tx/src/Hydra/Tx/Crypto.hs index 0548956859c..827f9b99d04 100644 --- a/hydra-tx/src/Hydra/Tx/Crypto.hs +++ b/hydra-tx/src/Hydra/Tx/Crypto.hs @@ -363,7 +363,7 @@ verify (HydraVerificationKey vk) (HydraSignature sig) a = -- | Naiively aggregated multi-signatures. newtype MultiSignature a = HydraMultiSignature {multiSignature :: [Signature a]} deriving stock (Eq, Show, Generic) - deriving newtype (Semigroup, Monoid) + deriving newtype (Semigroup, Monoid, ToCBOR, FromCBOR) deriving anyclass instance ToJSON a => ToJSON (MultiSignature a) deriving anyclass instance FromJSON a => FromJSON (MultiSignature a) diff --git a/hydra-tx/src/Hydra/Tx/DepositPeriod.hs b/hydra-tx/src/Hydra/Tx/DepositPeriod.hs index 24820a10b2c..def9a75b6be 100644 --- a/hydra-tx/src/Hydra/Tx/DepositPeriod.hs +++ b/hydra-tx/src/Hydra/Tx/DepositPeriod.hs @@ -9,7 +9,7 @@ import Text.Show (Show (..)) -- Nodes within the same Head must configure identical values. newtype DepositPeriod = DepositPeriod {toNominalDiffTime :: NominalDiffTime} deriving stock (Eq, Ord) - deriving newtype (Read, Num, Real, ToJSON, FromJSON) + deriving newtype (Read, Num, Real, ToJSON, FromJSON, ToCBOR, FromCBOR) instance Show DepositPeriod where show (DepositPeriod dt) = show (round dt :: Integer) <> "s" diff --git a/hydra-tx/src/Hydra/Tx/HeadId.hs b/hydra-tx/src/Hydra/Tx/HeadId.hs index 6072738d5ed..942aea15033 100644 --- a/hydra-tx/src/Hydra/Tx/HeadId.hs +++ b/hydra-tx/src/Hydra/Tx/HeadId.hs @@ -21,6 +21,7 @@ import PlutusLedgerApi.V3 (CurrencySymbol (..), toBuiltin) newtype HeadId = UnsafeHeadId ByteString deriving stock (Show, Eq, Ord, Generic) deriving (ToJSON, FromJSON) via (UsingRawBytesHex HeadId) + deriving newtype (ToCBOR, FromCBOR) instance SerialiseAsRawBytes HeadId where serialiseToRawBytes (UnsafeHeadId bytes) = bytes @@ -53,6 +54,7 @@ mkHeadId = UnsafeHeadId . serialiseToRawBytes newtype HeadSeed = UnsafeHeadSeed ByteString deriving stock (Show, Eq, Ord, Generic) deriving (ToJSON, FromJSON) via (UsingRawBytesHex HeadSeed) + deriving newtype (ToCBOR, FromCBOR) instance IsString HeadSeed where fromString = UnsafeHeadSeed . fromString diff --git a/hydra-tx/src/Hydra/Tx/HeadParameters.hs b/hydra-tx/src/Hydra/Tx/HeadParameters.hs index 89cc56c1569..f9ee6cb6d74 100644 --- a/hydra-tx/src/Hydra/Tx/HeadParameters.hs +++ b/hydra-tx/src/Hydra/Tx/HeadParameters.hs @@ -14,3 +14,10 @@ data HeadParameters = HeadParameters } deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) + +instance ToCBOR HeadParameters where + toCBOR HeadParameters{contestationPeriod, depositPeriod, parties} = + toCBOR contestationPeriod <> toCBOR depositPeriod <> toCBOR parties + +instance FromCBOR HeadParameters where + fromCBOR = HeadParameters <$> fromCBOR <*> fromCBOR <*> fromCBOR diff --git a/hydra-tx/src/Hydra/Tx/OnChainId.hs b/hydra-tx/src/Hydra/Tx/OnChainId.hs index a8e30527201..e0b9893fbaf 100644 --- a/hydra-tx/src/Hydra/Tx/OnChainId.hs +++ b/hydra-tx/src/Hydra/Tx/OnChainId.hs @@ -16,6 +16,7 @@ import Hydra.Cardano.Api ( newtype OnChainId = UnsafeOnChainId ByteString deriving stock (Show, Eq, Ord, Generic) deriving (ToJSON, FromJSON) via (UsingRawBytesHex OnChainId) + deriving newtype (ToCBOR, FromCBOR) instance SerialiseAsRawBytes OnChainId where serialiseToRawBytes (UnsafeOnChainId bytes) = bytes diff --git a/hydra-tx/src/Hydra/Tx/Snapshot.hs b/hydra-tx/src/Hydra/Tx/Snapshot.hs index 721c9fd61aa..00396b18328 100644 --- a/hydra-tx/src/Hydra/Tx/Snapshot.hs +++ b/hydra-tx/src/Hydra/Tx/Snapshot.hs @@ -134,6 +134,30 @@ instance IsTx tx => FromJSON (Snapshot tx) where let accumulator = Accumulator.buildFromSnapshotUTxOs utxo utxoToCommit utxoToDecommit pure $ Snapshot{headId, version, number, confirmed, utxo, utxoToCommit, utxoToDecommit, accumulator} +-- NOTE: Like the JSON encoding, the accumulator is not transmitted (only +-- derived data) and gets rebuilt from the UTxO sets on decode. +instance IsTx tx => ToCBOR (Snapshot tx) where + toCBOR Snapshot{headId, version, number, confirmed, utxo, utxoToCommit, utxoToDecommit} = + toCBOR headId + <> toCBOR version + <> toCBOR number + <> toCBOR confirmed + <> toCBOR utxo + <> toCBOR utxoToCommit + <> toCBOR utxoToDecommit + +instance IsTx tx => FromCBOR (Snapshot tx) where + fromCBOR = do + headId <- fromCBOR + version <- fromCBOR + number <- fromCBOR + confirmed <- fromCBOR + utxo <- fromCBOR + utxoToCommit <- fromCBOR + utxoToDecommit <- fromCBOR + let accumulator = Accumulator.buildFromSnapshotUTxOs @tx utxo utxoToCommit utxoToDecommit + pure Snapshot{headId, version, number, confirmed, utxo, utxoToCommit, utxoToDecommit, accumulator} + -- | All UTxOs represented by this snapshot: settled plus any pending commit/decommit. snapshotUTxO :: IsTx tx => Snapshot tx -> UTxOType tx snapshotUTxO Snapshot{utxo, utxoToCommit, utxoToDecommit} = @@ -155,6 +179,12 @@ data ConfirmedSnapshot tx deriving stock (Generic, Eq, Show) deriving anyclass (ToJSON, FromJSON) +instance IsTx tx => ToCBOR (ConfirmedSnapshot tx) where + toCBOR = genericToCBOR + +instance IsTx tx => FromCBOR (ConfirmedSnapshot tx) where + fromCBOR = genericFromCBOR + -- | Safely get a 'Snapshot' from a confirmed snapshot. -- -- NOTE: While we could use 'snapshot' directly, this is a record-field accessor diff --git a/typos.toml b/typos.toml index d4607882b62..211e6c8e0fe 100644 --- a/typos.toml +++ b/typos.toml @@ -9,6 +9,8 @@ "Compactible" = "Compactible" # 2[nd] ordinal suffix macro in spec/src/macros.tex "nd" = "nd" +# GHC.Type[Lits] module name +"Lits" = "Lits" [default] # Git commit hashes in docs get tokenized into letter runs (e.g. "ba" in