CBOR-1: Switch to cbor format in DB

CHANGELOG.md

@@ -88,6 +88,16 @@ changes.
   observed a block (so a stalled chain backend can be detected even while drift is
   frozen). [#2749](https://github.com/cardano-scaling/hydra/issues/2749)
 
+- **BREAKING**: Event payloads in the `hydra.db` persistence database are now
+  CBOR-encoded instead of JSON, reducing disk usage and speeding up event
+  append and replay. Existing databases are migrated automatically on first
+  start (one-shot re-encode of all rows followed by `VACUUM`); a corrupt row
+  aborts startup and leaves the database untouched. After migration, older
+  hydra-node versions refuse to open the database — there is no downgrade
+  path.
+
+## [2.3.0] - 2026.07.15
+
 - Add **selective partial fanout**: distribute a chosen subset of a closed
   head's UTxO instead of draining it all at once. Introduces the `PartialFanout`
   client input, the `HeadPartiallyFannedOut` server output (with a `fanoutMode`

docs/docs/dev/architecture/event-sourcing.md

@@ -21,7 +21,18 @@ The database contains a single `events` table:
 | Column       | Type    | Description                              |
 |-------------|---------|------------------------------------------|
 | `event_id`  | INTEGER | Primary key, matches the in-memory event id |
-| `event_data` | BLOB   | JSON-encoded event payload               |
+| `event_data` | BLOB   | CBOR-encoded event payload (`ToCBOR`/`FromCBOR`) |
+
+Event payloads use a constructor-name-tagged CBOR format: the constructor name
+as a CBOR text string, followed by the constructor fields in declaration order.
+Most instances are derived generically via `genericToCBOR` / `genericFromCBOR`
+from `Hydra.CBOR.Generic` (hydra-prelude), which makes the data type
+declaration itself the on-disk format: changing the order or types of an
+existing constructor's fields is a breaking change that requires a schema
+migration, while adding, removing or reordering constructors keeps existing
+data decodable thanks to the name tags. The golden tests in `Hydra.CBORSpec`
+lock the concrete bytes per constructor and fail on any accidental format
+change.
 
 The following connection pragmas are set on every open:
 
@@ -39,16 +50,18 @@ The last-seen event id is updated atomically at enqueue time (not write time), s
 
 #### Schema versioning
 
-The database schema is versioned using SQLite's built-in `PRAGMA user_version`. On startup, `initSchema` reads the current version and applies any pending migration steps incrementally up to `nextVersion`. Each step is defined as a case in `migrateStep`:
+The database schema is versioned using SQLite's built-in `PRAGMA user_version`. On startup, `initSchema` reads the current version and applies any pending migration steps incrementally up to `nextVersion`. Each step runs together with its version bump in a single transaction, so a crash or failure mid-migration rolls back to a well-defined version. The steps are defined as cases in `migrateStep`:
 
 ```haskell
-migrateStep conn = \case
+migrateStep conn reencodeRow = \case
   0 -> -- create the events table
-  1 -> -- (future) e.g. add an index or new column
+  1 -> -- re-encode all JSON event payloads to CBOR
 ```
 
 A fresh database starts at version 0 (SQLite default). After all migrations run, `user_version` is set to `nextVersion`. If the database has a version higher than `nextVersion` (e.g. from a newer release), the node refuses to start to prevent silent data corruption on downgrade.
 
+Version 1 databases (written by earlier hydra-node releases) store event payloads as JSON. Opening one automatically re-encodes every row to CBOR and runs `VACUUM` afterwards to reclaim the freed space. A row that fails to decode aborts the migration — and thereby node startup — leaving the database untouched at version 1, still usable by the previous hydra-node version. Note that after a successful migration there is no downgrade path: older releases refuse to open a version 2 database. The legacy `state` file migration also inserts CBOR (the file itself remains JSON lines).
+
 To add a new migration:
 
 1. Bump `nextVersion` in `Hydra.Events.SQLiteBased`

head-state-viewer/head-state-viewer.cabal

@@ -57,6 +57,7 @@ library
     , aeson
     , base
     , bytestring
+    , cardano-binary
     , containers
     , hydra-node
     , hydra-prelude

head-state-viewer/src/HydraVis/History.hs

@@ -6,8 +6,10 @@
 -- The Hydra node persists events using the schema defined by
 -- "Hydra.Events.SQLiteBased": a single @events@ table with
 -- @event_id INTEGER PRIMARY KEY@ and a @event_data BLOB@ column that holds a
--- JSON-encoded @StateEvent tx@. We open the database read-only (so a running
--- node can keep writing) and replay the events in event-id order.
+-- CBOR-encoded @StateEvent tx@ (JSON in version 1 databases; we decode CBOR
+-- first and fall back to JSON so pre-migration databases and archives still
+-- open). We open the database read-only (so a running node can keep writing)
+-- and replay the events in event-id order.
 module HydraVis.History (
   HistoryStep (..),
   loadHistoryFor,
@@ -18,6 +20,7 @@ module HydraVis.History (
 
 import Hydra.Prelude
 
+import Cardano.Binary (decodeFull')
 import Data.Aeson qualified as Aeson
 import Data.Text.Encoding qualified as TE
 import Database.SQLite.Simple (Only (..), SQLData (..), open, query, query_)
@@ -50,7 +53,7 @@ deriving stock instance (IsChainState tx, Show (NodeState tx)) => Show (HistoryS
 -- recover. If the file does not exist the underlying 'open' call throws.
 loadHistoryFor ::
   forall tx.
-  (FromJSON (StateEvent tx), IsChainState tx) =>
+  (FromCBOR (StateEvent tx), FromJSON (StateEvent tx), IsChainState tx) =>
   NodeState tx ->
   FilePath ->
   IO [HistoryStep tx]
@@ -62,13 +65,29 @@ loadHistoryFor initial path = do
       "SELECT event_data FROM events ORDER BY event_id ASC" ::
       IO [Only EventBlob]
   SQL.close conn
-  events <- forM rows $ \(Only (EventBlob blob)) ->
-    case Aeson.eitherDecodeStrict' blob of
+  events <- forM rows $ \(Only (EventBlob blob)) -> decodeEventBlob path blob
+  pure (buildHistory initial events)
+
+-- | Decode an event payload: CBOR (the current production format) first,
+-- falling back to JSON for version 1 databases and pre-rotation archives.
+decodeEventBlob ::
+  (FromCBOR (StateEvent tx), FromJSON (StateEvent tx)) =>
+  FilePath ->
+  ByteString ->
+  IO (StateEvent tx)
+decodeEventBlob path blob =
+  case decodeFull' blob of
+    Right e -> pure e
+    Left cborErr -> case Aeson.eitherDecodeStrict' blob of
       Right e -> pure e
-      Left err ->
+      Left jsonErr ->
         fail $
-          "HydraVis.History: failed to decode event in " <> path <> ": " <> err
-  pure (buildHistory initial events)
+          "HydraVis.History: failed to decode event in "
+            <> path
+            <> ": CBOR: "
+            <> show cborErr
+            <> "; JSON: "
+            <> jsonErr
 
 -- | Wrapper that accepts both BLOB (the production format) and TEXT (what
 -- a quick @sqlite3 CLI@ insert produces) columns when reading event payloads.
@@ -98,7 +117,7 @@ buildHistory initial events =
 -- Used by the follow loop to fetch only newly persisted rows on each poll.
 loadEventsAfter ::
   forall tx.
-  FromJSON (StateEvent tx) =>
+  (FromCBOR (StateEvent tx), FromJSON (StateEvent tx)) =>
   FilePath ->
   Maybe EventId ->
   IO [StateEvent tx]
@@ -117,12 +136,7 @@ loadEventsAfter path lastSeen = do
         (Only eid) ::
         IO [Only EventBlob]
   SQL.close conn
-  forM rows $ \(Only (EventBlob blob)) ->
-    case Aeson.eitherDecodeStrict' blob of
-      Right e -> pure e
-      Left err ->
-        fail $
-          "HydraVis.History: failed to decode event in " <> path <> ": " <> err
+  forM rows $ \(Only (EventBlob blob)) -> decodeEventBlob path blob
 
 -- | Extend an existing history with newly observed events, folding them
 -- through 'aggregateNodeState' starting from the final state of the existing

head-state-viewer/src/HydraVis/SampleDb.hs

@@ -3,14 +3,14 @@
 --
 -- Real Hydra nodes write to @<persistenceDir>/hydra.db@ via
 -- "Hydra.Events.SQLiteBased"; this module writes a file with the same shape
--- (the @events (event_id, event_data BLOB)@ schema, JSON-encoded
--- @StateEvent tx@ per row) using purely synthesised events so the
--- visualizer can be exercised without a running node.
+-- (the @events (event_id, event_data BLOB)@ schema, CBOR-encoded
+-- @StateEvent tx@ per row, schema version 2) using purely synthesised events
+-- so the visualizer can be exercised without a running node.
 module HydraVis.SampleDb (writeSampleDb, sampleStateEvents) where
 
 import Hydra.Prelude
 
-import Data.Aeson qualified as Aeson
+import Cardano.Binary (serialize')
 import Database.SQLite.Simple (close, execute, execute_, open)
 import Hydra.Events (EventId)
 import Hydra.HeadLogic (aggregateState, update)
@@ -62,18 +62,20 @@ sampleStateEvents =
      in (n, scs) : collect (n + 1) s' rest
 
 -- | Create (or recreate) a SQLite database at @path@ holding
--- 'sampleStateEvents'. Overwrites any existing rows.
+-- 'sampleStateEvents'. Overwrites any existing rows. Rows are CBOR-encoded
+-- and the schema version set to 2, matching what a current hydra-node writes.
 writeSampleDb :: FilePath -> IO ()
 writeSampleDb path = do
   conn <- open path
   execute_
     conn
     "CREATE TABLE IF NOT EXISTS events \
     \(event_id INTEGER NOT NULL PRIMARY KEY, event_data BLOB NOT NULL)"
+  execute_ conn "PRAGMA user_version = 2"
   execute_ conn "DELETE FROM events"
   forM_ sampleStateEvents $ \e ->
     execute
       conn
       "INSERT INTO events (event_id, event_data) VALUES (?, ?)"
-      (eventId e, toStrict (Aeson.encode e))
+      (eventId e, serialize' e)
   close conn

head-state-viewer/src/HydraVis/UI/Update.hs

@@ -242,7 +242,7 @@ applyAction a m = case a of
 -- | Background poll loop. Reads any new rows from the SQLite file at a
 -- fixed cadence and dispatches them as 'AppendEvents'.
 followSub ::
-  FromJSON (StateEvent tx) =>
+  (FromCBOR (StateEvent tx), FromJSON (StateEvent tx)) =>
   FilePath ->
   Maybe EventId ->
   Sub (Action tx)

hydra-cardano-api/src/Hydra/Cardano/Api/AddressInEra.hs

@@ -1,7 +1,10 @@
+{-# OPTIONS_GHC -Wno-orphans #-}
+
 module Hydra.Cardano.Api.AddressInEra where
 
 import Hydra.Cardano.Api.Prelude
 
+import Cardano.Api qualified as Api
 import Cardano.Ledger.Address qualified as Ledger
 import Cardano.Ledger.BaseTypes qualified as Ledger
 import Cardano.Ledger.Credential qualified as Ledger
@@ -14,6 +17,32 @@ import PlutusLedgerApi.V3 (
  )
 import PlutusLedgerApi.V3 qualified as Plutus
 
+-- * Orphans
+
+-- missing CBOR instances
+
+-- NOTE: Encoded as the bech32/base58 address text, consistent with the JSON
+-- representation.
+instance (IsShelleyBasedEra era, Typeable era) => ToCBOR (AddressInEra era) where
+  toCBOR = toCBOR . serialiseAddress
+
+instance (IsShelleyBasedEra era, Typeable era) => FromCBOR (AddressInEra era) where
+  fromCBOR = do
+    t <- fromCBOR
+    case deserialiseAddress (proxyToAsType $ Proxy @(AddressInEra era)) t of
+      Nothing -> fail $ "failed to deserialise AddressInEra from " <> show (t :: Text)
+      Just addr -> pure addr
+
+instance ToCBOR (Api.Address ByronAddr) where
+  toCBOR = toCBOR . serialiseToRawBytes
+
+instance FromCBOR (Api.Address ByronAddr) where
+  fromCBOR = do
+    bs <- fromCBOR
+    case deserialiseFromRawBytes (proxyToAsType $ Proxy @(Api.Address ByronAddr)) bs of
+      Left err -> fail (show err)
+      Right v -> pure v
+
 -- * Extras
 
 -- | Construct a Shelley-style address from a verification key. This address has

hydra-cardano-api/src/Hydra/Cardano/Api/ChainPoint.hs

@@ -10,3 +10,26 @@ getChainPoint header =
   ChainPoint slotNo headerHash
  where
   (BlockHeader slotNo headerHash _) = header
+
+-- * Orphans
+
+-- missing CBOR instances
+
+instance ToCBOR ChainPoint where
+  toCBOR = \case
+    ChainPointAtGenesis ->
+      toCBOR ("ChainPointAtGenesis" :: Text)
+    ChainPoint slotNo headerHash ->
+      toCBOR ("ChainPoint" :: Text) <> toCBOR slotNo <> toCBOR (serialiseToRawBytes headerHash)
+
+instance FromCBOR ChainPoint where
+  fromCBOR =
+    fromCBOR >>= \case
+      ("ChainPointAtGenesis" :: Text) -> pure ChainPointAtGenesis
+      "ChainPoint" -> do
+        slotNo <- fromCBOR
+        bytes <- fromCBOR
+        case deserialiseFromRawBytes (proxyToAsType $ Proxy @(Hash BlockHeader)) bytes of
+          Left err -> fail (show err)
+          Right headerHash -> pure $ ChainPoint slotNo headerHash
+      tag -> fail $ show (tag :: Text) <> " is not a proper CBOR-encoded ChainPoint"

hydra-cardano-api/src/Hydra/Cardano/Api/NetworkId.hs

@@ -25,3 +25,17 @@ instance FromJSON NetworkId where
       "Mainnet" -> pure Mainnet
       "Testnet" -> Testnet <$> o .: "magic"
       _ -> fail "Expected tag to be Mainnet | Testnet"
+
+-- missing CBOR instances
+
+instance ToCBOR NetworkId where
+  toCBOR = \case
+    Mainnet -> toCBOR ("Mainnet" :: Text)
+    Testnet (NetworkMagic magic) -> toCBOR ("Testnet" :: Text) <> toCBOR magic
+
+instance FromCBOR NetworkId where
+  fromCBOR =
+    fromCBOR >>= \case
+      ("Mainnet" :: Text) -> pure Mainnet
+      "Testnet" -> Testnet . NetworkMagic <$> fromCBOR
+      tag -> fail $ show tag <> " is not a proper CBOR-encoded NetworkId"

hydra-cardano-api/src/Hydra/Cardano/Api/PolicyAssets.hs

@@ -11,3 +11,27 @@ instance ToJSON PolicyAssets where
 
 instance FromJSON PolicyAssets where
   parseJSON v = PolicyAssets <$> parseJSON v
+
+-- missing CBOR instances
+
+instance ToCBOR AssetName where
+  toCBOR = toCBOR . serialiseToRawBytes
+
+instance FromCBOR AssetName where
+  fromCBOR = do
+    bs <- fromCBOR
+    case deserialiseFromRawBytes AsAssetName bs of
+      Left err -> fail (show err)
+      Right v -> pure v
+
+instance ToCBOR Quantity where
+  toCBOR (Quantity q) = toCBOR q
+
+instance FromCBOR Quantity where
+  fromCBOR = Quantity <$> fromCBOR
+
+instance ToCBOR PolicyAssets where
+  toCBOR (PolicyAssets assets) = toCBOR assets
+
+instance FromCBOR PolicyAssets where
+  fromCBOR = PolicyAssets <$> fromCBOR

hydra-cardano-api/src/Hydra/Cardano/Api/PolicyId.hs

@@ -9,6 +9,20 @@ import Cardano.Ledger.Mary.Value qualified as Ledger
 import Hydra.Cardano.Api.ScriptHash ()
 import PlutusLedgerApi.V3 (CurrencySymbol, fromBuiltin, unCurrencySymbol)
 
+-- * Orphans
+
+-- missing CBOR instances
+
+instance ToCBOR PolicyId where
+  toCBOR = toCBOR . serialiseToRawBytes
+
+instance FromCBOR PolicyId where
+  fromCBOR = do
+    bs <- fromCBOR
+    case deserialiseFromRawBytes AsPolicyId bs of
+      Left err -> fail (show err)
+      Right v -> pure v
+
 -- * Type conversions
 
 -- | Convert Cardano api 'PolicyId' to Cardano ledger `PolicyID`.

hydra-node/golden/DepositStatus.cbor

@@ -0,0 +1 @@
+ŸhInactivefActivegExpiredÿ

hydra-node/golden/FanoutProgressMode.cbor

@@ -0,0 +1 @@
+ŸnAutoFanningOutwAwaitingFanoutSelectionÿ

hydra-node/golden/HeadStatus.cbor

@@ -0,0 +1 @@
+ŸdIdledOpenfClosednFanoutPossiblejFanningOutÿ

hydra-node/golden/InvalidInput.cbor

@@ -0,0 +1 @@
+ŸlInvalidInputdKnWokQô€¸Œð¤£Ÿ=vÿ

hydra-node/golden/SyncedStatus.cbor

@@ -0,0 +1 @@
+ŸfInSyncjCatchingUpÿ

hydra-node/golden/WhichEtcd.cbor

@@ -0,0 +1 @@
+ŸlEmbeddedEtcdjSystemEtcdÿ