Skip to content
72 changes: 72 additions & 0 deletions docs/persistence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Data Persistence

By default, Warnet nodes use ephemeral storage, meaning all data is lost when a pod is deleted or restarted. This document describes how to enable persistent storage to be able to use warnet for persistent development environment, such that blockchain data, wallet information, and other node state can survive pod restarts and network redeployments. This is done with Kubernetes Persistent Volume Claims (PVCs), which persist independently of pod lifecycle.

Persistence is available for:
- **Bitcoin Core** nodes
- **LND** nodes
- **CLN** nodes

## Enabling Persistence

Persistence is configured per-node in the network graph definition. Add a `persistence` section to any node's configuration. This creates a new PVC for that node, which is then mounted to the appropriate data directory inside the container.

Also add `restartPolicy: Always` to the node's configuration to ensure that the pod is restarted if it is deleted or crashes. This is important to ensure proper restart after restart of the kubernetes cluster. If there is a risk of hard resets of the cluster, add `reindex=1` to bitcoin_core config to reindex the blockchain on startup and fix potential corrupted chain state.

### Bitcoin Core Node

```yaml
bitcoin_core:
image: bitcoincore-27.1:latest
restartPolicy: Always
persistence:
enabled: true
size: 20Gi # optional, default is 20Gi
storageClass: "" # optional, default is cluster default storage class
accessMode: ReadWriteOncePod # optional, default is ReadWriteOncePod. For compatibility with older Kubernetes versions, you may need to set this to ReadWriteOnce
config: |
reindex=1
```

### Lightning Node

```yaml
<lnd or cln>:
image:
tag: <node-version-tag>
restartPolicy: Always
persistence:
enabled: true
size: 10Gi # optional, default is 10Gi
storageClass: "" # optional, default is cluster default storage class
accessMode: ReadWriteOncePod # optional, default is ReadWriteOncePod. For compatibility with older Kubernetes versions, you may need to set this to ReadWriteOnce
```

## Existing PVCs

To use custom made PVC or PVC from previous deployment, use the `existingClaim` field to reference an existing PVC by name. If the network configuration or namespace did not change, there is no need to explicitly set the `existingClaim`. The existing PVC is used by default, since its generated name matches the default pattern. To explicitly use a PVC set the name like this:

```yaml
persistence:
enabled: true
existingClaim: "tank-0001.default-bitcoincore-data"
```

The generated PVC names follow the pattern:
`<pod-name>.<namespace>-<node-type>-data`

For example for a bitcoin core node:
`tank-0001.default-bitcoincore-data`

And for a LND node:
`tank-0001-ln.default-lnd-data`

Get the list of PVCs in the cluster with `kubectl get pvc -A` and delete any PVCs that are no longer needed with `kubectl delete pvc <pvc-name> -n <namespace>`.

## Mount Paths

When persistence is enabled, the following directories are persisted in the PVCs:

- **Bitcoin Core:** `/root/.bitcoin/`
- **LND:** `/root/.lnd/`
- **CLN:** `/root/.lightning/`
14 changes: 14 additions & 0 deletions resources/charts/bitcoincore/charts/cln/templates/pod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ spec:
- mountPath: /root/.lightning/config
name: config
subPath: config
- mountPath: /root/.lightning
name: cln-data
{{- with .Values.extraContainers }}
{{- toYaml . | nindent 4 }}
{{- end }}
Expand All @@ -80,6 +82,18 @@ spec:
- configMap:
name: {{ include "cln.fullname" . }}
name: config
- name: cln-data
{{- if .Values.persistence.enabled }}
{{- if .Values.persistence.existingClaim }}
persistentVolumeClaim:
claimName: {{ .Values.persistence.existingClaim }}
{{- else }}
persistentVolumeClaim:
claimName: {{ include "cln.fullname" . }}.{{ .Release.Namespace }}-cln-data
{{- end }}
{{- else }}
emptyDir: {}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 4 }}
Expand Down
19 changes: 19 additions & 0 deletions resources/charts/bitcoincore/charts/cln/templates/pvc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "cln.fullname" . }}.{{ .Release.Namespace }}-cln-data
labels:
{{- include "cln.labels" . | nindent 4 }}
annotations:
"helm.sh/resource-policy": keep
spec:
accessModes:
- {{ .Values.persistence.accessMode }}
resources:
requests:
storage: {{ .Values.persistence.size }}
{{- if .Values.persistence.storageClass }}
storageClassName: {{ .Values.persistence.storageClass }}
{{- end }}
{{- end }}
9 changes: 9 additions & 0 deletions resources/charts/bitcoincore/charts/cln/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,15 @@ startupProbe:
- "-c"
- "lightning-cli createrune > /working/rune.json"

# Node data persistence configuration. Create persistent volume claim, or use an existing one.
persistence:
enabled: false
storageClass: ""
accessMode: ReadWriteOncePod
size: 10Gi
# Use existing persistent volume claim instead of creating a new one.
existingClaim: ""

# Additional volumes on the output Deployment definition.
volumes:
- name: working
Expand Down
16 changes: 13 additions & 3 deletions resources/charts/bitcoincore/charts/lnd/templates/pod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ spec:
- mountPath: /root/.lnd/tls.cert
name: config
subPath: tls.cert
- name: shared-volume
- name: lnd-data
mountPath: /root/.lnd/
{{- with .Values.extraContainers }}
{{- toYaml . | nindent 4 }}
Expand All @@ -95,7 +95,7 @@ spec:
- "--macaroonpath=/root/.lnd/data/chain/bitcoin/{{ .Values.global.chain }}/admin.macaroon"
- "--httplisten=0.0.0.0:{{ .Values.circuitbreaker.httpPort }}"
volumeMounts:
- name: shared-volume
- name: lnd-data
mountPath: /root/.lnd/
- name: config
mountPath: /tls.cert
Expand All @@ -108,8 +108,18 @@ spec:
- configMap:
name: {{ include "lnd.fullname" . }}
name: config
- name: shared-volume
- name: lnd-data
{{- if .Values.persistence.enabled }}
{{- if .Values.persistence.existingClaim }}
persistentVolumeClaim:
claimName: {{ .Values.persistence.existingClaim }}
{{- else }}
persistentVolumeClaim:
claimName: {{ include "lnd.fullname" . }}.{{ .Release.Namespace }}-lnd-data
{{- end }}
{{- else }}
emptyDir: {}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 4 }}
Expand Down
19 changes: 19 additions & 0 deletions resources/charts/bitcoincore/charts/lnd/templates/pvc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "lnd.fullname" . }}.{{ .Release.Namespace }}-lnd-data
labels:
{{- include "lnd.labels" . | nindent 4 }}
annotations:
"helm.sh/resource-policy": keep
spec:
accessModes:
- {{ .Values.persistence.accessMode }}
resources:
requests:
storage: {{ .Values.persistence.size }}
{{- if .Values.persistence.storageClass }}
storageClassName: {{ .Values.persistence.storageClass }}
{{- end }}
{{- end }}
20 changes: 14 additions & 6 deletions resources/charts/bitcoincore/charts/lnd/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

podLabels:
podLabels:
app: "warnet"
mission: "lightning"

Expand Down Expand Up @@ -65,12 +65,11 @@ resources: {}
# cpu: 100m
# memory: 128Mi


livenessProbe:
exec:
command:
- pidof
- lnd
- pidof
- lnd
failureThreshold: 3
initialDelaySeconds: 60
periodSeconds: 5
Expand All @@ -85,6 +84,15 @@ readinessProbe:
port: 10009
timeoutSeconds: 1

# Node data persistence configuration. Create persistent volume claim, or use an existing one.
persistence:
enabled: false
storageClass: ""
accessMode: ReadWriteOncePod
size: 10Gi
# Use existing persistent volume claim instead of creating a new one.
existingClaim: ""

# Additional volumes on the output Deployment definition.
volumes: []
# - name: foo
Expand Down Expand Up @@ -128,6 +136,6 @@ defaultConfig: ""
channels: []

circuitbreaker:
enabled: false # Default to disabled
enabled: false # Default to disabled
image: carlakirkcohen/circuitbreaker:attackathon-test
httpPort: 9235
httpPort: 9235
10 changes: 10 additions & 0 deletions resources/charts/bitcoincore/templates/pod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,17 @@ spec:
{{- toYaml . | nindent 4 }}
{{- end }}
- name: data
{{- if .Values.persistence.enabled }}
{{- if .Values.persistence.existingClaim }}
persistentVolumeClaim:
claimName: {{ .Values.persistence.existingClaim }}
{{- else }}
persistentVolumeClaim:
claimName: {{ include "bitcoincore.fullname" . }}.{{ .Release.Namespace }}-bitcoincore-data
{{- end }}
{{- else }}
emptyDir: {}
{{- end }}
- name: config
configMap:
name: {{ include "bitcoincore.fullname" . }}
Expand Down
19 changes: 19 additions & 0 deletions resources/charts/bitcoincore/templates/pvc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "bitcoincore.fullname" . }}.{{ .Release.Namespace }}-bitcoincore-data
labels:
{{- include "bitcoincore.labels" . | nindent 4 }}
annotations:
"helm.sh/resource-policy": keep
spec:
accessModes:
- {{ .Values.persistence.accessMode }}
resources:
requests:
storage: {{ .Values.persistence.size }}
{{- if .Values.persistence.storageClass }}
storageClassName: {{ .Values.persistence.storageClass }}
{{- end }}
{{- end }}
15 changes: 11 additions & 4 deletions resources/charts/bitcoincore/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

podLabels:
podLabels:
app: "warnet"
mission: "tank"

Expand Down Expand Up @@ -61,12 +61,11 @@ resources: {}
# cpu: 100m
# memory: 128Mi


livenessProbe:
exec:
command:
- pidof
- bitcoind
- pidof
- bitcoind
failureThreshold: 12
initialDelaySeconds: 5
periodSeconds: 5
Expand All @@ -78,6 +77,14 @@ readinessProbe:
successThreshold: 1
timeoutSeconds: 10

# Node data persistence configuration. Create persisten volume claim, or use an existing one.
persistence:
enabled: false
storageClass: ""
accessMode: ReadWriteOncePod
size: 20Gi
# Use existing persistent volume claim instead of creating a new one.
existingClaim: ""

# Additional volumes on the output Deployment definition.
volumes: []
Expand Down
7 changes: 6 additions & 1 deletion resources/scenarios/commander.py
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,13 @@ def run_test(self):
@staticmethod
def ensure_miner(node):
wallets = node.listwallets()

if "miner" not in wallets:
node.createwallet("miner", descriptors=True)
allwallets = node.listwalletdir()
if "'miner'" in str(allwallets):
node.loadwallet("miner")
else:
node.createwallet("miner", descriptors=True)
return node.get_wallet_rpc("miner")

@staticmethod
Expand Down
Loading