Validator Backups and Identity Dumps

How to configure scheduled backups and identity dumps for validator nodes in CatalyX Blockchain Manager.

Overview

This guide explains how to configure scheduled backups and identity dumps for validator nodes in CatalyX Blockchain Manager.

Scheduled backups help ensure:

• Faster recovery after failures

• Protection against data loss

• Disaster recovery readiness

• Validator identity preservation

This document covers: validator database backups, validator identity dumps (IDDumps), restore procedures, and backup strategy recommendations.

---

Backup Types Overview

Backup Type	Includes	Does NOT Include	Recommended Usage
IDDump	Cryptographic keys and validator identity material	Database contents, ledger state, transaction history	Identity recovery and validator restoration
PVC/PV Backup	Validator and participant database state	Cryptographic keys and secrets	Stateful workload recovery
S3 Backup	Database dumps and snapshots stored externally	Runtime state and identities unless explicitly exported	Long-term disaster recovery

IDDumps alone cannot restore ledger/database state. Database backups alone cannot restore validator identity. Full disaster recovery requires both backup types.

---

Part 1 — Schedule Validator Backups

Prerequisites

Before enabling scheduled backups:

• Ensure sufficient cluster storage is available

• Verify backup retention requirements

• Confirm the validator is operational

If IntellectEU manages the cluster, backup snapshotting may already be enabled. Verify available resources before configuring additional scheduled backups.

Create a Backup Schedule

1

Open Validator Backups

Navigate to the Validators page, open the target validator, locate the Backups section, and click Schedule.

2

Configure the Schedule

Provide the following settings:

Cron Expression — defines how often backups run using standard Unix cron syntax.

Schedule	Cron
Every day at midnight	0 0 * * *
Every 6 hours	0 */6 * * *
Every Sunday at 02:00	0 2 * * 0

Max Backup Number — defines how many backups are retained. When the limit is exceeded, older backups are automatically deleted.

• Development: 3–5 backups

• Production: 7–30 backups

3

Confirm the Schedule

Click Confirm to activate the backup schedule. Once configured, backups will run automatically and the retention policy will be enforced.

View Existing Backups

To view backup history, navigate to Validators → Backups → Show history. The history page displays backup timestamps, status, and available restore actions.

---

1.1 Restore Validator from Backup

1

Open Backup History

Navigate to Validators → Backups → Show history.

2

Start the Restore

Open the Actions menu for the desired backup and select Restore. A restore status indicator will appear showing start time, current progress, and restore status.

3

Wait for Completion

When complete, the restore status changes to Completed.

Restoring from a database backup restores database state only. Validator identities and cryptographic material are not restored by database backups.

---

Part 2 — Schedule Validator Identity Dumps (IDDumps)

What is an Identity Dump?

An Identity Dump (IDDump) contains the validator's cryptographic keys, identity material, and validator access credentials.

IDDumps are required to recover validator ownership, restore wallet access, rejoin the network after failures, and recover balances. They do not contain database contents, transaction history, or ledger state.

Configure an Identity Dump Schedule

1

Open Identity Backup Settings

Navigate to the Validator details page, open the Management section, locate Identity dumps, and click Schedule.

2

Configure the Schedule

Cron Expression — defines how frequently IDDumps are created. Recommended: daily for production validators, weekly for development environments.

Max Dump Number — defines how many IDDumps are retained. Older dumps are automatically removed after reaching the configured limit.

• Production: minimum 14 dumps

• Critical infrastructure: 30+ dumps

3

Activate the Schedule

Click Confirm. After configuration, the next scheduled execution time is displayed along with schedule management options.

View Existing Identity Dumps

To list available IDDumps, open Validator details → Identity Backups → Show list. The list displays dump names, creation timestamps, and available actions.

---

2.1 Restore Validator from Identity Dump

Important Notes

When restoring a validator:

• Reuse the same validator configuration values

• Keep Keycloak users and clients

• Preserve onboarding secrets

• Use the same party hint and node identifier

The restored validator receives a new participant hash, reuses existing cryptographic keys, and regains access to balances and network identity.

Restore Procedure

1

Create an Identity Dump

Create an IDDump from the existing validator before deletion.

2

Delete the Validator Resource

Delete the validator resource only. Do not remove Keycloak users, Keycloak clients, or Kubernetes secrets.

3

Recreate the Validator

Create the validator again using the same onboarding secret, party hint, node identifier, and migration configuration. Include the following restore configuration:

"participantIdentitiesDumpImport": {
    "identitiesSecretName": "<iddump-secret-name>",
    "newParticipantIdentifier": "<new-identifier>"
},
"migrateValidatorParty": true

Disable auto-init during restore and choose a new participant identifier.

Verify Successful Restore

After restore, open the validator wallet UI and log in using the existing Keycloak wallet user. Confirm that wallet access works, balances are visible, and the validator is operational.

Use an incognito browser session to avoid Keycloak session conflicts with the CatalyX UI.

---

Best Practices

Use both scheduled database backups and scheduled identity dumps to ensure complete disaster recovery coverage.

Recommended Backup Frequency

Environment	Database Backup	IDDump
Development	Daily	Weekly
Staging	Every 12h	Daily
Production	Every 6h	Daily

Storage Recommendations

As of v1.11.7, backup volume sizing is automatic. The CAT-BM Operator calculates the required backup volume size using the formula:

backup volume size = number of backups × database storage size

The volume is automatically resized on the next scheduled backup run when the database storage size increases. Note that Kubernetes PVCs can only increase in size — reducing the Max Backup Number setting does not shrink the backup volume.

If IntellectEU manages your cluster, snapshot-based backup may already be enabled. Verify available storage capacity before configuring additional scheduled backups alongside existing snapshots.

Additional recommendations:

• Backup retention should align with your compliance requirements

• Configure off-cluster backup export for disaster recovery

• For production environments, allocate headroom beyond the calculated minimum to account for database growth between backup runs

Never Force-Delete a Database Pod

Domain, participant, and validator databases run as stateful Kubernetes workloads, each backed by its own storage. Deleting a database pod is sometimes necessary — for maintenance, node draining, or troubleshooting — but how it's done matters.

Never force-delete a database pod (kubectl delete pod --force --grace-period=0). A force-delete does not wait for the pod to fully terminate, and a replacement can be scheduled while the original is still shutting down — leaving two processes writing to the same database storage at once. This can corrupt the database, and the damage isn't always visible right away: it can go undetected until the affected data is read.

If a pod needs to be removed, delete it normally and allow it to terminate gracefully. If a pod is stuck terminating, contact CatalyX support before forcing removal.

Support Center

---

Reduce Restart-Related Corruption Risk (v1.11.17+)

As of v1.11.17, validator, participant, and domain databases can additionally run in a way that guards against the failure mode described above, preventing data corruption during pod restarts or configuration changes — even when a pod isn't cleanly terminated.

This is gated by the CANTON_OPERATOR_POSTGRES_USESTATEFULSET environment variable, set under spec.envVars (see Environment Variables).

Before upgrading to this release, stop all validators and participants, and restart them once the upgrade has completed. Enabling this variable also triggers a database server restart, incurring roughly 1 minute of downtime.

---

Use an External Database for Production

For production domains, participants, and validators, we recommend configuring an External database (see the storage type options on the Domains and Participants pages) instead of the in-cluster provisioned PostgreSQL option. An externally managed database service (e.g. AWS RDS) moves durability, backup, and failover responsibility to a managed service, reducing the operational risk of running the database as an in-cluster workload.

---

Troubleshooting

Issue	Cause	Resolution
Restore completes but wallet inaccessible	Missing IDDump	Restore identities separately
Validator cannot rejoin network	Wrong participant identifier	Use correct migration settings
Login fails after restore	Keycloak users removed	Preserve original users and clients
Backup creation fails	Insufficient storage on initial setup, or automatic volume resize not yet triggered	On v1.11.7+, the Operator resizes the backup volume automatically on the next scheduled run. If the issue persists on first setup, ensure sufficient cluster storage is available for the initial volume allocation.
Database pod stuck terminating	Pod not fully released before a replacement is scheduled	Do not force-delete (--force --grace-period=0) — this risks two processes writing to the same storage. Wait for graceful termination, or contact CatalyX support.