> For the complete documentation index, see [llms.txt](https://docs.catalyx.solutions/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.catalyx.solutions/catalyx-blockchain-manager/canton-network/version-1.11/validator-management/identity-backup-and-recovery.md).

# Validator Backups and Identity Dumps

How to configure scheduled backups and identity dumps for validator nodes in CatalyX Blockchain Manager.

## Overview

This guide explains how to configure scheduled backups and identity dumps for validator nodes in CatalyX Blockchain Manager.

Scheduled backups help ensure:

* Faster recovery after failures
* Protection against data loss
* Disaster recovery readiness
* Validator identity preservation

This document covers: validator database backups, validator identity dumps (IDDumps), restore procedures, and backup strategy recommendations.

***

## Backup Types Overview

| Backup Type   | Includes                                           | Does NOT Include                                        | Recommended Usage                           |
| ------------- | -------------------------------------------------- | ------------------------------------------------------- | ------------------------------------------- |
| IDDump        | Cryptographic keys and validator identity material | Database contents, ledger state, transaction history    | Identity recovery and validator restoration |
| PVC/PV Backup | Validator and participant database state           | Cryptographic keys and secrets                          | Stateful workload recovery                  |
| S3 Backup     | Database dumps and snapshots stored externally     | Runtime state and identities unless explicitly exported | Long-term disaster recovery                 |

{% hint style="warning" %}
IDDumps alone cannot restore ledger/database state. Database backups alone cannot restore validator identity. Full disaster recovery requires **both** backup types.
{% endhint %}

***

## Part 1 — Schedule Validator Backups

### Prerequisites

Before enabling scheduled backups:

* Ensure sufficient cluster storage is available
* Verify backup retention requirements
* Confirm the validator is operational

{% hint style="info" %}
If IntellectEU manages the cluster, backup snapshotting may already be enabled. Verify available resources before configuring additional scheduled backups.
{% endhint %}

### Create a Backup Schedule

{% stepper %}
{% step %}

#### Open Validator Backups

Navigate to the **Validators** page, open the target validator, locate the **Backups** section, and click **Schedule**.
{% endstep %}

{% step %}

#### Configure the Schedule

Provide the following settings:

**Cron Expression** — defines how often backups run using standard Unix cron syntax.

| Schedule              | Cron          |
| --------------------- | ------------- |
| Every day at midnight | `0 0 * * *`   |
| Every 6 hours         | `0 */6 * * *` |
| Every Sunday at 02:00 | `0 2 * * 0`   |

**Max Backup Number** — defines how many backups are retained. When the limit is exceeded, older backups are automatically deleted.

* Development: 3–5 backups
* Production: 7–30 backups
  {% endstep %}

{% step %}

#### Confirm the Schedule

Click **Confirm** to activate the backup schedule. Once configured, backups will run automatically and the retention policy will be enforced.
{% endstep %}
{% endstepper %}

### View Existing Backups

To view backup history, navigate to **Validators → Backups → Show history**. The history page displays backup timestamps, status, and available restore actions.

***

## 1.1 Restore Validator from Backup

{% stepper %}
{% step %}

#### Open Backup History

Navigate to **Validators → Backups → Show history**.
{% endstep %}

{% step %}

#### Start the Restore

Open the **Actions** menu for the desired backup and select **Restore**. A restore status indicator will appear showing start time, current progress, and restore status.
{% endstep %}

{% step %}

#### Wait for Completion

When complete, the restore status changes to **Completed**.
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
Restoring from a database backup restores database state only. Validator identities and cryptographic material are **not** restored by database backups.
{% endhint %}

***

## Part 2 — Schedule Validator Identity Dumps (IDDumps)

### What is an Identity Dump?

An Identity Dump (IDDump) contains the validator's cryptographic keys, identity material, and validator access credentials.

IDDumps are required to recover validator ownership, restore wallet access, rejoin the network after failures, and recover balances. They do **not** contain database contents, transaction history, or ledger state.

### Configure an Identity Dump Schedule

{% stepper %}
{% step %}

#### Open Identity Backup Settings

Navigate to the **Validator details** page, open the **Management** section, locate **Identity dumps**, and click **Schedule**.
{% endstep %}

{% step %}

#### Configure the Schedule

**Cron Expression** — defines how frequently IDDumps are created. Recommended: daily for production validators, weekly for development environments.

**Max Dump Number** — defines how many IDDumps are retained. Older dumps are automatically removed after reaching the configured limit.

* Production: minimum 14 dumps
* Critical infrastructure: 30+ dumps
  {% endstep %}

{% step %}

#### Activate the Schedule

Click **Confirm**. After configuration, the next scheduled execution time is displayed along with schedule management options.
{% endstep %}
{% endstepper %}

### View Existing Identity Dumps

To list available IDDumps, open **Validator details → Identity Backups → Show list**. The list displays dump names, creation timestamps, and available actions.

***

## 2.1 Restore Validator from Identity Dump

### Important Notes

When restoring a validator:

* Reuse the same validator configuration values
* Keep Keycloak users and clients
* Preserve onboarding secrets
* Use the same party hint and node identifier

The restored validator receives a new participant hash, reuses existing cryptographic keys, and regains access to balances and network identity.

### Restore Procedure

{% stepper %}
{% step %}

#### Create an Identity Dump

Create an IDDump from the existing validator before deletion.
{% endstep %}

{% step %}

#### Delete the Validator Resource

Delete the validator resource only. Do **not** remove Keycloak users, Keycloak clients, or Kubernetes secrets.
{% endstep %}

{% step %}

#### Recreate the Validator

Create the validator again using the same onboarding secret, party hint, node identifier, and migration configuration. Include the following restore configuration:

```json
"participantIdentitiesDumpImport": {
    "identitiesSecretName": "<iddump-secret-name>",
    "newParticipantIdentifier": "<new-identifier>"
},
"migrateValidatorParty": true
```

{% hint style="warning" %}
Disable auto-init during restore and choose a new participant identifier.
{% endhint %}
{% endstep %}
{% endstepper %}

### Verify Successful Restore

After restore, open the validator wallet UI and log in using the existing Keycloak wallet user. Confirm that wallet access works, balances are visible, and the validator is operational.

{% hint style="info" %}
Use an incognito browser session to avoid Keycloak session conflicts with the CatalyX UI.
{% endhint %}

***

## Best Practices

Use **both** scheduled database backups and scheduled identity dumps to ensure complete disaster recovery coverage.

### Recommended Backup Frequency

| Environment | Database Backup | IDDump |
| ----------- | --------------- | ------ |
| Development | Daily           | Weekly |
| Staging     | Every 12h       | Daily  |
| Production  | Every 6h        | Daily  |

### Storage Recommendations

As of **v1.11.7**, backup volume sizing is automatic. The CAT-BM Operator calculates the required backup volume size using the formula:

```
backup volume size = number of backups × database storage size
```

The volume is automatically resized on the next scheduled backup run when the database storage size increases. Note that Kubernetes PVCs can only increase in size — reducing the **Max Backup Number** setting does not shrink the backup volume.

{% hint style="info" %}
If IntellectEU manages your cluster, snapshot-based backup may already be enabled. Verify available storage capacity before configuring additional scheduled backups alongside existing snapshots.
{% endhint %}

Additional recommendations:

* Backup retention should align with your compliance requirements
* Configure off-cluster backup export for disaster recovery
* For production environments, allocate headroom beyond the calculated minimum to account for database growth between backup runs

### Never Force-Delete a Database Pod

Domain, participant, and validator databases run as stateful Kubernetes workloads, each backed by its own storage. Deleting a database pod is sometimes necessary — for maintenance, node draining, or troubleshooting — but how it's done matters.

{% hint style="danger" %}
Never force-delete a database pod (`kubectl delete pod --force --grace-period=0`). A force-delete does not wait for the pod to fully terminate, and a replacement can be scheduled while the original is still shutting down — leaving two processes writing to the same database storage at once. This can corrupt the database, and the damage isn't always visible right away: it can go undetected until the affected data is read.
{% endhint %}

If a pod needs to be removed, delete it normally and allow it to terminate gracefully. If a pod is stuck terminating, contact CatalyX support before forcing removal.

{% content-ref url="/pages/n4QmocInDV7pxmDuH8fz" %}
[Support Center](/support-center.md)
{% endcontent-ref %}

***

### Reduce Restart-Related Corruption Risk (v1.11.17+)

As of **v1.11.17**, validator, participant, and domain databases can additionally run in a way that guards against the failure mode described above, preventing data corruption during pod restarts or configuration changes — even when a pod isn't cleanly terminated.

This is gated by the `CANTON_OPERATOR_POSTGRES_USESTATEFULSET` environment variable, set under `spec.envVars` (see [Environment Variables](/catalyx-blockchain-manager/canton-network/version-1.11/validator-management/validator-custom-resource-definition.md#environment-variables)).

{% hint style="warning" %}
Before upgrading to this release, stop all validators and participants, and restart them once the upgrade has completed. Enabling this variable also triggers a database server restart, incurring roughly 1 minute of downtime.
{% endhint %}

***

### Use an External Database for Production

For production domains, participants, and validators, we recommend configuring an **External** database (see the storage type options on the [Domains](/catalyx-blockchain-manager/canton-network/version-1.11/network-and-node-management/domains.md) and [Participants](/catalyx-blockchain-manager/canton-network/version-1.11/network-and-node-management/participants.md) pages) instead of the in-cluster provisioned PostgreSQL option. An externally managed database service (e.g. AWS RDS) moves durability, backup, and failover responsibility to a managed service, reducing the operational risk of running the database as an in-cluster workload.

***

## Troubleshooting

| Issue                                     | Cause                                                                               | Resolution                                                                                                                                                                                                           |
| ----------------------------------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Restore completes but wallet inaccessible | Missing IDDump                                                                      | Restore identities separately                                                                                                                                                                                        |
| Validator cannot rejoin network           | Wrong participant identifier                                                        | Use correct migration settings                                                                                                                                                                                       |
| Login fails after restore                 | Keycloak users removed                                                              | Preserve original users and clients                                                                                                                                                                                  |
| Backup creation fails                     | Insufficient storage on initial setup, or automatic volume resize not yet triggered | On v1.11.7+, the Operator resizes the backup volume automatically on the next scheduled run. If the issue persists on first setup, ensure sufficient cluster storage is available for the initial volume allocation. |
| Database pod stuck terminating            | Pod not fully released before a replacement is scheduled                            | Do not force-delete (`--force --grace-period=0`) — this risks two processes writing to the same storage. Wait for graceful termination, or contact CatalyX support.                                                  |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.catalyx.solutions/catalyx-blockchain-manager/canton-network/version-1.11/validator-management/identity-backup-and-recovery.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.