> For the complete documentation index, see [llms.txt](https://docs.catalyx.solutions/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.catalyx.solutions/catalyx-blockchain-manager/canton-network/version-1.11/validator-management/disaster-recovery-from-a-failure-on-global-synchronizer.md).

# Disaster Recovery from a Failure on Global Synchronizer

This flow is adapted from the [Canton official guide](https://dev.network.canton.global/validator_operator/validator_backups.html#disaster-recovery-from-loss-of-the-cometbft-storage-layer-of-the-global-synchronizer), with additional instructions for Catalyst validator installations.

{% hint style="warning" %}
Follow these steps only when instructed to by your super validator sponsor or the IntellectEU team during a network recovery event.
{% endhint %}

{% stepper %}
{% step %}
**Set environment variables**

```bash
export NAME=<name of the validator>
export PASSWORD=<password for wallet>
export KEYCLOAK_URL=<keycloak url>
export REALM=<keycloak realm for catalyst>
```

{% endstep %}

{% step %}
**Log in to Keycloak and obtain a token**

```bash
curl --location "${KEYCLOAK_URL}/auth/realms/${REALM}/protocol/openid-connect/token" \
  --header "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "grant_type=password" \
  --data-urlencode "client_id=${NAME}-wallet-ui" \
  --data-urlencode "username=${NAME}_walletuser" \
  --data-urlencode "password=${PASSWORD}" > token.json
```

{% endstep %}

{% step %}
**Extract the access token**

```bash
export TOKEN=$(jq -r .access_token token.json)
```

{% endstep %}

{% step %}
**Obtain the snapshot from the wallet**

{% hint style="info" %}

* The wallet URL can be obtained from the UI page of the wallet application.
* The super validators on network recovery will provide the timestamp in the `#validators-ops` channel.
* The timestamp format is the same as the logs for the validator.
  {% endhint %}

```bash
export WALLET_URL=<url of the validator wallet application>
export TIMESTAMP=<timestamp on the same format as the logs>

curl -sSLf "${WALLET_URL}/api/validator/v0/admin/domain/data-snapshot?timestamp=${TIMESTAMP}&force=true" \
  -X GET \
  -H "authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" > dump_response.json
```

{% endstep %}

{% step %}
**Extract the data snapshot**

```bash
jq '.data_snapshot' dump_response.json > data_snapshot.json
```

{% endstep %}

{% step %}
**Copy the snapshot to the validator pod**

{% hint style="info" %}
The validator pod will have the name of the validator plus suffixes added by Kubernetes.
{% endhint %}

```bash
export NAMESPACE=<namespace of the validator>
export PODNAME=<pod of validator>

kubectl cp data_snapshot.json ${NAMESPACE}/${PODNAME}:/domain-upgrade-dump/domain_migration_dump.json
```

{% endstep %}

{% step %}
**Verify the file integrity (MD5 checksum)**

```bash
export LOCAL_DUMP_MD5SUM=$(md5sum data_snapshot.json)
export PODS_DUMP_MD5SUM=$(kubectl -n "${NAMESPACE}" exec -it ${PODNAME} -- md5sum /domain-upgrade-dump/domain_migration_dump.json)

if [ "$(echo $LOCAL_DUMP_MD5SUM | awk '{print $1}')" = "$(echo $PODS_DUMP_MD5SUM | awk '{print $1}')" ]; then
  echo 'MD5SUM checksums are equal.'
else
  echo 'MD5SUM checksums are not equal! Check the file copied is correct.'
fi
```

{% endstep %}

{% step %}
**Trigger the migration**

In the Catalyst UI validator page, **edit the validator**, set `migrationID` to a new migration ID, and toggle **migrating** to `true`.

Confirm and wait for the update to complete.
{% endstep %}
{% endstepper %}

***


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.catalyx.solutions/catalyx-blockchain-manager/canton-network/version-1.11/validator-management/disaster-recovery-from-a-failure-on-global-synchronizer.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.