Disaster Recovery from a Failure on Global Synchronizer

This flow is adapted from the Canton official guide, with additional instructions for Catalyst validator installations.

Follow these steps only when instructed to by your super validator sponsor or the IntellectEU team during a network recovery event.

1

Set environment variables

export NAME=<name of the validator>
export PASSWORD=<password for wallet>
export KEYCLOAK_URL=<keycloak url>
export REALM=<keycloak realm for catalyst>

2

Log in to Keycloak and obtain a token

curl --location "${KEYCLOAK_URL}/auth/realms/${REALM}/protocol/openid-connect/token" \
  --header "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "grant_type=password" \
  --data-urlencode "client_id=${NAME}-wallet-ui" \
  --data-urlencode "username=${NAME}_walletuser" \
  --data-urlencode "password=${PASSWORD}" > token.json

3

Extract the access token

export TOKEN=$(jq -r .access_token token.json)

4

Obtain the snapshot from the wallet

• The wallet URL can be obtained from the UI page of the wallet application.

• The super validators on network recovery will provide the timestamp in the #validators-ops channel.

• The timestamp format is the same as the logs for the validator.

export WALLET_URL=<url of the validator wallet application>
export TIMESTAMP=<timestamp on the same format as the logs>

curl -sSLf "${WALLET_URL}/api/validator/v0/admin/domain/data-snapshot?timestamp=${TIMESTAMP}&force=true" \
  -X GET \
  -H "authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" > dump_response.json

5

Extract the data snapshot

jq '.data_snapshot' dump_response.json > data_snapshot.json

6

Copy the snapshot to the validator pod

The validator pod will have the name of the validator plus suffixes added by Kubernetes.

export NAMESPACE=<namespace of the validator>
export PODNAME=<pod of validator>

kubectl cp data_snapshot.json ${NAMESPACE}/${PODNAME}:/domain-upgrade-dump/domain_migration_dump.json

7

Verify the file integrity (MD5 checksum)

export LOCAL_DUMP_MD5SUM=$(md5sum data_snapshot.json)
export PODS_DUMP_MD5SUM=$(kubectl -n "${NAMESPACE}" exec -it ${PODNAME} -- md5sum /domain-upgrade-dump/domain_migration_dump.json)

if [ "$(echo $LOCAL_DUMP_MD5SUM | awk '{print $1}')" = "$(echo $PODS_DUMP_MD5SUM | awk '{print $1}')" ]; then
  echo 'MD5SUM checksums are equal.'
else
  echo 'MD5SUM checksums are not equal! Check the file copied is correct.'
fi

8

Trigger the migration

In the Catalyst UI validator page, edit the validator, set migrationID to a new migration ID, and toggle migrating to true.

Confirm and wait for the update to complete.

---