Troubleshooting

Common problems, and the order in which to look at things.

A general method

Most problems resolve quickly if you work from the outside in.

1

Is it a configuration problem or a runtime problem?

Check the validator's Ready condition on the Status tab.

• InvalidSpec — configuration. The message names the field. Nothing else needs investigating.

• ReconcileError — the operator could not complete an action, usually because something external was unreachable.

• ApplicationNotReady — configuration is fine, a component has not come up.

2

Which component?

The Summary tab breaks readiness down per component. Expand the offending row for its conditions and per-replica state.

3

Kubernetes problem or Canton problem?

A component that is Ready but not working is a Canton-level problem. Go to the Participant tab.

4

Read the logs

The Logs link on the component's row opens Grafana, filtered to that component.

---

Installation

The API pod will not start — CreateContainerConfigError

Almost always the missing catalyx-api-credentials secret. It is mandatory and the pod cannot start without it.

kubectl create secret generic catalyx-api-credentials \
  --from-literal=client-secret=<catalyx-api-client-secret> \
  -n <namespace>

Validator UIs are unreachable, or serve a certificate error

Every ingress route the operator creates references the TLS secret named by operatorRuntime.tlsSecretName, which defaults to catalyx-app-tls. If that secret does not exist, those routes cannot serve HTTPS — even if you configured ingress.tls.secretName separately for the UI and API.

Check the secret exists in the validator's namespace, and that its certificate covers the hostnames derived from operatorRuntime.baseHostname.

Traffic does not reach the platform at all

Confirm Traefik is installed with entry points named exactly web and websecure, and that it is version 3.x — CAT-BM creates IngressRoute resources in the traefik.io API group.

A field I set after upgrading has no effect

Helm never upgrades the resources in a chart's crds/ directory, so a new field may not exist in the installed schema — in which case the API server prunes it silently.

kubectl apply -f <chart>/crds

Then re-apply your Validator.

---

Validators

Ready is False with reason InvalidSpec

The message names the field. The most common causes:

Message mentions	Cause
version or spec.network.spliceVersion must be set	A component has no version and no spliceVersion to fall back on
jvm.maxHeap … exceeds … limits.memory	The heap would not fit in the container. Lower the heap or raise the limit, leaving headroom for non-heap memory
is not a valid JVM heap size	You used Kubernetes units. Heap values need k, m, or g — 2048m, not 2Gi
pqsDatabase must be set	PQS is enabled without its database
walletGatewayStoreDatabase and walletGatewaySigningDatabase	The Wallet Gateway is enabled without both of its databases
partyHint	It does not match the required pattern — two alphanumeric segments and a numeric segment, hyphen-separated

Ready is False with reason ReconcileError

Something outside the resource failed. Check the operator logs:

kubectl -n <namespace> logs deploy/catalyx-canton-operator --tail=200

Usual causes: PostgreSQL unreachable or the credentials lacking CREATEDB; the identity provider unreachable; the realm or admin client missing; the database credentials secret missing or missing a key.

The database never provisions

The operator connects to the postgres maintenance database on your configured host and issues CREATE DATABASE. Confirm the host is reachable from the cluster, the postgres database exists, and the credentials have CREATEDB.

Managed authentication never completes

Auth Provisioned stays No on the Status tab. Check, in order:

1. The realm exists. CAT-BM never creates a realm.

2. The admin client exists in that realm — not in master — and its service account has realm-management permissions.

3. operator.keycloak.serverUrl, realm, and adminClientId are set on the release.

4. The catalyx-operator-keycloak secret exists with the expected key.

5. spec.network.partyHint is set on the validator.

The participant stays Initializing

Look at Waiting for on the Participant tab — it names what the node is blocked on.

During onboarding this is usually normal and resolves once the network accepts the validator. If it persists, check that the onboarding secret is present and has not expired — they are single-use and short-lived — and that the sequencer, Scan, and sponsoring Super Validator endpoints are reachable.

The participant was working and is now Disconnected

Check Connected Synchronizers on the Participant tab for the synchronizer's health, and confirm the sequencer endpoint is still reachable from the cluster.

Also check the Traffic tab: a participant that has exhausted both its base allowance and its purchased traffic is throttled by the synchronizer.

The Wallet Gateway component is missing

On a new validator using managed authentication, the Wallet Gateway waits for identity provider provisioning to finish before it starts. A missing or zero-replica Wallet Gateway on the first reconcile is expected — give it a cycle.

If it persists, the validator's Ready condition will name the missing field. The Wallet Gateway has more prerequisites than any other component; see Wallet Gateway.

KMS is configured but keys are still in the database

Check the spelling and case of spec.kms.provider. It is matched case-sensitively — Azure or AWS silently disables KMS, and the validator otherwise reconciles normally. Use lowercase azure or aws.

---

Parties

Allocation failed after the party signed

A named participant UID does not exist on the synchronizer. The signature is consumed; restart onboarding with a fresh party hint, and use the advisory lookup on every UID before signing.

A multi-hosted party is not usable

It needs authorization from every named host. Check which are outstanding, and note that all hosts must be on the same synchronizer — a participant that is absent from your topology store, including one whose validator has been offline long enough to fall behind, will not resolve.

I am named in a proposal but there is no Approve button

Nothing is required of you. A participant is only asked to authorize when it takes on more responsibility — when it is being added, or when its permission is being strengthened.

A hosting amendment signature is rejected

Check the Key algorithm selector in the Edit Hosting modal. It defaults to Ed25519 and does not detect the party's actual algorithm, so an ECDSA P-256 party needs it changed before signing.

---

UI

A validator does not appear in the list

The UI only shows validators in the namespace the API is configured to watch (api.namespace in the Helm values — see Platform Installation). A validator in another namespace is also not being reconciled, so this is usually a real problem rather than a display one.

A tab shows "Failed to load…"

These tabs read live from Canton through the API, so the failure is upstream rather than in the UI. Check that the component is Ready, then that the API can authenticate — the Identity tab failing to load its Authenticated user section points at the client secret, audience, or scope.

The Canton console will not connect

It attaches to a running participant pod. If none is running it reports so rather than connecting — check the validator's Summary tab first. Reconnect clears and re-establishes the session.

Nothing seems to update

The UI polls continuously — status every three seconds, metrics every five. There is no refresh button and no "last updated" indicator, so stale data means the API is not responding rather than that you need to refresh. The exception is the Parties tab, which has explicit Refresh buttons because reading the topology store is expensive.

---

Escalating

If none of the above helps, gather the following before raising a ticket:

kubectl -n <namespace> get validators -o wide
kubectl -n <namespace> get validator <name> -o yaml
kubectl -n <namespace> get applications
kubectl -n <namespace> logs deploy/catalyx-canton-operator --tail=500
kubectl -n <namespace> get events --sort-by=.lastTimestamp | tail -50

Review the output before attaching it. kubectl get validator -o yaml includes secret names but not values — that is fine. Never attach an identity dump, a client secret, or database credentials.

See Support & Resources for how to reach the service desk.