> For the complete documentation index, see [llms.txt](https://docs.catalyx.solutions/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.catalyx.solutions/catalyx-blockchain-manager/canton-network/version-2.0/support-and-resources/troubleshooting.md).

# Troubleshooting

## A general method

Most problems resolve quickly if you work from the outside in.

{% stepper %}
{% step %}
**Is it a configuration problem or a runtime problem?**

Check the validator's `Ready` condition on the [Status](/catalyx-blockchain-manager/canton-network/version-2.0/console-guide-canton/validators/status-and-specification.md) tab.

* `InvalidSpec` — configuration. The message names the field. Nothing else needs investigating.
* `ReconcileError` — the operator could not complete an action, usually because something external was unreachable.
* `ApplicationNotReady` — configuration is fine, a component has not come up.
  {% endstep %}

{% step %}
**Which component?**

The [Summary](/catalyx-blockchain-manager/canton-network/version-2.0/console-guide-canton/validators/summary.md) tab breaks readiness down per component. Expand the offending row for its conditions and per-replica state.
{% endstep %}

{% step %}
**Kubernetes problem or Canton problem?**

A component that is `Ready` but not working is a Canton-level problem. Go to the [Participant](/catalyx-blockchain-manager/canton-network/version-2.0/console-guide-canton/validators/participant-and-keys.md) tab.
{% endstep %}

{% step %}
**Read the logs**

The **Logs** link on the component's row opens Grafana, filtered to that component.
{% endstep %}
{% endstepper %}

***

## Installation

<details>

<summary>The API pod will not start — CreateContainerConfigError</summary>

Almost always the missing `catalyx-api-credentials` secret. It is mandatory and the pod cannot start without it.

```bash
kubectl create secret generic catalyx-api-credentials \
  --from-literal=client-secret=<catalyx-api-client-secret> \
  -n catalyx-system
```

</details>

<details>

<summary>Validator UIs are unreachable, or serve a certificate error</summary>

Every ingress route the operator creates references the TLS secret named by `operatorRuntime.tlsSecretName`, which defaults to `catalyx-app-tls`. If that secret does not exist, those routes cannot serve HTTPS — even if you configured `ingress.tls.secretName` separately for the console and API.

Check the secret exists in the validator's namespace, and that its certificate covers the hostnames derived from `operatorRuntime.baseHostname`.

</details>

<details>

<summary>Traffic does not reach the platform at all</summary>

Confirm Traefik is installed with entry points named exactly `web` and `websecure`, and that it is version 2.11 or later — CAT-BM creates `IngressRoute` resources in the `traefik.io` API group, which earlier 2.x releases do not have.

</details>

<details>

<summary>A field I set after upgrading has no effect</summary>

Helm never upgrades the resources in a chart's `crds/` directory, so a new field may not exist in the installed schema — in which case the API server prunes it silently.

```bash
kubectl apply -f <chart>/crds
```

Then re-apply your `Validator`.

</details>

***

## Validators

<details>

<summary>Ready is False with reason InvalidSpec</summary>

The message names the field. The most common causes:

| Message mentions                                              | Cause                                                                                                            |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `version or spec.network.spliceVersion must be set`           | A component has no version and no `spliceVersion` to fall back on                                                |
| `jvm.maxHeap … exceeds … limits.memory`                       | The heap would not fit in the container. Lower the heap or raise the limit, leaving headroom for non-heap memory |
| `is not a valid JVM heap size`                                | You used Kubernetes units. Heap values need `k`, `m`, or `g` — `2048m`, not `2Gi`                                |
| `pqsDatabase must be set`                                     | PQS is enabled without its database                                                                              |
| `walletGatewayStoreDatabase and walletGatewaySigningDatabase` | The Wallet Gateway is enabled without both of its databases                                                      |
| `partyHint`                                                   | It does not match the required pattern — two alphanumeric segments and a numeric segment, hyphen-separated       |

</details>

<details>

<summary>Ready is False with reason ReconcileError</summary>

Something outside the resource failed. Check the operator logs:

```bash
kubectl -n catalyx-system logs deploy/catalyx-canton-operator --tail=200
```

Usual causes: PostgreSQL unreachable or the credentials lacking `CREATEDB`; the identity provider unreachable; the realm or admin client missing; the database credentials secret missing or missing a key.

</details>

<details>

<summary>The database never provisions</summary>

The operator connects to the `postgres` maintenance database on your configured host and issues `CREATE DATABASE`. Confirm the host is reachable from the cluster, the `postgres` database exists, and the credentials have `CREATEDB`.

</details>

<details>

<summary>Managed authentication never completes</summary>

`Auth Provisioned` stays `No` on the Status tab. Check, in order:

1. The realm exists. CAT-BM never creates a realm.
2. The admin client exists **in that realm** — not in `master` — and its service account has realm-management permissions.
3. `operator.keycloak.serverUrl`, `realm`, and `adminClientId` are set on the release.
4. The `catalyx-operator-keycloak` secret exists with the expected key.
5. `spec.network.partyHint` is set on the validator.

</details>

<details>

<summary>The participant stays Initializing</summary>

Look at **Waiting for** on the [Participant](/catalyx-blockchain-manager/canton-network/version-2.0/console-guide-canton/validators/participant-and-keys.md) tab — it names what the node is blocked on.

During onboarding this is usually normal and resolves once the network accepts the validator. If it persists, check that the onboarding secret is present and has not expired — they are single-use and short-lived — and that the sequencer, Scan, and sponsoring Super Validator endpoints are reachable.

</details>

<details>

<summary>The participant was working and is now Disconnected</summary>

Check **Connected Synchronizers** on the Participant tab for the synchronizer's health, and confirm the sequencer endpoint is still reachable from the cluster.

Also check the [Traffic](/catalyx-blockchain-manager/canton-network/version-2.0/console-guide-canton/validators/balances-and-traffic.md) tab: a participant that has exhausted both its base allowance and its purchased traffic is throttled by the synchronizer.

</details>

<details>

<summary>The Wallet Gateway component is missing</summary>

On a new validator using managed authentication, the Wallet Gateway waits for identity provider provisioning to finish before it starts. A missing or zero-replica Wallet Gateway on the first reconcile is expected — give it a cycle.

If it persists, the validator's `Ready` condition will name the missing field. The Wallet Gateway has more prerequisites than any other component; see [Wallet Gateway](/catalyx-blockchain-manager/canton-network/version-2.0/validator-management/wallet-gateway.md).

</details>

<details>

<summary>KMS is configured but keys are still in the database</summary>

Check the spelling and case of `spec.kms.provider`. It is matched case-sensitively — `Azure` or `AWS` silently disables KMS, and the validator otherwise reconciles normally. Use lowercase `azure` or `aws`.

</details>

***

## Parties

<details>

<summary>Allocation failed after the party signed</summary>

A named participant UID does not exist on the synchronizer. The signature is consumed; restart onboarding with a fresh party hint, and use the advisory lookup on every UID before signing.

</details>

<details>

<summary>A multi-hosted party is not usable</summary>

It needs authorization from **every** named host. Check which are outstanding, and note that all hosts must be on the same synchronizer — a participant that is absent from your topology store, including one whose validator has been offline long enough to fall behind, will not resolve.

</details>

<details>

<summary>I am named in a proposal but there is no Approve button</summary>

Nothing is required of you. A participant is only asked to authorize when it takes on more responsibility — when it is being added, or when its permission is being strengthened.

</details>

<details>

<summary>A hosting amendment signature is rejected</summary>

Check the **Key algorithm** selector in the Edit Hosting modal. It defaults to Ed25519 and does not detect the party's actual algorithm, so an ECDSA P-256 party needs it changed before signing.

</details>

***

## Console

<details>

<summary>A validator does not appear in the list</summary>

The console only shows validators in the namespace the API watches — `catalyx-system` by default. A validator in another namespace is also not being reconciled, so this is usually a real problem rather than a display one.

</details>

<details>

<summary>A tab shows "Failed to load…"</summary>

These tabs read live from Canton through the API, so the failure is upstream rather than in the console. Check that the component is `Ready`, then that the API can authenticate — the [Identity](/catalyx-blockchain-manager/canton-network/version-2.0/console-guide-canton/validators/identity-and-endpoints.md) tab failing to load its **Authenticated user** section points at the client secret, audience, or scope.

</details>

<details>

<summary>The Canton console will not connect</summary>

It attaches to a running participant pod. If none is running it reports so rather than connecting — check the validator's Summary tab first. **Reconnect** clears and re-establishes the session.

</details>

<details>

<summary>Nothing seems to update</summary>

The console polls continuously — status every three seconds, metrics every five. There is no refresh button and no "last updated" indicator, so stale data means the API is not responding rather than that you need to refresh. The exception is the Parties tab, which has explicit **Refresh** buttons because reading the topology store is expensive.

</details>

***

## Escalating

If none of the above helps, gather the following before raising a ticket:

```bash
kubectl -n catalyx-system get validators -o wide
kubectl -n catalyx-system get validator <name> -o yaml
kubectl -n catalyx-system get applications
kubectl -n catalyx-system logs deploy/catalyx-canton-operator --tail=500
kubectl -n catalyx-system get events --sort-by=.lastTimestamp | tail -50
```

{% hint style="danger" %}
Review the output before attaching it. `kubectl get validator -o yaml` includes secret **names** but not values — that is fine. Never attach an identity dump, a client secret, or database credentials.
{% endhint %}

See [Support & Resources](/catalyx-blockchain-manager/canton-network/version-2.0/support-and-resources.md) for how to reach the service desk.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.catalyx.solutions/catalyx-blockchain-manager/canton-network/version-2.0/support-and-resources/troubleshooting.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.