[BUG] - Errors with keycloak on Nebari upgrade to 2026.3.1

[sumphlet]

Describe the bug

Hello, we attempted to upgrade our Nebari installation to 2026.3.1 but we receive errors when it attempts to run the keycloak pods. I have pasted the error in the section below.

Expected behavior

The upgrade should complete successfully without errors with the keycloak pods.

OS and architecture in which you are running Nebari

RHEL 9 x86_64

How to Reproduce the problem?

Performing the usual upgrade steps outlined at: https://nebari.dev/classic/how-tos/nebari-upgrade/

Command output

I have pasted the error below. These are taken from the log for the keycloak-keycloakx-0 pod.

dbchecker Waiting for Database to become ready...
dbchecker .Database OK ✓
stream closed: EOF for dev/keycloak-keycloakx-0 (dbchecker)
initialize-spi-metrics-jar Connecting to github.com (140.82.116.3:443)
initialize-spi-metrics-jar wget: note: TLS certificate validation not implemented
initialize-spi-metrics-jar Connecting to release-assets.githubusercontent.com (185.199.108.133:443)
initialize-spi-metrics-jar saving to '/data/keycloak-metrics-spi-7.0.0.jar'
initialize-spi-metrics-jar keycloak-metrics-spi 100% |********************************|  151k  0:00:00 ETA
initialize-spi-metrics-jar '/data/keycloak-metrics-spi-7.0.0.jar' saved
initialize-spi-metrics-jar /data/keycloak-metrics-spi-7.0.0.jar: OK
stream closed: EOF for dev/keycloak-keycloakx-0 (initialize-spi-metrics-jar)
keycloak Changes detected in configuration. Updating the server image.
keycloak Updating the configuration and installing your custom providers, if any. Please wait.
keycloak 2026-07-29 18:17:15,531 WARN  [org.keycloak.services[] (build-5) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
keycloak 2026-07-29 18:17:15,858 WARN  [org.keycloak.services[] (build-5) KC-SERVICES0047: metrics-listener (org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory) is implementing the internal SPI eventsListener. This SPI is internal and may change without notice
keycloak 2026-07-29 18:17:18,848 INFO  [io.quarkus.deployment.QuarkusAugmentor[] (main) Quarkus augmentation completed in 4816ms
keycloak Server configuration updated and persisted. Run the following command to review the configuration:
keycloak
keycloak     kc.sh show-config
keycloak
keycloak Next time you run the server, just run:
keycloak
keycloak     kc.sh start --optimized
keycloak
keycloak WARNING: Hostname v1 options [hostname-path, hostname-strict-https[] are still in use, please review your configuration
keycloak 2026-07-29 18:17:32,672 INFO  [org.keycloak.url.HostnameV2ProviderFactory[] (main) If hostname is specified, hostname-strict is effectively ignored
keycloak 2026-07-29 18:17:37,020 INFO  [org.keycloak.quarkus.runtime.storage.database.liquibase.QuarkusJpaUpdaterProvider[] (main) Updating database. Using changelog META-INF/jpa-changelog-master.xml
keycloak 2026-07-29 18:18:04,520 INFO  [org.keycloak.storage.datastore.DefaultMigrationManager[] (main) Migrating older model to 18.0.0
keycloak 2026-07-29 18:18:04,878 INFO  [org.keycloak.spi.infinispan.impl.embedded.JGroupsConfigurator[] (main) JGroups JDBC_PING discovery enabled.
keycloak 2026-07-29 18:18:06,014 INFO  [org.keycloak.spi.infinispan.impl.embedded.JGroupsConfigurator[] (main) JGroups Encryption enabled (mTLS).
keycloak 2026-07-29 18:18:06,485 INFO  [org.infinispan.CONTAINER[] (main) Virtual threads support enabled
keycloak 2026-07-29 18:18:06,625 INFO  [org.keycloak.jgroups.certificates.CertificateReloadManager[] (main) Starting JGroups certificate reload manager
keycloak 2026-07-29 18:18:06,762 INFO  [org.infinispan.CONTAINER[] (main) ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller'
keycloak 2026-07-29 18:18:06,933 INFO  [org.infinispan.CLUSTER[] (main) ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping`
keycloak 2026-07-29 18:18:06,935 INFO  [org.jgroups.JChannel[] (main) local_addr: 00000000-0000-0000-0000-000000000001, name: keycloak-keycloakx-0-53167
keycloak 2026-07-29 18:18:06,945 INFO  [org.jgroups.protocols.FD_SOCK2[] (main) server listening on *:57800
keycloak 2026-07-29 18:18:06,974 INFO  [org.infinispan.CLUSTER[] (main) ISPN000094: Received new cluster view for channel ISPN: [keycloak-keycloakx-0-53167|0] (1) [keycloak-keycloakx-0-53167[]
keycloak 2026-07-29 18:18:06,977 INFO  [org.keycloak.jgroups.certificates.CertificateReloadManager[] (main) Reloading JGroups Certificate
keycloak 2026-07-29 18:18:07,022 INFO  [org.infinispan.CLUSTER[] (main) ISPN000079: Channel `ISPN` local address is `keycloak-keycloakx-0-53167`, physical addresses are `[10.42.2.111:7800[]`
keycloak 2026-07-29 18:18:08,878 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory[] (main) Node name: keycloak-keycloakx-0-53167, Site name: null
keycloak 2026-07-29 18:18:09,433 INFO  [org.keycloak.storage.datastore.DefaultMigrationManager[] (main) Migrating older model to 20.0.0
keycloak 2026-07-29 18:18:09,512 INFO  [org.keycloak.storage.datastore.DefaultMigrationManager[] (main) Migrating older model to 21.0.0
keycloak 2026-07-29 18:18:09,517 INFO  [org.keycloak.storage.datastore.DefaultMigrationManager[] (main) Migrating older model to 22.0.0
keycloak 2026-07-29 18:18:09,611 INFO  [org.keycloak.migration.migrators.RealmMigration[] (main) migrated realm nebari to 22.0.0
keycloak 2026-07-29 18:18:09,659 INFO  [org.keycloak.migration.migrators.RealmMigration[] (main) migrated realm master to 22.0.0
keycloak 2026-07-29 18:18:09,659 INFO  [org.keycloak.storage.datastore.DefaultMigrationManager[] (main) Migrating older model to 23.0.0
keycloak 2026-07-29 18:18:09,677 INFO  [org.keycloak.migration.migrators.RealmMigration[] (main) migrated realm nebari to 23.0.0
keycloak 2026-07-29 18:18:09,685 INFO  [org.keycloak.migration.migrators.RealmMigration[] (main) migrated realm master to 23.0.0
keycloak 2026-07-29 18:18:09,685 INFO  [org.keycloak.storage.datastore.DefaultMigrationManager[] (main) Migrating older model to 24.0.0
keycloak 2026-07-29 18:18:09,762 INFO  [com.arjuna.ats.jbossatx[] (main) ARJUNA032014: Stopping transaction recovery manager
keycloak 2026-07-29 18:18:09,870 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler[] (main) ERROR: Failed to start server in (production) mode
keycloak 2026-07-29 18:18:09,870 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler[] (main) ERROR: ldapErrorEditModeMandatory
keycloak 2026-07-29 18:18:09,870 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler[] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
stream closed: EOF for dev/keycloak-keycloakx-0 (keycloak)

The deploy eventually fails after repeating this line:

Still waiting... (0/1 ready)

and with:

  ⚠ Warning: Timed out waiting for StatefulSet to be ready
  The StatefulSet may still be starting up

✓ Renamed backup file to /home/nebari/nebari-infra/nebari/keycloak-backup.sql.restored
================================================================================

Versions and dependencies used.

No response

Compute environment

None

Integrations

Keycloak

Anything else?

No response

[sumphlet]

I have done a backup of data per this guide: https://nebari.dev/classic/how-tos/manual-backup

I'm wondering if we can just do a nebari destroy and re-deploy from scratch, and just re-import this data manually.

[sumphlet]

I believe I fixed the issue. I had to add a line to my keycloak-backup.sql file in the section for importing data to the public.component_config table (I just generated the UUID on the first column)

ca76cb52-9d26-4084-b4a9-3ed2aced29dd    9cbe440b-362c-47dc-a1c3-bf9d8bc2e6a4    editMode        READ_ONLY

I am guessing this is a setting or attribute that is not checked with the old version of keycloak used on Nebari, and since our SAML config didn't have it it was failing. Perhaps there should be a check for this setting for other folks using SAML as their Identity Provider that are upgrading.