Password logging not working properly after 2.14 upgrade

[doloban]

After upgrade to version 2.14 the password logging stopped working. I get the following message when I click on Login button in the CZERTAINLY login page:

The error message from DEBUG logs is as follows:
2025-01-06 15:04:58,298 WARN [org.keycloak.events] (executor-thread-75) type="LOGIN_ERROR", realmId="xyz", clientId="czertainly", userId="null", ipAddress="xyz", error="client_not_found"
2025-01-06 15:04:58,298 DEBUG [WebApplicationException] (executor-thread-75) Restarting handler chain for exception exception: org.keycloak.services.ErrorPageException: HTTP 500 Internal Server Error
at org.keycloak.protocol.oidc.endpoints.AuthorizationEndpoint.checkClient(AuthorizationEndpoint.java:241)
at org.keycloak.protocol.oidc.endpoints.AuthorizationEndpoint.process(AuthorizationEndpoint.java:138)
at org.keycloak.protocol.oidc.endpoints.AuthorizationEndpoint.buildGet(AuthorizationEndpoint.java:113)
at org.keycloak.protocol.oidc.endpoints.AuthorizationEndpoint$quarkusrestinvoker$buildGet_4b690b27439f19dd29733dc5fd4004f24de0adb6.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)

Our theory is that the problem is with client id, which is in the redirect URL (from the page in the screenshot) in all small letters (czertainly) and in default KC configuration is in all large letters (CZERTAINLY). It should be related to changes in the new version.

When we tried changing the client ID to the corresponding one in the redirect URL, so from "CZERTAINLY" to "czertainly", in the KC configuration menu, we received different error saying this: An error occurred: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: 401 Unauthorized: [no body].

[3keyroman]

Hello @doloban , did you follow the upgrade instructions here?:
https://docs.czertainly.com/docs/certificate-key/installation-guide/deployment/deployment-helm/upgrading#to-2140

In connection with new support of multiple OAuth 2.0 providers configuration, there was a change in the configuration of internal Keycloak. If you are upgrading, you should run the script to update the Keycloak client. Sample script can be found here: https://github.com/CZERTAINLY/CZERTAINLY-Helm-Charts/tree/master/charts/keycloak-internal/scripts/update_realm_from_2.7.0_to_2.14.0.py

In case of fresh deployment, you do not have to update Keycloak client.

Did you update your Keycloak configuration according to the upgrading documentation?

[doloban]

Hello,
I somehow missed the upgrade instructions, will take them into consideration next time.

I already did successful upgrade to new version in Virtual Appliance by raising version number to 2.14 in TUI and then installing CZERTAINLY.

Is it safe to run the script after complete upgrade from 2.13 to 2.14?

[3keyroman]

Yes, it should be safe. You can see what has been changed inside the python script and you can do it manually, if you prefer. The script should update the client automatically, providing the URL and credentials.

If you are running appliance and does not have publicly trusted certificates, you may need to skip the certificate validation as was described by @semik in this PR: CZERTAINLY/CZERTAINLY-Helm-Charts#214

[doloban]

We run the script and we got a bit further - the redirect after clicking on Login button works and user sees the page with the login form. Interestingly after running the script successfully the password logging started working only for me, not for the other Czertainly users though. Me and other users have the exactly same role, so it is odd. Other users get after an attempt to login through KC a red color "Internal Server Error" text on a blank white page after redirecting. We believe, that other users are successfully logged in the Keycloak, but the Czertainly has problems accessing users to the Czertainly app. We even see users in the
new "czertainly" client log sessions.

Any ideas? We did not find any errors or issue in the keycloak debug logs, in the core debug logs i only saw a json code showing connections from different IP addresses (other users) and localhost, together with connection response codes.

[3keyroman]

We did not experience such behavior. Especially, when it is working for one of the users.

Are you able to share the debug logs? (you can also share it privately on Discord)
Can you check settings for different users in CZERTAINLY and also in Keycloak?
Do you see any fields that are set for working users and not for non-working users or opposite?

[doloban]

I will send everything to you through Discord.