Installing Eclipse Che on the Azure Kubernetes Service (AKS)

We installed Eclipse Che on AKS (Azure Kubernetes Service) cluster integrated with AAD (Azure Active Directory) according to these old (7.41) instructions (there is no recent version of AKS installation documentation). And to our surprise it did not work. Here at Epam Systems we love complex challenges, so we decided not to stop and deep dived into Eclipse Che world to improve interaction with Microsoft Azure.

Note	Taking into account that Eclipse Che calls Kubernetes API server using user’s identity tokens generated by Identity Provider, AKS needs to be integrated with AAD, otherwise AKS will reject the token. Details how to integrate AKS with AAD can be found here.

Architecture diagram below can be used to get a context about our case. Please note that some of the components are omitted for the sake of simplicity. But it should be enough to get up to speed and dive into Eclipse Che world together with us.

Figure 1. Eclipse Che components deployed on AKS

Challenge 1: AKS does not accept `id_token`

OAuth2 Proxy is configured to send an id_token to Traefik for installations made for Kubernetes. Traefik, in turn, sends the id_token to all upstream services, including components such as Che server and User Dashboard. Che server and User Dashboard call AKS API Server using the provided id_token. But when AKS is configured to use AAD, its API Server does not accept id_token (it only accepts access_token).

How did we solve the first challenge

We need a way to ask OAuth2 Proxy to send an access_token instead of an id_token. Token replacement is a OAuth2 Proxy alpha feature, hence it cannot be used in production. The alternative is to configure it to pass access_token via X-Forwarded-Access-Token header, but the id_token in the Authorization header won’t be replaced. So, first step is to add access_token via X-Forwarded-Access-Token header:

pass_access_token = true

Then we need somehow to replace Authorization header on X-Forwarded-Access-Token header. Fortunately, we have Traefik in place. So, let’s ask Traefik to make a necessary replacement:

http:
  middlewares:
    che-header-rewrite:
      plugin:
        header-rewrite:
          from: X-Forwarded-Access-Token
          prefix: 'Bearer '
          to: Authorization

And voila, we made it work, like magic. But how can we make it usable for other users who wants to install Eclipse Che on AKS? We already mentioned, that at Epam Systems we love complex challenges. And of course we love coding. So, we opened the che-operator enhancements and proposed the improvements.

As a consequence of the proposed improvements, a new configuration field identityToken has been introduced into Eclipse Che starting from version 7.50. A Eclipse Che administrator can easily configure what token will be passed to upstream servicees in CheCluster CR (Custom Resource):

spec:
  networking:
    auth:
      identityToken: access_token

There are two types of token supported: id_token and access_token. Default value is id_token.

Note	The field `identityToken` is specific to Eclipse Che installations made for Kubernetes only and ignored for OpenShift.

Challenge 2: No way to obtain a proper access token accepted by AKS

This is a follow up of Challenge 1: after configuring Eclipse Che to pass an access_token instead of an id_token we need to make sure that AKS considers it valid.

When AKS integration with AAD is enabled, AKS expects OpenID Connect (OIDC) tokens to be issued from an application, pre-registered in AAD: 'Azure Kubernetes Service AAD Server'.

Figure 2. Azure Kubernetes Service AAD Server application

Eclipse Che Gateway OAuth2 Proxy with the default settings uses a standard set of OIDC scopes and AAD will return an access_token that can be used against Microsoft Graph but not against AKS.

In order to return a token that will be accepted by AKS, OAuth2 Proxy should be configured with an additional OIDC scope, representing Azure Kubernetes Service AAD Server application:

6dae42f8-4368-4678-94ff-3960e28e3630/user.read

where the ID is the Application ID of the AKS instance registered in AAD.

How did we solve the second challenge

A new configuration field named oAuthScope has been introduced into Eclipse Che starting from version 7.50. That has been specified by the already opened che-operator enhancements and implemented through the provided improvement. An Eclipse Che administrator can easily configure authorization scopes in CheCluster CR:

spec:
  networking:
    auth:
      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read

Note	The field `oAuthScope` is specific to Eclipse Che installations made for Kubernetes only and ignored for OpenShift.

Challenge 3: Che server requires mandatory 'email' claim which is absent in AKS access_token

Besides passing OIDC token to AKS for authorization purposes, Che server also uses it for creating a 'user' record in PostgreSQL database. There are two pieces of information (claims) about current user that Che server expects to extract from the token: name and email. Che allows configuring of user name claim via CheCluster CR in AKS (CHE_OIDC_USERNAME__CLAIM according to documentation), but email claim name is hardcoded to email and cannot be changed. This hardcode becomes a problem for AKS setup since access_token returned by AAD for AKS does not contain email claim (see current structure below), and customization is not possible here since AKS application registration in AAD is maintained by Microsoft.

{
  "typ": "JWT",
  "alg": "RS256",
  "x5t": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI",
  "kid": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI"
}.{
  "aud": "6dae42f8-4368-4678-94ff-3960e28e3630",
  "iss": "https://sts.windows.net/f974903a-f693-4ca2-aa63-45e5bd2d1318/",
  "iat": 1656412511,
  "nbf": 1656412511,
  "exp": 1656417802,
  "acr": "1",
  "aio": "AUQAu/8TAABB1o0F/h6L8SJWLibrQqPKrVF7qpdIFiQ0qeIj/wNUfcLkM87v+wgvQ+I8uj0WC2Y6WEoHlBcsJdt0m+kJ4dFeNw==",
  "amr": [
    "pwd"
  ],
  "appid": "1c0b88a0-8ec2-4cca-a2b0-4f468110ef86",
  "appidacr": "1",
  "family_name": "Kluklu",
  "given_name": "Tratata",
  "groups": [
    "73fdccdf-e10d-45f9-b7f6-31848842999f",
    "5988d043-3af9-4e81-b041-90b3456f9f4e",
    "bddcb049-0337-4dec-bbaf-7600b8c12623"
  ],
  "ipaddr": "10.123.51.3",
  "name": "Tratata Kluklu",
  "oid": "2cf0521c-c76d-4e7c-b41f-863674057db3",
  "onprem_sid": "S-2-4-31-6364504-298352422-13854118-387761",
  "puid": "3213CDFA3CAF2IA5",
  "rh": "0.AQkA1NIbtJ39JkuKaqbJ82fCHscCrm1oG3hTlP47YOQHDjAJAHg.",
  "scp": "user.read",
  "sub": "qweBLvHX49QA5WlXpJzq_erXQ2NldnSqpgY93oALLDY",
  "tid": "a385e78a-aedc-4033-82ba-e6ef88120591",
  "unique_name": "Tratata.Kluklu@gmail.com",
  "upn": "Tratata.Kluklu@gmail.com",
  "uti": "lfZmPsgcWmS3dG78GpMjRA",
  "ver": "1.0",
  "wids": [
    "c79abafb-610b-4a34-82e2-ef7a293db6ca"
  ]
}.[Signature]

How did we solve the third challenge

As for the previous challenges, we need some enhancements on Eclipse Che side here too. We want to allow administrators to configure what token claim need to be used to extract user email. As we did it before, we opened the che-server enhancement and proposed the improvement.

Now Eclipse Che adminstrators can configure the email claim to be used when parsing the JWT token:

spec:
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_EMAIL__CLAIM: unique_name

If not defined, the fallback value is email.

Conclusion

In this post, we walked through the challenges we faced to install Eclipse Che on AKS and how we contributed back to the project to address the issues.

Now, user has all needed things configurable to be able to run successfully Eclipse Che on AKS. For example, in our particular case we prepared yaml file that overrides the default values in CheCluster CR.

spec:
  networking:
    auth:
      identityProviderURL: https://sts.windows.net/{TENANT_ID}/v2.0/
      identityToken: access_token
      oAuthClientName: {CLIENT_ID}
      oAuthSecret: {CLIENT_SECRET}
      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://sts.windows.net/{TENANT_ID}/v2.0/
        CHE_OIDC_EMAIL__CLAIM: unique_name

TENANT_ID - Directory (tenant) ID, see Figure 3.
CLIENT_ID - Application (client) ID, see Figure 3.
CLIENT_SECRET - Client secret, you can manage it in 'Certificates & secret' section

Figure 3. Registered Che application

Warning

Don’t forget to configure API permissions to authorize your application to call AKS Server API.

Figure 4. AKS API permissions

After the Eclipse Che App configuration in Azure is completed, the command chectl server:deploy can be used to install Eclipse Che on AKS using the YAML file above:

chectl server:deploy \
       --platform=k8s \
       --installer=operator \
       --che-operator-cr-patch-yaml=che.yaml \
       --skip-oidc-provider-check \
       --skip-cert-manager \
       --domain=eclipse-che-demo.mydomain.com

Note	In our case we already configured `cert-manager` and created `domain` according to the old (7.41) instructions.

Challenge 1: AKS does not accept id_token