vSphere with Tanzu API and Ingress Certificate Replacement

I wrote about enabling vSphere with Tanzu here. Much hasn’t changed about the pre-reqs and the procedure when it comes to the latest version of vSphere 7.0 U3, except for a few cosmetic changes. I haven’t tested it on vSphere 8 yet. If there are changes, I would love to blog about it too 🙂

But today, I intend to cover about the certs that we see on the following UI. You can get to this UI by clicking on Cluster > Configure > Supervisor Cluster > Certificates.

Before diving deeper into the topic, I would like to highlight what these certs are used for.

Workload Platform Management

This is the tile where we find information about the kube-api server certificate that we use to connect to our Supervisor Cluster that we enabled. Basically, this certificate is the one that gets referred to when we do a “kubectl login“

If you have enabled workload management with the defaults, you should see a self-signed certificate (like the one in the image below) when you click View Details on the Workload Management tile.

Now let’s see what happens when I try to login to the supervisor cluster using the kubectl plugin.

The classic case of “x509: certificate signed by unknown authority”. This is because I haven’t added the vCenter CA as a trusted root on my jump box (more about it here). Per VMware documentation, though using –insecure-skip-tls-verify is supported, it is NOT a recommended practice. Thus, the blog 🙂

However, using the flag will let you login to the cluster as shown below.

Also, failing to replace a expired/invalid kube-api server certificate could impact the internal Kubernetes communication and could bring down the entire cluster – a situation no day 2 administrator would want to put themselves in!

NSX Load Balancer

This tile gives us info about the certificate that will be used for ingress objects within the supervisor cluster which can be retrieved through “kubectl get ing -A” command.

Once an ingress object is created, from a NSX standpoint, an LB with two virtual servers (one for HTTP and one for HTTPS) is created and this certificate is assigned to the LB. As you can see from the screenshot below, the VMCA cert is generated with defaultingress.local as the common name.

Setting up my CA server

For the context of this demo, I will use one of my Linux Jump boxes to sign the CSR’s that we will later generate as part of the procedure. To know more about setting up your own CA server, you can use this write up at deliciousbrains (strictly not for production environments). You can send the CSR’s to your in-house or custom CA for it to be signed.

Workload Management Certificate Replacement

Step 1: Click on “Generate CSR“

Step 2: Key in the required values and generate the CSR by clicking on Next. Copy/Download the CSR and send it to your CA for a signed cert. You can skip to step 5 if you are not signing the CSR using your own CA.

Step 3: Create a x509 V3 certificate extension config file with the content below.

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = tanzuuatapiserver.xxx.xxx 
IP.1  = 192.168.xx.xx 

As you can see above, I have also added the IP SAN as part of the cert config file just in case the cluster needs to be accessed via IP.

Step 4: Copy the downloaded csr to your CA server and execute the following command to sign the cert and get the new certs issued.

openssl x509 -req -in tanzuuatapiserver.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out kubeapiserver.crt -days 825 -sha256 -extfile caconf.ext

where caconf.ext is the name of the file created in Step 3 and the other flags’ values should be available if you had set up your CA server per deliciousbrains.

You will be prompted for the pass phrase that you configured while generating your CA Key file. Once entered, then you should have your kubeapiserver.crt in the location.

Step 5: Once you have the crt file, you can either cat its content and paste it on to the UI or upload the certificate using the upload option provided. Click on Replace Certificate as shown below and either copy or upload the cert file.

Once done, you should see a “Certificate replaced successfully“ message and now when you connect to the supervisor cluster without the insecure flag, you should be able to connect without any issues.

Ingress Certificate Replacement

VMware KB 87649 talks about the procedure to replace the cert and I am going to walk you through this with the screenshots from NSX.

If there are no ingress resources in your cluster, you will find this cert on NSX to be unused like shown below.

Step 1: Create an ingress resource manifest file as shown below. Using the manifest we will then deploy the ingress resource.

Note: Ignoring this step will result in NSX creating two certs for ingress when replaced through UI and there by creating a stale entry which then needs to be cleaned up via APIs.

apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: cafe-ingress
    annotations:
      kubernetes.io/ingress.class: "nsx"
  spec:
    tls:
    - hosts:
      - cafe.example.com
      secretName: cafe-secret
    rules:
    - host: cafe.example.com
      http:
        paths:
        - path: /tea
          pathType: ImplementationSpecific
          backend:
            service:
              name: tea-svc
              port:
                number: 80 

Step 2: kubectl apply -f ingress.yaml && kubectl get ingress -A

On the NSX UI, you should now see the ingress certificate being used.

Step 3: Click on “Generate CSR“

Step 4: Key in the required values and generate the CSR by clicking on Next. Copy/Download the CSR and send it to your CA for a signed cert. You can follow the steps above in the API cert replacement section to sign this CSR as well. Modify the config file accordingly.

Step 5: Upload the signed certificate using the Replace Certificate option. You can either copy paste the content into the box provided or upload the certificate.

Step 6: Once the cert is upload, you should see a “Certificate replaced successfully” message pop-up on the tile. Your NSX UI should like the one below.

This confirms that the cert has been replaced and I conclude the blog with this 🙂

PS: If someone is willing to write a code to automate the process by checking the validity of the certs and replacing it near expiry, we can explore the options available together, as codes and logics are not my cup of tea 😀

Also, if at all there is something that’s bothering about operationalizing your Tanzu/Kubernetes environment, please feel free to comment and I can help as much as I can! I hope that’s how we grow 🙂 Happy learning!