Connaisseur v2.2 - Improving Usability of Container Signature Validation in Kubernetes

Connaisseur is a Kubernetes admission controller for container image signature verification. The latest release v2.2 improves usability and compatibility.

Connaisseur allows you to verify the signatures of container images before deployment to a Kubernetes cluster. To learn more about Connaisseur, checkout the docs or read our previous articles:

If you want to get started using Connaisseur, there is - as always - a quick demo below ...

So what's new?

While the detailed changes in version 2.2 can be found in the release log, the overarching scope is usability and compatibility:

  • More native Helm integration
  • Simplified and more stable administration (deploy, update, delete)
  • Better compatibility with different flavors of Kubernetes (e.g. OpenShift)
  • Better compatibility with different versions of Kubernetes (v1.16+)
  • Improved KMS support for Cosign

We are pleased to announce that Connaisseur Helm charts are now published in a Connaisseur Artifact Hub repository. As a consequence, Connaisseur can now be administered directly via Helm without the need to clone the whole repository, greatly facilitating integration into the CI/CD pipeline.

The public Helm chart repository for Connaisseur on Artifact Hub
The public Helm chart repository for Connaisseur on Artifact Hub

Furthermore, we reworked the way Connaisseur is deployed, updated and deleted in order to make administration faster and considerably more stable. Specifically, changing the Connaisseur configuration at runtime via helm upgrade was improved and now works with zero downtime. The default configuration has been adjusted to allow compatibility with different flavors and versions of Kubernetes. Tests for Kubernetes v1.16 and higher were implemented to ensure future compatibility. Finally, the key management service (KMS) support of Cosign for a range of Providers (e.g. AWS, Azure, GCP, HashiCorp, Kubernetes) has been experimentally integrated into Connaisseur.

So let's take a look at some of those improvements ...

Demo Time 🚀

Within the demo, we will install and customize Connaisseur to use our own KMS via the public Helm repository. Here is a screencast of the demo to get a feeling for Connaisseur in action:

[The video shows a screencast of adding Connaisseur’s Helm repository and installing the admission controller in default configuration. Next, the configuration is adjusted to use Kubernetes secret as a KMS for container image signature verification. The configurations are tested by creating pods with signed/unsigned images that are admitted/denied.](NLS8SsGJHiRL1nvmmVccSg.gif "Note: The lower section displays the Connaisseur Kubernetes resources.")

For the demo, you will need a test (!) Kubernetes cluster and image registry of your choice. Furthermore, Helm, Cosign and Docker should be installed and configured.

We start by adding the Connaisseur Helm repository and installing it in the connaisseur namespace:

$ helm repo add connaisseur https://sse-secure-systems.github.io/connaisseur/charts

\"connaisseur\" has been added to your repositories

$ helm install connaisseur connaisseur/connaisseur --atomic --create-namespace --namespace connaisseurNAME: connaisseur

LAST DEPLOYED: Sun Oct 17 14:39:58 2021
NAMESPACE: connaisseur
STATUS: deployed
REVISION: 1
TEST SUITE: None

You can verify the installation by checking the deployed resources via kubectl get all -n connaisseur or simply deploying some test images:

# unsigned image is denied
$ kubectl run demo1 --image=docker.io/securesystemsengineering/testimage:unsigned  # is denied

Error from server: admission webhook \"connaisseur-svc.connaisseur.svc\" denied the request: Unable to find signed digest for image docker.io/securesystemsengineering/testimage:unsigned.

# signed image is admitted
$ kubectl run demo2 --image=docker.io/securesystemsengineering/testimage:signed  # is accepted

pod/demo2 created

Connaisseur is now successfully enabled and you verified your first image! 🚀

Let's customize the configuration to verify a container image signed by ourselves, but let's do this as a KMS instead of directly configuring the public key. For simplicity, we will use a Kubernetes secret as a "KMS" here, since no external service or vendor is required. The secret can be created directly using Cosign:

$ cosign generate-key-pair k8s://connaisseur/cosign-keys

Enter password for private key: 
Enter password for private key again: 
Successfully created secret cosign-keys in namespace connaisseur
Public key written to cosign.pub

# check if keys were created
$ kubectl get secrets -n connaisseur cosign-keys

NAME          TYPE     DATA   AGE
cosign-keys   Opaque   3      92s

Connaisseur cannot, by default, access Kubernetes secrets, so we will have to manually grant access via a role and role binding:

$ cat << EOF | kubectl apply -f -         
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:                   
  name: connaisseur-kms-role                                                             
  namespace: connaisseur
  labels:                              
    app.kubernetes.io/name: connaisseur
rules:            
- apiGroups: [\"*\"]      
  resources: [\"secrets\"]
  verbs: [\"get\"]
---                                     
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:                          
  name: connaisseur-kms-rolebinding                                                      
  namespace: connaisseur
  labels:                              
    app.kubernetes.io/name: connaisseur
subjects:             
- kind: ServiceAccount            
  name: connaisseur-serviceaccount                                            
  namespace: connaisseur
roleRef:    
  kind: Role                
  name: connaisseur-kms-role         
  apiGroup: rbac.authorization.k8s.io
EOFrole.rbac.authorization.k8s.io/connaisseur-kms-role created
rolebinding.rbac.authorization.k8s.io/connaisseur-kms-rolebinding created

Next, we create the actual custom configuration for Connaisser in a values.yaml file:

validators:
# our new custom validator
- name: my-validator
  type: cosign
  trust_roots:
  - name: my-key
    key: k8s://connaisseur/cosign-keys
# the preconfigured Connaisseur public key is required!
- name: dockerhub-basics
  type: notaryv1
  host: notary.docker.io
  trust_roots:
  - name: securesystemsengineering-official
    key: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsx28WV7BsQfnHF1kZmpdCTTLJaWe
      d0CA+JOi8H4REuBaWSZ5zPDe468WuOJ6f71E7WFg3CVEVYHuoZt2UYbN/Q==
      -----END PUBLIC KEY-----policy:
# using the custom validator to validate all non-Connaisseur images
- pattern: \"*:*\"
  validator: my-validator
  with:
    trust_root: my-key
- pattern: \"docker.io/securesystemsengineering/*:*\"
  validator: dockerhub-basics
  with:
    trust_root: securesystemsengineering-official

Note that in case you use a registry that requires authentication, you also need to add an auth configuration to my-validator.

Let's update Connaisseur to use our new custom configuration:

$ helm upgrade connaisseur connaisseur/connaisseur -n connaisseur --wait -f values.yaml

Release \"connaisseur\" has been upgraded. Happy Helming!
NAME: connaisseur
LAST DEPLOYED: Sun Oct 17 15:06:31 2021
NAMESPACE: connaisseur
STATUS: deployed
REVISION: 2
TEST SUITE: None

To test our changes, we need a suitable image in a registry. Either use your own, or simply re-tag a public image:

# give it a name via IMG environment variable
$ export IMG=<registry>/<your-repo>/demo:v1
$ docker pull docker.io/library/hello-world
$ docker tag docker.io/library/hello-world $IMG
$ docker push $IMG

However, we have not signed this image so far, and if we try deploying it, it will be denied:

$ kubectl run my-img --image=$IMG

Error from server: admission webhook \"connaisseur-svc.connaisseur.svc\" denied the request: No trust data for image \"<registry>/<your-repo>/demo:v1\".

So let`s sign it and try again:

$ cosign sign -key k8s://connaisseur/cosign-keys $IMG
$ kubectl run my-img --image=$IMGpod/my-img created

Congratulations! You have rolled out your own custom configuration and verified your first container images.🎉

To clean up, delete the test containers and uninstall Connaisseur via:

$ helm uninstall connaisseur -n connaisseur

release \"connaisseur\" uninstalled

Why the Community is so important to us

What is the common denominator of the changes in version 2.2? Right: they were all inspired, requested or even directly contributed by the community, and that has been immensely helpful in steering the development of Connaisseur. While we do our best and run a significant number of linters, integration and unit tests on all changes, it is sheer impossible to check the behavior in all different environments and use-cases. Furthermore, feedback helps us to identify the critical features for future releases.

Besides the direct feedback on GitHub, we have also seen a lot of mentions and reviews in newsletters, blogs and podcasts which has been a great inspiration.

In essence: when developing an open source project, it`s difficult to understand what people think and what the community really needs. In that respect, GitHub stars give you motivation while issues give you focus.

Therefore, thanks for all the support 🙏. Please continue sharing your feedback via discussions and issues, and if you like Connaisseur, give it a ⭐️ on GitHub!

Dr. Christoph Hamsen
Christoph is part of our Defensive Security Team supporting our clients to design, build and operate secure solutions.