The Kubernetes API is really quite a beautiful thing: a RESTful API provided via HTTP, using consistent HTTP verbs and URIs for accessing API resources, and allowing for deployment and versioning of multiple APIs.
And it even allows for extension… via the API itself. If you’re reading this, you probably already know that, but every time I remember this it still impresses me!
The Kubernetes API resource used to declare API extensions is called
CRD for short) and it is in the
apiextensions.k8s.io/v1 API group/version. When a
CRD resource is created, the Kubernetes API server dynamically handles endpoints that follow the consistent API semantics as mentioned above. An API resource defined via a
CRD feels just as native as the core Kubernetes APIs. The
CRD defines the structure of the API resource for the Kubernetes API server to serve.
The Kubernetes API server will ensure that any requests to create or update instances (we’ll call this a
CR for the remainder of this post to distinguish from the definition,
CRD of the API itself) of the newly defined resource are valid: they contain only properties defined in the CRD and those properties contain valid values. Validation failures will cause the invalid request to be denied and failure messages returned to the client.
There are a few ways to validate the contents when creating an instance of this CRD:
- OpenAPI schema validation, usually defined via comments in code and generated into YAML manifests via
controller-gen(more details below). Validation via OpenAPI schema is performed in-process by the Kubernetes API server, directly returning errors to the client.
- Webhook validation, defined in code and deployed as part of a controller manager pod (not discussed in this article). Validation happens by the API server sending requests to webhooks configured via the API, aggregating failures, and returning these to the client.
- Common Expression Language validation, usually defined via comments in code and generated into YAML manifests via
controller-gen(more details below). Similar to OpenAPI schema validation, validation via CEL is performed in-process by the Kubernetes API server, directly returning errors to the client.
OpenAPI schema validation is the most basic as it only allows for validating types, formats, required, etc., and only can validate a single property in isolation. This is simple to understand, implement, and has the possibility of client-side support by virtue of using the widely supported OpenAPI schema to define validation rules.
Webhook validation (also know as validating admission webhooks) is the most complex, requiring writing and deploying code, but with the complexity comes the most power: you can add any validation logic in your code, validate the whole resource, reach out to external systems… basically anything!
Common Expression Language validation will be discussed below and allows for complex validation without writing and deploying code (well, other than CEL code). It does not have the same level of flexibility as webhook validation, but allows for complex validations not supported by OpenAPI schema validation.
Let’s work through an example, gradually adding validation rules. Skip straight ahead to the section on CEL validation if you’re familiar with CRD development - we’re going to go slow and build up to validation via CEL.
Demo Project Set Up
Let’s create a project using
kubebuilder (change any values you want to for your environment):
And create an API definition, our first CRD:
This will create files containing boilerplate for your CRD, which in this case is called
Placeholder. Create the deployment manifests, generated from the source code definition, via:
Now you have everything you need to deploy your first CRD.
Take a look at the source code for the definition of the
Placeholder struct that defines our CRD in Go code:
Note the kubebuilder annotations in Go comments starting with
//+kubebuilder:. These are what
controller-gen uses to generate the CRD definition when we ran
As you can see, all this is generated from the Go code comments.
Validating Properties via OpenAPI Schema
CRDs support OpenAPI schema validation. We can use kubebuilder annotations to add some simple validations, in this case let’s mark the
Foo field of the
PlaceholderSpec as an enum that only accepts
Regenerate the CRD manifests:
And looking at the generated manifests again the enum is added to the OpenAPI spec:
Deploy and Test Out the Validation
If you don’t have a cluster to use, then first create one with KinD:
Deploy the CRD and controller on to the cluster:
Test out the validation by trying to create an invalid CR:
Creating a valid resource:
As discussed above, OpenAPI schema validation is perfect for simple, single property validation. However, if you need to do more complex validations, then you’re going to need something more powerful. We could use webhook validation, but we’re going to explore CEL validation instead.
Validating Properties via Common Expression Language (CEL)
The validation rules feature (via the
CustomResourceValidationExpressions feature gate) moved to beta (and therefore enabled by default) in Kubernetes v1.25. Validation rules enable the use of CEL to validate custom resource values.
CEL describes itself as implementing
We’ll work through some examples below to try to highlight the benefits of using CEL for validation, but the major benefit is not having to maintain and deploy extra code for validations that can be expressed clearly and concisely via CEL. Note that CEL is really powerful and these examples are not exhaustive. Take a look at the Kubernetes documentation for further reading.
We can add CEL validation rules via kubebuilder annotations, just as we did with OpenAPI schema validation. Let’s add a couple of properties,
current, and add validation rules that
min <= max, and
min <= current <= max.
make manifests and look at the generated CRD manifest (showing just the relevant parts):
Our build annotations above combined both OpenAPI validations to add minimum, maximum, and defaults to the
current properties, and add the CEL validation rules via the
x-kubernetes-validations schema extension. This is used by the API server to run the specified validations.
Notice how we set the validation rule on the
PlaceholderSpec object in our Go code. This sets the scope of the rule, i.e. the
self variable in the CEL validation will be set to the value of the property that is annotated when the rule is evaluated. This allows for access to any properties below this point in the object, but not up. Be mindful of this when specifying your validation rules, and limit the scope as much as necessary by placing your validation rules at the appropriate level in your object.
Let’s test these validation rules out by first deploying them:
And then updating the CR we created earlier:
Again our validations worked! The error message is what we specified on the Kubebuilder annotation. Validation rules actually support a wider range of options related to the client response, but these are not directly supported by Kubebuilder annotations at the time of writing. See the validation rules for more details - you can of course edit the generated CRD manifest (or use Kustomize patches) if you want to add more options to the validation rules.
Kubernetes CEL validation also includes something called transition rules, which enables comparing the requested state with the previous state. This allows for some very common use cases that could previously only be achieved by creating validating admission webhooks, in particular immutable properties and enforcing what values are allowed to be set considering the previous value, i.e. only allowing valid requested state changes.
A transition rule is implicitly created by referencing the
oldSelf variable. Let’s implement this to make the
max property immutable by adding the kubebuilder annotation directly on the
Max field in our struct, limiting the scope of the rule as much as possible:
Generating and deploying our updated CRD as we did before:
And testing it out, first by creating a valid CR:
Then trying to change the
This post gives a brief introduction to using CEL for validating your CRDs. The CEL implementation in Kubernetes has a pretty extensive library of validation functions to use in your CEL expressions - it is pretty powerful stuff.
As we have slowly started migrating to CEL in our CRDs, we have been able to delete the code for a number of our validating webhooks, without sacrificing data integrity or expressiveness. While CEL is not necessarily simple for non-developers to understand, it also has the benefit of appearing in the CRD spec, which adds a layer of transparency compared to the opaque nature of validating webhooks.
As CEL is becoming more widely used in Kubernetes (e.g. validating admission policy graduating to beta in Kubernetes v1.28), it is a good time to familiarize yourself with it and see where you can apply it to simplify and enhance your application deployments.