Using Cert Manager to Deploy TLS for Postgres on Kubernetes
There is no cure-all when it comes to security. Making things open to the public internet can leave you vulnerable to various security risks, including zero-day vulnerabilities or various password attacks. You can put your services in a VPN or a private network, but you can still be susceptible to an internal compromise.
While security for most people isn't a "fun" thing to spend time on, it's a necessity. You shouldn't have to trade off between security and ease-of-use. This was one big reason for an important design decision in v5 of Crunchy Postgres for Kubernetes.
One of the cornerstones in network security is encrypting traffic between two services. TLS is a cryptographic protocol that allows data to be securely transmitted from point A to point B. In an ever-changing environment like Kubernetes where you have lots of systems communicating with each other, knowing who you're talking to and that your conversation is secure seems like an obvious choice.
By default, you get your own native TLS management for Postgres with PGO, the open source Postgres Operator from Crunchy Data, which for most of you should be set and forget. However, you may want to set up your own TLS for your Postgres cluster. Maybe it's that you want your own CA, maybe it's due to organizational requirements; whatever the reason, you have flexibility so you can secure communication however you see fit.
Let's look at how we can use cert-manager, a popular open source utility for certificate management and how we can use it to deploy TLS on Postgres clusters. Major kudos to my colleague Brian Pace for researching how to accomplish this.
Getting Started with cert-manager
I recommend reading the cert-manager installation documentation for details on how to install cert-manager. For the purposes of this exercise, we'll use the one-liner:
kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
This will installer cert-manager into a namespace called cert-manager
.
Once cert-manager is installed, we need to create a certificate issuer. A certificate issuer is responsible for managing certificates, and can be configured to manage certificates in a single Kubernetes namespace or across a whole Kubernetes cluster.
There is an example of a cluster-wide certificate issuer In the Postgres Operator Examples repository. Fork this repository (we'll need it for the next example) and run the following command:
kubectl apply -k kustomize/certmanager/certman
This will create the following objects:
- A cluster-wide (
ClusterIssuer
) self-signed certificate issuer. - A common certificate authority (CA) certificate.
- A cluster-wide (
ClusterIssuer
) CA certificate issuer using the generated CA certificate.
The CA certificate issuer is an important piece for setting up TLS in Postgres
clusters. Postgres requires a reference to a CA when setting up TLS and will be
used as the ca.crt
component in the
PGO custom TLS Secret.
The CA certificate issuer is important as the Postgres components require that the ca.crt be the same for the certificates generated to support Postgres.
Now that we have cert-manager set up, let's use cert-manager to generate custom, managed TLS certificates for a Postgres cluster.
Setting Up cert-manager TLS Certificates for Postgres
A PGO-managed Postgres cluster will have at least two certificates. One certificate is for the Postgres cluster itself and is used to both identify the cluster and encrypt communications. The second certificate is used for replication authentication.
The cluster certificate should contain values that can identify the cluster,
such as its DNS name. You can connect to Postgres primary instance within a
Kubernetes using the <clusterName>-primary
service.For example, if my cluster
is named hippo
and is deployed in a namespace called postgres-operator
, the
hostname you would connect to is hippo-primary.postgres-operator.svc
. You want
that name to be represented in the DNS section of your certificate.
The replication certificate needs to have a specific common name
(CN) in it: _crunchyrepl
. This is a PGO managed user account used
for replication, a key part of
Postgres high availability.
Postgres uses the common name (CN) as part of its
certificate-based authentication
system.
You can store the TLS certificates in Kubernetes Secrets in the same namespace as your Postgres cluster. For more information on setting up custom TLS certificates with PGO, please refer to the documentation.
Let's create some cert-manager managed certificates as described above. These
certificates will be part of a Postgres cluster named hippo
. If you want to
skip ahead, you can run the
example
in the PGO examples repository:
https://github.com/CrunchyData/postgres-operator-examples/tree/main/kustomize/certmanager
First, let's create a certificate for the Postgres cluster:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: hippo-certmanager
spec:
# Secret names are always required.
secretName: hippo-tls
duration: 2160h # 90d
renewBefore: 360h # 15d
subject:
organizations:
- crunchydata
commonName: hippo-primary
isCA: false
privateKey:
algorithm: ECDSA
size: 256
usages:
- digital signature
- key encipherment
# Set DNS names to the names of the Postgres hosts
dnsNames:
- hippo-primary
- hippo-primary.postgres-operator.svc
issuerRef:
name: ca-issuer
kind: ClusterIssuer
group: cert-manager.io
Adjust the values to match the names and fields for your cluster.
Now, create a certificate for the _crunchyrepl
user:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: hippo-repl-certmanager
spec:
# Secret names are always required.
secretName: hippo-repl-tls
duration: 2160h # 90d
renewBefore: 360h # 15d
subject:
organizations:
- crunchydata
# This needs to be set to _crunchyrepl to work with Postgres
# cert auth
commonName: _crunchyrepl
isCA: false
privateKey:
algorithm: ECDSA
size: 256
usages:
- digital signature
- key encipherment
# At least one of a DNS Name, URI, or IP address is required, so we'll
# have it match the CN for the benefit of Postgres
dnsNames:
- _crunchyrepl
issuerRef:
name: ca-issuer
kind: ClusterIssuer
group: cert-manager.io
You now have two cert-manager managed certificates for Postgres cluster. The final step is to create your Postgres cluster. Below is an example of a high availability Postgres cluster that uses the two certificates we just created:
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
name: hippo
spec:
image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.3-0
postgresVersion: 13
customReplicationTLSSecret:
name: hippo-repl-tls
customTLSSecret:
name: hippo-tls
instances:
- replicas: 2
dataVolumeClaimSpec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: 1Gi
backups:
pgbackrest:
image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.33-0
repoHost:
dedicated: {}
repos:
- name: repo1
volume:
volumeClaimSpec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: 1Gi
And that's it, you've now created a Postgres cluster with TLS certificates generated and managed by cert-manager!
Conclusion
While the Postgres Operator makes it as safe and convenient as possible to have TLS in your cluster, you or your organization may have your own PKI or certificate requirements. We get it! That's why we made it simple to bring your own TLS to Postgres clusters.
![Avatar for Brian Pace](webp/brian-pace.png-n7tmywmv.webp)