Using Cert Manager to Deploy TLS for Postgres on Kubernetes

There is no cure-all when it comes to security. Making things open to the public internet can leave you vulnerable to various security risks, including zero-day vulnerabilities or various password attacks. You can put your services in a VPN or a private network, but you can still be susceptible to an internal compromise.

While security for most people isn't a "fun" thing to spend time on, it's a necessity. You shouldn't have to trade off between security and ease-of-use. This was one big reason for an important design decision in v5 of Crunchy Postgres for Kubernetes.

One of the cornerstones in network security is encrypting traffic between two services. TLS is a cryptographic protocol that allows data to be securely transmitted from point A to point B. In an ever-changing environment like Kubernetes where you have lots of systems communicating with each other, knowing who you're talking to and that your conversation is secure seems like an obvious choice.

By default, you get your own native TLS management for Postgres with PGO, the open source Postgres Operator from Crunchy Data, which for most of you should be set and forget. However, you may want to set up your own TLS for your Postgres cluster. Maybe it's that you want your own CA, maybe it's due to organizational requirements; whatever the reason, you have flexibility so you can secure communication however you see fit.

Let's look at how we can use cert-manager, a popular open source utility for certificate management and how we can use it to deploy TLS on Postgres clusters. Major kudos to my colleague Brian Pace for researching how to accomplish this.

Getting Started with cert-manager

I recommend reading the cert-manager installation documentation for details on how to install cert-manager. For the purposes of this exercise, we'll use the one-liner:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml

This will installer cert-manager into a namespace called cert-manager.

Once cert-manager is installed, we need to create a certificate issuer. A certificate issuer is responsible for managing certificates, and can be configured to manage certificates in a single Kubernetes namespace or across a whole Kubernetes cluster.

There is an example of a cluster-wide certificate issuer In the Postgres Operator Examples repository. Fork this repository (we'll need it for the next example) and run the following command:

kubectl apply -k kustomize/certmanager/certman

This will create the following objects:

• A cluster-wide (ClusterIssuer) self-signed certificate issuer.
• A common certificate authority (CA) certificate.
• A cluster-wide (ClusterIssuer) CA certificate issuer using the generated CA certificate.

The CA certificate issuer is an important piece for setting up TLS in Postgres clusters. Postgres requires a reference to a CA when setting up TLS and will be used as the ca.crt component in the PGO custom TLS Secret.

The CA certificate issuer is important as the Postgres components require that the ca.crt be the same for the certificates generated to support Postgres.

Now that we have cert-manager set up, let's use cert-manager to generate custom, managed TLS certificates for a Postgres cluster.

Setting Up cert-manager TLS Certificates for Postgres

A PGO-managed Postgres cluster will have at least two certificates. One certificate is for the Postgres cluster itself and is used to both identify the cluster and encrypt communications. The second certificate is used for replication authentication.

The cluster certificate should contain values that can identify the cluster, such as its DNS name. You can connect to Postgres primary instance within a Kubernetes using the <clusterName>-primary service.For example, if my cluster is named hippo and is deployed in a namespace called postgres-operator, the hostname you would connect to is hippo-primary.postgres-operator.svc. You want that name to be represented in the DNS section of your certificate.

The replication certificate needs to have a specific common name (CN) in it: _crunchyrepl. This is a PGO managed user account used for replication, a key part of Postgres high availability. Postgres uses the common name (CN) as part of its certificate-based authentication system.

You can store the TLS certificates in Kubernetes Secrets in the same namespace as your Postgres cluster. For more information on setting up custom TLS certificates with PGO, please refer to the documentation.

Let's create some cert-manager managed certificates as described above. These certificates will be part of a Postgres cluster named hippo. If you want to skip ahead, you can run the example in the PGO examples repository:

https://github.com/CrunchyData/postgres-operator-examples/tree/main/kustomize/certmanager

First, let's create a certificate for the Postgres cluster:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: hippo-certmanager
spec:
 # Secret names are always required.
 secretName: hippo-tls
 duration: 2160h # 90d
 renewBefore: 360h # 15d
 subject:
    organizations:
    - crunchydata
  commonName: hippo-primary
  isCA: false
  privateKey:
    algorithm: ECDSA
    size: 256
  usages:
    - digital signature
    - key encipherment
 # Set DNS names to the names of the Postgres hosts
  dnsNames:
  - hippo-primary
  - hippo-primary.postgres-operator.svc
 issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
 group: cert-manager.io

Adjust the values to match the names and fields for your cluster.

Now, create a certificate for the _crunchyrepl user:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hippo-repl-certmanager
spec:
 # Secret names are always required.
  secretName: hippo-repl-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
   organizations:
    - crunchydata
  # This needs to be set to _crunchyrepl to work with Postgres
 # cert auth
  commonName: _crunchyrepl
  isCA: false
  privateKey:
    algorithm: ECDSA
    size: 256
  usages:
    - digital signature
    - key encipherment
  # At least one of a DNS Name, URI, or IP address is required, so we'll
 # have it match the CN for the benefit of Postgres
 dnsNames:
  - _crunchyrepl
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
    group: cert-manager.io

You now have two cert-manager managed certificates for Postgres cluster. The final step is to create your Postgres cluster. Below is an example of a high availability Postgres cluster that uses the two certificates we just created:

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: hippo
spec:
  image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.3-0
 postgresVersion: 13
  customReplicationTLSSecret:
    name: hippo-repl-tls
  customTLSSecret:
    name: hippo-tls
  instances:
    - replicas: 2
      dataVolumeClaimSpec:
       accessModes:
        - "ReadWriteOnce"
        resources:
          requests:
            storage: 1Gi
 backups:
    pgbackrest:
      image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.33-0
      repoHost:
        dedicated: {}
      repos:
      - name: repo1
        volume:
          volumeClaimSpec:
          accessModes:
            - "ReadWriteOnce"
            resources:
              requests:
               storage: 1Gi

And that's it, you've now created a Postgres cluster with TLS certificates generated and managed by cert-manager!

Conclusion

While the Postgres Operator makes it as safe and convenient as possible to have TLS in your cluster, you or your organization may have your own PKI or certificate requirements. We get it! That's why we made it simple to bring your own TLS to Postgres clusters.