Using GitOps to Self-Manage Postgres in Kubernetes

"GitOps" is a term that I've been seeing come up more and more. The concept was first put forward by the team at Weaveworks as a way to consolidate thought around deploying applications. In essence: your deployment topology lives in your git repository. You can update your deployment information by adding a new commit. Likewise, if you need to revert your system's state, you can rollback to the commit that you want to represent your production environment. Any changes to your deployment topology should be reconciled in your production environment.

A lot of the conversations around GitOps came around the Postgres Operator for Kubernetes and how to apply these principles. Platforms like Kubernetes make it relatively seamless to apply the ideas of GitOps to the stateless pieces of applications (e.g., your web application). More work needs to be done with a stateful service such as PostgreSQL, as each stateful application can have unique requirements. Let's take something like replication: PostgreSQL's replication system is both configured and managed differently than in other database systems, and any GitOps management tool, such as Helm or Kustomize, would have to account for this.

This is the beauty of the Operator pattern. It provides a generic framework for developers to allow for GitOps style management of a stateful application in a way that can be "simply" configured. I say "simply" because there is still a lot to consider when running a complex stateful application like a database in a production environment, but an Operator can make the overall management and application of changes easier.

There are many ways to support GitOps workflows in current versions of the PostgreSQL Operator, from Kubernetes YAML files to Helm charts to Kustomize manifests.

Let's work through a GitOps style workflow. The examples below will assume that you have installed the Postgres Operator.

Deploying a HA PostgreSQL Cluster

For the first example, let's create a HA PostgreSQL cluster called hippo using a Kubernetes YAML file. In your command-line environment, set the following environmental variables to where you want your PostgreSQL cluster deployed.

For this example, the namespace uses the same one created as part of the quickstart. You can copy and paste the below into your environment to try the example out. You can also modify the custom resource manifest to match your specific environment. Note that if your environment does not support environmental variables, you can find/replace the below values in your manifest file.

From your command line, execute the following:

# this variable is the namespace the cluster is being deployed into
export cluster_namespace=pgo
# this variable is the name of the cluster being created
export pgo_cluster_name=hippo
# this variable sets the default disk size
export cluster_disk_size=5Gi

cat <<-EOF > "${pgo_cluster_name}-pgcluster.yaml"
apiVersion: crunchydata.com/v1
kind: Pgcluster
metadata:
  annotations:
    current-primary: ${pgo_cluster_name}
  labels:
    crunchy-pgha-scope: ${pgo_cluster_name}
    deployment-name: ${pgo_cluster_name}
    name: ${pgo_cluster_name}
    pg-cluster: ${pgo_cluster_name}
    pgo-version: 4.6.0
  name: ${pgo_cluster_name}
  namespace: ${cluster_namespace}
spec:
  BackrestStorage:
    accessmode: ReadWriteOnce
    size: ${cluster_disk_size}
    storagetype: dynamic
  PrimaryStorage:
    accessmode: ReadWriteOnce
    name: ${pgo_cluster_name}
    size: ${cluster_disk_size}
    storagetype: dynamic
  ReplicaStorage:
    accessmode: ReadWriteOnce
    size: ${cluster_disk_size}
    storagetype: dynamic
  ccpimage: crunchy-postgres-ha
  ccpimageprefix: registry.developers.crunchydata.com/crunchydata
  ccpimagetag: centos8-13.1-4.6.0
  clustername: ${pgo_cluster_name}
  database: ${pgo_cluster_name}
  exporterport: "9187"
  limits: {}
  name: ${pgo_cluster_name}
  namespace: ${cluster_namespace}
  pgDataSource: {}
  pgbadgerport: "10000"
  pgoimageprefix: registry.developers.crunchydata.com/crunchydata
  podAntiAffinity:
    default: preferred
    pgBackRest: preferred
    pgBouncer: preferred
  port: "5432"
  replicas: "1"
  user: ${pgo_cluster_name}
  userlabels:
    pgo-version: 4.6.0
EOF

kubectl apply -f "${pgo_cluster_name}-pgcluster.yaml"

This manifest tells the PostgreSQL Operator to create a high availability PostgreSQL cluster. It will take a few moments to get everything provisioned. You can check on the status using kubectl get pods or using the pgo test command if you have installed the PostgreSQL Operator client. For example, the below demonstrates that there are two PostgreSQL instances in the hippo cluster available:

kubectl -n "${cluster_namespace}" get pods  --selector="pg-cluster=${pgo_cluster_name},pgo-pg-database"
NAME                                          READY   STATUS      RESTARTS   AGE
hippo-585bb4f797-f76bl                        1/1     Running     0          3m32s
hippo-gftf-7f55674d78-gsjx7                   1/1     Running     0          2m53s

Let's log into our newly provisioned database cluster. In our example above, we created a database named hippo along with a user called hippo. The Postgres Operator creates user credentials for several defaults users, including any users we specify. To get the credentials, assuming you still have the environmental variables set from the earlier step, you can use the following command:

kubectl -n "${cluster_namespace}" get secrets "${pgo_cluster_name}-${pgo_cluster_name}-secret" -o jsonpath="{.data.password}" | base64 -d

For convenience for connecting to the cluster, we can store the user's password directly in an environmental variable that psql recognizes:

export PGPASSWORD=$(kubectl -n jkatz get secrets "${pgo_cluster_name}-${pgo_cluster_name}-secret" -o jsonpath="{.data.password}" | base64 -d)

In a separate terminal window, open up a port-forward:

export cluster_namespace=pgo
export pgo_cluster_name=hippo
kubectl port-forward -n "${cluster_namespace}" "svc/${pgo_cluster_name}" 5432:5432

Now, back in the original window, you can now connect to the cluster:

psql -h localhost -U "${pgo_cluster_name}" "${pgo_cluster_name}"
psql (13.1)
Type "help" for help.

hippo=>

Success!

Adding More Resources to a Deployed Cluster

Part of the GitOps principle is the ability to modify (or version) a configuration file and have the changes reflected in your environment. This is typical of "Day 2" type of operations, such as requiring more memory / CPU resources as the workload on a database increases.

Let's say we want to raise our memory limits to 2Gi and our CPU limit to 2.0 cores. Open up the file you created in the previous step, and add the following block to the spec:

limits:
  memory: 2Gi
  cpu: 2.0

Your file should look similar to this:

apiVersion: crunchydata.com/v1
kind: Pgcluster
metadata:
  annotations:
    current-primary: hippo
  labels:
    crunchy-pgha-scope: hippo
    deployment-name: hippo
    name: hippo
    pg-cluster: hippo
    pgo-version: 4.6.0
  name: hippo
  namespace: pgo
spec:
  BackrestStorage:
    accessmode: ReadWriteOnce
    size: 5Gi
    storagetype: dynamic
  PrimaryStorage:
    accessmode: ReadWriteOnce
    name: hippo
    size: 5Gi
    storagetype: dynamic
  ReplicaStorage:
    accessmode: ReadWriteOnce
    size: 5Gi
    storagetype: dynamic
  ccpimage: crunchy-postgres-ha
  ccpimageprefix: registry.developers.crunchydata.com/crunchydata
  ccpimagetag: centos8-13.1-4.6.0
  clustername: hippo
  database: hippo
  exporterport: '9187'
  limits:
    cpu: 2.0
    memory: 2Gi
  name: hippo
  namespace: pgo
  pgDataSource: {}
  pgbadgerport: '10000'
  pgoimageprefix: registry.developers.crunchydata.com/crunchydata
  podAntiAffinity:
    default: preferred
    pgBackRest: preferred
    pgBouncer: preferred
  port: '5432'
  replicas: '1'
  user: hippo
  userlabels:
    pgo-version: 4.6.0

After saving your changes, apply the updates to your Kubernetes environment:

kubectl apply -f "${pgo_cluster_name}-pgcluster.yaml"

Upon detecting the change, the Postgres Operator uses a rolling update strategy to apply the resource changes to each PostgreSQL instance in a way that minimizes downtime. Wait a few moments for the changes to be apple, and then run a describe on the Pods to see the changes (output truncated):

kubectl -n jkatz describe pods --selector="pg-cluster=${pgo_cluster_name},pgo-pg-database"
Name:         hippo-585bb4f797-f76bl
Namespace:    pgo
Containers:
  database:
    Limits:
      cpu:     1000m
      memory:  2Gi
    Requests:
      cpu:      1000m
      memory:   2Gi

Name:         hippo-gftf-7f55674d78-gsjx7
Namespace:    pgo
Containers:
  database:
    Limits:
      cpu:     1000m
      memory:  2Gi
    Requests:
      cpu:      1000m
      memory:   2Gi

Excellent! And using GitOps principles, if I wanted to revert these changes, I could do so by removing the "limits" clause and the Postgres Operator would update the cluster as such.

For a full list of attributes that can be updated, please see the custom resources section of the Postgres Operator documentation.

Evolution of GitOps and Stateful Services

When coupled with a tool like the PostgreSQL Operator, GitOps principles can be extended to work for stateful services. Applying a GitOps mindset to managing PostgreSQL workloads can help make it easier to manage production-grade Postgres instances on Kubernetes. GitOps principles can certainly make it easier for deploying a range of PostgreSQL topologies, from single instances to multi-zone, fault tolerant clusters.

Upcoming posts will look at how some of the other Kubernetes toolsets can make it even easier to work with the PostgreSQL Operator in a GitOps manner.