Accelerating Spatial Postgres: Varnish Cache for pg_tileserv using Kustomize
Recently I worked with one of my Crunchy Data PostgreSQL clients on implementing caching for pg_tileserv. pg_tileserv is a lightweight microservice to expose spatial data to the web and is a key service across many of our geospatial customer sites. pg_tileserv can generate a fair amount of database server load, depending on the complexity of the map data and number of end users, so putting a proxy cache such as Varnish in front of it is a best practice. Using Paul Ramsey's Production PostGIS Vector Tiles Caching as a starting point, I started to experiment with client standard toolset of Kustomize, OpenShift, and Podman. I found this was an easy and fast way to deploy Crunchy Data Kubernetes components and caching.
In this blog I would like to share with you a step-by-step guide on the process of deploying Varnish Caching for pg_tileserv using Kustomize in Redhat Openshift Container Platform and PGO, the Postgres Operator from Crunchy Data (PostgreSQL 13.x, PostGIS, pg_tileserv).
Prerequisites
- Access to Redhat Openshift Container Platform, with permission to deploy Kubernetes objects in the Namespace.
- Access to Openshift CLI 4.9 and knowledge of
oc
and Kubernetes commands. - Access to deployed PostgreSQL Operator for Kubernetes (PGO) (PostgreSQL 13.x, PostGIS, pg_tileserv) environment able to view deployed components information.
- pg_tileserv pod/deployment, services are deployed, and in our environment pg_tileserv service is named tileserv.
Deployment Steps
Varnish Image
Find the right Varnish Cache image for your use case and upload it to your private client repository. Below are the steps we used in our client environment:
podman login registry.redhat.io
podman pull registry.redhat.io/rhel8/varnish-6:1-191.1655143360
podman tag registry.redhat.io/rhel8/varnish-6:1-191.1655143360 client registry/foldername/varnish-6:1-191.1655143360
podman push client registry/foldername/varnish-6:1-191.1655143360
Create the VCL file
Create the default.vcl
file using a starting template from Varnish website.
VCL stands for Varnish Configuration Language and is the domain-specific
language used by Varnish to control request handling, routing, caching, and
several other things.
Configure backend .host
in default segment set to pg_tileserv service object,
in our case it was tileserv.
.host = tileserv
.port = 7800
Sample default.vcl
vcl 4.0;
backend default { .host = "tileserv"; .port = "7800"; }
Create varnish_deployment.yaml
I will highlight few things that are important and provide our example below.
- Configure the image in the client private repository
(
client private_registry/ varnish-6:1-191.1655143360
), use image to pull the secret if the private repository requires it. - Mount
tmpfs
,mountPath /var/lib/varnish, Volumes emptyDir: {}
. - Volume mount the varnish
configMap
, to/etc/varnish/default.vcl
. - Configure
containerPort
to8080
. - Configure Container
resources
, if this is not configured it will take default configuration configured at the cluster level, e.g. cpu request: 1M. - Configure pod commands to execute
varnishd
andvarnishncsa
.varnishcsa
command below adds varnish request info to the pod logs, this provides cache hit or miss information.
command: ["/bin/sh"]
args:
- -c
- |
varnishd -a :8080 -a :6081 -f /etc/varnish/default.vcl -s malloc,512M;
varnishncsa -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" % Varnish:handling}x %{Varnish:hitmiss}x'
Sample Varnish deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: varnish
labels:
app: varnish
spec:
replicas: 1
selector:
matchLabels:
app: varnish
template:
metadata:
labels:
app: varnish
spec:
containers:
- name: varnish
image: privateclientregistry/crunchydata/varnish-6:1-191.1655143360
imagePullPolicy: Always
command: ["/bin/sh"]
args:
- -c
- |
varnishd -a :8080 -a :6081 -f /etc/varnish/default.vcl -s malloc,512M;
varnishncsa -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" % Varnish:handling}x %{Varnish:hitmiss}x'
ports:
- containerPort: 8080
- containerPort: 6081
resources:
limits:
cpu: 200m
memory: 1024Mi
requests:
cpu: 100m
memory: 512Mi
volumeMounts:
- mountPath: /etc/varnish/default.vcl
name: varnish-config
subPath: default.vcl
defaultMode: 0777
- mountPath: /var/lib/varnish
name: tmpfs
defaultMode: 0777
volumes:
- name: varnish-config
configMap:
name: varnish-config
defaultMode: 0777
items:
- key: default.vcl
path: default.vcl
- name: tmpfs
emptyDir: {}
serviceAccount: pgo-default
securityContext:
runAsNonRoot: true
fsGroupChangePolicy: "OnRootMismatch"
terminationGracePeriodSeconds: 30
Create varnish_service.yaml
for the Kubernetes varnish service with the ClusterIP
Below is our example:
---
kind: Service
apiVersion: v1
metadata:
name: varnish-svc
labels:
name: varnish
spec:
selector:
app: varnish
ports:
- protocol: TCP
port: 8080
targetPort: 8080
type: ClusterIP
Create varnish_route.yaml
for a secure Varnish route
In our case we have created edge route with no certificates to enable https. Our recommendation is to follow your use case security guidelines and Openshift documentation. Below is an example:
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: varnish
labels:
name: varnish
spec:
tls:
insecureEdgeTerminationPolicy: Redirect
termination: edge
to:
kind: Service
name: varnish-svc
weight: 100
port:
targetPort: 8080
wildcardPolicy: None
Create kustomize.yaml
Here is an example manifest:
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- varnish_deployment.yaml
- varnish_service.yaml
- varnish_route.yaml
configMapGenerator:
- name: varnish-config
files:
- default.vcl
Deploy
The final step is to list files and deploy kustomize manifest using Openshift
cli. Assumption here is that kustomize.yaml
is in current directory.
$ ls
# default.vcl
# kustomization.yaml
# varnish_deployment.yaml
# varnish_route.yaml
# varnish_service.yaml
oc apply -k .
Troubleshooting
Adjust Default VCL Cookies
After the initial deployment the Varnish pod log was showing most of the pages
as a miss on the cache hit/miss. After analyzing, I found that set-cookie header
was added to the request and response. Add default.vcl
unset the cookies:
sub vcl_recv {
if (req.url ~ "^[^?]*\.(pbf)(\?.*)?$") {
unset req.http.Cookie;
unset req.http.Authorization;
# Only keep the following if VCL handling is complete
return(hash);
}
}
sub vcl_backend_response {
if (bereq.url ~ "^[^?]*\.(pbf)(\?.*)?$") {
unset beresp.http.Set-Cookie;
set beresp.ttl = 1d;
}
}
Redeploy Varnish, assumption kustomize.yaml is in current directory.
# delete the deployment
oc delete -k .
# Deploy
oc apply -k .
Encountered Errors and Resolution
Errors | Resolution |
---|---|
Permission denied cannot create <some path> vsm? and pod restarting with CrashLoopbackoff error. | Check tmpfs , set mountPath to the path mentioned in the error, also map the Volumes to emptyDir: {} |
503 Backend fetch failed | default.vcl backend .host and .port in default segment is not mapped correctly to pg_tileserv Openshift service |
Port 80 permission denied | Configure ContainerPort to 8080 |
Pod logs show all Cache pages are MISS | In default.vcl Configure vcl_recv and vcl_backend_response functions to unset req.http.Cookie and beresp.http.Set-Cookie, after our deployment we found that pages were not caching as our webserver was setting cookies on every page request this meant that every page was a new page and the hash value calculated by the Varnish cache is going to be different. |
Closing Thoughts
Our clients have a complex mapping data. pg_tileserv was generating a fair amount of database server load and rendering could be slow at times. Implementing Varnish caching improved map rendering performance by 25% to 30% and equally reduced load on our database. I hope this blog was helpful, and we at Crunchy Data wish you happy learnings.
![Avatar for Rekha Khandhadia](webp/rekha-khandhadia.png-ovt7vyji.webp)