I had to Google this today and was kind of surprised by how many odd workarounds are used for this one, and most of them seem wrong.

What is it?

The error will create a traceback and it will look similar to this:

Fontconfig error: No writable cache directories
time=2024-08-05T17:57:28.924000 level=ERROR location=__init__.py:75:handle_exception msg="Error" request_id="" exception="Traceback (most recent call last):
  File \"/var/www/.cache/pypoetry/virtualenvs/pdf-maker-xS3fZVNL-py3.12/lib/python3.12/site-packages/flask/app.py\", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File \"/var/www/.cache/pypoetry/virtualenvs/pdf-maker-xS3fZVNL-py3.12/lib/python3.12/site-packages/flask/app.py\", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspace/pdf_maker/__init__.py\", line 127, in generate
    font_config = FontConfiguration()
                  ^^^^^^^^^^^^^^^^^^^
  File \"/var/www/.cache/pypoetry/virtualenvs/pdf-maker-xS3fZVNL-py3.12/lib/python3.12/site-packages/weasyprint/text/fonts.py\", line 108, in __init__
    self._folder = Path(mkdtemp(prefix='weasyprint-'))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/tempfile.py\", line 373, in mkdtemp
    prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/tempfile.py\", line 126, in _sanitize_params
    dir = gettempdir()
          ^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/tempfile.py\", line 126, in _sanitize_params
    dir = gettempdir()
          ^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/tempfile.py\", line 315, in gettempdir
    return _os.fsdecode(_gettempdir())
                        ^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/tempfile.py\", line 308, in _gettempdir
    tempdir = _get_default_tempdir()
              ^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/tempfile.py\", line 223, in _get_default_tempdir
    raise FileNotFoundError(_errno.ENOENT,
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/workspace']"

In our case, the error is triggered by rendering a PDF through Weasyprint, which is what we do in an internal service on Kubernetes. But generally: this error will happen when a Python library attempts to write to a temporary directory and fails.

But how’s this error triggered since /tmp and /var/tmp exist in the container?

Security

When we deploy on Kubernetes, we run containers without root and with a read-only (root) filesystem.

In most cases, the container will not write anything to disk. Logs are flushed to standard out, and persistent storage is usually an object storage. This allows the container to be truly ephemeral: TL;DR no state, aka “still the easiest way to scale”.

In addition, both of these measures try to make it harder to abuse a container and the host system if an attacker gains access.

Over the years, it’s been a good habit to use these measures for all services - internal and public. So whatever happens, happens — but the idea is to make it harder and not provide a root shell so easily.

Non root

To start a container non-root, you need to adjust the Pod definition - some settings omitted for brevity.

apiVersion: v1
kind: Pod
metadata:
  name: app
  namespace: namespace
spec:
  volumes:
    - name: tmp-volume
      emptyDir:
        sizeLimit: 800Mi
  containers:
    - name: app
      image: image:version
      ports:
        - name: http
          containerPort: 5000
          protocol: TCP
      env:
        - name: FLASK_APP
          value: pdf_maker
        - name: FLASK_ENV
          value: production
      volumeMounts:
        - name: tmp-volume
          mountPath: /tmp
      securityContext:
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        readOnlyRootFilesystem: true
  restartPolicy: Always
  securityContext:
    fsGroup: 33
    fsGroupChangePolicy: Always
    seccompProfile:
      type: RuntimeDefault

Let’s take a look in more detail.

containers: securityContext

securityContext:
  capabilities:
    drop:
      - ALL
  runAsNonRoot: true
  readOnlyRootFilesystem: true

There’s usually too much yaml - but thankfully this yaml is speaking for itself:

  • we set the container to run as a non-root user (requires e.g. USER 33:33 in your Dockerfile)
  • we enforce the root (/) to be mounted read-only

User 33 in our image is www-data; Kubernetes currently requires numeric IDs to be able to resolve it as an attacker could otherwise craft an image with a user that maps to 0.

Almost every Docker image comes with a nobody user which is suitable as well — inspect your /etc/passwd to find out more.

How do you fix this specific error?

The fix is included in the yaml above — we mount an emptyDir volume into /tmp and made sure we write to it exclusively.

Take a look at spec.volumes to see the definition of the emptyDir and spec.containers[0].volumeMounts for directions to mount it to /tmp.

Set your size limit accordingly — to whatever you expect to need. If you exceed an emptyDir, it’ll usually evict the pod and start from scratch. Bonus points if your application doesn’t fall over.

If you need persistence (“need files to stick around”), you have to use a PVC instead of an emptyDir or configure a hostPath if you run a single-node Kubernetes - ymmv.

spec: securityContext

This is usually more of a nice to have, but especially helpful in this case: we mount storage.

We set the same group (remember USER 33:33 above) and ensure permissions on volumes are updated to match the group permissions and therefor become writable.

Fin

Thanks for reading.

Always remember: good friends don’t let friends run containers with root. ;)