So, here’s a fun problem I ran into at work:

We use Github Actions to deploy all our services to Kubernetes. And in order to do a deployment, we have kube configs (as secrets) setup to connect to a Kubernetes cluster when a workflow runs.

The Kubernetes config has an expiration date and will become invalid when the Kubernetes certificates are rotated (k0s takes great care of that). When they expire, the deploys fail until we update the secret on Github Actions.

How we solved it

Once we understood the problem, the goal seemed straight forward:

  1. We use k0s(ctl) to download a new Kubernetes configuration
  2. We update the secrets on Github Actions

Details

To elaborate… once we understood the actual problem (expiration of the configuration), we looked into ways to automate the secret updates on Github. And what better way to update a secret, than another workflow as it minimizes the required infrastructure and credential handling on our end.

In order to update the secret, we are using the following tools:

k0s

Without going into great detail here. We usek0s (and k0sctl) to manage our Kubernetes clusters. One of the advantages of the tool is that it will take care of the entire lifecycle of a cluster — installation, maintenance and updates. Said maintenance includes rotation all the internal SSL certificates that are used when we kubectl.

k0sctl works based on a configuration file (which is also subject to another blog post). It connects to your cluster (via ssh) and has a command to download a configuration — and that’s what we use to download a new configuration.

Using k0sctl posed some difficulties, so for example, it took me more than a while to catch a bug because the entire output is piped to a file. Which then included the error (and made for an invalid kube config). The DEBUG environment variable is a bit too noisy and I did not have the time to figure out if the tool is really aware of stdout and stderr, or if the logs etc. are all a in one.

Working through this required lots of run: echo ./file until I got it right.

Github Apps

Talking to the Github API always requires a token. There are various options to use:

  • Developer Token: The first option to use Github’s new developer tokens (where the tokens have an expiration date) — great in theory, but also yet another thing you need to rotate. They are also also not great for control as a token is bound to a user account/seat. Either yourself, or a shared account.
  • Classic Token: The second option is a classic token. Classic tokens are also bound to an account/seat. And never expire. These tokens are bound to stay around for forever which is also not a great look.
  • Github Application: The third option is a Github Application.

The setup for a Github Application is relatively straight forward (if you ignore most of what you read about them).

The (for me) most confusing part is that after creating the app, I needed to install it into my organization.

Once I moved past this, all I needed to use it in a workflow are the application id and a private key (as secrets — GH_APP_ID and GH_APP_PRIVATE_KEY below) — both are provided when I created it.

To follow along:

  1. go to your org settings (https://github.com/organizations/YOUR-ORG/settings/apps)
  2. select the permissions to access (organization) secrets
  3. install the application for your organization

The advantages of a Github application are:

  • bound to your organization
  • not bound to an account/seat
  • bonus: by default, generated tokens are invalidated after the workflow ran

Github CLI

Last but not least — the Github CLI.

After I looked at various Actions (in the marketplace), I decided to use the cli instead.

Primarily because it’s already installed on the runner. But other reasons include that a third-party action implies trust with its author and also the additional burden of the potential maintenance on our end.

The command to update an organization secret (for Github Actions) is:

gh secret set SECRET_NAME \
  --app actions \
  --org YOUR-ORG \
  --visibility private \
  < ./file-name

Adding a repository to the command, would allow you to set/update a secret that is bound to a repository. Flags like --visibility designate if the secret can be used for public repositories — which we do not want. Inspect the output of gh secret set --help to learn more.

Workflow

Putting it all together, take a look at the following job (as part your workflows):

jobs:
  rotation:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - run: # setup ssh key
    - run: # install k0sctl
    - name: Pull kube config
      run: k0sctl kubeconfig --config ./cluster-name.yaml > ./k0s-kube.config
      env:
        SSH_KNOWN_HOSTS: /dev/null
    - run: # mask the config file
    - name: Generate token to update secrets
      id: generate-token
      uses: actions/create-github-app-token@v1
      with:
        app-id: ${{ secrets.GH_APP_ID }}
        private-key: ${{ secrets.GH_APP_PRIVATE_KEY }}
        owner: ${{ github.repository_owner }}
    - name: Update org secret
      run: |
        gh secret set KUBE_CONFIG \
          --app actions \
          --org ${{ github.repository_owner }} \
          --visibility private \
          < ./k0s-kube.config        
      env:
        GH_TOKEN: ${{ steps.generate-token.outputs.token }}
    - run: rm -f ./k0s-kube.config

Please note: I omitted a couple things in the workflow, so please don’t copy and paste.

What’s next? Maybe add a schedule (cron) for the workflow to run:

on:
  workflow_dispatch:
  schedule:
  - cron: '30 2 7,14,21,28 * *'

For bonus points add a failure notification in slack to know when it stops working. :-)

Fin

Thanks for reading! And welcome to my new Hugo blog!