Using skaffold.dev with helmfile deployments on minikube

October 7, 2021 0 By addshore

Recently I setup a small skaffold.dev development environment for a platform deployed using helmfile, deployed to a GKE cluster in production that is moving towards also having a local setup using minikube.

There is an open Github issue requesting helmfile support, so my first fear was that this wouldn’t be possible. Though after a little hacking around this seemed fairly easy by using the skaffold pre deploy hooks.

Here is a quick introduction to this stack, and a solution for using skaffold with helmfile deployments on minikube before “native” support is added.

The stack

Helmfile is a declarative spec for deploying helm charts.

https://github.com/roboll/helmfile

Helmfile allows simple declarative use of helm and helm charts to deploy resources to a Kubernetes cluster.

Skaffold handles the workflow for building, pushing and deploying your application, allowing you to focus on what matters most: writing code.

https://skaffold.dev/

Skaffold ultimately aims to allow you to write code in a standard way, while quickly and easily being able to see your altered code in a deployed cluster, be that local using minikube or elsewhere.

It exists as a simple CLI tool, and the one key command here is dev which will start a loop watching for resource changes and going through the process of rebuilding needed container images, injecting code or files and updating Kubernetes resources.

The Skaffold Config

The skaffold config below covers reloading a helmfile deployment for a single deployed service in the local minikube context.

apiVersion: skaffold/v2beta23
kind: Config
profiles:
- name: local
  activation:
    - kubeContext: minikube
build:
  artifacts:
    - image: local/skaffold/ui
      context: ./../../ui
deploy:
  kubeContext: minikube-wbaas
  helm:
    releases:
      - name: ui
        chartPath: ./../../deploy/charts/ui
        valuesFiles:
          - ".tmp.values.ui.yaml"
          - NeverPull.yaml
        artifactOverrides:
          image: local/skaffold/ui
        imageStrategy:
          helm: {}
    hooks:
      before:
        - host:
            command: [ "./helmfile-values", "-e", "local", "-r", "ui", "-n" ]Code language: JavaScript (javascript)

skaffold will use the build.artifacts.context to locate and build a Dockerfile while watching the context for changes. The resulting image will end up having the name local/skaffold/ui and be stored within the minikube docker context.

When things change skaffold will then deploy the alternate image to the minikube cluster, overwriting the image and tag with the appropriate one from the build step. The imageStrategy here is needed as my chart uses {{ .Values.image.repository }}:{{ .Values.image.tag}}. You can read more about other strategies in the docs.

skaffold does not know about the helmfile values out of the box, and for this we use a simple on host hook that runs before deployment happens. This script ultimately uses a helmfile tool and write-values command to output a values.yaml file for use in the deployment.

#!/usr/bin/env bash

# Set variable defaults
TARGET_RELEASE="unset"
TARGET_ENVIRONMENT="unset"
REMOVE_IMAGE_YAML=0

# Get the options
while getopts ":r:e:n" option; do
   case $option in
      r)
         TARGET_RELEASE=$OPTARG;;
      e)
         TARGET_ENVIRONMENT=$OPTARG;;
      n)
         REMOVE_IMAGE_YAML=1;;
   esac
done

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
OUTPUT_YAML=$SCRIPT_DIR/.tmp.values.$TARGET_RELEASE.yaml

# Run the commands
echo "Targetting environment: $TARGET_ENVIRONMENT and release: $TARGET_RELEASE"
cd $SCRIPT_DIR/../k8s/helmfile && \
helmfile --environment $TARGET_ENVIRONMENT --selector name=$TARGET_RELEASE write-values --skip-deps --output-file-template $OUTPUT_YAML

if [[ "$REMOVE_IMAGE_YAML" == 1 ]]; then
   yq eval 'del(.image)' --no-colors $OUTPUT_YAML > $OUTPUT_YAML.altered
   mv $OUTPUT_YAML.altered $OUTPUT_YAML
fi

cat $SCRIPT_DIR/.tmp.values.$TARGET_RELEASE.yamlCode language: PHP (php)

This script also optionally removes the specified image information from the values file as this seems to conflict with the override that skaffold is trying to perform.

This is in turn combined with a simple NeverPull.yaml values file by skaffold to ensure that the cluster does not try to pull the image that we have just built, as it doesn’t exist in any registries, only in the local context that it was built in.

 image:
  pullPolicy: Never

All of this, while running skaffold run looks something like this…

Gotchas

The pullPolicy really had me stuck for a while until I realize that this should be Never. Without using Never your cluster will try to pull the image when the new container definition reaches the cluster, fail, and never continue.

Including the image values in the values file to be passed to skaffold also seemed to give me problems, hence this is stripped out by the bash script.

The key within artifactOverrides (image in the example above) needs to relate to the key in the values of your chart that holds the image information. I initially through this was tied to the strategy, but after reading the docs more and more while debugging I realized this is meant to refer to the cart.

Links

https://skaffold.dev/

https://skaffold.dev/docs/

https://skaffold.dev/docs/pipeline-stages/deployers/helm/

And while figuring out this solution I did a fair amount of GitHub issue reading: