Recently I setup a small skaffold.dev development environment for a platform deployed using helmfile, deployed to a GKE cluster in production that is moving towards also having a local setup using minikube.

There is an open Github issue requesting helmfile support, so my first fear was that this wouldn’t be possible. Though after a little hacking around this seemed fairly easy by using the skaffold pre deploy hooks.

Here is a quick introduction to this stack, and a solution for using skaffold with helmfile deployments on minikube before “native” support is added.

The stack

Helmfile is a declarative spec for deploying helm charts.

https://github.com/roboll/helmfile

Helmfile allows simple declarative use of helm and helm charts to deploy resources to a Kubernetes cluster.

Skaffold handles the workflow for building, pushing and deploying your application, allowing you to focus on what matters most: writing code.

https://skaffold.dev/

Skaffold ultimately aims to allow you to write code in a standard way, while quickly and easily being able to see your altered code in a deployed cluster, be that local using minikube or elsewhere.

It exists as a simple CLI tool, and the one key command here is dev which will start a loop watching for resource changes and going through the process of rebuilding needed container images, injecting code or files and updating Kubernetes resources.

The Skaffold Config

The skaffold config below covers reloading a helmfile deployment for a single deployed service in the local minikube context.

apiVersion: skaffold/v2beta23 kind: Config profiles: - name: local activation: - kubeContext: minikube build: artifacts: - image: local/skaffold/ui context: ./../../ui deploy: kubeContext: minikube-wbaas helm: releases: - name: ui chartPath: ./../../deploy/charts/ui valuesFiles: - ".tmp.values.ui.yaml" - NeverPull.yaml artifactOverrides: image: local/skaffold/ui imageStrategy: helm: {} hooks: before: - host: command: [ "./helmfile-values", "-e", "local", "-r", "ui", "-n" ]
Code language: JavaScript (javascript)

skaffold will use the build.artifacts.context to locate and build a Dockerfile while watching the context for changes. The resulting image will end up having the name local/skaffold/ui and be stored within the minikube docker context.

When things change skaffold will then deploy the alternate image to the minikube cluster, overwriting the image and tag with the appropriate one from the build step. The imageStrategy here is needed as my chart uses {{ .Values.image.repository }}:{{ .Values.image.tag}}. You can read more about other strategies in the docs.

skaffold does not know about the helmfile values out of the box, and for this we use a simple on host hook that runs before deployment happens. This script ultimately uses a helmfile tool and write-values command to output a values.yaml file for use in the deployment.

#!/usr/bin/env bash # Set variable defaults TARGET_RELEASE="unset" TARGET_ENVIRONMENT="unset" REMOVE_IMAGE_YAML=0 # Get the options while getopts ":r:e:n" option; do case $option in r) TARGET_RELEASE=$OPTARG;; e) TARGET_ENVIRONMENT=$OPTARG;; n) REMOVE_IMAGE_YAML=1;; esac done SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" OUTPUT_YAML=$SCRIPT_DIR/.tmp.values.$TARGET_RELEASE.yaml # Run the commands echo "Targetting environment: $TARGET_ENVIRONMENT and release: $TARGET_RELEASE" cd $SCRIPT_DIR/../k8s/helmfile && \ helmfile --environment $TARGET_ENVIRONMENT --selector name=$TARGET_RELEASE write-values --skip-deps --output-file-template $OUTPUT_YAML if [[ "$REMOVE_IMAGE_YAML" == 1 ]]; then yq eval 'del(.image)' --no-colors $OUTPUT_YAML > $OUTPUT_YAML.altered mv $OUTPUT_YAML.altered $OUTPUT_YAML fi cat $SCRIPT_DIR/.tmp.values.$TARGET_RELEASE.yaml
Code language: PHP (php)

This script also optionally removes the specified image information from the values file as this seems to conflict with the override that skaffold is trying to perform.

This is in turn combined with a simple NeverPull.yaml values file by skaffold to ensure that the cluster does not try to pull the image that we have just built, as it doesn’t exist in any registries, only in the local context that it was built in.

image: pullPolicy: Never

All of this, while running skaffold run looks something like this…

Gotchas

The pullPolicy really had me stuck for a while until I realize that this should be Never. Without using Never your cluster will try to pull the image when the new container definition reaches the cluster, fail, and never continue.

Including the image values in the values file to be passed to skaffold also seemed to give me problems, hence this is stripped out by the bash script.

The key within artifactOverrides (image in the example above) needs to relate to the key in the values of your chart that holds the image information. I initially through this was tied to the strategy, but after reading the docs more and more while debugging I realized this is meant to refer to the cart.

Links

https://skaffold.dev/

https://skaffold.dev/docs/

https://skaffold.dev/docs/pipeline-stages/deployers/helm/

And while figuring out this solution I did a fair amount of GitHub issue reading: