Upgrade to Headscale v0.28.0.

[Gerrit91]

Description

References metal-stack/metal-apiserver#205.

Actions Required

Upgrade headscale from `0.23.0` -> `v0.28.0` with all minor versions in between prior to updating the metal-stack control plane. You can use the following approach:

Creating firewalls with working VPN does not work until both headscale and metal-stack control plane were updated. There is a possibility that with `v0.27.1` the connected clients loose connectivity. This can be solved by restarting tailscaled on the clients, e.g. firewalls. With https://github.com/metal-stack/firewall-controller/pull/220 this can be done fleet-wide as platform admin from all seeds.

#!/usr/bin/env bash

# async start a loop to fix broken postgres migration (see https://github.com/juanfont/headscale/issues/2351)
while true; do k exec headscale-db-0 -- psql -U postgres headscale -c "alter table users add constraint uni_users_name unique (name);"; done

# cycle through headscale versions, after every update check the headscale logs and see if it comes up properly
NAMESPACE=metal-control-plane

for VERSION in v0.24.3 v0.25.1 v0.26.1 v0.27.1 v0.28.0
do
    echo "upgrade to ${VERSION}"
    kubectl patch deployments.apps -n "${NAMESPACE}" headscale \
        --patch="{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\": \"headscale\",\"image\":\"docker.io/headscale/headscale:${VERSION}\"}]}}}}"
    kubectl delete pod -n "${NAMESPACE}" -l app=headscale
    kubectl wait --for=condition=Ready pod -l app=headscale
    read -p "check that headscale is running properly, press enter to continue"
done

# recreate headscale api token for metal-api 
kubectl delete secret headscale-api-key

echo "done, now run the metal-stack deployment"

Used AI-Tools ✨

• None used for generation

[majst01]

Like it more than my #649 :-)

maybe go straight to v0.28.0 as there are no api changes in between

[majst01]

headscale-674b6cb9cc-r5xsd headscale 2026-06-18T09:34:51Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init database: migration failed: automigrating types.Route: ERROR: constraint \"uni_users_name\" of relation \"users\" does not exist (SQLSTATE 42704)"

[majst01]

when trying to create a new authkey:

m vpn key --project b5f26a3b-9a4d-48db-a6b3-d1dd4ac4abec

metal-api-69ff6d7c5f-7rcn4 metal-api {"time":"2026-06-18T10:58:34.839492285Z","level":"ERROR","msg":"finished handling rest call","rqid":"ca2ee627206d3f0254dbd7f15f82959a","remoteaddr":"10.240.2.63:37162","method":"POST","uri":"/metal/v1/vpn/authkey","route":"/metal/v1/vpn/authkey","useremail":"stefan.majer@extern.f-i-ts.de","username":"ize0540","usertenant":"fits","status":500,"content-length":148,"duration":"409.790834ms","response":"\"{\\\"statuscode\\\":500,\\\"message\\\":\\\"failed to create new auth key: failed to create new Auth Key: rpc error: code = Unknown desc = user not found\\\"}\""}

But:

 k exec -it headscale-b478d9958-c4p6k -- headscale user list |grep b5f26a3b-9a4d-48db-a6b3-d1dd4ac4abec
6   | b5f26a3b-9a4d-48db-a6b3-d1dd4ac4abec | 2022-09-15 10:52:24

[vknabel]

Can we use a return user, true here? Or do we always need the last match?

[majst01]

Important:

insert into migrations values(202501311657);

alter table users add constraint uni_users_name unique (name);

[majst01]

while true; do k exec headscale-db-0 -- psql -U postgres headscale -c "alter table users add constraint uni_users_name unique (name);"; done