A deploy that proves it deployed

Our health check passed while production kept serving the old image. The fix is one field and one comparison.

The pipeline was green. The container answered its health check. Production was still running last week's code.

Docker Swarm had been asked to roll a new image and, for reasons of its own, kept the old one. The post-deploy check polled /healthz, got a 200, and passed. It was only ever proving that a container was up, which is the one thing that stays true when a deploy silently doesn't happen.

The fix

Bake the commit into the image, serve it, and compare.

# Dockerfile, runtime stage
ARG CI_COMMIT_SHA
ENV BUILD_SHA=$CI_COMMIT_SHA

GET /healthz
{"status":"ok","build_sha":"9f2c…"}

After the stack update, the deploy job polls every running replica for up to 150 seconds and passes only when all of them report the SHA that was pushed:

for i in $(seq 1 30); do
  all_ok=1
  for cid in $(docker ps -q -f name=api); do
    live=$(docker exec "$cid" wget -qO- localhost:8080/healthz | sed -n 's/.*"build_sha":"\([^"]*\)".*/\1/p')
    [ "$live" = "$CI_COMMIT_SHA" ] || { all_ok=0; break; }
  done
  [ "$all_ok" = 1 ] && exit 0
  sleep 5
done
echo "deployed SHA $CI_COMMIT_SHA never became live (last saw: ${live:-none})"
exit 1

Every replica, not the first one that answers: halfway through a rolling update, half of production is still the old version.

Three ways to get it wrong

The ENV in the wrong stage. An ENV does not cross a FROM. Set it in the build stage of a multi-stage Dockerfile and the runtime image reports unknown.
Short SHA against full SHA. Images are usually tagged with the short one. Compare like with like or the check never passes.
The probe behind auth. On a frontend, the same check fetched /version.txt, and the auth middleware redirected it to the sign-in page. It would have failed every pipeline. Exempt the probe path.

What it still doesn't prove

The check runs inside the container over SSH. It proves the orchestrator rolled the image. It says nothing about DNS, TLS or the CDN in front. An external poll of the public URL covers those, as a second assertion, not a replacement: when it fails you want to know which half broke.