Nidhi's blog

Production Engineering • SRE • Kubernetes

Container pinning illustration

Pinning Metabase in Kubernetes: Stop Using Latest

June 12, 2026

The case for pinning OSS images

latest is a lie. It’s a mutable pointer that can change under you any time the maintainer pushes a new build. On a pod restart - during a node drain, a rollout, a crash recovery - Kubernetes pulls the image again, gets whatever latest points to now, and your workload is running a version you never tested.

For stateless apps this is tolerable. For stateful apps - anything that runs schema migrations on startup - it’s a problem.

Liquibase, Flyway, and similar tools modify your database on startup. They’re designed to be forward-only. A migration that creates a column, renames a table, or adds an index doesn’t have an automatic undo. If the new version’s migration fails halfway through, you can end up in a state where the old image sees “already migrated” and the new image sees “migration failed” - and neither can start cleanly.

The rule is simple: pin every stateful OSS workload to an explicit version tag.

What happened

We were running Metabase in Kubernetes for internal analytics. The image was not pinned to an explicit patch version. A routine upgrade to v0.63.2 went out, the pod restarted, and it never came back up.

The pod kept crashlooping. Logs showed Liquibase failing during the migration phase on startup. Metabase couldn’t complete its migration checks, threw an exception, and exited - over and over.

We rolled back the image to v0.63.1.2. The pod still wouldn’t start. The DATABASECHANGELOG table - Liquibase’s record of which migrations have run - was now in a partially inconsistent state from the failed v0.63.2 migration attempt. The old image couldn’t reconcile it either.

We had to manually inspect and repair the table before either version could start cleanly.

Why it failed: Liquibase internals

Metabase uses Liquibase to manage its database schema. Liquibase tracks every migration it has run in a DATABASECHANGELOG table, keyed by (id, author, filename).

In Metabase’s migrations/063/ directory, most files use their physical path as the filename in DATABASECHANGELOG. But two files - 20260604_curated_search_entries.yaml and 20260623_osi_ai_context.yaml - have a logicalFilePath attribute set to migrations/063_update_migrations.yaml. This tells Liquibase to record them under that consolidated filename instead of their physical path.

v0.63.2 introduced a consolidation step: a bulk UPDATE to rename all DATABASECHANGELOG entries matching filename LIKE 'migrations/063/%' to the consolidated path. The problem: two changesets, ghy_3747 and ghy_3933, share the same (id, author) pair. When the bulk UPDATE tried to move both to the same consolidated filename, it hit a unique constraint violation. Liquibase aborted. Metabase couldn’t start.

This is a bug in Metabase’s migration authoring - duplicate (id, author) pairs that were never caught because the consolidation code path was never triggered until v0.63.2. It’s also the kind of thing you cannot predict by reading release notes.

The fix: pin and repair

Step 1: repair DATABASECHANGELOG.

We needed a stable mixed state that both satisfies Liquibase’s lookup logic and avoids the constraint violation on every future startup:

  • 4 entries with filename = 'migrations/063_update_migrations.yaml' - the two files that declare logicalFilePath (Liquibase looks them up by consolidated name)
  • 59 entries with individual migrations/063/... filenames - everything else
  • Both ghy_3747 and ghy_3933 present as individual-path entries

With this layout, on every restart: the consolidation UPDATE silently fails (constraint violation on the duplicates), Liquibase falls back to reading individual JAR paths, finds all migrations already recorded, sees 0 unrun migrations, and Metabase starts cleanly.

Step 2: pin the image.

image: metabase/metabase:v0.63.1.2
imagePullPolicy: IfNotPresent

IfNotPresent means: if the image is already cached on the node, don’t pull again. Combined with an explicit version tag, this gives you two layers of protection against silent upgrades. If you want even stronger guarantees, pin by digest (image: metabase/metabase@sha256:...) - a digest is immutable, a tag is not.

Step 3: add a startupProbe with enough headroom.

Metabase can take a while to run migrations on first start or after an upgrade. Without a startupProbe, Kubernetes’s default liveness/readiness probes can kill the pod before migrations finish, creating a crashloop that looks like an application bug.

startupProbe:
  httpGet:
    path: /api/health
    port: 3000
  failureThreshold: 60
  periodSeconds: 5

This gives Metabase 5 minutes (60 × 5s) to pass its health check before Kubernetes declares it failed. After startup succeeds, normal liveness and readiness probes take over.

How to upgrade safely

Now that we’re pinned, upgrades are deliberate:

  1. Check the release notes for any mention of schema migrations, Liquibase changes, or breaking changes to the application database. For Metabase, pay attention to any new files in migrations/ and whether they use logicalFilePath.

  2. Test in staging against a copy of prod data. Restore a recent prod DB snapshot into staging, update the image tag, and let it run. Watch the startup logs for Liquibase errors.

  3. Snapshot prod before upgrading. If the migration fails and leaves DATABASECHANGELOG in an inconsistent state, you need to restore - not just roll back the image tag.

  4. Deploy and watch startup logs live. Don’t wait for the health check to fail. kubectl logs -f the pod from the moment it starts and watch Liquibase’s output.

  5. If it fails: image tag rollback + DB restore. Rolling back only the image tag is not enough if migrations partially ran. Restore the DB snapshot and then roll back the image.

When it breaks

Pod crashlooping after an image update.

Check the logs immediately - if you see Liquibase errors, stop restarting the pod. Each restart may attempt (and partially apply) more migrations. Roll back the image tag first, then inspect DATABASECHANGELOG before doing anything else.

Rolled back image tag but pod still won’t start.

The migration table is inconsistent. The old image sees entries it doesn’t recognise or can’t reconcile. You need to either restore a DB snapshot (cleanest) or manually repair DATABASECHANGELOG to match what the old image expects.

Startup probe keeps failing, pod gets killed before it’s ready.

Migrations are taking longer than the probe allows. Increase failureThreshold on the startupProbe. Don’t increase the liveness probe timeout - that controls the steady-state, not startup.

Unsure which image version is actually running.

kubectl get pod <metabase-pod> -o jsonpath='{.spec.containers[0].image}'

If you’re seeing a digest instead of a tag, cross-reference it against the registry to confirm the version.

imagePullPolicy: Always with a pinned tag.

Even with an explicit version tag, Always tells Kubernetes to check the registry on every start. If the tag is mutable (the maintainer re-pushed to the same tag), you can still get a different image. Use IfNotPresent or pin by digest.