Skip to main content

The 10-Hour Journey to a 10-Second Solution: K8s Database Migrations

· 2 min read
Max Kaido
Max Kaido
Architect

TL;DR​

# The simple solution we ended up with:
kubectl -n mercury port-forward svc/postgres 15432:5432
DATABASE_URL="postgresql://mercury:mercury_secure_password_123@localhost:15432/mercury" pnpm migration:up:mercury

The Journey​

What started as a seemingly simple task - "we need to run database migrations in Kubernetes" - turned into a 10-hour adventure through various approaches, each more complex than the last. Here's what we learned.

Attempt 1: The "Proper" Kubernetes Way​

First, we tried to be "enterprise-grade" with a dedicated migrations container:

FROM node:20-slim
# ... tons of setup ...
COPY . .
CMD ["pnpm", "migration:up:mercury"]

Problems:

  • Complex multi-stage builds
  • Dependency management in monorepo
  • TypeScript compilation issues
  • Missing types and imports
  • Large image size

Attempt 2: The "Dev Environment" Approach​

Then we thought: "Let's just replicate our dev environment in a container!"

FROM node:20-slim
RUN apt-get update && apt-get install -y build-essential python3 git
# ... even more setup ...

Problems:

  • Even larger image
  • More complexity
  • Still fighting with monorepo structure
  • "It works on my machine" but in reverse

The Epiphany​

After 10 hours, we had our "aha" moment:

  1. We already have a working development environment
  2. Kubernetes has built-in secure tunneling with port-forward
  3. Why not use what we know works?

The Solution​

Two commands. That's it:

  1. Create a secure tunnel to the database:

    kubectl -n mercury port-forward svc/postgres 15432:5432

  2. Run migrations from your local environment:

    DATABASE_URL="postgresql://user:pass@localhost:15432/db" pnpm migration:up:mercury

Lessons Learned​

  1. Start Simple: Don't begin with "enterprise patterns" - start with what you know works
  2. Use Built-in Tools: Kubernetes' port-forward is there for a reason
  3. Local Development is Valid: Your local environment is often the best environment
  4. Security is Built-in: port-forward provides a secure tunnel without exposing your database

Why This Matters​

This journey exemplifies a common pattern in software development - the tendency to overcomplicate solutions before finding the elegant simple one. We went from:

  • Custom Docker images
  • Kubernetes Jobs
  • Init containers
  • Complex monorepo setups
  • Multi-stage builds

To:

  • Two commands
  • Zero new infrastructure
  • Existing tools
  • Secure by default

Sometimes the best solution isn't about adding more - it's about realizing what you already have.

Bonus: Why Port 15432?​

A small but important detail - we used port 15432 instead of 5432 because:

  1. Local Postgres often runs on 5432
  2. Avoiding port conflicts is easier than debugging them
  3. The explicit non-standard port reminds us we're working with a forwarded connection

Conclusion​

Next time you're deep in the weeds of a "proper enterprise solution", take a step back and ask: "What's the simplest thing that could possibly work?"

The answer might just save you 9 hours and 50 minutes.