Skip to main content

From Zero to Hero: Our Kubernetes Journey with Mercury Bot

· 2 min read
Max Kaido
Max Kaido
Architect

Setting up a production-ready Kubernetes cluster can be a challenging journey, filled with unexpected twists and turns. Today, we want to share our experience of deploying the Mercury Bot on Kubernetes, complete with proper networking, HTTPS, and all the fun debugging that came with it.

The Initial Setup​

Our journey began with a basic Kubernetes installation on a Hetzner server. The goal was simple: deploy our Mercury Bot service and make it accessible to the world. But as with many things in tech, what seems simple at first can quickly become an adventure.

Challenge #1: Pod Networking​

The first major hurdle we encountered was pod networking. Our pods were stuck in a ContainerCreating state with the cryptic error:

Failed to create pod sandbox: failed to setup network for sandbox: plugin type="flannel" failed (add): loadFlannelSubnetEnv failed

The issue? Flannel wasn't properly configured. The solution involved:

  1. Loading the necessary kernel modules (br_netfilter and overlay)
  2. Configuring system parameters for networking
  3. Ensuring the Flannel CNI plugin was properly installed

Challenge #2: Ingress Controller Setup​

With pod networking solved, we moved on to setting up the ingress controller. We chose nginx-ingress-controller for its robustness and flexibility. However, we faced several issues:

  1. Port conflicts with a system-level Nginx installation
  2. Admission webhook configuration issues
  3. Default backend routing problems

The solution was a methodical approach:

  1. Disabling the system Nginx service
  2. Cleaning up the admission webhooks
  3. Configuring the ingress controller with proper host networking

Challenge #3: SSL/TLS Configuration​

The final piece of the puzzle was setting up HTTPS. Initially, we had a self-signed certificate, which worked but wasn't ideal for production. We had Let's Encrypt certificates on the host but weren't using them in Kubernetes.

The solution was elegant:

  1. Creating a Kubernetes TLS secret from existing Let's Encrypt certificates
  2. Configuring the ingress resource to use this secret
  3. Restarting the ingress controller to pick up the new certificate

The Final Architecture​

Our final setup includes:

  • Kubernetes cluster with Flannel networking
  • nginx-ingress-controller for traffic routing
  • Let's Encrypt certificates for HTTPS
  • Mercury Bot service running with proper networking and security

Key Learnings​

  1. Layer by Layer: Solve problems one layer at a time - networking, then ingress, then SSL
  2. Debug Methodically: Use kubectl describe, logs, and system tools to understand issues
  3. Security First: Proper SSL certificates are crucial for production deployments
  4. Documentation: Keep track of configurations and solutions for future reference

Conclusion​

While the journey had its challenges, we now have a robust, secure, and properly configured Kubernetes setup. The Mercury Bot is accessible via HTTPS at mercury.arcana.cfd, and we have a solid foundation for future services.

Remember: Kubernetes complexity comes with great power, but with methodical debugging and persistence, any issue can be solved.

Next Steps​

  • Set up monitoring and logging
  • Implement automatic certificate renewal
  • Add horizontal pod autoscaling
  • Document deployment procedures

Stay tuned for more updates on our Kubernetes journey!