Guide: Setting Up OpenTelemetry Collector, Rancher Monitoring, and Thanos with MinIO on an External Server (Docker-based MinIO)

This guide outlines the steps to deploy OpenTelemetry Collector, enable Rancher Monitoring with Thanos, and configure MinIO as external object storage running on a separate server using Docker. The Kubernetes cluster is already running and imported into Rancher.

1. Prerequisites

• A Kubernetes cluster running and imported into Rancher.
• Rancher is installed and accessible.
• kubectl is configured to interact with the cluster.
• Helm is installed (helm version).
• An external server (running Linux/Armbian) where MinIO will be deployed via Docker.
• Docker installed on the MinIO server.

---

2. Deploy MinIO on an External Server (Standalone Mode)

MinIO will act as an S3-compatible object store for Thanos.

2.1 Install Docker on the MinIO Server

If Docker is not installed, install it on the external server:

sudo apt update && sudo apt install -y docker.io

2.2 Deploy MinIO

Run MinIO with a persistent volume mounted on the external server:

docker run -d --name minio \
  -p 9000:9000 -p 9090:9090 \
  -e "MINIO_ROOT_USER=admin" \
  -e "MINIO_ROOT_PASSWORD=adminpassword" \
  -v /mnt/storage/minio:/data \
  quay.io/minio/minio server /data --console-address ":9090"

2.3 Verify MinIO

1. Open http://<minio-server-ip>:9090 in a browser.
2. Login with admin / adminpassword.
3. Create a bucket named thanos.

---

3. Deploy Rancher Monitoring with Thanos

Rancher Monitoring is a Helm chart that includes Prometheus, Grafana, and Thanos.

3.1 Add the Rancher Helm Chart Repository

helm repo add rancher-charts https://charts.rancher.io
helm repo update

3.2 Create a Thanos Object Storage Secret in Kubernetes

Create a Kubernetes secret with MinIO configuration:

apiVersion: v1
kind: Secret
metadata:
  name: thanos-object-storage
  namespace: cattle-monitoring-system
type: Opaque
stringData:
  objstore.yml: |
    type: s3
    config:
      bucket: thanos
      endpoint: <minio-server-ip>:9000
      access_key: admin
      secret_key: adminpassword
      insecure: true

Apply the secret:

kubectl apply -f thanos-object-storage.yaml

3.3 Install Rancher Monitoring with Thanos

Deploy Prometheus, Thanos, and Grafana with Helm:

helm install rancher-monitoring rancher-charts/rancher-monitoring \
  --namespace cattle-monitoring-system \
  --set prometheus.thanos.create=true \
  --set prometheus.thanos.objectStorageConfig.secretName=thanos-object-storage \
  --set prometheus.thanos.objectStorageConfig.secretKey=objstore.yml

3.4 Verify Installation

Check that Prometheus, Thanos, and Grafana are running:

kubectl get pods -n cattle-monitoring-system

Expected output:

NAME                                      READY   STATUS    RESTARTS   AGE
prometheus-rancher-monitoring-0           2/2     Running   0          2m
thanos-query-0                            1/1     Running   0          2m
thanos-storegateway-0                     1/1     Running   0          2m
grafana-0                                 2/2     Running   0          2m

---

4. Deploy OpenTelemetry Collector

OpenTelemetry Collector will collect and export telemetry data to Prometheus.

4.1 Create an OpenTelemetry Collector Configuration

Create a ConfigMap for the collector:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: monitoring
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:

    processors:
      batch:

    exporters:
      logging:
      prometheus:
        endpoint: "0.0.0.0:8889"

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus]

Apply the configuration:

kubectl apply -f otel-collector-config.yaml

4.2 Deploy OpenTelemetry Collector

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:latest
          args: ['--config=/etc/otel/config.yaml']
          volumeMounts:
            - name: config-volume
              mountPath: /etc/otel
      volumes:
        - name: config-volume
          configMap:
            name: otel-collector-config

Apply the deployment:

kubectl apply -f otel-collector.yaml

4.3 Verify OpenTelemetry Collector

Check logs:

kubectl logs -l app=otel-collector -n monitoring

---

5. Configure Grafana for Monitoring

5.1 Access Grafana

kubectl port-forward -n cattle-monitoring-system svc/grafana 3000:80

Open http://localhost:3000 and login (default: admin / admin).

5.2 Add Thanos as a Data Source

1. Open Grafana → Configuration → Data Sources.
2. Add a new Prometheus data source.
3. Set the URL to:

   http://thanos-query.cattle-monitoring-system.svc.cluster.local:9090
4. Click Save & Test.

5.3 Verify Data in Grafana

• Open Explore in Grafana.
• Run PromQL queries such as:

  rate(node_cpu_seconds_total[5m])
  container_memory_usage_bytes

---

6. Summary

Component	Purpose
MinIO (Docker on external server)	S3-compatible storage for Thanos
Prometheus (Rancher Monitoring)	Collects cluster metrics
Thanos (Rancher Monitoring)	Stores and queries long-term metrics
Grafana (Rancher Monitoring)	Visualizes Prometheus & Thanos data
OpenTelemetry Collector	Collects and exports traces & metrics

---

7. Next Steps

1. Confirm if all components are running:

   kubectl get pods -n cattle-monitoring-system
2. Fine-tune retention:
   • Modify storage.tsdb.retention.time for Prometheus.
   • Adjust MinIO bucket lifecycle policies if needed.
3. Secure MinIO:
   • Use SSL certificates.
   • Configure firewall rules to restrict access.

---

This setup ensures scalable, long-term observability using OpenTelemetry, Prometheus, Thanos, and MinIO with Rancher. Let me know if you need modifications or troubleshooting steps.

more info: https://chatgpt.com/c/67d82dcc-ce18-8002-87ee-a85efd18952e