Skip to main content

Expose Prometheus metrics from your Kubernetes pods to Mimir

You want to monitor your application and you don't want to run a Prometheus. Good news: on France Nuage Kubernetes, Alloy, Mimir and Grafana are already there. You add three annotations on your pod, and your metrics flow into Mimir automatically. No Helm chart to install, no scraper to configure, no tenant to create.

What you need​

  • A namespace on your France Nuage Kubernetes cluster
  • A France Nuage Grafana instance (see the Grafana Stack guide)
  • An application that exposes — or can expose — a /metrics endpoint in Prometheus format

That's it. No observability component to deploy on your side.

Prometheus vs Mimir: what's the difference?​

Prometheus is the standard for collecting and storing metrics. It's excellent as a single instance, but it becomes awkward at scale: long retention, high availability, multi-tenancy — each of those takes work.

Mimir is the scalable flavor of Prometheus. Same query language (PromQL), same metric format, but built for long-term retention, high availability and multi-tenancy. Your dashboards and alerts keep working exactly the same — Mimir looks like Prometheus from the client side. For more details, read the Mimir guide.

On France Nuage Kubernetes, we run:

  • Grafana Alloy as the collector: it auto-discovers which pods to scrape via annotations
  • Mimir as the storage backend: one tenant per namespace, hard isolation
  • Grafana already wired to Mimir with your tenant pre-configured

Step 1: expose a /metrics endpoint​

If your app already exposes /metrics, skip to step 2.

Otherwise, here's a minimal example with Node.js and prom-client:

npm install express prom-client
// server.js
const express = require('express');
const client = require('prom-client');

const register = new client.Registry();
register.setDefaultLabels({ app: 'my-service' });
client.collectDefaultMetrics({ register });

const httpRequests = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests received',
  labelNames: ['method', 'status'],
});
register.registerMetric(httpRequests);

const app = express();

app.get('/', (req, res) => {
  httpRequests.inc({ method: 'GET', status: '200' });
  res.send('hello');
});

app.get('/metrics', (req, res) => {
  register.metrics().then((metrics) => {
    res.set('Content-Type', register.contentType);
    res.end(metrics);
  });
});

app.listen(9090, () => console.log('listening on :9090'));

You now have an app listening on port 9090 and exposing its metrics at /metrics. Test it locally:

curl http://localhost:9090/metrics

You should see a bunch of default metrics (process_cpu_seconds_total, nodejs_heap_size_used_bytes, etc.) followed by your http_requests_total.

Step 2: add annotations to your Pod​

This is the heart of the tutorial. Alloy continuously watches the cluster and scrapes every pod that carries these annotations:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"       # port of /metrics
    prometheus.io/path: "/metrics"   # optional, defaults to /metrics

Important: these annotations must live on the Pod, not on the Deployment or Service. In practice, that means spec.template.metadata.annotations inside a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
  namespace: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: app
          image: registry.gitlab.com/my-group/my-service:latest
          ports:
            - name: http
              containerPort: 9090

Apply it:

kubectl apply -f deployment.yaml -n my-app

If you deploy through a CI/CD pipeline, check out the Deploy to Kubernetes from GitLab CI tutorial to wire helm upgrade to every push.

Alloy picks up the new pod, starts scraping /metrics every 15 seconds, and pushes the samples to Mimir with the header X-Scope-OrgID: my-app. The Mimir tenant is derived from your namespace name — you have nothing to configure.

Step 3: query your metrics in Grafana​

Open your France Nuage Grafana instance. The Mimir datasource is already configured and wired to your tenant.

  1. Go to Explore
  2. Pick the Mimir datasource
  3. Type a PromQL query, for example:
http_requests_total{app="my-service"}

Or to see the request rate over 5 minutes:

sum(rate(http_requests_total{app="my-service"}[5m])) by (status)

You can also check the scrape is healthy by looking at the up metric:

up{namespace="my-app"}

A value of 1 means scrape succeeded. A value of 0 means Alloy can't reach your endpoint — jump to the troubleshooting section.

Troubleshooting​

The up metric returns 0​

Alloy sees your pod but can't scrape the endpoint. Check, in order:

  1. The port is right: the prometheus.io/port annotation must point to the container's actual listening port, not the Service port
  2. The path is right: if your endpoint isn't /metrics, set prometheus.io/path: "/your-path"
  3. The container listens on all interfaces: 0.0.0.0:9090, not 127.0.0.1:9090, otherwise Alloy can't reach it from outside the pod

The up metric doesn't show up at all​

Alloy isn't seeing your pod. This is almost always an annotation issue:

  • Annotations are on the Pod (spec.template.metadata.annotations), not on the Deployment itself
  • Values are strings: "true", "9090", not true or 9090
  • The prometheus.io/scrape annotation is exactly "true" (not "1" or "yes")

Check with:

kubectl get pod -n my-app -l app=my-service -o jsonpath='{.items[0].metadata.annotations}'

My metrics don't show up in Grafana​

If up works but your custom metrics don't come through:

  • Make sure you're on the right datasource (Mimir, not some local Prometheus)
  • Remove any overly restrictive filters from your PromQL query
  • Wait 15-30 seconds: that's the scrape interval

ERR cardinality limit exceeded​

Mimir protects tenants against cardinality explosions. If you see this error, one of your metrics has too many distinct label combinations. See the next section.

Best practices​

Keep label cardinality under control​

This is pitfall number one with Prometheus and Mimir. Each unique label combination creates a distinct time series, and every series consumes memory.

Absolutely avoid:

  • user_id, request_id, trace_id as labels: infinite cardinality, your tenant will blow up
  • Raw url: /user/42, /user/43... templatize it into /user/:id
  • timestamp or any time-derived value

Prefer:

  • Bounded-cardinality labels: method, status, endpoint (templatized), env
  • A consistent app and env across all your metrics, so queries stay simple

Name your metrics per Prometheus conventions​

  • _total suffix for counters
  • _seconds suffix for durations (not _ms)
  • _bytes suffix for sizes
  • snake_case, no uppercase

One app, one port, one endpoint​

Exposing /metrics on the same port as your main app is simpler than using a dedicated port. If you have a reason to separate them (security, firewalling), open a second port in your container and adjust prometheus.io/port.

Going further​

Questions? [email protected]