Prometheus

Connect Prometheus as a metrics data source for real-time cluster insights

Connect your Prometheus instance to Ankra to view metrics directly in the platform. This powers resource graphs, the AI Assistant’s metrics awareness, and cluster health indicators.

Why Connect Prometheus?

When you connect Prometheus to Ankra:

• Resource Metrics - View CPU, memory, and network graphs on pods, deployments, and nodes
• AI-Powered Analysis - The AI Assistant can query your metrics to diagnose issues
• Cluster Overview - See aggregate metrics on the cluster dashboard
• Troubleshooting - Correlate metrics with logs and events in one place

---

Supported Endpoints

Ankra connects to any Prometheus-compatible endpoint:

Provider	Example Endpoint
Self-hosted Prometheus	http://prometheus.monitoring.svc.cluster.local:9090
Grafana Cloud	https://prometheus-prod-xx-xxx.grafana.net
Thanos Query	http://thanos-query.monitoring.svc.cluster.local:9090
Victoria Metrics	http://victoria-metrics.monitoring.svc.cluster.local:8428

Amazon Managed Prometheus, Google Cloud Managed Prometheus and Azure Monitor appear in the connection dialog but cannot be selected yet.

---

Connecting Prometheus

1. Open Integrations

   Navigate to your cluster → Settings → Integrations.

2. Select Metrics

   Choose Metrics in the sidebar.

3. Pick a provider

   Under Provider, select Prometheus for a self-hosted instance, or Grafana Cloud / Mimir or VictoriaMetrics for a hosted Prometheus-compatible endpoint. The managed cloud providers (AWS, Google, Azure) are marked Coming soon.

4. Enter the endpoint

   Click Connect Existing and enter your Prometheus endpoint URL. For a self-hosted Prometheus, Ankra also scans the cluster for Prometheus services and offers the ones it finds.

   For in-cluster Prometheus, use the internal service URL:

   http://prometheus-server.monitoring.svc.cluster.local:9090
   

   For kube-prometheus-stack:

   http://prometheus-operated.monitoring.svc.cluster.local:9090
   
5. Add credentials (if required)

   If your Prometheus requires authentication, expand Authentication, enter a Credential Name, and pick the Authentication Method:

   • API Token - For bearer token authentication
   • User Account - For basic auth (username/password)
6. Test and save

   Click Test Connection to verify Ankra can reach your Prometheus instance, then Submit to store the configuration.

---

Authentication Options

No Authentication

For Prometheus instances without authentication (common for in-cluster deployments): Simply enter the endpoint URL and save. No credentials needed.

API Token

For Prometheus instances using bearer token authentication:

1. Expand Authentication in the metrics form and enter a Credential Name
2. Choose the API Token method
3. Enter your token value

Ankra sends the token as: Authorization: Bearer <token>

Basic Auth

For Prometheus instances using username/password:

1. Expand Authentication in the metrics form and enter a Credential Name
2. Choose the User Account method
3. Enter the username and password

---

In-Cluster vs External Prometheus

In-Cluster Prometheus

If Prometheus runs inside the same cluster:

1. Use the internal Kubernetes service URL
2. No authentication is typically needed
3. The Ankra agent forwards requests through the cluster network

Common service URLs:

# Standard Prometheus
http://prometheus-server.monitoring.svc.cluster.local:9090

# kube-prometheus-stack
http://prometheus-operated.monitoring.svc.cluster.local:9090

# Prometheus Operator
http://prometheus-k8s.monitoring.svc.cluster.local:9090

External Prometheus

If Prometheus runs outside the cluster (cloud managed, separate infrastructure):

1. Use the external URL (must be reachable from your cluster)
2. Configure authentication as required by your provider
3. Ensure network connectivity and firewall rules allow access

---

Credentials

In the portal, the credential is created with the metrics source, in the Authentication section of the form:

• Credential Name - A descriptive name like prometheus-prod
• Authentication Method - API Token for bearer token auth, User Account for basic auth

Ankra keeps its own copy of the secret and sends it with every query and with the health probe that runs every five minutes. It never reads the credential from the cluster - not from a Kubernetes Secret, not from the Prometheus configuration - so rotating the password on the Prometheus side does not update Ankra, and updating Ankra does not change Prometheus.

A saved credential cannot be edited in place. To rotate a token or password, delete the credential from Settings → Integrations → Metrics and create a new one.

To keep the gap short, rotate Prometheus first, then Ankra: once the new secret is live on the Prometheus side, open Edit on the metrics source, type the new credentials into the Authentication section and click Test Connection. The test uses what you typed without saving it, so a green result proves the new secret opens the door before you replace the saved credential. Queries fail with a 401 for the time between Prometheus dropping the old secret and Ankra receiving the new one.

Rotating by API

To rotate from a script, send the cluster’s import request with only the new credential. The call replaces the saved metrics credential and leaves everything else on the cluster as it is - as long as the body carries no stacks, because a stack in the body replaces the saved stack of that name.

curl -sS -X POST "https://platform.ankra.app/api/v1/clusters/import?wait=true" \
  -H "Authorization: Bearer $ANKRA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "<cluster-name>",
    "spec": {
      "prometheus_credential": {
        "name": "<credential-name>",
        "username": "<username>",
        "password": "<new-password>"
      }
    }
  }'

• Use a REST token - a personal access token created without --scopes, or a service token with no MCP scope, whose role can change the cluster. A token with mcp:read or mcp:write is refused with 403.
• Send the whole block. name, username and password for basic auth, or name and token for bearer auth. An omitted username is saved as empty and basic auth stops working.
• Keep the name the Metrics panel shows, so the old secret is overwritten rather than left behind.
• Use wait=true, so the call answers with the outcome - check its errors list - instead of 202 before the work is done.
• If the call answers 422 Failed to push changes to Git repository, the credential is already saved: Ankra has not yet synced your latest commit to the cluster’s repository. Do not retry; check with the query below.

This call is the only scripted way to change the secret: ankra cluster apply and the Git sync both ignore a prometheus_credential block. To check the whole path afterwards - the saved credential, the agent and Prometheus - run ankra cluster metrics query 'up' against the cluster. Results are cached for ten seconds.

---

Managing the Connection in Git

If the cluster is connected to a GitOps repository, the metrics data source is part of the cluster document at clusters/<cluster-name>-<cluster-id>/<cluster-name>.yaml. Adding or editing the spec.prometheus_metrics block connects and updates the source on the next sync, exactly as the settings page does:

spec:
  prometheus_metrics:
    endpoint: http://prometheus-operated.monitoring.svc.cluster.local:9090
    flavor: self_hosted
    credential_name: prometheus-prod

Accepted fields are endpoint, credential_name, flavor and verify_ssl - see the ImportCluster schema for what each one means. Any other top-level field is refused with an error naming it, rather than ignored. A block without credential_name keeps using the credential already saved on the cluster, if there is one; the Metrics panel under Settings → Integrations shows which.

Credentials never go in Git. A username, password, token or auth_scheme inside the block is refused by name: silently ignoring it would leave a live secret in your repository while you believe it is in effect. Create the credential in the platform and reference it with credential_name.

Removing the block does not disconnect Prometheus. Git can create and update the source, never disconnect it - the source survives the sync and the block reappears the next time Ankra regenerates the file. Disconnect from Settings → Integrations instead.

---

Verifying the Connection

After connecting Prometheus:

1. Check Integrations - The Metrics entry in the sidebar shows a green dot, and the panel shows the last health check
2. View Resource Metrics - Go to any pod or deployment and check for metric graphs
3. Ask the AI - Press ⌘+J and ask “What’s the CPU usage of this pod?”

If metrics aren’t showing:

• Verify the endpoint URL is correct
• Check that Prometheus has data for your cluster
• Re-test the connection from Settings → Integrations → Metrics → Edit
• Check network connectivity between your cluster and Prometheus

---

Troubleshooting

Connection Test Fails

1. Verify the URL is correct and includes the port
2. For in-cluster Prometheus, ensure the service exists:
   kubectl get svc -n monitoring
   
3. Check if Prometheus is running:
   kubectl get pods -n monitoring -l app=prometheus
   
4. Test connectivity from within the cluster:
   kubectl run curl --rm -it --image=curlimages/curl -- curl http://prometheus-server.monitoring:9090/api/v1/query?query=up
   

Authentication Errors

1. Verify the credential is correctly configured
2. For API tokens, ensure the token hasn’t expired
3. For basic auth, verify the username and password are correct
4. Check Prometheus logs for authentication failures

Metrics Not Appearing

1. Prometheus may not have metrics for the resource yet (wait for scrape interval)
2. Verify Prometheus is scraping the correct targets
3. Check the Prometheus UI targets page for scrape errors
4. Ensure the metric names match what Ankra expects (standard Kubernetes metrics)

Slow Queries

1. Large time ranges can be slow. Try narrower ranges.
2. Consider using recording rules for frequently-queried metrics
3. Check Prometheus resource usage (may need more CPU/memory)

---

What Metrics Does Ankra Use?

Ankra queries standard Kubernetes and node metrics:

Metric	Source	Used For
container_cpu_usage_seconds_total	cAdvisor	Pod/container CPU graphs
container_memory_working_set_bytes	cAdvisor	Pod/container memory graphs
node_cpu_seconds_total	node-exporter	Node CPU usage
node_memory_MemAvailable_bytes	node-exporter	Node memory usage
kube_pod_status_phase	kube-state-metrics	Pod status indicators
kube_deployment_status_replicas	kube-state-metrics	Deployment health

If you’re using kube-prometheus-stack, all these metrics are included by default.

---

Next Steps

Build a Monitoring Stack

Deploy Prometheus and Grafana with one Stack.

AI Assistant

Use the AI to query and analyze your metrics.

Kubernetes Insights

Explore built-in cluster dashboards.

Credentials

Manage authentication for all integrations.