> For the complete documentation index, see [llms.txt](https://docs.cybus.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cybus.io/2-6-0/monitoring/connectware-metrics/scraping-metrics.md).

# Scraping Connectware Metrics with Self-Managed Prometheus

Configure a self-managed Prometheus instance to scrape Connectware metrics endpoints using static targets or Kubernetes Service Discovery, without relying on Prometheus Operator.

For Kubernetes deployments with Prometheus Operator, use `ServiceMonitor` resources through the Helm chart instead. See [Configuring Prometheus Metrics](/2-6-0/cybus-helm-charts/working-with-cybus-helm-charts/metrics.md).

## Prerequisites

* Metrics are enabled for the component you want to scrape. See [Enabling Metrics for a Single Component](/2-6-0/cybus-helm-charts/working-with-cybus-helm-charts/metrics.md#enabling-metrics-for-a-single-component).
* [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) is installed and configured for your cluster.
* You know the namespace of your Connectware installation. See [Obtaining the Name, Namespace, and Version of Your Installation](/2-6-0/cybus-helm-charts/working-with-cybus-helm-charts.md#obtaining-the-name-namespace-and-version-of-your-installation).

## Metrics Endpoints

Each component with `metrics.enabled` set to `true` creates a dedicated Kubernetes `Service` for its metrics endpoint, separate from its main Service. The Service name always follows the pattern `<app-name>-metrics`, regardless of your installation name or namespace. All endpoints use HTTP and the path `/metrics`.

| Component                | App Name                   | Metrics Service                    | Port   |
| ------------------------ | -------------------------- | ---------------------------------- | ------ |
| Auth Server              | `auth-server`              | `auth-server-metrics`              | `9184` |
| Broker                   | `broker`                   | `broker-metrics`                   | `8888` |
| NATS                     | `nats`                     | `nats-metrics`                     | `7777` |
| Protocol Mapper          | `protocol-mapper`          | `protocol-mapper-metrics`          | `9184` |
| Resource Status Tracking | `resource-status-tracking` | `resource-status-tracking-metrics` | `9184` |
| Service Manager          | `service-manager`          | `service-manager-metrics`          | `9184` |

{% hint style="warning" %}

## Scraping horizontally scaled Workloads

Workloads which are horizontally scaled, like Broker, NATS, or auth-server deliver metrics per node. Each replica exposes its own metrics endpoint and the cluster does not aggregate them. The Kubernetes Service load-balances across nodes, so scraping it by its Service name or ClusterIP returns metrics from only one node per scrape. To capture every node, scrape the individual pod endpoints, for example by using the `endpoints` role instead of the `service` role in `kubernetes_sd_configs`.

For node-level broker metrics and calculating cluster-level aggregates at query time, see [Monitoring & Analytics](/2-6-0/broker/cybusmq/configuration/monitoring-and-analytics.md).
{% endhint %}

## Verifying an Endpoint with curl

To confirm that a metrics endpoint responds before configuring Prometheus:

1. Forward the component's metrics Service to your local machine. The following example targets Protocol Mapper:

{% code lineNumbers="true" %}

```bash
kubectl port-forward -n ${NAMESPACE} svc/protocol-mapper-metrics 9184:9184
```

{% endcode %}

Replace `protocol-mapper-metrics` and `9184` with the Service name and port for your component from [Metrics Endpoints](#metrics-endpoints). Leave this command running in its own terminal.

2. In a separate terminal, request the endpoint:

{% code lineNumbers="true" %}

```bash
curl http://localhost:9184/metrics
```

{% endcode %}

The endpoint returns metrics in the [OpenMetrics](https://openmetrics.io) text format, which Prometheus scrapes natively. The following shows a sample from Protocol Mapper:

{% code lineNumbers="true" %}

```
# HELP agent_connections_deployed Current number of deployed connections by protocol and status
# TYPE agent_connections_deployed gauge
agent_connections_deployed{protocol="opcua",status="connected"} 3
agent_connections_deployed{protocol="mqtt",status="connected"} 1

# HELP agent_endpoints_messages_total Total number of messages processed by endpoints
# TYPE agent_endpoints_messages_total counter
agent_endpoints_messages_total{protocol="opcua",operation="subscribe"} 42180
agent_endpoints_messages_total{protocol="mqtt",operation="write"} 14

# HELP agent_health Health status of the agent components
# TYPE agent_health gauge
agent_health{component="control-plane"} 1
agent_health{component="data-plane"} 1
```

{% endcode %}

## Configuring Prometheus

Add a scrape job to your `prometheus.yml` file. Choose one of the following approaches.

### Static Targets

Use a static target when you know the exact Service address you want to scrape. The following example targets Protocol Mapper:

{% code title="prometheus.yml" lineNumbers="true" %}

```yaml
scrape_configs:
  - job_name: connectware-protocol-mapper
    static_configs:
      - targets:
          - protocol-mapper-metrics.${NAMESPACE}.svc.cluster.local:9184
```

{% endcode %}

Add one job per component, replacing the Service name and port from [Metrics Endpoints](#metrics-endpoints). This approach requires updating `prometheus.yml` whenever you change a component's metrics configuration.

### Kubernetes Service Discovery

Every metrics Service carries `prometheus.io/scrape`, `prometheus.io/port`, and `prometheus.io/path` annotations once `metrics.enabled` is `true`. If your Prometheus instance discovers targets through `kubernetes_sd_configs`, configure a scrape job that filters on these annotations:

{% code title="prometheus.yml" lineNumbers="true" %}

```yaml
scrape_configs:
  - job_name: connectware
    kubernetes_sd_configs:
      - role: service
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels:
          - __address__
          - __meta_kubernetes_service_annotation_prometheus_io_port
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        regex: (.+)
        target_label: __metrics_path__
```

{% endcode %}

This job automatically picks up every Connectware metrics Service in the cluster without requiring a `ServiceMonitor`.

{% hint style="info" %}
For the full relabeling syntax, see the Prometheus documentation on [`kubernetes_sd_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config).
{% endhint %}

## Avoiding the Scrape Rate Limit

Auth Server, Protocol Mapper, Resource Status Tracking, and Service Manager limit how many scrape requests their metrics endpoint accepts within a time window. If you set a short scrape interval or scrape a component from multiple Prometheus jobs simultaneously, you may exceed `rateLimit.count`. Increase `rateLimit.count` or `rateLimit.windowMs` for the affected component, or reduce your scrape frequency. See [Configuring Prometheus Metrics](/2-6-0/cybus-helm-charts/working-with-cybus-helm-charts/metrics.md#configuration-values) for the full Helm values reference.

Broker and NATS do not have a configurable rate limit.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.cybus.io/2-6-0/monitoring/connectware-metrics/scraping-metrics.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.