Aembit Edge Prometheus-compatible metrics

Many organizations rely on and use various data collection and visualization tools to monitor components in their environment. This information provides users with the ability to quickly be alerted to any potential issues that may arise, and troubleshoot those issues.

Aembit Edge Components expose various Prometheus-compatible metrics so you have greater visibility into each of these components (Agent Controller, Agent Proxy, Agent Injector).

Prometheus configuration

Aembit exposes Prometheus-compatible metrics in several different deployment models, including Kubernetes and Virtual Machines. The installation and configuration steps for both of these deployment models are described below, but please note that you may select any observability tool you wish, as long as it can to scrape Prometheus-capable metrics.

Configuring Prometheus (Kubernetes)

These steps described below show an example of how you can configure a “vanilla” Prometheus instance in a Kubernetes cluster. Depending on your own Kubernetes cluster configuration, you may need to perform a different set of steps to configure Prometheus for your cluster.

Open a terminal window in your environment and run the command shown below.

kubectl edit configmap prometheus-server

Edit the prometheus.yaml configuration file by adding the following code snippet before the kubernetes-pods section:

    - honor_labels: true
      job_name: kubernetes-pods-aembit
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
          - __meta_kubernetes_pod_annotation_aembit_io_metrics_scrape
      - action: replace
        regex: (.+)
        source_labels:
          - __meta_kubernetes_pod_annotation_aembit_io_metrics_path
        target_label: __metrics_path__
      - action: replace
        regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
        replacement: "[$2]:$1"
        source_labels:
          - __meta_kubernetes_pod_annotation_aembit_io_metrics_port
          - __meta_kubernetes_pod_ip
        target_label: __address__
      - action: replace
        regex: (\d+);((([0-9]+?)(\.|$)){4})
        replacement: $2:$1
        source_labels:
          - __meta_kubernetes_pod_annotation_aembit_io_metrics_port
          - __meta_kubernetes_pod_ip
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: drop
        regex: Pending|Succeeded|Failed|Completed
        source_labels:
          - __meta_kubernetes_pod_phase
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: node

The example code block shown above allows for the automatic detection of Aembit annotations so Prometheus can automatically scrape Agent Proxy metrics.

Save your changes in the prometheus.yaml configuration file.

Kubernetes Annotations

Agent Controller and Agent Proxy come with standard Prometheus annotations, enabling Prometheus to automatically discover and scrape metrics from these Aembit Edge Components.

Since the Agent Proxy runs as part of the Client Workload, which may already expose Prometheus metrics and have its own annotations, a new set of annotations was introduced. These annotations can be added to Client Workload pods without conflicting with existing annotations.

The following annotations have been introduced, which are automatically added to the Client Workload where the Agent Proxy is injected:

aembit.io/metrics-scrape - Default value is true.
aembit.io/metrics-path - Default value is /metrics.
aembit.io/metrics-port - Default value is 9099. This is a default metrics port used by Agent Proxy to expose metrics.

You may override these annotations, aembit.io/metrics-port to adjust metrics port on Agent Proxy.

Helm Variables

The following Helm variables control whether metrics are enabled or disabled:

agentController.metrics.enabled
agentInjector.metrics.enabled
agentProxy.metrics.enabled

Configuring Prometheus (Virtual Machine)

You need to configure which Virtual Machines you want to scrape for metrics and data by editing the /etc/prometheus/prometheus.yml YAML file and replacing example.vm.local:port with the Agent Controller and Agent Proxy VM hostname, and port number on which the metrics servers are listening. For Agent Controller, set the port number to 9090. For Agent Proxy, set the port number to 9099.

scrape_configs:
  - job_name: 'vm-monitoring'
    static_configs:
      - targets: ['example.vm.local:<port>']

Virtual Machine Environment Variables

These environment variables can be passed to the Agent Controller installer to manage the metrics functionality.

AEMBIT_METRICS_ENABLED - enabled for both Agent Controller and Agent Proxy
AEMBIT_METRICS_PORT - available only for Agent Proxy

Aembit Edge Prometheus Metrics

Aembit Edge Components expose Prometheus-compatible metrics that can be viewed using an observability tool that is capable of scraping these types of metrics.

The sections below list the various Prometheus-compatible metrics that Aembit Edge Components expose, along with the labels you can use to filter results and drill down into specific data.

Agent Proxy Metrics

The Agent Proxy Prometheus-compatible metrics listed below may be viewed in a dashboard.

aembit_agent_proxy_incoming_connections_total - The total number of incoming connections (connections established from a Client Workload to the Agent Proxy).
- labels:
  - application_protocol: http, snowflake, postgres, redshift, mysql, redis, unspecified
  - resource_set_id (optional): <resource_set_id>
  - client_workload_id (optional): <client_workload_id>
  - server_workload_id (optional): <server_workload_id>
aembit_agent_proxy_active_incoming_connections - The number of active incoming connection (connections established from a Client Workload to the Agent Proxy).
- labels:
  - application_protocol: http, snowflake, postgres, redshift, mysql, redis, unspecified
  - resource_set_id (optional): <resource_set_id>
  - client_workload_id (optional): <client_workload_id>
  - server_workload_id (optional): <server_workload_id>
aembit_agent_proxy_credentials_injections_total - The total number of credentials injected by Agent Proxy.
- labels:
  - application_protocol: http, snowflake, postgres, redshift, mysql, redis, unspecified
  - success: success, failure.
aembit_agent_proxy_token_expiration_unix_timestamp - The expiration timestamp for Aembit Agent Proxy Token (to access Aembit Cloud).
aembit_agent_proxy_aembit_cloud_connection_status - The current connection status between Agent Proxy and Aembit Cloud. If the connection is up, the result is “1” (Connected). If the status is down, the result is “0” (Disconnected).
aembit_agent_proxy_credentials_cached_entries_total - The total number of unexpired credentials currently cached by Agent Proxy.
- labels:
  - resource_set_id (optional): <resource_set_id>
aembit_agent_proxy_directives_cached_entries_total - The total number of unexpired directives currently cached by Agent Proxy.
- labels:
  - resource_set_id (optional): <resource_set_id>
version - The Agent Proxy version.
- labels:
  - component: aembit_agent_proxy
  - version: version: <version>
process_cpu_second_total - The Amount of CPU seconds used by the Agent Proxy. This value could be more than the wall clock time if Agent Proxy used more than one core. This metric is useful in conjunction with machine_cpu_cores to calculate CPU % usage.
- labels:
  - component: aembit_agent_proxy
  - hostname: hostname: <hostname>
machine_cpu_cores - The number of CPU cores available to Agent Proxy.
- labels:
  - component: aembit_agent_proxy
  - hostname: hostname: <hostname>
process_memory_usage_bytes - The amount of memory (in bytes) used by Agent Proxy.
- labels:
  - component: aembit_agent_proxy
  - hostname: hostname: <hostname>

Agent Controller Metrics

The Agent Controller Prometheus-compatible metrics listed below may be viewed in a dashboard.

aembit_agent_controller_token_expiration_unix_timestamp - The expiration timestamp for Aembit Agent Controller Token (to access Aembit Cloud).
aembit_agent_controller_access_token_requests_total - The number of Agent Controller requests to get access token (for Agent Controller use).
- label
  - Result: success, failure
  - Agent_Controller_Id: <AgentControllerID>
aembit_agent_controller_proxy_token_requests_total - The number of Agent Proxy requests received by the Agent Controller to get access token.
- labels
  - Result: success, failure
  - Agent_Controller_Id (optional): <AgentControllerID>
aembit_agent_controller_registration_status - The Agent Controller registration status. Status can be either: 0 (Not Registered) or 1 (Registered).
- labels
  - Agent_Controller_Id (optional): <AgentControllerID>
version - The Agent Controller version.
- labels
  - component: aembit_agent_controller
  - version: <version>

Agent Injector metrics

The Agent Injector Prometheus-compatible metrics listed below may be viewed in a dashboard.

aembit_injector_pods_seen_total - The number of pods proceeded by the Agent Injector.
aembit_injector_pods_injection_total - The number of pods into which Aembit Edge Components were injected.
- label
  - success: “success” or “failure”