We manage a few telephone systems based on Kamailio and other open-source components for our customers. This requires a proactive approach to ensure seamless operations. Our experience highlighted the importance of the observability of every aspect of our systems (and not simply relying on end-user feedback).

To achieve that, you have to gather metrics, logs, and traces from all components and consolidate them for analysis and alerting. We advocate for a robust stack based on Prometheus, Alert Manager, Loki, and Grafana. This has been our choice for years.

This article will look at how we can collect Kamailio metrics using Prometheus.

Where do you start?

Several methods exist for exposing Kamailio's metrics:

  • Pushing metrics to a Statsd server using the stastd module and use the statsd_exporter
  • Expose the metrics using the xhttp_prom module
  • Deploy an exporter that interacts with Kamailio using the ctl module

We settled for the latter as it gives a lot of flexibility in terms of deployment. The florentchauveau and pascomnet Kamailio exporters are robust options, and we ran both of them in production at one point. The pascomnet exporter, in particular, offers additional metrics collection capabilities, including metrics created using the xhttp_prom module and RTPengine metrics. Unfortunately, it is missing some advanced metrics. So we decided to fork and refactor it to bring it closer to the other Prometheus exporters. We've added the collection of additional metrics and extra functionalities (more on this later).

The project is available here: angarium-cloud/kamailio_exporter

Configure Kamailio for BINRPC Connection

These exporters need a BINRPC connection to Kamailio.

  1. Load and configure the CTL module in Kamailio loadmodule "ctl.so"
  2. If Kamailio and the Exporter run on the same machine, you can use a unix socket. The default path is unix:/var/run/kamailio/kamailio_ctl.
  3. If you are running Kamailio in a Docker container or want to deploy the exporter on a remote server, you will need to configure a TCP socket. For example, add this line in your Kamailio config: modparam("ctl", "binrpc", "tcp:172.16.106.10:2046").

Collect metrics using the Kamailio Exporter

  1. Download the kamailio_exporter binary from the project release page.
  2. Start the exporter. By default, it connects to Kamailio using the unix domain socket at /var/run/kamailio/kamailio_ctl Check the README of the project for more options or run the exporter with the --help flag.
  3. The exporter exposes metrics on port 9494 at the HTTP path /metrics.
  4. Test that the exporter can connect to Kamailio by running curl -s http://localhost:9494/metrics | grep kamailio_up. The output should return this:
# HELP kamailio_up kamailio_exporter: Whether the Kamailio endpoint is up.
# TYPE kamailio_up gauge
kamailio_up 1

If the value of the kamailio_up metrics is 1, the exporter can connect to Kamailio, and it collects further metrics.

  1. Configure your Prometheus server to scrape the metrics from the exporter. Assuming it runs on a server with the IP 172.16.106.10, the config should look like this:
- job_name: "kamailo"
    static_configs:
      - targets: ["172.16.106.10:9494"]  
    honor_timestamps: false
  1. Using the Prometheus or Grafana UI, verify that Prometheus has scraped the metrics. You are now ready to start building dashboards and creating alerts.

Which metrics are available?

The exporter gathers a comprehensive range of metrics, including:

  • Core metrics (Shared memory, Request/Reply, TCP, Dialog SL, TMX)
  • Uptime and core information about Kamailio
  • Status of each process running
  • Extra TCP metrics
  • Dispatcher list status
  • Dialog metrics
  • Number of dialogs belonging to a profile.
  • Htable status and metrics
  • Extra Private memory metrics
  • RTPengine status
  • Additional SL module Stats
  • Additional TM module Stats
  • TLS metrics

It will check that a module is loaded before collecting the metrics.

Advanced features

Our exporter provides a couple of advanced features that need extra configuration.

Dialog profile

By default, the exporter collects active dialogs metrics. We wanted to go deeper and get the current number of active calls in a dialog profile. To do this, you can pass the list of profiles to the command line using the --collector.dialog.profiles= flag.

Let's say you have the following profile definition in Kamailio: modparam("dialog", "profiles_with_value", "SBC;PROVIDER_A_IN;PROVIDER_A_OUT;PROVIDER_B_IN;PROVIDER_B_OUT;") and set the current dialog with a profile like this set_dlg_profile("PROVIDER_A_IN","$fu");.

You can run the exporter as follow: kamalio_exporter --collector.dialog.profiles="PROVIDER_A_IN" --collector.dialog.profiles="PROVIDER_A_OUT" --collector.dialog.profiles="PROVIDER_B_IN" --collector.dialog.profiles="PROVIDER_B_OUT"

The exporter will return the following metrics:

kamailio_dlg_profile_get_size_dialog{profile="PROVIDER_A_IN"} 10
kamailio_dlg_profile_get_size_dialog{profile="PROVIDER_A_OUT"} 2
kamailio_dlg_profile_get_size_dialog{profile="PROVIDER_B_IN"} 19
kamailio_dlg_profile_get_size_dialog{profile="PROVIDER_B_OUT"} 5

Dispatcher mapping

Our exporter allows you to map the dispatcher ID to a Name. This is useful when building a dynamic Dashboard in Grafana.

If you have the following dispatcher.list:

# dispatcher destination sets (groups)
# line format
# setid(int) destination(sip uri) flags(int,opt) priority(int,opt) attributes(str,opt)
# Carrier 1
200 sip:172.16.105.138:5070;transport=tcp 0 0 socket=tcp:172.16.105.10:5060 Carrier 1
# Carrier 2
400 sip:172.16.106.128:5060 0 50 socket=udp:172.16.106.10:5060 Carrier 2 SBC1
400 sip:172.16.106.129:5060 0 50 socket=udp:172.16.106.10:5060 Carrier 2 SBC2

You can configure the exporter with the following flags: --collector.dispatcher.mapping="200:Carrier 1" --collector.dispatcher.mapping="400:Carrier 2". The This will result in the following metrics:

kamailio_dispatcher_list_target{destination="sip:172.16.105.138:5070;transport=tcp",set_id="200",set_name="Carrier 1"} 1
kamailio_dispatcher_list_target{destination="sip:172.16.106.128:5060",set_id="400",set_name="Carrier 2"} 1
kamailio_dispatcher_list_target{destination="sip:172.16.106.129:5060",set_id="400",set_name="Carrier 2"} 1

Conclusion

This exploration into Kamailio metrics collection using Prometheus sets the stage for advanced monitoring. In our following article, we'll dive into leveraging this data for creating insightful Grafana dashboards and effective alerting mechanisms, ensuring your telephony systems perform optimally.