UPDATE: HarperDB now has an official Prometheus Exporter

‍

Monitoring a database is important to identify performance bottlenecks and areas for optimization. Currently HarperDB exposes its metrics via the `system_information` endpoint and visualizes them on HarperDB Studio under the `status` tab:

‍

However, if you wish to export these metrics to your existing monitoring stack, currently there are no native plugins for popular monitoring tools including Prometheus, Datadog, or Splunk. In this post, we’ll explore how to export HarperDB metrics to Prometheus, utilizing the Push Gateway to periodically update metrics. 

‍

HarperDB Setup

First, we will run a local instance of HarperDB with default configurations:

```
docker run -d \                                                                          
  -v $(pwd):/home/harperdb/hdb \
  -e HDB_ADMIN_USERNAME=HDB_ADMIN \
  -e HDB_ADMIN_PASSWORD=password -e \
  -p 9925:9925 \
  -p 9926:9926 \
  harperdb/harperdb
```

‍

Once the database is ready, we can check for the system information:

```
curl --location --request POST 'localhost:9925' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic SERCX0FETUlOOnBhc3N3b3Jk' \
--data '{
    "operation": "system_information"
}'
```

‍

NOTE: the response format differs slightly from the API docs so make sure to use the format returned by the version of HarperDB you are running locally. 

‍

Prometheus Setup

We are now ready to also deploy Prometheus locally for testing. First, we need to add a configuration for Prometheus server to also scrape Push Gateway. 

‍

Create a new directory called `prometheus` and add a new file called `prometheus.yml` with the following contents:

lobal:
 scrape_interval:     15s
 evaluation_interval: 15s


# A scrape configuration containing exactly one endpoint to scrape.
scrape_configs:
 - job_name: 'prometheus'
   scrape_interval: 10s
   static_configs:
     - targets: ['localhost:9090']


 - job_name: 'pushgateway'
   scrape_interval: 10s
   honor_labels: true
   static_configs:
     - targets: ['pushgateway:9091']

Then we can create our docker compose file:

services:
 prometheus:
   image: prom/prometheus:v2.43.0
   container_name: prometheus
   restart: unless-stopped
   volumes:
     - ./prometheus:/etc/prometheus
   ports:
     - "9090:9090"


 pushgateway:
   image: prom/pushgateway:v1.5.1
   container_name: pushgateway
   restart: unless-stopped
   ports:
     - "9091:9091"

Here we are mounting the prometheus config file to the server and running both server and pushgateway containers with ports mapped to localhost. 

‍

You can navigate to http://localhost:9090/targets to verify that both components are up and running:

‍

‍

Exporting Metrics to Prometheus

Finally, we’re ready to start exporting metrics to Prometheus. First, we need to store our basic auth token to `.env`. If you used the above docker command to run HarperDB, you can copy the BASIC_AUTH variable and paste it to `.env`: 

BASIC_AUTH=SERCX0FETUlOOnBhc3N3b3Jk

‍

Then create `index.js` with the following contents:

const axios = require('axios');
const client = require('prom-client');
require('dotenv').config();


const headers = {
 'Content-Type': 'application/json',
 'Authorization': "Basic " + process.env.BASIC_AUTH
};


// Prometheus Setup
const Registry = client.Registry;
const register = new Registry();
const gateway = new client.Pushgateway('http://127.0.0.1:9091', [], register);
const prefix = 'harperdb';


const cpuLoad = new client.Gauge({
 name: 'harperdb_cpu_load',
 help: 'Current cpu load'
});


const usedMemory = new client.Gauge({
 name: 'harperdb_used_memory',
 help: 'Memory usage'
})


register.registerMetric(cpuLoad)
register.registerMetric(usedMemory)


async function getMetrics() {
 const { data } = await axios.post('http://127.0.0.1:9925', { operation: 'system_information' }, { headers });
 const { cpu, memory } = data;


 cpuLoad.set(cpu.current_load.currentLoad);
 usedMemory.set(memory.used);


 try {
   console.log(`Publishing metrics to Push Gateway: cpu ${cpu.current_load.currentLoad} and memory ${memory.used}`)
   await gateway.push({ jobName: prefix });
 } catch(err) {
   console.log(`Error: ${err}`);
 }
}


setInterval(() => {
 getMetrics()
}, 5000);

‍

Let’s take a deeper look at this code:

1. We are registering two metrics `cpuLoad` and `usedMemory` and creating a Prometheus Gauge. 
2. Every 5 seconds, we poll the system information from HarperDB and we set the gauge values accordingly.
3. Finally, we publish those metrics to Push Gateway via the `gateway.push` method. 
4. These metrics are registered to the Push Gateway’s register, which is in turn scraped by Prometheus server as specified in the config file we mounted earlier. 

‍

Run this code and you should see log messages like:

```
Publishing metrics to Push Gateway: cpu 2.348993288590604 and memory 4006883328
Publishing metrics to Push Gateway: cpu 2.5662529352566255 and memory 4012720128
Publishing metrics to Push Gateway: cpu 2.56797583081571 and memory 4012437504
Publishing metrics to Push Gateway: cpu 2.683663200268366 and memory 4008710144
```

Visualizing Data

Navigate to localhost:9090/graph to start querying our data.

‍

For example, for CPU load, we can search for `harperdb_cpu_load`:

‍

Likewise, we can also graph memory usage:

‍

‍

Conclusion

In this post, we saw how we can use the Prometheus client library to export HarperDB metrics available via its system_information endpoint. We registered two example CPU load and memory usage metrics. You can easily extend this example with other metrics available in HarperDB such as disk information or more granular CPU/memory statistics. 

‍

‍