The Essence of Observability

Observability is more than just a trendy term; it's foundational for building dependable and reliable systems. This practice is vital for understanding system performance, ensuring its health, and resolving issues swiftly.
‍

Companies today invest heavily in system observability to safeguard their digital infrastructure, from small apps to large-scale systems. It revolves around the systematic collection and analysis of data from diverse sources within systems, applications, and infrastructure, offering unparalleled insights into their performance, health, and behaviors

Observability with Respect to Databases

Databases are the backbone of many applications, responsible for efficient data collection, storage, and retrieval. In simple database systems with a single cluster and instance, observability can be more straightforward. Metrics like query response times, error rates, and resource utilization offer direct insights into system performance.

However, observing the health and performance of databases becomes more intricate when dealing with multiple database clusters in a complex system. Various clusters within this ecosystem serve distinct purposes, such as handling user data, managing application logs, or supporting analytics workloads. To achieve comprehensive observability, it's crucial to tailor your metrics collection strategy to each cluster's specific role.
‍

The diagram below visually represents this concept. It showcases different clusters, including "User Data," "Logs," and "Analytics," each contributing specific metrics like "User Metrics" or "Log Data" to the central point labeled "Metrics Collection System." The arrows in the diagram illustrate the flow of data, underscoring the importance of customizing metrics collection for comprehensive observability within a diverse ecosystem.

‍

‍

For instance, a user data cluster's observability metrics may prioritize transaction throughput, latency, and data consistency to ensure smooth and reliable user interactions. In contrast, an analytics cluster's observability may focus on query execution times, data processing rates, and resource utilization to optimize data-driven decision-making. These tailored metrics provide valuable insights into the daily operations of the system.

‍

Availability vs. Performance Metrics

As you delve into the world of observability, take a look at the graph below, which highlights some of the most critical observability metrics.

‍

These metrics can be further divided into two main categories: availability metrics and performance metrics.
‍

Availability Metrics

These metrics revolve around ensuring that a database cluster remains accessible and operational. They include:

1. Uptime
2. Downtime
3. Error Rates
   ‍

Performance Metrics

Performance metrics dive deeper into the database's efficiency and responsiveness. These metrics encompass:

1. Query Execution Times 
2. CPU Utilization 
3. Disk I/O
4. Throughput
5. Latency
6. Response Time
   ‍

Having said that, what are some real-life observability and monitoring tools that can actually assist us in monitoring such important metrics? And how can we utilize such tools for peak system observability?

Prometheus

Source

‍

The first observability tool on our list is Prometheus. Prometheus is essential for system and database tracking, as it gathers metrics to provide deep insights into our system’s health. The Prometheus tool pulls data from throughout our system in order to monitor and better understand the internal states of a system, ensuring everything runs smoothly. 
‍

Prometheus tracks critical performance metrics, such as error rate, cpu, and memory usage among other metrics, enabling proactive and immediate issue resolution. It goes without saying, that such an optimization enhances system reliability and elevates the overall user experience. (HarperDB actually has a Prometheus exporter if you’d like to check it out). 

‍

PromQL

PromQL, a mighty query language tailored specifically for Prometheus, empowers users to query and analyze time-series data efficiently, gaining valuable insights into system performance and behavior. This capability aligns well with the flexibility offered by databases like HarperDB.
‍

Each query is based on an operation (such as sum), a metric (such as cpu usage), and labels that specify different filters or properties on that metric (such as mode).
‍

PromQL offers a range of basic aggregate functions, including max, min, avg, sum, and count, among others. For example, consider the below query:

sum by (instance) (cpu_usage{mode="user"})

This query calculates the sum of CPU usage in user mode for each instance. The output would be a time series for each instance showing the total CPU usage in user mode.
‍

Additionally, PromQL allows us to measure aggregations over a specified time period such as the avg_over_time query.

avg_over_time(<metric_name>[<time_range>])

This function computes the average of a metric over a defined time range, similar to the avg function but operating over a range of values.
‍

Now, let's explore rate queries. Rate queries are designed to calculate the rate of change of a counter metric within a specified time span. To use rate queries effectively, you must specify the metric of interest and the specific time range you wish to analyze.

rate(<metric_name>[<time_range>])

The following query calculates the per-second rate of API requests for a HarperDB server's 'data-api' endpoint over the last 5 minutes. The output would be a time series representing the request rate.

rate(harperdb_requests_total{endpoint="data-api"}[5m])

For example, if the query returns a value of 10, it means that, on average, there were 10 API requests per second for the HarperDB instance's 'data-api' endpoint during the last 5 minutes. This user-defined rate metric is valuable for assessing HarperDB instance performance, tracking spikes in data interaction, and setting alert thresholds for abnormal request rates.

Furthermore, to visualize the metrics collected by Prometheus in an intuitive and user-friendly manner, many teams integrate it with Grafana. Grafana transforms the raw data from Prometheus into comprehensive dashboards, allowing users to glean important insights at a glance. This combination of Prometheus and Grafana ensures not only the efficient tracking of system metrics but also a clear visualization of system performance.

‍

Grafana graphs(Source)

‍

These functions, in combination with appropriate metric selection and label filtering, provide a powerful toolkit for querying, aggregating, and transforming Prometheus metrics to gain insights into the behavior and performance of systems and applications.

OpenTelemetry - Tracing the Path

Source

‍

Our second observability solution is OpenTelemetry. OpenTelemetry is an open-source project that focuses on providing a set of APIs, libraries, agents, and instrumentation to enable observability in software applications and infrastructures. It facilitates the collection of telemetry data, which includes metrics, traces, and logs, from applications and services. 
‍

OpenTelemetry offers libraries for various programming languages (e.g., Java, Python, Go, JavaScript) that developers can integrate into their code. These libraries automatically capture telemetry data, such as traces (distributed traces to track requests across services) and metrics (performance and operational data).
‍

That being said, OpenTelemetry emerges as another invaluable asset for effective metric collection.

‍

Alerting to Slack, Emails, and PagerDuty

Alerting to platforms like Slack, emails, and PagerDuty is a crucial component of database server and system observability. These alerting mechanisms ensure that relevant stakeholders are promptly informed of any issues, anomalies, or critical events in the database and system performance. 
‍

Imagine a real-life scenario, where you are the software engineer responsible for a popular e-commerce platform gearing up for the holiday season, including the Black Friday shopping frenzy. During this time, the system experiences an unusually high load due to a surge in online shoppers. Here's how the alerting process comes into play.

‍

At the first sign of trouble, your monitoring tools trigger alerts that pop up in a dedicated Slack channel. Your team members and on-call personnel are actively monitoring this channel. They quickly spot the alerts, indicating that server response times are slowing down due to the increased load. The alerts will be triggered based on the metrics we collected through Prometheus meeting certain conditions

‍

While addressing the issue in Slack, your team realizes that this surge in traffic might require additional server resources. In order to keep your customers happy, you decide to scale up the system. As part of the process, detailed incident reports and instructions are sent via email to all relevant team members, outlining the steps needed for scaling and load balancing.
‍

Despite the initial efforts, the system load continues to rise, and it becomes clear that this situation demands specialized expertise. PagerDuty comes into play, automatically routing alerts to an on-call expert in system scaling and performance optimization. The PagerDuty platform empowers this expert to take charge and fine-tune the system for peak performance during peak load seasons such as Black Fridays.
‍

Conclusion

Observability is like our system's detective. It aids in the seamless operation of both modest and massive applications, serving as an early warning system for issues. Its purpose extends beyond ensuring availability to also optimizing the performance of databases and our system.
‍

Tools like Prometheus and OpenTelemetry augment this capability. And even when things go wrong, we get alerts on Slack, email, or PagerDuty to fix them fast. In the digital realm, observability serves as our compass, ensuring excellence prevails.

‍