Overview

‍

According to a HashiCorp - Forrester report, 94% of enterprises overspend in the cloud, even if they don’t always know it. And based on our discussions, 9 out of 10 enterprises don’t measure their disk utilization, essential for understanding your storage spend. But what is disk utilization and why is it important?

Amazon Elastic Block Store (Amazon EBS) is a block-level storage service designed for use with Amazon Elastic Compute Cloud (Amazon EC2). It provides highly available, scalable, and reliable storage for Amazon EC2 instances. Amazon EBS volumes are highly flexible and can be easily attached or detached from Amazon EC2 instances.

Disk utilization is a measure of how much of the available storage space is being used in a particular disk or volume. It is an important metric to keep track of because it allows users to optimize their storage resources and avoid unnecessary costs. By monitoring disk utilization, users can identify trends and patterns in their storage usage and make informed decisions about when to add or remove storage capacity. It also helps users avoid over-provisioning, which can result in wasted resources and increased costs.

‍

‍

Finding your AWS EBS storage metrics

‍

The native way of collecting Amazon EBS metrics is to use Amazon’s built-in CloudWatch monitoring service. CloudWatch provides a convenient and standard overview of your volume fleet.

However, CloudWatch's method of gathering metrics through hypervisor instead of directly reporting from your volumes, may not capture all the EBS metrics that you may want to monitor. Eg. disk space statistics such as disk utilization, free disk space, used disk space, and so on.

One way to fill this gap is to install an agent on your instances that can collect system-level information such as disk utilization metrics. An example of this is Amazon’s CloudWatch Agent. CloudWatch treats metrics forwarded by its agent as custom metrics, meaning that by default it collects them at a one-minute resolution and has the ability to go as high as one second.

Note, however, additional charges will accrue for custom metrics. This is where observability services offered by Datadog can help customers.

Datadog is an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. 

Datadog’s AWS integration lets you connect CloudWatch to Datadog in order to automatically collect metrics from multiple AWS services—including EBS. For even greater visibility into your Amazon EBS volumes and your entire infrastructure, you can install the Datadog Agent on your instances. This enables you to gather system-level metrics from your volumes, including disk usage, at a 15-second resolution.

‍

‍

Steps to collect average disk utilization across your EBS volumes using Datadog

You can capture the average disk utilization across your EBS volumes using two methods.

‍

Method 1: Using Metrics Explorer (Manual)

‍

1. Open Datadog from https://app.datadoghq.com/ and log in.

2. From the left pane, click on Metrics>Explorer.

‍

‍

3. In Metric explorer, click on Add query > click on the code icon at the end of the corresponding line.

‍

‍

4. Copy and paste the below query.

‍

avg:system.disk.in_use{!device:/dev/loop*,device:/dev/*}

‍

Repeat steps 3 and 4, and replace the query with the one below.

‍

avg:system.disk.in_use{device:*:}

‍

‍

5. Click on Save to Dashboard and select New Dashboard.

‍

‍

6. Click on View Dashboard to navigate to the new Dashboard created.

‍

‍

The dashboard will contain:

Average Disk utilization across all Linux instances running Datadog

Average Disk utilization across all Windows instances running Datadog

‍

‍

Method 2: Using a JSON template (Automated)

‍

Download this file and store it locally. This will be used in the next steps.

‍

1. Open Datadog from https://app.datadoghq.com/ and log in.

‍

2. From the left pane, click on Dashboards>New Dashboard.

‍

‍

3. In the Create a Dashboard window, Select New Dashboard.

‍

‍

4. Once the new dashboard window loads, go to Settings from the right side corner and select Import dashboard JSON.

‍

‍

5. In the browse section, select the file downloaded in step 1.

‍

‍

6. A confirmation window pops up asking whether you want to replace the existing dashboard. Select "Yes, Replace".

‍

‍

7. Lucidity Disk Assessment dashboard loads up. It provides 2 metrics:

a. Average Disk utilization across all Linux instances running Datadog

b. Average Disk utilization across all Windows instances running Datadog

‍

‍

P.S Few other metrics you can track for your EBS volumes using Datadog are:

‍

• system.disk.free - The amount of disk space that is free. Shown in bytes.

• system.disk.used - The amount of disk space in use. Shown in bytes.

• system.disk.total - The total amount of disk space. Shown in bytes.

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍