How to configure low-disk health notifications on Platform.sh

How-to Guides
, , , ,
#1

Goal

Set up health notifications for low-disks on project containers. These will notify when free disk space drops below 20% and below 10%, and will issue an all-clear notification when the free disk space returns above 20%.

Assumptions

Problems

Container storage can silently fill due to any number of reasons. This will often cause an application to experience erratic errors or cease to function entirely.

Steps

1. Email Notifications

  1. Add the integration for email using the Platform CLI:
platform integration:add --type health.email --from-address you@example.com --recipients them@example.com --recipients others@example.com

Note that the --from-address can be any email address you like (including one of the recipients), and you can define one or more recipients by optionally adding more recipient flags.

2. Slack Notifications

  1. Create a custom bot user for your Slack group, configure it for the appropriate channels, and note the API Token.
  2. Add the integration for slack using the Platform CLI:
platform integration:add --type health.slack --token YOUR_API_TOKEN --channel '#channelname'

The bot will then post the notification to the channel specified by the --channel flag.

3. PagerDuty Notifications

  1. Create a PagerDuty integration using the Events API v2. Copy the Integration Key to use as the --routing-key flag in step (2).
  2. Add the integration for PagerDuty using the Platform CLI:
platform integration:add --type health.pagerduty --routing-key YOUR_ROUTING_KEY

4. What to do when you get a low-disk notification

There are several different options for resolving low-disk situations. For a database, depending on the application set up, you may be able to truncate a table in MySQL that is performing a logging or caching function. For an application container, you can use du -sh <path> to work your way through your filesystem to find if there are any particular files that are taking a lot of space that can be removed.

Alternatively, if the space used is necessary and will continue to grow, you can change the size of your disk to resolve the low-disk situation. For the application disk, you can change this with the disk: key in your .platform.app.yaml file. For any other services, their disk sizes can be adjusted from the relevant disk: key in your .platform/services.yaml file.

Conclusion

Using low-disk health notification with your Platform.sh environment can help you catch outages before they happen, and can be a red flag if they do lead to an outage.