How to forward Platform.sh logs to Logz.io

Goal

To forward Platform.sh logs to Logz.io.

Assumptions

Problems

Forwarding Platform.sh to a third-party indexer such as Logz.io is useful, especially when integrating log monitoring across other projects, including those that are not hosted on Platform.sh. Logz.io uses Filebeat to create configuration, auth and forwarding files when shipping logs, which must be taken into account on Platform.sh’s read-only file system.

Logz.io recommends using Filebeat to forward logs to it, however there are a number of other options outlined in the “Log Forwarding” section on the Logz.io dashboard.

The following How-to is written to ship logs from a Python application, but the steps do not require that.

More information regarding accessing Platform.sh logs can be found in the public documentation.

Steps

1. Configure routes and services

The project locally will ultimately have the following structure:

├── .platform
│   ├── local
│   │   ├── .gitignore
│   │   ├── README.txt
│   │   └── project.yaml
│   ├── routes.yaml
│   └── services.yaml
├── .platform.app.yaml
├── Pipfile
├── Pipfile.lock
├── README.md
├── config
│   └── filebeat
│       └── scripts
│           ├── config.sh
│           └── install.sh
├── filebeat.yml
└── server.py

The project is a simple “Hello, world!” application using Flask. Routes must be configured in the .platform/routes.yaml:

# .platform/routes.yaml

https://{default}/:
  type: upstream
  upstream: "app:http"

as well as its services in .platform/services.yaml, which is defined here with a single MySQL/MariaDB service:

# .platform/services.yaml

mysql:
  type: "mysql:10.2"
  disk: 1024

2. Set up the Platform.sh Application configuration

Modify .platform.app.yaml to define writable mounts for Filebeat:

# .platform.app.yaml

mounts:
    '/.filebeat':
        source: local
        source_path: filebeat

Mounts are utilized in this configuration because after Filebeat is installed, it will attempt to write files to the project so that logs can be forwarded to Logz.io. Since Platform.sh is designed to provide a read-only file system, mounts can be used for this purpose.

Modify the build hook:

# .platform.app.yaml

hooks:
    build: |
        if [ ! -z $LOGZ_CONFIG ]; then
          ./config/filebeat/scripts/install.sh
        fi
        pipenv install --system --deploy

This section runs an install script (defined below) that installs Filebeat if it does not detect a project variable that we will define after the installation has completed.

Modify the deploy hook:

    deploy: |
        if [ ! "$(ls -A filebeat)" ]; then
             ./config/filebeat/scripts/config.sh
        fi
        ./filebeat/filebeat run --once

The deploy hook runs a configuration script on the installation if the filebeat directory is empty.

The final command runs Filebeat until the logs are up to date with those previously shipped and then forwards them to Logz.io on every deployment.

3. Install Filebeat

In config/filebeat/scripts/ modify a script install.sh:

# config/filebeat/scripts/install.sh

#!/usr/bin/env bash

TEMP_SPLUNK_HOME=config/splunk/build

# Install Splunk Universal Forwarder
#!/usr/bin/env bash

TEMP_BEAT_HOME=config/filebeat/build

[ ! -d $TEMP_BEAT_HOME ] && mkdir -p $TEMP_BEAT_HOME
cd $TEMP_BEAT_HOME

# Install Filebeat
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.7.0-linux-x86_64.tar.gz
tar xzvf filebeat-6.7.0-linux-x86_64.tar.gz
rm filebeat-6.7.0-linux-x86_64.tar.gz

# Download the certificate
wget https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt

mkdir -p filebeat-6.7.0-linux-x86_64/pki/tls/certs
cp COMODORSADomainValidationSecureServerCA.crt filebeat-6.7.0-linux-x86_64/pki/tls/certs/

The script will define a temporary location within the config/filebeat subdirectory to install Filebeat. During the build phase mounts are not yet available, so this becomes a necessary step before the mounts are fully defined.

The initial configuration requires that a certificate is downloaded along with the installation and placed in the appropriate directory. Details regarding this step are available on the Filebeat steps of the Log Forwarding section of the dashboard.

Make sure to replace the installation link with the most recent version listed on the Filebeat downloads page.

4. Move installation to mount point

Write a config.sh script within config/filebeat/scripts that will be used to control the initial configuration:

# config/filebeat/scripts/config.sh

#!/usr/bin/env bash

# Move filebeat to mount with write access
cd $PLATFORM_HOME
cp -v -r config/filebeat/build/filebeat-6.7.0-linux-x86_64/* filebeat
mkdir filebeat/registry

This script is run in the deploy hook when the mounts defined in .platform.app.yaml are now available, so Filebeat can be moved from the temporary installation directory config/filebeat/build/ to app/filebeat.

5. Finish the configuration

The main configuration file for Filebeat is filebeat.yml, which should be placed in the application directory outside of filebeat.

On Logz.io within the configuration steps for Filebeat, the Wizard will construct this file based on the desired inputs that are given in its fields. An example for forwarding all file in /var/log that end with .log

# filebeat.yml
############################# Filebeat #####################################

filebeat.inputs:

- type: log
  paths:
    - /var/log/*.log
  fields:
    logzio_codec: plain
    token: <generated token>
    type: Platform.sh
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h

registry_file: /filebeat/registry

############################# Output ##########################################

output:
  logstash:
    hosts: ["listener.logz.io:5015"]
    ssl:
      certificate_authorities: ["/app/filebeat/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt"]

filebeat.yml configures both the forwarder’s inputs and its output location using Logstash to Logz.io.

6. Push to Platform.sh

Commit the changes and push to the Platform.sh remote:

$ git push platform <environment name>

Filebeat will be installed during the build hook and configured to Logz.io during the deploy hook.

Each time Filebeat is run at the end of each deployment, it will detect changes to the log files within /var/log and ship those differences to Logz.io.

7. Define a project variable

To ensure that the installation in the build hook does not run on every build, add a project environment variable to the project that will be visible during the build process that will keep this from happening.

$ platform variable:create --level project --name LOGZ_CONFIG --value 'true'

This step can be automated by installing the Platform.sh CLI into the web container itself, although it will force a redeploy of the application when the variable is added.

Conclusion

Platform.sh is set up to provide a read-only file system for application containers so that consistent builds and deployments are ensured. Utilizing writable mounts however provides a simple way of setting up Filebeat to ship Platform.sh logs to Logz.io.

Note: if at any time you would like to remove Splunk from your application configuration, make sure to manually remove the associated files from each mount. Simply removing the mount will not eradicate the files on it and they will exist on disk until manually removed. Remember to remove the project variable created as well if keeping the build hook described above.