How to forward Platform.sh logs to a Splunk Indexer

Goal

To forward Platform.sh logs into a Splunk indexer.

Assumptions

  • An active Platform.sh project
  • Platform.sh CLI tool installed locally
  • SSH key configured with the project account
  • A Splunk instance installed on the indexing server that is configured to receive logs on a specified port (default 9997) and a configured admin login.

Problems

Forwarding Platform.sh to a third-party indexer such as Splunk is useful, especially when integrating log monitoring across other projects, including those that are not hosted on Platform.sh. Splunk creates configuration, auth and forwarding files when shipping logs, which must be taken into account on Platform.sh’s read-only file system.

Splunk log forwarding is, in its simplest configuration, dependent on configuring two components: a Receiver/Indexer and a Forwarder. The Forwarder will be installed on a writable mount on a Platform.sh project and is responsible for collecting and shipping the desired log files, whereas the Indexer receives those files and provides a dashboard for indexing, searching, and analyzing them. Splunk actions can be executed either through that dashboard or through its CLI.

The following How-to is written to ship logs from a Python application, but the steps do not require that.

More information regarding accessing Platform.sh logs can be found in the public documentation.

Steps (Splunk Indexer)

This How-to is based on installing Splunk Enterprise on a local machine, but an Indexer can be created using another Splunk product that is set-up remotely. Consult the Splunk documentation to determine the correct Indexer installation.

A Splunk receiver has to be enabled to receive forwarded logs on a specified port. By default, the dedicated port for Splunk indexer’s is 9997, and it can be enabled to listen on that port either through the dashboard or using the CLI. Consult the Splunk documentation for full details on how to do that: Enable a receiver.

Steps (Splunk Universal Forwarder)

1. Configure routes and services

The project locally will ultimately have the following structure:

.
├── .platform
│   ├── routes.yaml
│   └── services.yaml
├── .platform.app.yaml
├── Pipfile
├── Pipfile.lock
├── README.md
├── config
│   └── splunk
│       ├── scripts
│       │   ├── config.sh
│       │   ├── install.sh
│       │   └── uninstall.sh
│       └── seeds
│           ├── inputs.conf
│           ├── outputs.conf
│           ├── splunk-launch.conf
│           └── user.conf
└── server.py

The project is a simple “Hello, world!” application using Flask. Routes must be configured in the .platform/routes.yaml:

# .platform/routes.yaml

https://{default}/:
  type: upstream
  upstream: "app:http"

as well as its services in .platform/services.yaml, which is defined here with a single MySQL/MariaDB service:

# .platform/services.yaml

mysql:
  type: "mysql:10.2"
  disk: 1024

2. Set up the Platform.sh Application configuration

Modify .platform.app.yaml to define writable mounts for the Splunk Universal Forwarder:

# .platform.app.yaml

mounts:
    'splunk':
        source: local
        source_path: splunk
    '/.splunk':
        source: local
        source_path: splauths

Mounts are utilized in this configuration because after the Splunk Universal Forwarder is installed, Splunk will still attempt to write files to the project so that logs can be forwarded to the indexer. Since Platform.sh is designed to provide a read-only file system, mounts can be used for this purpose.

Splunk writes modified log files and authorization files when a connection is made to an indexer, and those files will be written in the directories splunk and .splunk respectively.

Modify the build hook:

# .platform.app.yaml

hooks:
    build: |
        if [ ! -z $SPLUNK_CONFIG ]; then
            ./config/splunk/scripts/install.sh
        fi
        pipenv install --system --deploy

This section runs an install script (defined below) that installs the Forwarder if it does not detect a project variable that we will define after the installation has completed.

Modify the deploy hook:

    deploy: |
        if [ ! "$(ls -A splunk)" ]; then
             ./config/splunk/scripts/config.sh
        fi
        ./splunk/splunkforwarder/bin/splunk restart

The deploy hook checks to see if the splunk directory is empty, and runs a configuration script on the Forwarder if it is. The final command actually restarts the Forwarder on every deploy and is used to ship the logs to the Indexer.

3. Installing the Splunk Universal Forwarder

In config/splunk/scripts/ modify a script install.sh:

# config/splunk/scripts/install.sh

#!/usr/bin/env bash

TEMP_SPLUNK_HOME=config/splunk/build

# Install Splunk Universal Forwarder
[ ! -d $TEMP_SPLUNK_HOME ] && mkdir -p $TEMP_SPLUNK_HOME
cd $TEMP_SPLUNK_HOME
wget -O splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.5.1&product=universalforwarder&filename=splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz&wget=true'
tar xvzf splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz
rm splunkforwarder-7.2.5.1-962d9a8e1586-Linux-x86_64.tgz

The script will define a temporary location within the config/splunk subdirectory to install the Splunk forwarder. During the build phase mounts are not yet available, so this becomes a necessary step before the mounts are fully defined.

Make sure to replace the installation link with the most recent version listed on Splunk’s Download page.

4. Configuring the Splunk Universal Forwarder

Write a config.sh script within config/splunk/scripts that will be used to control the initial configuration:

# config/splunk/scripts/config.sh

#!/usr/bin/env bash

cd $PLATFORM_APP_DIR

TEMP_SPLUNK_HOME=config/splunk/build/*
SPLUNK_HOME=$PLATFORM_APP_DIR/splunk/splunkforwarder

# Copy temp build to writable storage
cp -v -r $TEMP_SPLUNK_HOME splunk

# Migrate used-seed.conf to the forwarder
cp -v config/splunk/seeds/user.conf $SPLUNK_HOME/etc/system/local/user-seed.conf

# Start Splunk for the first time, accepting license
./splunk/splunkforwarder/bin/splunk start --accept-license

# Update outputs.conf with receiver address seed
cp -v $PLATFORM_APP_DIR/config/splunk/seeds/outputs.conf $PLATFORM_APP_DIR/splunk/splunkforwarder/etc/system/local/outputs.conf

# Update inputs.conf with monitor inputs seed
cp -v $PLATFORM_APP_DIR/config/splunk/seeds/inputs.conf $PLATFORM_APP_DIR/splunk/splunkforwarder/etc/system/local/inputs.conf

This script is run in the deploy hook when the mounts defined in .platform.app.yaml are now available, so the forwarder can be moved from the temporary installation directory config/splunk/build/ to app/splunk.

After that, the installation is set-up with a collection of configuration seed files located in config/splunk/seeds that are also moved to the mounted installation.

The first user.conf is a user seed that sets up an admin user for the forwarder. There is already one set up upon install, but unless another admin password is set up, the CLI will not be accessible remotely.

# config/splunk/seeds/user.conf

[user_info]
USERNAME = admin
PASSWORD = testpass

Once the user seed is moved to the final mounted installation location, the admin credentials can be passed to the Splunk CLI to accept its license and start it for the first time.

Next, an outputs.conf seeds the outputs configuration for the forwarder, which identifies the IP address and listening port for the receiver.

# config/splunk/seeds/outputs.conf

[tcpout]
defaultGroup=default

[tcpout:default]
server=<receiver ip>:9997

[tcpout-server://<receiver ip>:9997]

Then an inputs.conf seeds the inputs for the forwarder. In this case, the entire var/logs Platform.sh log directory is added to the forwarder’s list of inputs.

# config/splunk/seeds/inputs.conf

[monitor://var/log/]
disabled = false

Like the admin user seed, outputs.conf and inputs.conf are moved to the final installation directory splunk.

5. Push to Platform.sh

Commit the changes and push to the Platform.sh remote:

$ git push platform <environment name>

The Forwarder will be installed during the build hook and configured to the indexer during the deploy hook. At the end of the deploy hook the Splunk forwarder will be restarted during each deployment. Each time it is restarted, it will detect changes to the log files within /var/log and ship those differences to the indexer.

6. Final Configuration

The installation is finished, and now that the application has successfully deployed and the indexer is listening on the configured port, logs will be sent there on every deployment.

To ensure that the forwarder installation in the build hook does not run on every build, add a project environment variable to the project that will be visible during the build process that will keep this from happening.

$ platform variable:create --level project --name SPLUNK_CONFIG --value 'true'

This step can be automated by installing the Platform.sh CLI into the web container itself, although it will force a redeploy of the application when the variable is added.

SSH into the environment to modify the admin credentials or add new users using the Splunk CLI.

Conclusion

Platform.sh is set up to provide a read-only file system for application containers so that consistent builds and deployments are ensured. Utilizing writable mounts however provides a simple way of setting up a Splunk Universal Forwarder to ship Platform.sh logs with an Indexer.

Note: if at any time you would like to remove Splunk from your application configuration, make sure to manually remove the associated files from each mount. Simply removing the mount will not eradicate the files on it and they will exist on disk until manually removed. Remember to remove the project variable created as well if keeping the build hook described above.