Preventing stuck deployments caused by long-running cron jobs

The default way of deploying to production on Platform.sh is to simply push master to the platform remote. However, if a cron job is running, the application and services will be stopped, and the deployment will not finish until the cron job is finished. When this happens, the application goes down. In order to prevent this from happening, you can use a custom deployment script that first makes sure that no Django management commands are running in the production environment.

In the example below, we are checking if any Django management commands are running. If so, we abort the deployment. I have tested the script using Python 3.7.5, but it should work on 3.6.0 and later. If you’re not running Python, you can use it as inspiration for porting it to your language of choice. If you do, please consider sharing it with the community, for example as a reply to this post.

As a caveat, note that if a cron job starts after the build has started, it might still block the deployment. The probability of this increases if your build takes a long time.

#!/usr/bin/env python
import sys
from os import getenv
from subprocess import CompletedProcess, run


def git(command: str) -> None:
    run(f"git {command}", shell=True)


def main() -> None:
    git("checkout master")
    git("pull origin")
    print("Getting running processes in order to prevent stuck deployment…")
    completed_process: CompletedProcess = run(
        f"ssh {getenv('PLATFORM_SH_SSH_DESTINATION')} 'ps auxf'",
        capture_output=True,
        shell=True,
    )
    output = str(completed_process.stdout)
    print("Making sure that process list was received…")
    if "/app/.global/bin/gunicorn" not in output:
        sys.exit("Gunicorn process not found. Are you able to connect using SSH?")
    print("Process list received. Checking for running Django management commands…")
    if "manage.py" in output:
        sys.exit("Running Django management command detected. Not safe to continue.")
    print(
        "No running commands detected. Pushing changes to the production environment…"
    )
    git("push platform")


if __name__ == "__main__":
    main()

To use the script:

  1. Create a Python file, e.g. deploy.py, in the root of your project and paste the script contents into it.
  2. Set the environment variable PLATFORM_SH_SSH_DESTINATION to the user@host destination of your production environment.
  3. Check if there are any modifications you need to make. For example, you may be running other kinds of jobs than just Django management commands, so you may need to add other strings in addition to manage.py
  4. Run ./deploy.py when you’re ready to deploy.
  5. Commit the script so you don’t lose it.