How to interactively debug Python applications on Platform.sh

How-to Guides
, , ,
#1

Goal

Interactively debug python applications running on Platform.sh.

Assumptions

Problems

Effectively debugging web applications isn’t trivial, especially when an HTTP request goes through multiple layers until it reaches the web app, which is usually the case in cloud platforms.

Debugging in a local environment is easier, but the bug might only trigger when interacting with data present on production.

Note: The debugging procedure should only be performed on a non-production environment. So, if necessary clone the production environment, with the data too if need be, depending on the preconditions that trigger the bug you’re investigating.

The existence of bugs is sometimes manifested by an unhandled exception being raised and bubbling up to the top of your application, or in other cases, there are no exceptions, but the actions that a request handler is supposed to perform do not occur the way you’d expect.

Let’s explore the different set of debugging tools we can use and learn when to use what.

Steps (dropping in a REPL shell)

Sometimes identifying a bug is possible by just looking at the stack trace of the exception or getting the values of some variables at some stack in the backtrace.

This is exactly what the DebuggedApplication WSGI wrapper from the werkzeug library does. To use it:

1. Add the new werkzeug dependency.

$ pipenv install werkzeug

2. Wrap and expose the WSGI application

Add the script wsgi-debug.py to the project directory. It wraps our WSGI application with DebuggedApplication and exposes it.

import werkzeug.debug
        
# For the template, it is assumed wsgi.py is in project/myapp/,     
# edit to match your project accordingly.
    
import myapp.wsgi
    
application = werkzeug.debug.DebuggedApplication(
    myapp.wsgi.application,
    evalex=True,    
)

3. Setup the debug script

Add the script gunicorn-debug.sh that stops the “app” service in the environment and spawns a new gunicorn instance that’s pointed to the wrapped WSGI application.

#!/usr/bin/env sh
et -ex
                   
# Stop the application.       
sv stop app
                   
# When this script exits, start it again.       
trap "exit 0" INT       
trap "sv start app" EXIT
                   
# Run a gunicorn and point it towards the wrapped WSGI application.       
# It's important to run in sync mode, and a single worker, because werkzeug       
# DebuggedApplication doesn't like forking.       
# Set a high enough timeout that gives us time to debug.       
gunicorn \
    --worker-class sync \
    --workers 1 \
    --timeout 3600 \
    --bind unix:$SOCKET \
    wsgi-debug:application

And make it executable.

$ chmod +x gunicorn-debug

4. Allow exceptions to propagate

Tell Django to let exceptions bubble up, by setting these in your myapp/settings.py file.

DEBUG = False      
DEBUG_PROPAGATE_EXCEPTIONS = True

5. Commit and push these changes.

6. Run the script

After the deployment succeeds, ssh into the environment and run the gunicorn-debug script.

$ ./gunicorn-debug

Now issue the request that makes it raise the unhandled exception, and lo and behold, a beautiful debugging interface appears.

The stack trace of the exception can be seen when a line of code is hovered over, and a “shell” button appears at the right that will spawn a Python shell at that line where variables (or anything else can be run, for that matter) can be printed.

Note: A PIN code is required to enable the shell, which is printed in the console of the gunicorn-debug script. This is for security reasons, to prevent unauthorized access in case it is accidentally enabled on production.

Steps (Debug the application code step by step with pdb.)

The previous approach will help to understand why an exception is raised so that it can be fixed, but if the bug occurs at an intermediate stage before the exception is raised, this will not be enough. The werkzeug debugging interface only captures the latest state of the app and the values of each variable (i.e., it cannot go back in time).

To gain extra insight into what code path is entered when the request is handled, a pdb trace point can be set at some point in the code, and then interactively execute it line by line and inspect variables as you go. See the pdb official documentation for more information.

1. Add a pdb trace

While still running the application with our gunicorn-debug script, add a pdb trace (import pdb; pdb.set_trace()) at some point in the code, which can even be inside a conditional.

For example, to test this methodology, the following can be included in a Django application’s views.py:

if request.GET.get('pdb', False) == '1':
        import pdb; pdb.set_trace()

2. Use the PDB shell

Issue a request that makes it reach the pdb.set_trace() and then code execution will pause and the PDB shell will appear in the terminal where the gunicorn-debug script is running.

From there, pdb commands can be issued, for example, to execute code line by line, to print variables, or run any arbitrary code desired.

Conclusion

Debugging complex bugs that are only reproduced with the data that are present on production is non trivial, but thanks to development environments data cloning capabilities, and the dynamic nature of Python, we have great tools at our disposal to properly investigate these kind of bugs.