Set up XDebug on Dedicated (Pro) server clusters

How-to Guides
, , , ,
#1

Use your local development environment to do breakpoint debugging on your remote server(s)

This document is mostly about the nuts-and-bolts of establishing communications between your local development environment and the remote servers. It’s not a HOWTO use your IDE, it’s about how to diagnose network or service issues that are specific to the Platform hosting environment.

Assumptions

You will need:

  • A local copy of your project site, with all code files available
  • The platform CLI utility installed and configured to connect to the hosting API
  • An IDE such as PHPStorm with integrated XDebug support.
  • XDebug extensions installed and activated on your environments

Conceptual overview

Before getting started, it’s helpful to understand what happens at every step in an xdebugging process.
You normally don’t have to worry about some of these layers, but if any one of them goes wrong, nothing will work, so this may help to narrow in on what does and doesn’t work, when you are setting up for the first time.

Once all the server configurations are in place, and your IDE and the tunnels are set up to listen, what happens is this:

  1. You initiate an xdebug session from your browser, by requesting http://your.site?XDEBUG_SESSION_START=yourxdebugkey
  2. This request is routed to the outer (cache) layer of the hosting platform, where the presence of the XDEBUG_SESSION_START key should tell the system not to cache it.
  3. The request next passes to the load-balancing router, which forwards it to one of three web heads.
  4. It reaches the (nginx) webserver, which also recognises the XDEBUG_SESSION_START key, and so passes it to a dedicated xdebug-enabled php service for processing.
    This is to avoid the performance cost of running xdebug resident or on-demand inside a production PHP service.
  5. PHP script execution begins, and this starts sending stack trace messages to a socket on that server ( /run/platform/${PROJECTID}${SUFFIX}/xdebug.sock ).
  6. Your development environment has a tunnel open to each of the three web heads, which listens to that remote socket(s), and relays these messages to a local socket (port 9000) on your development machine.
  7. Your IDE is configured to listen to that port, so the incoming message triggers IDE debugging.
  8. Your IDE is configured to map the paths of php code files on the server to your local project folders, so it’s able to show you where in the local code, the remote process is currently stepping through.
  9. You can use your IDE to step through the execution stack, evaluate state, or run to breakpoints. This sends messages back over the tunnel to the running process, which then executes on the server.
  10. Eventually, execution and page build completes, and the response is sent back from the server, and your browser displays the page.
  11. For subsequent browser requests, an XDEBUG_SESSION cookie should have been set, and should provide the same effect as the XDEBUG_SESSION_START parameter for subsequent requests.

That is what is supposed to happen when all is well.

The routing, tunneling and the multi-head delegation of requests are the quirks specific to this hosting environment that you may need to know. Other tutorials on how to xdebug Magento or remote breakpoint debugging with PHPStorm should be referred to for deeper HOWTOs.

For reference, the config settings on the server that make this happen can be inspected on the server at

/etc/platform/$USER/php-fpm.xdebug.conf
/etc/platform/$USER/php.xdebug.ini

Activity logs are kept separate from the usual access logs, and are seen at

/var/log/platform/$USER/xdebug.access.log
/var/log/platform/$USER/php5-fpm-xdebug.log

Remember, you are sometimes talking to three different servers at once.

A Dedicated (Previously known as “Pro”, “Platform Enterprise” or “PE”) site with integrated deployment management has several web heads in the cluster.
This makes connecting to “the server” indeterminate, as any one of three may be the server for a request, so keep this in mind as we go forwards.

Getting started

Getting the server configured

For a “Dedicated” cluster, you need to have requested to have xdebug enabled via a support ticket to Platform.sh.

This may already be done for you, so please check before raising another ticket.
They will have set xdebug_enable: true on your project, and provided you with a unique xdebug_key to use to initiate the session.

Your xdebug_key is usually different between your production and your staging environments. Take note of which you are using.

If xdebug has already been enabled, a record of your key may have been helpfully left in a text file /mnt/shared/ for your reference.

If it’s not there, you can usually retrieve the xdebug key with a command like:

platform ssh --environment=staging 'grep -A 3  XDebug /etc/platform/$USER/nginx.conf'

     # XDebug Configuration
     ##
     map "$cookie_xdebug_session$arg_xdebug_session_start$arg_xdebug_session_stop" $php_backend {
       "Gd6QdPZaqnnSet32"                                      "unix:///run/platform/myproject_stg/php5-xdebug.sock";

= Your XDEBUGKEY=Gd6QdPZaqnnSet32

If you can’t find that key in your nginx.conf file either, then xdebug is probably not yet enabled for you, and you should raise the request.

Local environment

Assume any debugging should be happening on staging in the first case.

    PROJECTID=[projectID from ticket]
    BRANCH='staging'

Or if you already have the project cloned locally:

    PROJECTID=$(platform project:info id)
    HOSTNAME=$(platform environment:info edge_hostname)

XDebug talking to multiple webheads

To listen to multiple possible sources of incoming xdebug connections,
XDebug communication from all three heads need to be tunneled back to a port (9000) on our local environment.

Most public XDebug tutorials won’t allow for this multiple-head issue.

Open tunnels to the servers

Here’s a small script that sets up 3 simultaneous tunnels:


# Set these:
PROJECTID="xxxxxxxxxxxxx"
XDEBUGKEY="yyyyyyyyyyyyy"

# Optionally change these:
BRANCH="staging"; # or 'master'
PORT=9000
# These are the per-instance configurations you may need to change.

# Review these
URL=$( platform --project=${PROJECTID} --environment=${BRANCH} route:get 'https://{default}/' --property=url )
[ $BRANCH = 'staging' ] && SUFFIX='_stg' || SUFFIX=''
SOCKETPATH="/run/platform/${PROJECTID}${SUFFIX}/xdebug.sock"

WEBHEADS=( $(platform --project=${PROJECTID} --environment=${BRANCH} ssh --pipe --all) )
# These are settings to be used for setting up the tunnels

# Now open the tunnels
for WEBHEAD in $WEBHEADS ; do  
  echo "Will listen to xdebug on webhead ${WEBHEAD}"
  echo "Clearing old xdebug socket on instance."
  ssh ${WEBHEAD} "rm ${SOCKETPATH}"
  echo "Opening port forwarding to xdebug socket. Listening on port ${PORT} in the background."
  ssh -R ${SOCKETPATH}:localhost:${PORT} -N ${WEBHEAD} &
done

At this point, the remote server(s) should be sending you messages back down that tunnel.

You next need to connect a listener on your end to do something with that info.

To close the tunnels

kill $(jobs -p)

If using zsh, then “jobs -p” doesn’t work as expected. Instead, “kill %1 %2 %3 %4” may work.

In my experience, the ssh tunnels time out on their own in about 10 minutes if idle.

Timeout, server ssh.platform.cloud not responding.
[1]   Exit 255                ssh -R ${SOCKETPATH}:localhost:${PORT} -N ${WEBHEAD}

To start debugging.

Launch your browser with the key in the URL

open "${URL}?XDEBUG_SESSION_START=${XDEBUGKEY}"

Diagnostic: Ensure the request is hitting the server(s) correctly

When that key is used (and is correct) during a browser session, transactions will be getting logged in xdebug.access.log
on the server(s).

You can check activity in these files to ensure that the request is even getting through - that it’s not being cached, and that the parameter is being passed through the router without being stripped.

The following command will summarize the most recent xdebug.access.log from all webheads at once.

WEBHEADS=$(platform ssh --project=$PROJECTID --environment=staging --pipe --all)
echo $WEBHEADS | xargs -I% ssh % 'tail /var/log/platform/$USER/xdebug.access.log'
Reasons the XDebug session may not be getting logged in xdebug.access.log
  • You have three web heads, the load balancer may be sending your request to any one of them. You need to check all logs on all instances at once, not just one.
  • Your $XDEBUGKEY is wrong. Double-check against the value in /etc/platform/$USER/nginx.conf.
  • The URL you used was for a different branch than the one you are looking at
  • The outside cache layer is intercepting the request. You can check if it’s cached using wget -I
  • The inside router is not recognising or honoring the XDEBUGKEY
  • Nginx is performing a redirect or an access denied before the request can be routed to php. You’ll need your IP to be whitelisted if using HTTP access controls.
  • The xdebug php service (site-$USER-xdebug-php) is not running or responding. You should see it in the server process list (ps -axf).

Investigating

Things don’t always go smoothly, so here is a process of elimination to ensure that all things are set up as expected.

To verify that xdebug configs have been deployed on the host(s)

You can see the settings on the servers in /etc/platform/${PROJECTID}${SUFFIX}/php.xdebug.ini,
looking like

    xdebug.remote_host = unix:///run/platform/xxxxxxxxxxxxx_stg/xdebug.sock

To verify that xdebug is being loaded by PHP

You may be able to check out a phpinfo diagnostic from within your web application and confirm xdebug is running.
In Drupal this can be found underneath reports.

Don’t verify using php -m command

Note: Running basic diagnostics like commandline php -i on the server
may not show that xdebug is enabled as php can be configured to use different settings
for commandline than it does for web requests. The files php-fpm.conf, php-cli.conf, and php-fpm.xdebug.conf
(deployed into in the apps etc folder) are different in that way, and are each used depending upon context of the request.

To verify that the xdebug process is active

xdebug.access.log

The xdebug.conf tells us that the logs are at /var/log/platform/$USER/xdebug.access.log

Tailing that log should show some current activity when a browser session activates xdebug with the xdebug key.

When connected to an instance,

 tail -f /var/log/platform/${USER}/xdebug.access.log

Or called directly from your environment:

echo tail /var/log/platform/${PROJECTID}${SUFFIX}/xdebug.access.log | platform --project=${PROJECTID} --environment=${BRANCH} ssh

echo tail /var/log/platform/${PROJECTID}${SUFFIX}/error.log | platform --project=${PROJECTID} --environment=${BRANCH} ssh

Beware, the requests will actually happen on more than one instance, so you will only see some of the requests. Use tmux or similar tools to watch them all simultaneously.

Gotcha if using non-standard project names

Most dedicated hosting plans name your docroot after your project ID. Such as qazqaz234qaz or qazqaz234qaz_stg. Thus is the assumption used in the tunnel script that is configured to listen to the xdebug socket /run/platform/qazqaz234qaz/php5-xdebug.sock . However, if you are using a non-standard or legacy docroot name, some of these paths need to be updated. The socket may instead be something like /run/platform/shoppingsite/php5-xdebug.sock.

Diagnosing if the cache layer is interfering

If your outer cache layer (eg Fastly) is returning a previously cached version of the page, you will seem to be getting debuggable transactions at first, but later requests will fail to debug, sometimes unpredictably.
Use curl -I "${URL}?XDEBUG_SESSION_START=${XDEBUGKEY}" a couple of times in a row, and look for x-cache: HIT, HIT in the response.

If this is happening, you need to bypass the cache. This may be possible by disabling your cache, or configuring it to use the presence of the XDEBUG_SESSION_START argument to prevent caching - using the cache management configurations if you have access to them.

To verify that the socket is open for communication with the debugger client

After the client sets up a tunnel from their development side, the socket file mentioned in the configs
should be seen on each of the servers.

ls -la /run/platform/${USER}/xdebug.sock

You should check to see that the last-modified date on it is recent - that reflects the last time the socket was set up.

Beware timezones on the server are likely to be quite different to your own!
Compare the date against the server time!

echo $(( $(date +%s) - $(stat -c%Y "/run/platform/${USER}/xdebug.sock") )) seconds old.`

Note: Don’t get distracted by php5-xdebug.sock seen in the same directory, that’s an internal socket used for communication between php-fpm and nginx

To verify that messages are being sent down the pipe to the debugger client

Listen for a bit

When the tunnels are active, port 9000 on the developers machine is a window into the xdebug process.

If it seems that xdebug is not firing at all, on the server you may sniff what’s happening on port 9000 with something like

As a very basic test, running

netcat --listen --local-port 9000

on the developer machine, and then visiting the website with the XDEBUG_SESSION_START key in the URL (or the XDEBUG_CONFIG set in a CLI environment) should result in the first raw xdebug message being shown on your console.
It’s an XML packet in the DBGp protocol.

Doing this will stall the server, as you will not be able to respond with the expected sort of acknowledgements (just exit out)
but if you get any sort of initial packet sent to that port, it shows that something xdebug is happening, and you need to work on the tunnels.

It seems that unless you acknowledge that first message appropriately, no subsequent ones will be sent, and the server will hang there until you kill one end of the conversation, so there is a limit to what can be done without a real debugging tool, but this may at least prove that messages are getting through to the developers desktop.
Dephpugger is a quick CLI tool for this, but you probably want to just go straight to using a real IDE.

Using an IDE to listen to xdebug messages

With something like PHPStorm, you can just ‘start listening’ to port 9000 and when the first message arrives from the server, the wizard will ask you to match the incoming request (eg /app/${PROJECTID}_stg/index.php )
to a local file to begin breakpoint debugging. If you don’t have a local checkout of the project, well, you need to go get one to proceed now.

XDebug on cli

Note that xdebugging on the CLI does NOT log into the access log (not even the xdebug.access.log which is for web requests)
so looking for clues there will not help.

You can trigger xdebug behaviour on the CLI

  • using a custom php.ini,
  • by setting an environment variable export XDEBUG_CONFIG="remote_enable=1"
  • or by specifying everything up front in the commandline arguments

…though all methods ALSO need XDEBUG_CONFIG to at least be set to SOMETHING.

To test XDebug is working in a snippet

As a single command is the most straightforward for testing, XDebug can be triggered minimally with:

# On the host:
SOCKETPATH="unix:///run/platform/${USER}/xdebug.sock"
export XDEBUG_CONFIG="remote_enable=1 remote_host=$SOCKETPATH"
php \
  -dzend_extension=xdebug.so \
  index.php

If you have a listener open on port 9000 on your local dev, it’ll start getting messages.

I haven’t been able to find a way to get logs of these transactions, so it’s up to you to be listening correctly.

To use the php.xdebug.ini

To work as designed however, a php.xdebug.ini has been provided.
To use that, you should invoke php, source the special ini, and also must set XDEBUG_CONFIG to non-null in your session.

PHPXDEBUG=/etc/platform/${USER}/php.xdebug.ini
export XDEBUG_CONFIG=true
php -c $PHPXDEBUG index.php

… and stuff should be coming down the socket.

Interesting snippets:

PHPXDEBUG=/etc/platform/$USER/php.xdebug.ini

php -c $PHPXDEBUG -r ‘echo(ini_get(“xdebug.idekey”));’

Go and trace your project

The real fun begins after the tunnels are set up and the XDebug communications are happening. Other tutorials from your IDE will probably be more helpful than can be covered here.