Rate limit connections to your application using Varnish

How-to Guides
#1

Rate limit connections to your application using Varnish

Problem

Bots, bad users, bad actors can often use up PHP workers and CPU time by constantly making bad queries to your application.

This causes 502/503 errors for the end users as documented in https://docs.platform.sh/development/troubleshoot.html#http-responses-502-bad-gateway-or-503-service-unavailable

We can use some automation to block these IPs but it would be great if we could do it automatically.

https://community.platform.sh/t/diagnosing-and-resolving-issues-with-excessive-bot-access/792

Solution

Lets add Varnish to our environment, it can check the traffic coming into the application and forward it on if it is within the rate limits we set.

Follow the instructions below to implement Varnish on your environment.

In the source code of your project.

  • Edit the .platform/services.yaml file and add a Varnish service.
varnish:
    type: varnish:6.0
    relationships:
        main: 'app:http'
    configuration:
        vcl: !include
            type: string
            path: varnish.vcl
  • Edit the .platform/routes.yaml file and edit the routes to point to the Varnish service.

Instead of

https://{default}/:
    type: upstream
    upstream: app:http

adjust your routes to

https://{default}/:
    type: upstream
    upstream: varnish:http
  • Create a file in that directory .platform/varnish.vcl with the following contents
import vsthrottle;

sub vcl_recv {

        # Varnish will set client.identity for you based on client IP.

        if ( req.url !~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|ogg|ogm|opus|otf|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$" && vsthrottle.is_denied(req.http.X-Client-IP, 10, 15s, 123s)) {
                # Client has exceeded 10 reqs per 15s.
                # When this happens, block altogether for the next 123s.
                return (synth(429, "Too Many Requests"));
        }

        # Only allow a few POST/PUTs per client.
        if (req.method == "POST" || req.method == "PUT") {
                if (vsthrottle.is_denied("rw" + req.http.X-Client-IP, 3, 10s, 123s)) {
                        return (synth(429, "Too Many Requests"));
                }
        }
        set req.backend_hint = main.backend();
}
  • git add the new file and git commit the changes and push them to your environment

Finally

Once implemented, any single IP trying to make more than 30 requests in a 15-second window, will be blocked for 127 seconds, and any IP making more than 3 POST or PUT requests to the endpoint within a 10-second window will be banned for 61 seconds.

These values can be adjusted as you need, and the key which is used by the vsthrottle module on can also be adjusted.

https://docs.varnish-software.com/varnish-cache-plus/vmods/vsthrottle/

The Platform.sh router will provide the real client IP as req.http.X-Client-IP and not client.identity.

4 Likes