Friday, September 23, 2016

Varnish Security Firewall (VSF) heatmap with the Elastic Stack


I am using Varnish reverse-proxy in order to increase overall performance to my website, by caching objects in memory and serving them at amazingly fast; thus reducing requests to my backend web-servers.

Recently I discovered VSF (Varnish Security Firewall) which is basically a set of rules that written in VCL  (Varnish Configuration Language) that you include in your own VCL. These rules will make a bunch of check (you can modify them, or add your own custom rules if needed) for a variety of things; such as file extensions, empty user-agent strings, cross site scripting and much more). If a request to your website, you can display an error and block the request.

With Varnish - I am also using varnishncsa to log all requests, response times and so on to a log file on my server. I use a filebeat (part of beats) from the good people at Elastic to ship in (near) real-time all entries, as they come, to logstash (and subsequently into elasticsearch).

This allows me to produce the nice dashboard pictured above with kibana. Since the dashboard in linked to the indexed log files, I am able to dynamically change the heatmap. I can make a rectangular selection on the map (geo coordinates) and the heatmap will show blocked requests for this only. As seen on the screenshot below



So how does it all work?

I assume you have a working Varnish, and a working logstash, elasticsearch and kibana environment running; if not, there plenty of tutorials available that can help you. You can probably also install VSF  and filebeat.

I run FreeBSD on my systems, you might choose to run something else.

Start by including the VSF rules before vcl_recv() in your default VCL.

/usr/local/etc/varnish/default.vcl:

[...]
include "/usr/local/etc/varnish/security/vsf.vcl";
sub vcl_recv  {
[...]

and reload your vcl:
# varnishadm vcl.load vsftest default.vcl && varnishadm vcl.use vsftest

if everything works, you should see:
VCL compiled.

VCL 'vsftest' now active


Next, make sure that the rule description is being logged in varnishncsa. On FreeBSD, the varnishncsa format is defined in /etc/rc.conf - so you need to add this the right place on your system:

/etc/rc.conf:
varnishd_enable="YES"
varnishd_config="/usr/local/etc/varnish/default.vcl"
varnishd_storage="malloc,52G"
varnishd_admin=":81"
varnishncsa_enable="YES"
varnishncsa_pidfile="/var/run/varnishncsa.pid"
varnishncsa_file="/var/log/varnishncsa.log"
varnishncsa_logformat="%{X-Forwarded-For}i %u %t %m '%{Host}i' '%U' '%q' %s %b '%{Referer}i' '%{User-agent}i' %{Varnish:time_firstbyte}x %{Varnish:handling}x '%{X-VSF-RuleName}i'"


You can find an overview of  the different formatters for the logformat here.
The one we are interested in right now is "%{X-VSF-RuleName}i" - which is set by VSF whenever a request is caught by a rule.
Here's an example:

123.123.255.248 - [23/Sep/2016:00:26:36 +0200] PROPFIND '5.5.5.5' '/webdav/' '' 403 279 '-' 'WEBDAV Client' 0.000060 synth '-' 'Method Not Allowed'

We need to ship this to logstash. As mentioned earlier, I use filebeat:

filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /var/log/varnishncsa.log
      input_type: log
      document_type: varnish

output.redis:
    hosts: ["172.16.0.1"]
    port: 6379
    password: "xxxxmysecurestringxxx"
    index: "filebeat"
    db: 0
    timeout: 5



I ship all my entries to a redis instance (broker) that my different logstash instances will poll continuously.

So far so good. We now have to configure logstash to match and manipulate the log entries before inserting them into elasticsearch

Here is the relevant section from my configuration.

/usr/local/etc/logstash/logstash.conf
        if [type] == "varnish" {
                grok {
                        patterns_dir => "/usr/local/etc/logstash/patterns"
                        match => [
                                "message", "%{VARNISH}"
                        ]
                        named_captures_only => true
                }
                geoip {
                        source => "ip1"
                        target => "geoip"
                        database => "/usr/local/etc/logstash/GeoLiteCity.dat"
                        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
                }
                mutate {
                        convert => [ "[geoip][coordinates]", "float"]
                        convert => [ "bytes", "integer" ]
                        convert => [ "berespms", "integer" ]
                }
        }

The geoip section, takes the ip address in the field "ip1" and makes a lookup in GeoLiteCity.dat - that will return geo coordinates into the field geoip.coordinates.

Here is my grok pattern for the Varnish log:

VARNISH %{IP:ip1} - \[%{HTTPDATE:timestamp}\] %{WORD:method} '%{NOTSPACE:host}' '%{NOTSPACE:path}' '(?:%{URIPARAM:param}|)' %{NUMBER:http_status} (?:%{NUMBER:bytes}|-') '(?:%{NOTSPACE:referrer}|-)' %{QS:agent} %{BASE10NUM:berespms} %{WORD:cache_handling} (?:%{QS:vsfvuln}|-)


Once this is in place, and you have restarted / reloaded all the different new configurations, you should be able to see vsfvuln and the GeoIP information in the indexed data:



I hope this was helpful, feel free to ask questions in the comment section below.

Thanks to all the people at Varnish Cache for providing this incredibly versatile and powerful software.





Friday, August 26, 2016

Fancy a dip?

Looks rough, right? 

Never happy..


So these days it is scorching..  I think we hit an all-year high in Paris yesterday with temperatures up to 37c. It was hot as balls; I love it.

All year the weather has been abysmal.  Cold, rainy, grey, windy - basically like a never-ending autumn. Everyone, and I mean everyone, have been bitching and complaining all year about the shitty weather.

Now, we finally get a spot of sun and warmth..  and how do people react? You guessed it - they bitch and complain.. "oh! it's too hot - I can't sleep .." etc etc ..  

People need to shut the fuck up, and enjoy the few days of summer we will have this year. 

We are still alive...

Hi,

It has been a while. We are still alive, and still struggling to come up with a plan to move to the south of France. I know we have been quiet for a while, but hoping the change this with this blog post. 

My good compadre Dimbo and I just spent a couple of days around Nîmes - and I can only reiterate what I have already stated a plethora of times: "What the hell are we still doing in Paris... ?"

Warm weather, fresh pool, great food and, of course, lots of aperos. What is not to like?


Waking up in the morning to this... 

As per tradition, we naturally had what I like to call "Pizza Galore Festival '16"  - the pizza oven was cracking at a comfortable 450c - and those pies hit the spot! 


Burn, baby, burn! 

While exposed to extreme heat, it is of course imperative to keep hydrated - that is why they have these adult-size rosé bottles : 

Enfin, je dis ça - je dis rien. 

Have a nice weekend everyone..