Friday, September 23, 2016

Varnish Security Firewall (VSF) heatmap with the Elastic Stack


I am using Varnish reverse-proxy in order to increase overall performance to my website, by caching objects in memory and serving them at amazingly fast; thus reducing requests to my backend web-servers.

Recently I discovered VSF (Varnish Security Firewall) which is basically a set of rules that written in VCL  (Varnish Configuration Language) that you include in your own VCL. These rules will make a bunch of check (you can modify them, or add your own custom rules if needed) for a variety of things; such as file extensions, empty user-agent strings, cross site scripting and much more). If a request to your website, you can display an error and block the request.

With Varnish - I am also using varnishncsa to log all requests, response times and so on to a log file on my server. I use a filebeat (part of beats) from the good people at Elastic to ship in (near) real-time all entries, as they come, to logstash (and subsequently into elasticsearch).

This allows me to produce the nice dashboard pictured above with kibana. Since the dashboard in linked to the indexed log files, I am able to dynamically change the heatmap. I can make a rectangular selection on the map (geo coordinates) and the heatmap will show blocked requests for this only. As seen on the screenshot below



So how does it all work?

I assume you have a working Varnish, and a working logstash, elasticsearch and kibana environment running; if not, there plenty of tutorials available that can help you. You can probably also install VSF  and filebeat.

I run FreeBSD on my systems, you might choose to run something else.

Start by including the VSF rules before vcl_recv() in your default VCL.

/usr/local/etc/varnish/default.vcl:

[...]
include "/usr/local/etc/varnish/security/vsf.vcl";
sub vcl_recv  {
[...]

and reload your vcl:
# varnishadm vcl.load vsftest default.vcl && varnishadm vcl.use vsftest

if everything works, you should see:
VCL compiled.

VCL 'vsftest' now active


Next, make sure that the rule description is being logged in varnishncsa. On FreeBSD, the varnishncsa format is defined in /etc/rc.conf - so you need to add this the right place on your system:

/etc/rc.conf:
varnishd_enable="YES"
varnishd_config="/usr/local/etc/varnish/default.vcl"
varnishd_storage="malloc,52G"
varnishd_admin=":81"
varnishncsa_enable="YES"
varnishncsa_pidfile="/var/run/varnishncsa.pid"
varnishncsa_file="/var/log/varnishncsa.log"
varnishncsa_logformat="%{X-Forwarded-For}i %u %t %m '%{Host}i' '%U' '%q' %s %b '%{Referer}i' '%{User-agent}i' %{Varnish:time_firstbyte}x %{Varnish:handling}x '%{X-VSF-RuleName}i'"


You can find an overview of  the different formatters for the logformat here.
The one we are interested in right now is "%{X-VSF-RuleName}i" - which is set by VSF whenever a request is caught by a rule.
Here's an example:

123.123.255.248 - [23/Sep/2016:00:26:36 +0200] PROPFIND '5.5.5.5' '/webdav/' '' 403 279 '-' 'WEBDAV Client' 0.000060 synth '-' 'Method Not Allowed'

We need to ship this to logstash. As mentioned earlier, I use filebeat:

filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /var/log/varnishncsa.log
      input_type: log
      document_type: varnish

output.redis:
    hosts: ["172.16.0.1"]
    port: 6379
    password: "xxxxmysecurestringxxx"
    index: "filebeat"
    db: 0
    timeout: 5



I ship all my entries to a redis instance (broker) that my different logstash instances will poll continuously.

So far so good. We now have to configure logstash to match and manipulate the log entries before inserting them into elasticsearch

Here is the relevant section from my configuration.

/usr/local/etc/logstash/logstash.conf
        if [type] == "varnish" {
                grok {
                        patterns_dir => "/usr/local/etc/logstash/patterns"
                        match => [
                                "message", "%{VARNISH}"
                        ]
                        named_captures_only => true
                }
                geoip {
                        source => "ip1"
                        target => "geoip"
                        database => "/usr/local/etc/logstash/GeoLiteCity.dat"
                        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
                }
                mutate {
                        convert => [ "[geoip][coordinates]", "float"]
                        convert => [ "bytes", "integer" ]
                        convert => [ "berespms", "integer" ]
                }
        }

The geoip section, takes the ip address in the field "ip1" and makes a lookup in GeoLiteCity.dat - that will return geo coordinates into the field geoip.coordinates.

Here is my grok pattern for the Varnish log:

VARNISH %{IP:ip1} - \[%{HTTPDATE:timestamp}\] %{WORD:method} '%{NOTSPACE:host}' '%{NOTSPACE:path}' '(?:%{URIPARAM:param}|)' %{NUMBER:http_status} (?:%{NUMBER:bytes}|-') '(?:%{NOTSPACE:referrer}|-)' %{QS:agent} %{BASE10NUM:berespms} %{WORD:cache_handling} (?:%{QS:vsfvuln}|-)


Once this is in place, and you have restarted / reloaded all the different new configurations, you should be able to see vsfvuln and the GeoIP information in the indexed data:



I hope this was helpful, feel free to ask questions in the comment section below.

Thanks to all the people at Varnish Cache for providing this incredibly versatile and powerful software.





Friday, August 26, 2016

Fancy a dip?

Looks rough, right? 

Never happy..


So these days it is scorching..  I think we hit an all-year high in Paris yesterday with temperatures up to 37c. It was hot as balls; I love it.

All year the weather has been abysmal.  Cold, rainy, grey, windy - basically like a never-ending autumn. Everyone, and I mean everyone, have been bitching and complaining all year about the shitty weather.

Now, we finally get a spot of sun and warmth..  and how do people react? You guessed it - they bitch and complain.. "oh! it's too hot - I can't sleep .." etc etc ..  

People need to shut the fuck up, and enjoy the few days of summer we will have this year. 

We are still alive...

Hi,

It has been a while. We are still alive, and still struggling to come up with a plan to move to the south of France. I know we have been quiet for a while, but hoping the change this with this blog post. 

My good compadre Dimbo and I just spent a couple of days around Nîmes - and I can only reiterate what I have already stated a plethora of times: "What the hell are we still doing in Paris... ?"

Warm weather, fresh pool, great food and, of course, lots of aperos. What is not to like?


Waking up in the morning to this... 

As per tradition, we naturally had what I like to call "Pizza Galore Festival '16"  - the pizza oven was cracking at a comfortable 450c - and those pies hit the spot! 


Burn, baby, burn! 

While exposed to extreme heat, it is of course imperative to keep hydrated - that is why they have these adult-size rosé bottles : 

Enfin, je dis ça - je dis rien. 

Have a nice weekend everyone.. 






Friday, April 20, 2012

Thursday, April 19, 2012

ttytter, perl and fortune

Ttytter is a fully fledged Twitter client written in Perl - it is operated in CLI only.. I love it.

I am not an avid Twitter user, though I try to be - so in order to increase my post statistics, I decided to write a little script that would post something clever everyday.
Enter the fortune game.. pretty much resident on most unix flavors.

I've written a small shellscript that looks like this:


#!/bin/sh
while [ 1 ]
do
FOO=`/usr/games/fortune -a -s`
if [ ${#FOO} -lt 140 ]; then
        /usr/local/bin/ttytter -curl=/usr/local/bin/curl -status="$FOO"
        exit
else
fi
done

It basically runs a while-loop -that finds a fortune with less than 140 characters (Twitter message limit), and then posts it to my Twitter account.
If the fortune found is more than 140 characters, it simple fetches another one.

I've added the script to run in cron(1) once a day..

Here's the result from today!

Mazel, mazel...


Wednesday, April 18, 2012

Vistaprint (those S-O-Bs)

So the other day, I purchased a new printer from Fnac.. I bought it online (like most of my purchases nowadays) - and it was shipped to a delivery point near my apartment..  Great.. All is good.

In the box from Fnac, I found a voucher for Vistaprint - offering free business cards, and a company stamp. Since I was in the need of both, I decided to use the voucher..

So going through their "wizard of hell" with more sale offers on everything from t-shirts to websites - I finally pass my order - something that was initially "free" ended up costing me around 30EUR..  Oh well..  I was happy about my order nonetheless.

I got a mail saying that the goods would be delivered the 18/4/12 at the latest. Yeah right.

Went down to the delivery point place today, and of course the package hadn't arrived. So I call up Vistaprint customer service - and after 10 minutes with IVR hell - I finally get connected to some nitwit..
The nitwit can't find any trace of the delivery, and offers to ship a new order (so another waiting 7 days) - I ask him how he intends on making sure that this one will be delivered - to which, of course, he has no answer.

After a few more minutes of me complaining, and he finally offers to ship the order express (3 days) but directly to my apartment - and not the pickup point..

Anyways... I am convinced the whole thing is a scam - and Vistaprint can go SABO..  (Fnac as well for that matter)

Grrrr

A Kuna Matata

Oyé Oyé! We is back!

Only one thing to say.

PLEEEASE let there not be a Socialist president in France - oh PLEASE!

The last thing this country needs is more socialism.

On that fine note - I'd gladly go home, have a spot of rosé, smoke some ribs, but it's fucking pissing down.

oh well.

And we're back...


After a longer hiatus from our part, we are back with the bitchin' and whining..   The original "Ghetto-Net" has since then gone bust, and Dimitri and myself are now working different places..

The weather still sucks, we still love to cook (or more appropriately; eat) - and we'd still rather go live in the south of France..

It's election time again in France, and our buddy Mr. Sarkozy is running for the second term - or at least trying to..  It's not looking good..

Well - that's it for now. Will try to write some more soon.

It's apero time.

Wednesday, January 14, 2009

Fried chicken breast with a champignon de Paris veloute:: DONE!

Poach the chicken.

Rest the chicken, while you sautee mushrooms, shallots and garlic

Fry the chicken in olive oil - and baste in butter at the end

Final result! It was gooooood!

Chicken:
-----------
- chicken
- carrots
- leeks
- celery
- salt & pepper
- olive oil
- butter
- sprig of thyme
- bay leaves
- garlic
- fresh black pepper corns


Veloute:
------------
- champignon
- shallots
- garlic
- sprig of thyme
- dry white wine
- double cream
- chicken stock

Sunday, January 04, 2009

Homemade pasta with shrimps fried in Pastis and a light red pepper sauce:: DONE!



- flour
- eggs
- shrimps / prawns
- pastis
- half a lemon
- Red bell pepper
- sour creme
- salt & pepper
- shallots
- garlic
- butter
- olive oil
- fresh basel

We are back!

After a long hiatus - I've decided to start blogging again. Mainly about food and other good stuff - so hang tight.

Also - I've invited a new author, Peter, that will write some interesting (hopefully) posts.

Happy New Year everybody!!