Ghetto-Net :: The dump
The bitching of two guys, that rather go and live in the south of France...
Friday, September 23, 2016
Varnish Security Firewall (VSF) heatmap with the Elastic Stack
I am using Varnish reverse-proxy in order to increase overall performance to my website, by caching objects in memory and serving them at amazingly fast; thus reducing requests to my backend web-servers.
Recently I discovered VSF (Varnish Security Firewall) which is basically a set of rules that written in VCL (Varnish Configuration Language) that you include in your own VCL. These rules will make a bunch of check (you can modify them, or add your own custom rules if needed) for a variety of things; such as file extensions, empty user-agent strings, cross site scripting and much more). If a request to your website, you can display an error and block the request.
With Varnish - I am also using varnishncsa to log all requests, response times and so on to a log file on my server. I use a filebeat (part of beats) from the good people at Elastic to ship in (near) real-time all entries, as they come, to logstash (and subsequently into elasticsearch).
This allows me to produce the nice dashboard pictured above with kibana. Since the dashboard in linked to the indexed log files, I am able to dynamically change the heatmap. I can make a rectangular selection on the map (geo coordinates) and the heatmap will show blocked requests for this only. As seen on the screenshot below
So how does it all work?
I assume you have a working Varnish, and a working logstash, elasticsearch and kibana environment running; if not, there plenty of tutorials available that can help you. You can probably also install VSF and filebeat.
I run FreeBSD on my systems, you might choose to run something else.
Start by including the VSF rules before vcl_recv() in your default VCL.
/usr/local/etc/varnish/default.vcl:
[...]
include "/usr/local/etc/varnish/security/vsf.vcl";
sub vcl_recv {
[...]
and reload your vcl:
# varnishadm vcl.load vsftest default.vcl && varnishadm vcl.use vsftest
if everything works, you should see:
VCL compiled.
VCL 'vsftest' now active
Next, make sure that the rule description is being logged in varnishncsa. On FreeBSD, the varnishncsa format is defined in /etc/rc.conf - so you need to add this the right place on your system:
/etc/rc.conf:
varnishd_enable="YES"
varnishd_config="/usr/local/etc/varnish/default.vcl"
varnishd_storage="malloc,52G"
varnishd_admin=":81"
varnishncsa_enable="YES"
varnishncsa_pidfile="/var/run/varnishncsa.pid"
varnishncsa_file="/var/log/varnishncsa.log"
varnishncsa_logformat="%{X-Forwarded-For}i %u %t %m '%{Host}i' '%U' '%q' %s %b '%{Referer}i' '%{User-agent}i' %{Varnish:time_firstbyte}x %{Varnish:handling}x '%{X-VSF-RuleName}i'"
You can find an overview of the different formatters for the logformat here.
The one we are interested in right now is "%{X-VSF-RuleName}i" - which is set by VSF whenever a request is caught by a rule.
Here's an example:
123.123.255.248 - [23/Sep/2016:00:26:36 +0200] PROPFIND '5.5.5.5' '/webdav/' '' 403 279 '-' 'WEBDAV Client' 0.000060 synth '-' 'Method Not Allowed'
We need to ship this to logstash. As mentioned earlier, I use filebeat:
filebeat.yml
filebeat:
prospectors:
-
paths:
- /var/log/varnishncsa.log
input_type: log
document_type: varnish
output.redis:
hosts: ["172.16.0.1"]
port: 6379
password: "xxxxmysecurestringxxx"
index: "filebeat"
db: 0
timeout: 5
I ship all my entries to a redis instance (broker) that my different logstash instances will poll continuously.
So far so good. We now have to configure logstash to match and manipulate the log entries before inserting them into elasticsearch
Here is the relevant section from my configuration.
/usr/local/etc/logstash/logstash.conf
if [type] == "varnish" {
grok {
patterns_dir => "/usr/local/etc/logstash/patterns"
match => [
"message", "%{VARNISH}"
]
named_captures_only => true
}
geoip {
source => "ip1"
target => "geoip"
database => "/usr/local/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
convert => [ "bytes", "integer" ]
convert => [ "berespms", "integer" ]
}
}
The geoip section, takes the ip address in the field "ip1" and makes a lookup in GeoLiteCity.dat - that will return geo coordinates into the field geoip.coordinates.
Here is my grok pattern for the Varnish log:
VARNISH %{IP:ip1} - \[%{HTTPDATE:timestamp}\] %{WORD:method} '%{NOTSPACE:host}' '%{NOTSPACE:path}' '(?:%{URIPARAM:param}|)' %{NUMBER:http_status} (?:%{NUMBER:bytes}|-') '(?:%{NOTSPACE:referrer}|-)' %{QS:agent} %{BASE10NUM:berespms} %{WORD:cache_handling} (?:%{QS:vsfvuln}|-)
Once this is in place, and you have restarted / reloaded all the different new configurations, you should be able to see vsfvuln and the GeoIP information in the indexed data:
I hope this was helpful, feel free to ask questions in the comment section below.
Thanks to all the people at Varnish Cache for providing this incredibly versatile and powerful software.
Friday, August 26, 2016
Never happy..
So these days it is scorching.. I think we hit an all-year high in Paris yesterday with temperatures up to 37c. It was hot as balls; I love it.
All year the weather has been abysmal. Cold, rainy, grey, windy - basically like a never-ending autumn. Everyone, and I mean everyone, have been bitching and complaining all year about the shitty weather.
Now, we finally get a spot of sun and warmth.. and how do people react? You guessed it - they bitch and complain.. "oh! it's too hot - I can't sleep .." etc etc ..
People need to shut the fuck up, and enjoy the few days of summer we will have this year.
We are still alive...
Hi,
It has been a while. We are still alive, and still struggling to come up with a plan to move to the south of France. I know we have been quiet for a while, but hoping the change this with this blog post.
My good compadre Dimbo and I just spent a couple of days around Nîmes - and I can only reiterate what I have already stated a plethora of times: "What the hell are we still doing in Paris... ?"
Warm weather, fresh pool, great food and, of course, lots of aperos. What is not to like?
As per tradition, we naturally had what I like to call "Pizza Galore Festival '16" - the pizza oven was cracking at a comfortable 450c - and those pies hit the spot!
While exposed to extreme heat, it is of course imperative to keep hydrated - that is why they have these adult-size rosé bottles :
It has been a while. We are still alive, and still struggling to come up with a plan to move to the south of France. I know we have been quiet for a while, but hoping the change this with this blog post.
My good compadre Dimbo and I just spent a couple of days around Nîmes - and I can only reiterate what I have already stated a plethora of times: "What the hell are we still doing in Paris... ?"
Warm weather, fresh pool, great food and, of course, lots of aperos. What is not to like?
Waking up in the morning to this... |
As per tradition, we naturally had what I like to call "Pizza Galore Festival '16" - the pizza oven was cracking at a comfortable 450c - and those pies hit the spot!
Burn, baby, burn! |
Enfin, je dis ça - je dis rien. |
Have a nice weekend everyone..
Friday, April 20, 2012
Thursday, April 19, 2012
ttytter, perl and fortune
Ttytter is a fully fledged Twitter client written in Perl - it is operated in CLI only.. I love it.
I am not an avid Twitter user, though I try to be - so in order to increase my post statistics, I decided to write a little script that would post something clever everyday.
Enter the fortune game.. pretty much resident on most unix flavors.
I've written a small shellscript that looks like this:
It basically runs a while-loop -that finds a fortune with less than 140 characters (Twitter message limit), and then posts it to my Twitter account.
If the fortune found is more than 140 characters, it simple fetches another one.
I've added the script to run in cron(1) once a day..
Here's the result from today!
Mazel, mazel...
I am not an avid Twitter user, though I try to be - so in order to increase my post statistics, I decided to write a little script that would post something clever everyday.
Enter the fortune game.. pretty much resident on most unix flavors.
I've written a small shellscript that looks like this:
#!/bin/sh while [ 1 ] do FOO=`/usr/games/fortune -a -s` if [ ${#FOO} -lt 140 ]; then /usr/local/bin/ttytter -curl=/usr/local/bin/curl -status="$FOO" exit else fi done
It basically runs a while-loop -that finds a fortune with less than 140 characters (Twitter message limit), and then posts it to my Twitter account.
If the fortune found is more than 140 characters, it simple fetches another one.
I've added the script to run in cron(1) once a day..
Here's the result from today!
Mazel, mazel...
Wednesday, April 18, 2012
Vistaprint (those S-O-Bs)
So the other day, I purchased a new printer from Fnac.. I bought it online (like most of my purchases nowadays) - and it was shipped to a delivery point near my apartment.. Great.. All is good.
In the box from Fnac, I found a voucher for Vistaprint - offering free business cards, and a company stamp. Since I was in the need of both, I decided to use the voucher..
So going through their "wizard of hell" with more sale offers on everything from t-shirts to websites - I finally pass my order - something that was initially "free" ended up costing me around 30EUR.. Oh well.. I was happy about my order nonetheless.
I got a mail saying that the goods would be delivered the 18/4/12 at the latest. Yeah right.
Went down to the delivery point place today, and of course the package hadn't arrived. So I call up Vistaprint customer service - and after 10 minutes with IVR hell - I finally get connected to some nitwit..
The nitwit can't find any trace of the delivery, and offers to ship a new order (so another waiting 7 days) - I ask him how he intends on making sure that this one will be delivered - to which, of course, he has no answer.
After a few more minutes of me complaining, and he finally offers to ship the order express (3 days) but directly to my apartment - and not the pickup point..
Anyways... I am convinced the whole thing is a scam - and Vistaprint can go SABO.. (Fnac as well for that matter)
Grrrr
In the box from Fnac, I found a voucher for Vistaprint - offering free business cards, and a company stamp. Since I was in the need of both, I decided to use the voucher..
So going through their "wizard of hell" with more sale offers on everything from t-shirts to websites - I finally pass my order - something that was initially "free" ended up costing me around 30EUR.. Oh well.. I was happy about my order nonetheless.
I got a mail saying that the goods would be delivered the 18/4/12 at the latest. Yeah right.
Went down to the delivery point place today, and of course the package hadn't arrived. So I call up Vistaprint customer service - and after 10 minutes with IVR hell - I finally get connected to some nitwit..
The nitwit can't find any trace of the delivery, and offers to ship a new order (so another waiting 7 days) - I ask him how he intends on making sure that this one will be delivered - to which, of course, he has no answer.
After a few more minutes of me complaining, and he finally offers to ship the order express (3 days) but directly to my apartment - and not the pickup point..
Anyways... I am convinced the whole thing is a scam - and Vistaprint can go SABO.. (Fnac as well for that matter)
Grrrr
A Kuna Matata
Oyé Oyé! We is back!
Only one thing to say.
PLEEEASE let there not be a Socialist president in France - oh PLEASE!
The last thing this country needs is more socialism.
On that fine note - I'd gladly go home, have a spot of rosé, smoke some ribs, but it's fucking pissing down.
oh well.
Only one thing to say.
PLEEEASE let there not be a Socialist president in France - oh PLEASE!
The last thing this country needs is more socialism.
On that fine note - I'd gladly go home, have a spot of rosé, smoke some ribs, but it's fucking pissing down.
oh well.
And we're back...
After a longer hiatus from our part, we are back with the bitchin' and whining.. The original "Ghetto-Net" has since then gone bust, and Dimitri and myself are now working different places..
The weather still sucks, we still love to cook (or more appropriately; eat) - and we'd still rather go live in the south of France..
It's election time again in France, and our buddy Mr. Sarkozy is running for the second term - or at least trying to.. It's not looking good..
Well - that's it for now. Will try to write some more soon.
It's apero time.
Wednesday, January 14, 2009
Fried chicken breast with a champignon de Paris veloute:: DONE!
Poach the chicken.
Rest the chicken, while you sautee mushrooms, shallots and garlic
Fry the chicken in olive oil - and baste in butter at the end
Final result! It was gooooood!
Rest the chicken, while you sautee mushrooms, shallots and garlic
Fry the chicken in olive oil - and baste in butter at the end
Final result! It was gooooood!
Chicken:
-----------
- chicken
- carrots
- leeks
- celery
- salt & pepper
- olive oil
- butter
- sprig of thyme
- bay leaves
- garlic
- fresh black pepper corns
Veloute:
------------
- champignon
- shallots
- garlic
- sprig of thyme
- dry white wine
- double cream
- chicken stock
-----------
- chicken
- carrots
- leeks
- celery
- salt & pepper
- olive oil
- butter
- sprig of thyme
- bay leaves
- garlic
- fresh black pepper corns
Veloute:
------------
- champignon
- shallots
- garlic
- sprig of thyme
- dry white wine
- double cream
- chicken stock
Sunday, January 04, 2009
Homemade pasta with shrimps fried in Pastis and a light red pepper sauce:: DONE!
We are back!
After a long hiatus - I've decided to start blogging again. Mainly about food and other good stuff - so hang tight.
Also - I've invited a new author, Peter, that will write some interesting (hopefully) posts.
Happy New Year everybody!!
Also - I've invited a new author, Peter, that will write some interesting (hopefully) posts.
Happy New Year everybody!!
Subscribe to:
Posts (Atom)