Our My first DDoS attack

Our My first DDoS attack
Velocity Europe 2011 – Berlin
Cosimo Streppone
Operations Lead
<video of Mr. Wolf going to Jimmy's house in Pulp Fiction>
this couldn't fit in the PDF... sorry.
http://www.youtube.com/watch?v=hsKv5d0sIlU
my.opera.com/Ao-Trang-Oi/blog/
nginx – secret sauces?
# Pavel's secret gzip tuning sauce
gzip on;
gzip_disable msie6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_comp_level 3;
gzip_types text/plain application/xml
application/x-javascript text/css;
nginx – secret sauces?
# Michael's secret file cache sauce
open_file_cache max=1000 inactive=20s;
open_file_cache_valid
30s;
open_file_cache_min_uses 2;
open_file_cache_errors
on;
nginx – antidos.conf
# More on https://calomel.org/nginx.html
client_header_timeout
5;
client_body_timeout
10;
ignore_invalid_headers
on;
send_timeout
10;
# To limit slowloris-like attacks
client_header_buffer_size
4k;
large_client_header_buffers
4 4k;
nginx – drop client connections
# Cut abusive established connections,
# forcing clients to reconnect
location ~ ^/Ao-Trang-Oi/blog/ {
return 444;
}
nginx – varnish caching
nginx
varnish
backends
iptraf
tcpdump of anomalous traffic
GET /Ao-Trang-Oi/blog/show.dml/14715682 HTTP/1.1
User-Agent: 1.{RND 10}.{RND 10}
Referrer: http://my.opera.com/Ao-Trang-Oi/
Cache-Control: no-cache
Cookie: __utma=218314117.745395330 […]
__utmz=218314117.1286774593. […]
utmcsr=google|utmccn= […]
utmctr=cach%20de%20hoc%20mon […]
<... random high speed junk follows ...>
tcpdump of anomalous traffic
GET /Ao-Trang-Oi/blog/?startidx=1295 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US;) Gecko/20030624 Netscape/7.1 (ax)
Accept: Accept=text/html,application/xhtml+xml,...
Accept-Language: Accept-Language=en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: Accept-Charset=ISO-8859-1,...
Referer: http://my.opera.com/Ao-Trang-Oi/blog/
Pragma: no-cache
Keep-Alive: 300
ua-cpu: x86
Connection: close
#nginx, 14th October 2010
cosimo: we're seeing a pretty "interesting" problem within our
nginx
fronts
cosimo: there's a few hosts sending a legitimate HTTP GET
request
cosimo: followed by a binary stream of random bytes that never
ends
cosimo: this is just 1 request going on and on
cosimo: is there some way to alter the nginx config to shut
down these
client connections?
cosimo: the client is sending something like:
cosimo: GET /blah HTTP/1.1
cosimo: Host: ...
cosimo: Etc: etc...
cosimo: and then random bullshit
vr: :)
vr: this is nkiller2
vr: haproxy can fight this
vr: you can set a timeout http-request
vr: don't know if nginx can do this
cosimo: cool
BLAH BLAH BLAH
BLAH BLAH BL
BLAH BLAH BLAH
OMGWTFBBQ!!!!11111
“this is nkiller2”
PHRACK#66
tcp window zero?
iptables -A -m u32
--u32 “6&0xFF=0x6 &&
4&0x1FFF=0 &&
0>>22&0x3C () 12&0xFFFF=0x0000”
-j ZERO_WINDOW_RECENT
u32 zero window filter
6 &
0xFF =
0x6
u32 zero window filter
4 &
0x1FFF =
0x0
u32 zero window filter
0>>22 &
0x3C ()
12 &
0xFFFF
=
0x0
u32 zero window filter
0>>22 &
0x3C ()
12 &
0xFFFF =
0x0
??
0>>22&0...@12&0xFFFF=0x0000
0>>22&0x3C@12&0xFFFF=0x0000
0>>22& [EMAIL PROTECTED]
&0xFFFF=0x0000
0>>22&0x3C@12&0xFFFF=0x0000
u32 zero window filter
0>>22 &
0x3C @
12 &
0xFFFF
=
0x0
iptables rules - logging
$ipt -N ZERO_WINDOW_RECENT
$ipt -A INPUT -m u32
--u32 "6&0xFF=0x6 &&
4&0x1FFF=0 &&
0>>22&0x3C@12&0xFFFF=0x0000"
-j ZERO_WINDOW_RECENT
$ipt -A ZERO_WINDOW_RECENT -m recent --set --name
ZERO_WINDOW
$ipt -A ZERO_WINDOW_RECENT -m recent --update
--seconds 60 --hitcount 20
--name ZERO_WINDOW -j LOG
--log-level info --log-prefix "ZeroWindow"
~18k distinct IPs
iptables rules - blocking
$ipt -N ZERO_WINDOW_RECENT
$ipt -A INPUT -m u32
--u32 "6&0xFF=0x6 &&
4&0x1FFF=0 &&
0>>22&0x3C@12&0xFFFF=0x0000"
-j ZERO_WINDOW_RECENT
$ipt -A ZERO_WINDOW_RECENT -m recent –set
--name ZERO_WINDOW
$ipt -A ZERO_WINDOW_RECENT -m recent –update
--seconds 60 --hitcount 20
--name ZERO_WINDOW -j DROP
shields-up.vcl
cacheable content
nginx
varnish
non-cacheable content
backends
shields-up.vcl
all HTTP content
varnish
nginx
HTTPS-only traffic
backends
nginx feels better
Pingdom response time
20s
10s
0s
End 29-Oct-2010
Packets/s seen by firewall
Start 13-Oct-2010
End 29-Oct-2010
¿Questions?
What can we, as Ops, do better?
●
Embrace failures and learn from them
●
Be fast (no panic/blame, think Mr. Wolf)
●
Coordinate (#ops, war rooms, ...)
●
Take notes
●
Learn TCP/IP
●
Know your tools
(tcpdump, tcpflow, strace, nc, iptraf, …)
my base_packages puppet module
class base_packages {
$packagelist = [
"ack-grep", "colordiff", "curl", "facter",
"git-core", "htop", "iftop", "iptraf",
"jed", "joe", "libwww-perl", "logrotate", "lsof",
"make", "mc", "oprofile", "psmisc", "rsync",
"screen", "svn", "sysstat", "tcpdump", "tcpflow",
"telnet", "unzip", "vim", "zip"
]
package { $packagelist:
ensure => "installed",
}
}
Thanks to...
●
ithilgore (sock-raw.org) for writing nkiller2
●
@vr in #nginx for pointing us at nkiller2
●
David Falloon for his great “untested” idea
●
marc.info for correctly handling “@” in ml
●
SANS Institute for the TCP/IP references
●
My team at Opera
Danke!