Advertising
Is your server exploited (Part 2)

12.02.2011 - 14:18:32

Hi Rod,

AFAIK, CSF uses iptables

 Quote
The idea with csf, as with most iptables firewall configurations, is to block
everything and then allow through only those connections that you want. This is
done in iptables by DROPPING all connections in and out of the server on all
protocols.

Source: http://www.configserver.com/free/csf/readme.txt

Maybe you`re using another csf, but this is the one i found.
Please let me know if this is your version.

please also try, if iptables is installed in your server correctly
f.e. iptables -L
to show current rules.

Regards
Volker
aka
Schnoog

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


12.02.2011 - 20:25:23

yea its the right CSF version i use.
Sorry for ask you this because manage linux (fedora 13 in my case) is not easy for me becuase i only use it to have the ET clan servers/ webhost and voice servers (mumble) running and everything i have done in it i need to start by the beggining and need to search and learn a lot. But by this time im afraid to do something wrong and block all the server.

Regrads,

Rod.

  zer0o0
Private First Class

User Pic

Posts: 37
Registred: 26.01.2011

   

0 approved this posting.


12.02.2011 - 21:40:04

Could you provide me more information about your system:

Please give me the output of the following commands:

 Code
1:
2:
3:
 iptables --version
iptables -L
which tcpdump


To response the output, please enter it between [ code ] tages.

I`ll try to help you if I`m able to do so.

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


12.02.2011 - 23:30:50

iptables v1.4.7

Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP tcp -- anywhere xxx.xxx.xxx.xxx tcp dpt:http STRING match "GET /w00tw00t.at.ISC.SANS." ALGO name bm TO 70
DROP udp -- 67.212.93.253 anywhere udp dpt:27960
DROP udp -- 67.212.93.253 anywhere udp dpt:27961

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

/usr/sbin/tcpdump

Cant find how to put between codes because all is in deutch.

Regards,
Rod.


OBS: I have disabled the csf due to problems.


last changed by zer0o0 am 12.02.2011 - 23:32:50

  zer0o0
Private First Class

User Pic

Posts: 37
Registred: 26.01.2011

   

0 approved this posting.


13.02.2011 - 00:28:31

Thean it should work.
But you should ensure, that:
IPTABLESBIN=/usr/sbin/iptables
and
TCPDUMPBIN=/usr/sbin/tcpdump
are set to the output of
which iptables
and
which tcpdump

Furthermore you need to have perl installed, since it uses perl to cut away the port from tcpdump-output.

But I`ll install a server similar to your ones to figure out the problem. Maybe csf manages the iptables rules without exceptions. Maybe I`m able to link it to csf.

Regards
schnoog

Edit: I have to rewrite a part of the script, because csf isn`t able to block ports to specific ips.
Will check it out.


last changed by schnoog am 13.02.2011 - 00:37:54

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


13.02.2011 - 15:30:09

OK, its done:

You can download the script here: http://wolffiles.de/index.php?filebase&fid=4210

NOTE: Use this script only if you`re useing configserverfirewall (csf) !!!!

 Code
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:
108:
109:
110:
111:
112:
113:
114:
115:
116:
117:
118:
119:
120:
121:
122:
123:
124:
125:
126:
127:
128:
129:
130:
131:
132:
133:
134:
135:
136:
137:
138:
139:
140:
141:
142:
143:
144:
145:
146:
147:
148:
149:
150:
151:
152:
153:
154:
155:
156:
157:
158:
159:
160:
161:
162:
163:
164:
165:
166:
167:
168:
169:
170:
171:
172:
173:
174:
175:
176:
 
#!/bin/bash

##################################################################################
##################################################################################
#                                                                                #
#  Q3-Engine-GetStatus-Flood-Fixer - ConfigServerFirewall                        #
#  Version 1.0                                                                   #
#  !USE THIS ONLY IF YOU ALSO USE CSF (http://www.configserver.com/cp/csf.html)! #
#                                                                                #
# Versioncontrol:                                                                #
# 1.0 - rework to fit csf                                                        #
#                                                                                #
#                                                                                #
#                                                                                #
# Thanks to benny from Hirntot for the regexp                                    #
##################################################################################
########################       DESCRIPTION     ###################################
#                                                                                #
# This script try to get rid of the Q3-Engine getstatus-Abuse-Exploit            #
#                                                                                #
# About the exploit                                                              #
#                                                                                #
# This exploit use the getstatus udp query command with a spoofed sender         #
# adress. This 14 byte long command enganges the gameserver engine to            #
# respond the whole status of the gameserver ( > ~600 bytes on empty server)     #
# This amplification is visible if you use a tool like vnstat (vnstat -l)        #
# The outgoint traffic exceeds the incoming traffic with a faktor > 3            #
#                                                                                #
#                                                                                #
# About the script                                                               #
# This script capture the count of tcp packets set in config and search for      #
# answered getstatus queries. For any requesting IP adress the count of          #
# those packets is compared to the defind limit.                                 #
# If the limit is broken by an IP adress, the access for this IP will ne denied  #
# with iptables by adding a ConfigServerFirewall deny entry.                     #
#                                                                                #
# Configuration                                                                  #
#                                                                                #
# By default, the script don`t need any configuration                            #
# The default values                                                             #
# CSFBIN=/usr/sbin/csf                                                           #
# TCPDUMPBIN=/usr/sbin/tcpdump                                                   #
# CAPTURETIME=3                                                                  #
# MAXPERSECONDS=4                                                                #
#                                                                                #
#                                                                                #
#                                                                                #
# will enforces drops in the case that:                                          #
#                                                                                #
# Each IP which requests more than 2 getstatus respones per second,              #
# averaged over 3 seconds                                                        #
#                                                                                #
# Requirements                                                                   #
#                                                                                #
# This script use a few open source tools to offence against this attacks        #
# csf - ConfigServerFirewall http://www.configserver.com/cp/csf.html             #
# tcpdump - capture incoming packets: http://www.tcpdump.org/                    #
# iptables - the most used linux firewall (packet filter)                        #
# GNU-tools - grep, cat                                                          #
#                                                                                #
# THIS SCRIPT COMES WITH NO WARRANTY! USE IT AT YOUR OWN RISK ONLY!              #
#                                                                                #
##################################################################################
######################       CONFIGURATION      ##################################
##################################################################################

CAPTURETIME=3
MAXPERSECONDS=4
CSFBIN=/usr/sbin/csf
TCPDUMPBIN=/usr/sbin/tcpdump


##################################################################################
##################################################################################
##################################################################################
mygoto=`dirname $0`
cd $mygoto
captsec=$CAPTURETIME
mylimit=$MAXPERSECONDS
cntout=file_cntout
tmpout=file_tmpout
tmpout2=file_tmpout2
tmpout3=file_tmpout3
banlist=file_bans

#fix: 31.01.2011 21:45
touch $banlist
#

MYIFS=$IFS
IFS="
"


rm $cntout 2>/dev/null
rm $tmpout 2>/dev/null
rm $tmpout2 2>/dev/null
rm $tmpout3 2>/dev/null

touch $cntout
touch $tmpout
touch $tmpout2
touch $tmpout3

$TCPDUMPBIN -f -c 100000 -A >$tmpout 2>$cntout &
pid=$!
sleep $captsec
kill $pid
sleep 1

maxcnt=$((mylimit*captsec))
grep -B2 Respon $tmpout | grep UDP | awk '{print $5}' | cut -d '.' -f 1,2,3,4 > $tmpout3

allIPs=`grep -B2 Respon $tmpout | grep UDP | awk '{print $5}' | cut -d '.' -f 1,2,3,4 | sort -u`
for ip in $allIPs
do

IPCNT=`grep "$ip" $tmpout3 | wc -l`
#PORTS=`grep -B2 Respon $tmpout | grep UDP | cut -d "." -f 4 | cut -d " " -f 1 | sort -u`
PORTS=`grep -B2 Respon file_tmpout | grep UDP |awk '{print $3}' | perl -e 'while (<STDIN>) {print "$1\n" if ($_ =~ /\.(\d+)$/)}' | sort -u`
if [ $IPCNT -gt $maxcnt ]
then
for PORT in $PORTS
do
ts=`date --utc +%s`
rps=$((IPCNT/captsec))
tshr=`date --utc --date "1970-01-01 $ts sec" "+%Y-%m-%d %T"`
out="$ts $ip $PORT $IPCNT = $rps ReqPerSec banned $tshr"
$CSFBIN -d $ip
echo $out >>$banlist
done
else
DEBUGOUT="no ban"
fi
done


####AUTO-UNBAN


LINES=`cat $banlist | sort -u`
for LINE in $LINES
do
LIP=`echo $LINE | awk '{print $2}'`
LPORT=`echo $LINE | awk '{print $3}'`

LCNT=`grep "$LIP" $tmpout | wc -l`
if [ "$LCNT" != "0" ]
then
DEBUGOUT="No release $LIP $LPORT"
else
#RELEASE
#IPTBLN=`$CSFBIN -L -v --line-numbers | grep "$LIP" | grep "$LPORT" | awk '{print $1}'`
#CHECK=`$CSFBIN -L -v --line-numbers | grep "$LIP" | grep "$LPORT" | wc -l`
#DEBUGOUT="RELEASE $LIP $LPORT BUG"

$CSFBIN -dr $LIP
DEBUGOUT="RELEASE $LIP $LPORT"
grep -v "$LIP" $banlist > tmp001
mv tmp001 $banlist

#echo $DEBUGOUT
fi
done




rm $tmpout3 2>/dev/null
rm $cntout 2>/dev/null
rm $tmpout 2>/dev/null
rm $tmpout2 2>/dev/null
IFS=$MYIFS




NOTE: Use this script only if you`re useing configserverfirewall (csf) !!!!

Because CSF blocks ALL Ports, I recommend you to run this script at least hourly. Otherwise 2 opened HLSW windows could ban you, even you ssh.
Also you should thinking about raising the limit, maybe to 10/s , and a scantime of 10s.

Please report any problems.

Regads
Volker
aka
Schnoog

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


13.02.2011 - 21:52:55

Thx a lot!! Youre the man!!
If the world got more people like you, i have sure we will live at a better place!!!

Regards,
Rod.

  zer0o0
Private First Class

User Pic

Posts: 37
Registred: 26.01.2011

   

0 approved this posting.


23.02.2011 - 01:16:28

Hi everybody!
First of all I wish to thank schnoog and all the others for their interest in this matter and for their contribution in making this world safer for gamers and server-admins.

I tried the script and it didn't work for me though :P well, not as it was anyway, and it took me over an hour to figure out why, so now I am here to tell about my trouble and how I got around it.

You should know that my server has multiple network adapters, each connecting to different networks, as I suspect many other servers do. The external NIC (or interface) in my server, the one facing the Internet, happens to be called eth3, but it turns out that your script assumes it's dealing with the first (or only) interface, i.e. eth0, and so no suitable traffic gets logged in file_bans.
So my first suggestion is that you create one more user configurable constant in the "CONFIGURATION" section, something like this:

IF=eth0 # the name of your external interface, the one connected to the Internet

and then modify the tcpdump line to monitor that specific interface:

$TCPDUMPBIN -f -c 100000 -A -i $IF >$tmpout 2>$cntout &

The second problem I ran into was that my firewall is pretty complex and it already had a bunch of DROP, REJECT and ACCEPT rules in the INPUT chain, so when your script added a few more DROP rules at the bottom of the chain (because of the -A), the bans had no effect, because they were trying to drop some packets that were already accepted by the ACCEPT rules.
So my suggestion is that you modify the iptables rule to do "$IPTABLESBIN -I ... " instead of "$IPTABLESBIN -A ... ", so that the rules get inserted at the top of the INPUT chain and can no longer be overruled by any ACCEPT rules that may follow in the firewall.

Once I applied the second adjustment the script worked for me too :D I hope this helps!


last changed by Wussie am 23.02.2011 - 13:15:30

  Wussie
Private

User Pic

Posts: 2
Registred: 23.02.2011

  

0 approved this posting.


23.02.2011 - 11:04:06

Wussie, thank you very much !!!

I`ll rewrite the script ASAP to fit also other adapters than eth0.

Hopefully I`ll find the time for it this week ;)


Regards
Schnoog

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


23.02.2011 - 21:24:47

Hey, nice to see someone is listening!
I was just thinking tonight: once the script has isolated one "bad" IP and it's about to drop all UDP packets coming from it on one specific port, why stop there?
What if I have more than one Q3 or W:ET servers running on the same machine, but on different UDP ports? Wouldn't I want all my game servers protected against abuse from that "bad" IP? So once we have established that bad things come from that IP, why not just drop all UDP packets coming from it on every port? What do we have to lose?
Nothing if you ask me! We'll just end up with a slightly simpler script and better protected servers.
Just my humble opinion...

  Wussie
Private

User Pic

Posts: 2
Registred: 23.02.2011

  

0 approved this posting.


24.02.2011 - 06:46:19

Thanks for your input, its an idea ;)

I had a reason to only drop udp packets to a specific port instead of dropping all...
But I couldn`t remember atm. (Shame on me).. maybe on later hours today I`m able to remember at least my name :D
I`ll have a look on it.

Regards
Schnoog

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


09.05.2011 - 21:57:29

After some check time I discovered that as more the file "file_bans" increases as dimensions, the script need more time to sort out everything, and it seems to "hang" .

I tried to set a cronjob to launch the script every 30 minutes. The command - ps aux will show a lot of this script running in the background, stuck or something like that.

If I delete the file_bans the script will return to be faster and so it's job without weird delays.


last changed by old-owl am 09.05.2011 - 22:19:43

  old-owl
Private

User Pic

Posts: 13
Registred: 06.02.2011

   

0 approved this posting.


09.05.2011 - 22:28:07

thanks for this information. Will try to find the bug in fix it.

Regards
Schnoog

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


18.05.2011 - 15:19:16

I notice the same due to increase of attacks recently i got like almost 20 ips per day banned and im running 2 servers in the ports 27960 / 27961 and it got 2 entries in the file_bans increasing the scans time, maybe if it not records an entrry per port will reduce the time of the scans.
I dont know if i write it clear so i ill make an exemple.
my file bans:
1305694832 98.18.104.97 27960 750 = 25 ReqPerSec banned 2011-05-18 05:00:32
1305694832 98.18.104.97 27961 750 = 25 ReqPerSec banned 2011-05-18 05:00:32
1305723327 85.65.244.254 27960 965 = 32 ReqPerSec banned 2011-05-18 12:55:27
1305723327 85.65.244.254 27961 965 = 32 ReqPerSec banned 2011-05-18 12:55:27

it get two entries from the same ip in the file_bans the only difference is the port, but when the script starts it checks the same ip twice because the 2 entries, other problem i see is the script take almost 1 minute to check 1 entry and when the file_bans got increased it takes more time creating many instances of script running.

Regards,
Rod.


last changed by zer0o0 am 18.05.2011 - 15:19:45

  zer0o0
Private First Class

User Pic

Posts: 37
Registred: 26.01.2011

   

0 approved this posting.


18.05.2011 - 15:24:54

Will rework the script, to catch the not-clearance of banlist. The goal of port-based drops were that not the whole IP is blocked (http still available).

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


20.05.2011 - 03:57:14

i find a script to block these ips today maybe it helps someone too, i upload it so if someone want to check it here is the link:
http://www.te666.com/serverdl/monitor-et-ddos.sh

Regards,
Rod.

  zer0o0
Private First Class

User Pic

Posts: 37
Registred: 26.01.2011

   

0 approved this posting.


27.05.2011 - 13:47:10

Thanks Zer0o0 for the link you've provided, but before test it I would like to wait the fixed version from Schnoog, hopefully soon.

Anyway I am curious if you tried it and how it works.. since actually I have to login on the root every few days, then killall the multiple instances and reset the file_bans file.

  old-owl
Private

User Pic

Posts: 13
Registred: 06.02.2011

   

0 approved this posting.


27.05.2011 - 15:44:43

My fixed version is in test-phase at the moment (runs since monday in debug-mode, no error or problem yet=
When no problem occoures till sunday, I `ll release it.
(I modded it, so hopefully it will work in spec ;) )

  schnoog
First Sergeant

User Pic

Posts: 294
Registred: 08.12.2010
Origin: Südbaden

    

0 approved this posting.


27.05.2011 - 16:52:53

im running it and it works very wel the ip whish stops the attacks are wel done deleted from the bans file and it takes only 5 seconds to run no mater how many ips it is blocking, im running that scripts for more then 1 week i think and not saw any problems with it, and i dont know if im wrong or not but seems my server got a better connection now.
try it out!!!

Regards,
Rod.

  zer0o0
Private First Class

User Pic

Posts: 37
Registred: 26.01.2011

   

0 approved this posting.


28.05.2011 - 16:42:09

I tried the script from te666, and it is fast but doesn't catch all IP's that the Schnoog script does.
So I prefer to wait the next version of the script.

The script ban Ip by port, and this is nice since all other services are still reachable, but in this way, since I run 4 servers on the same root, the IPTABLES have 4 entries per IP...
is not possible to specify something like an IP range? Example: ban the attacker IP by some ports range (user defined) i.e. 27960-27970 ?


last changed by old-owl am 28.05.2011 - 16:43:32

  old-owl
Private

User Pic

Posts: 13
Registred: 06.02.2011

   

0 approved this posting.




Images by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Powered by IlchBB Forum 3.1 © 2010 Weblösungen Florian Körner