| 解释: 分析日志文件/etc/httpd/logs/yfddos_log中,在过去5s内的用户访问日志,如果有IP在这5s内访问量>=4,守护进程yfddosd将视其为资源滥用者，然后将这个IP加入到黑名单文件/etc/yfddos/web-yf-search.b中，此条黑名单的作用时间为10s，在10s之后，守护进程yfddosd将删除此黑名单条目。 |
例子: analyze_and_insert_black "150" "2700" "905"
| 解释: 分析日志文件/etc/httpd/logs/yfddos_log中,在过去905s内的用户访问日志,如果有IP在这905s内访问量>=150,守护进程yfddosd将视其为资源滥用者，然后将这个IP加入到黑名单文件/etc/yfddos/web-yf-search.b中，此条黑名单的作用时间为2700s，在2700s之后，守护进程yfddosd将删除此黑名单条目。 |
简记为: analyze_and_insert_black "limit" "ttl" "time"
| 解释: 分析日志文件/etc/httpd/logs/yfddos_log中,在过去(time)s内的用户访问日志,如果有IP在这(time)s内访问量>=limit,守护进程yfddosd将视其为资源滥用者，然后此IP将会被加入到黑名单文件/etc/yfddos/web-yf-search.b中，作用时间为(ttl)s，在(ttl)s之后，守护进程yfddosd将自动删除此条目。 |
从上述中可看出，守护进程yfddosd至少需要完成如下三个任务:
◆分析日志文件/etc/httpd/logs/yfddos_log中指定时间内的用户访问记录
◆将资源滥用者的IP加入文件/etc/yfddos/web-yf-search.b，并设置封杀TTL参数值
◆将/etc/yfddos/web-yf-search.b中已经过期的条目全部及时删除
守护进程yfddosd是如何实现上面三个逻辑的:
◆分析日志文件/etc/httpd/logs/yfddos_log中指定时间内的用户访问记录:
(1) 取出/etc/httpd/logs/yfddos_log中过去time秒的访问日志数据，使用二分法将这一操作的时间复杂度压缩到K*log2(N)以内，其中N为/etc/httpd/logs/yfddos_log中日志总行数，K为一次测试的耗时量，一般为1ms以内，即如有1048576条访问记录，这一操作将仅需要20*1ms。
(2) 使用正则RE对这些数据进行二次处理，过滤出所有访问指定URL的用户IP(这个URL为想要防御的http服务url，例如在http://www.yfdc.org系统中，所防御的就是/shell/yf,这个服务向访问者提供信息的search与get服务)，再次使用sort与uniq对这些IP进行处理，以统计出每个IP的访问次数并进行高低排序。
◆将资源滥用者的IP加入文件/etc/yfddos/web-yf-search.b，并设置封杀TTL参数值
将所有访问次数超过阈值limit的IP更新到黑名单文件/etc/yfddos/web-yf-search.b中,每个黑名单条目的封杀时间为ttl秒
◆将/etc/yfddos/web-yf-search.b中已经过期的条目全部及时删除
遍历/etc/yfddos/web-yf-search.b中所有黑名单条目，结合当前时间戳，将所有已经过期的条目一一删去
下面是守护进程yfddosd状态机的伪代码:(略去了一些处理细节)
#init and FSM start work...
counter=0
while true
do
sleep 5
counter=counter+1
delete obsolete items #将/etc/yfddos/web-yf-search.b中已经过期的条目全部删除
if # every 5 seconds : 5s
then
analyze_and_insert_black "6" "10" "5"
# 分析在过去5s内访问的用户 如果有人其访问量大于等于6 系统将视其为资源滥用者
# 遂将其加入服务黑名单 其作用时间为10s 在10s之后 daemon进程自动删除这个ip黑名单条目
fi
if #every 5*3 seconds : 15s
then
analyze_and_insert_black "14" "45" "15"
fi
if #every 5*3*4+5 seconds : 65s
then
analyze_and_insert_black "40" "840" "65"
fi
if #every 5*3*4*3*5+5 seconds : 905s : 15min
then
analyze_and_insert_black "150" "2700" "905"
fi
if #every 5*3*4*3*5*4+5 seconds : 3605s : 1h
then
analyze_and_insert_black "300" "7200" "3605"
fi
if #every 5*3*4*3*5*4*3+5 seconds : 10805s : 3h
then
analyze_and_insert_black "400" "21600" "10805"
if #在每天的00:01-04:59时间区间 一天仅执行一次
then
#备份日志
fi
fi
done
防御者应斟酌调整每个检测时间点的参数值(封杀时间ttl与判定阈值limit)，以调节系统应对CC攻击到来时的反应时间。
0x02 源代码
- ##################################### vim /usr/local/bin/yfddosd.sh :
- ##################################### nohup bash /usr/local/bin/yfddosd.sh &>"/etc/yfddos/""yfddosd-log-`date +%Y-%m-%d`" &
- ##################################### yfddos daemon
- mkdir /etc/yfddos
- yfddos_blackfilePath='/etc/yfddos/web-yf-search.b'
- yfddos_accesslogPath='/etc/httpd/logs/yfddos_log'
- ### refresh tll
- logtmpfile=`mktemp`
- stamp=`date +%s`
- touch "$yfddos_blackfilePath"
- if grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' "$yfddos_blackfilePath" &>/dev/null
- then
- cat "$yfddos_blackfilePath" | while read i
- do
- deadstamp=`echo "$i" | grep -Po '[0-9]+$'`
- if [ "$stamp" -le "$deadstamp" ]
- then
- echo "$i" >>"$logtmpfile"
- fi
- done
- fi
- chmod o+r "$logtmpfile"
- mv -f "$logtmpfile" "$yfddos_blackfilePath"
- if ! grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' "$yfddos_blackfilePath" &>/dev/null
- then
- echo '255.255.255.255 0 0' >> "$yfddos_blackfilePath"
- fi
- function analyze_and_insert_black() {
- # analyze_and_insert_black() :
- # $1:max frequency(seems as abuse if above that) $2:blackip-ttl,time to live,unit is seconds (s)
- # $3:the access log ${3} seconds before will be analyzed to generate the abuse ip lists that we will block
- # example : analyze_and_insert_black "limit" "ttl" "time"
- # example : analyze_and_insert_black "4" "10" "5"
- # 分析在过去5s内的用户访问日志 如果有人在这5s内访问量>=4 系统将视其为资源滥用者 将其加入服务黑名单
- # 一条黑名单的作用时间为10s 即在10s之后 系统自动删除此黑名单条目 服务则继续向其开放
- # global vars:
- # stamp logtmpfile yfddos_blackfilePath
- local threshold="$1"
- local ttl="$2"
- local stamp_pre="$3"
- local i=0
- local num=""
- local fre=0
- local ip=0
- local localbuf=0
- local linenum=0
- local deadstamp=0
- stamp_pre="$((stamp-stamp_pre))"
- #二分查找初始化
- local temp=0
- local yf_x='1'
- local yf_y=`cat "$logtmpfile" | wc -l`
- if [ "$yf_y" -le "1" ]
- then
- yf_y=1
- fi
- local yf_I=$(((yf_x+yf_y)/2))
- temp=`cat "$logtmpfile" | wc -l`
- if [ "$temp" -gt "0" ]
- then
- temp=`sed -n '$p' "$logtmpfile" | grep -Po '[0-9]+ $'`
- if [ "$temp" -lt "$stamp_pre" ]
- then
- num=""
- else
- while true #使用二分查找的方法 快速地分析访问日志
- do
- temp=`sed -n "${yf_x}p" "$logtmpfile" | grep -Po '[0-9]+ $'`
- if [ "$temp" -ge "$stamp_pre" ]
- then
- break
- fi
- if [ "$((yf_y-yf_x))" -le "1" ]
- then
- yf_x="$yf_y"
- break
- fi
- temp=`sed -n "${yf_I}p" "$logtmpfile" | grep -Po '[0-9]+ $'`
- if [ "$temp" -lt "$stamp_pre" ]
- then
- yf_x="$yf_I"
- yf_y="$yf_y"
- yf_I="$(((yf_x+yf_y)/2))"
- continue
- fi
- yf_x="$yf_x"
- yf_y="$yf_I"
- yf_I="$(((yf_x+yf_y)/2))"
- continue
- done
- temp=`sed -n "${yf_x}p" "$logtmpfile" | grep -Po '[0-9]+ $'`
- if [ "$temp" -ge "$stamp_pre" ]
- then
- num="$yf_x"
- else
- num=""
- fi
- fi
- if [ -n "$num" ]
- then
- sed -n "${num},\$p" "$logtmpfile" | grep -Po '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | sort -n | uniq -c | sort -rn | while read i
- do
- fre=`echo "$i" | grep -Po '[0-9]+' | head -1`
- ip=`echo "$i" | grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' `
- if [ "$fre" -ge "$threshold" ]
- then #insert illegal ips : cat "$yfddos_blackfilePath"
- # ip add-stamp rmv-stamp
- #1.2.3.4 1416046335 1416046395
- temp=`grep -Pn "${ip//./\\.} " "$yfddos_blackfilePath"`
- if [ -n "$temp" ]
- then
- linenum=`echo "$temp" | grep -Po '^[0-9]+' | head -1`
- deadstamp=`echo "$temp" | grep -Po '[0-9]+$' | sort -rn | head -1 `
- if [ "$((stamp+ttl))" -gt "$deadstamp" ]
- then
- sed -i "${linenum}s/.*/${ip} ${stamp} $((stamp+ttl))/g" "$yfddos_blackfilePath"
- fi
- else
- sed -i "\$a ${ip} ${stamp} $((stamp+ttl))" "$yfddos_blackfilePath"
- fi
- else
- break
- fi
- done
- fi
- fi
- }
- #init and yfddosd's FSM start work...
- counter=0
- while true
- do
- sleep 5
- counter=$((counter+1))
- echo -n `date +%Y-%m-%d\ %H:%M:%S`" ""counter ${counter}:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- echo -n "refresh tll:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- ### refresh tll
- #refresh ttl: analyze file: "$yfddos_blackfilePath" if some items'ttl has been reach the date , we will remove it and open service to the ip had been banned before.
- #insert illegal ips : cat "$yfddos_blackfilePath"
- # ip add-stamp rmv-stamp
- #1.2.3.4 1416046335 1416046395
- #sed -i "/^.* $((stamp-5))$/d;/^.* $((stamp-4))$/d;/^.* $((stamp-3))$/d;/^.* $((stamp-2))$/d;/^.* $((stamp-1))$/d;/^.* $((stamp))$/d;/^$/d" "$yfddos_blackfilePath"
- logtmpfile=`mktemp`
- stamp=`date +%s`
- touch "$yfddos_blackfilePath"
- if grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' "$yfddos_blackfilePath" &>/dev/null
- then
- cat "$yfddos_blackfilePath" | while read i
- do
- deadstamp=`echo "$i" | grep -Po '[0-9]+$'`
- if [ "$stamp" -le "$deadstamp" ]
- then
- echo "$i" >>"$logtmpfile"
- fi
- done
- fi
- chmod o+r "$logtmpfile"
- mv -f "$logtmpfile" "$yfddos_blackfilePath"
- if ! grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' "$yfddos_blackfilePath" &>/dev/null
- then
- echo '255.255.255.255 0 0' >> "$yfddos_blackfilePath"
- fi
- logtmpfile=`mktemp`
- stamp=`date +%s`
- cat "$yfddos_accesslogPath" | grep -P ' "/shell/yf" ' >"$logtmpfile"
- if true # every 5 seconds : 5s
- then
- echo -n "analyze_and_insert_black 6 10 5:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- #analyze yfddos log : analyze_and_insert_black() $1:max frequency(seems as abuse if above that) $2:blackip-ttl $3:the access log ${3} seconds before will be analyzed to generate the abuse ips that we will block
- analyze_and_insert_black "6" "10" "5"
- # 分析在过去5s内访问的用户 如果有人其访问量大于等于6 系统将视其为资源滥用者 遂将其加入服务黑名单 其作用时间为10s 在10s之后 daemon进程自动删除这个ip黑名单条目
- fi
- if [ "$((counter%(3)))" -eq "0" ] #every 5*3 seconds : 15s
- then
- echo -n "analyze_and_insert_black 14 45 15:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "10" "45" "15"
- fi
- if [ "$((counter%(3*4+1)))" -eq "0" ] #every 5*3*4+5 seconds : 65s
- then
- echo -n "analyze_and_insert_black 40 840 65:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "25" "840" "65"
- fi
- if [ "$((counter%(3*4*3*5+1)))" -eq "0" ] #every 5*3*4*3*5+5 seconds : 905s : 15min
- then
- echo -n "analyze_and_insert_black 150 2700 905:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "150" "2700" "905"
- fi
- if [ "$((counter%(3*4*3*5*4+1)))" -eq "0" ] #every 5*3*4*3*5*4+5 seconds : 3605s : 1h
- then
- echo -n "analyze_and_insert_black 300 7200 3605:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "300" "7200" "3605"
- fi
- if [ "$((counter%(3*4*3*5*4*3+1)))" -eq "0" ] #every 5*3*4*3*5*4*3+5 seconds : 10805s : 3h
- then
- echo -n "analyze_and_insert_black 400 21600 10805:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "400" "21600" "10805"
- #### "${yfddos_accesslogPath}" backup : 在每天的00:01-04:59时间区间内 备份日志一次
- if [ "`date +%H`" -le "5" ] && ! [ -f "${yfddos_accesslogPath}-`date +%Y-%m-%d`" ]
- then
- service httpd stop
- mv "${yfddos_accesslogPath}" "${yfddos_accesslogPath}-`date +%Y-%m-%d`"
- service httpd start
- fi
- fi
- rm -fr "$logtmpfile"
- echo "sleep:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- done
如果您喜欢本文请分享给您的好友，谢谢！本文来源:燕云 乌云知识库