yurisk.info » Awk weekly

List of valid domain names for load testing DNS

Yuri — Sat, 14 Aug 2010 09:45:19 +0000

I am currently running a bunch of tests on DNS resolver software called Unbound to see what it is worth and for that needed a list of valid domain names in different but controllable TLDs. The only resource to download such list I could find was 3 million records file from Nominum Sample query data file for use with resperf . Only that it contains all kinds of record types : A, PTR, AAAA and I want list of domain names where I can modify query type but also that it will be of a specific TLD sample.
Say all domains in .ASIA only TLD . To compile such list I took a word list , added to each word specific extensions and then run against some DNS server. Then I filtered the answers to include only existing resolvable domains that return at least 1 answer to query ANY. So far I did it for extensions : .ASIA .COM .CA .BIZ .EDU .EU .FR .INFO .MIL .NET .ORG .RU and it brought 831903 valid domains.
You can download the final list of those domains here : Domain list 831903 domains

Top 10 usernames used in SSH brute force

Yuri — Fri, 04 Jun 2010 09:08:23 +0000

In continuation to yesterday’s post I thought it would be interesting to know statistics of the usernames used in those bruteforce probes. I thought and I did . Find below awk/sed script to get usernames for failed ssh login attempts and sort it for statistics and also list of the usernames I got from my server. The full list of usernames can be found at the end.
The script:

awk '/Failed password for/ ' /var/log/secure* | sed 's/.* $[[:print:]]\+$ from .*/ \1 /g ' | sort | uniq -c | sort -n -k1

And the winners are:

The table listing top 10 usernames used in real cracking attampts on SSH service
Username	Number of times seen
mysql	232
info	252
postgres	317
guest	435
nagios	452
user	459
oracle	598
admin	884
test	1017
root	22058

Full list of the usernames Usernames.log

SSH brute force is on the rise

Yuri — Thu, 03 Jun 2010 19:31:43 +0000

SSH brute forcing is still in high demand. I have , for my own testing and pleasure, virtual servers scattered around the world. All of them being of the Linux/BSD family I manage by SSH. The other quirk of mine is that I have on purpose no static IP at home for various reasons (saving me money being one of them). And to manage those servers by SSH I implement a very simple security rule – from Any to SSH port allow. Port is left to be standard one – 22. After all that time my server was broken into just once , when I gave access by SSH to the colleague of mine and later he changed the password to something crackable in 5 secs. Since then I – first don’t give ssh access to colleagues , and second – look from time to time at ssh failed attempts logs for amusement.

My observations so far are :
– ssh brute forcing is still/yet/again extremely popular and increasing . On average after unfirewalled access to port 22 is discovered it goes to ~ 5000-6000 attempts per day .
– crackers do have some means of communicating between them (market economy ?) – my servers have static IPs and first days after its set up brute force login attempts are as low as 2-10 a day. But once the server IP has been discovered by determined crackers it goes up in numbers very quickly.
– origins of the attacks correlate pretty well with the known sources of Spam/Malware : Brazil, China, US etc.
If you’d like to look at your SSH logs and do some stats on failed attempts here is the awk one-liner I use. Enjoy.

awk --re-interval '/authentication failure/ {}
/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]/ {match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]/,IP); IPS[IP[0]]++ } END { for (cracker_ips in IPS) print cracker_ips " " IPS[cracker_ips]}' /var/log/secure.1 | sort -n -k2

190.202.85.3 1
194.192.14.7 1
212.111.199.3 1
222.124.195.1 1
210.71.71.1 2
89.138.195.1 5
212.156.65.7 25
202.117.51.2 32
210.51.48.7 32
115.146.138.5 47
60.191.98.5 88
174.120.208.5 107
61.129.60.2 165
202.103.180.4 175
213.251.192.2 239
91.82.101.4 242
220.173.60.6 264
12.11.210.3 271
144.16.72.1 291
212.118.5.1 360
66.11.122.1 384
211.160.160.1 703
190.12.66.1 999
83.19.184.3 1176
67.213.8.2 4955
199.187.120.2 5312
95.0.180.2 6680
85.131.163.5 7685

NB Crackers IPs are not sanitized

Schedule fw monitor to run unattended

Yuri — Sat, 29 May 2010 08:43:48 +0000

Not a groundbreaking idea but worth remembering that you can also run scheduled fw monitor using the cron. In case you have some problem occurring at the late night hours or you want to run debug at night when system is loaded less or put your case here this is one of the ways to do it.
First, the script named timed_fw_monitor.sh that starts the fw monitor:

#!/bin/bash
# We have to source Checkpoint environment variables for fw monitor to work
. /etc/profile.d/CP.sh
/opt/CPsuite-R71/fw1/bin/fw monitor -o /home/lambada/capture.cap -e ‘accept icmp or port(25);’

Then of course I will want to stop fw monitor , here is the script named stop_fw_monitor.sh that I also put in cron jobs that stops previously started fw monitor :

#!/bin/bash
ps ax | grep ‘capture.cap’ | grep -v grep | awk ‘{ print (“kill -s 3 ” $1) | “/bin/bash” }’

Now my crontab looks like this:

# DO NOT EDIT THIS FILE – edit the master and reinstall.
# (/tmp/crontab.4760 installed on Sat May 29 11:00:22 2010)
# (Cron version — $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
03 23 * * * /home/lambada/timed_fw_monitor.sh > /dev/null
17 23 * * * /home/lambada/stop_fw_monitor.sh > /dev/null

Alert on change of SOA in domain

Yuri — Sat, 22 May 2010 12:27:57 +0000

This comes from unpleasant experience of mine. One of my clients’ domain records (MX for the case involved) was mistakenly changed. While it was a human error and trying to fix humans is rolling the rock of Sisyphus,damage would be much lesser had I known about the change immediately . To take care of this side of the story I wrote awk one-liner that when invoked by cron compares SOA of the domain with the one saved locally in the file. If there is a discrepancy the mail is sent.
I tried to find more elegant solution on Google but found zillions of tools too complex for such a simple task.

#!/bin/bash
awk ‘ BEGIN {“dig +short soa yurisk.info” | getline
SOA_NOW=$3
getline SOA < “serial-yurisk.info”
if (SOA_NOW != SOA) { print ( ” mail -c yurisk@yurisk.info -s \”SOA of domain yurisk.info has changed\” president@whitehouse.gov”) | “/bin/bash” }}’

serial-yurisk.info:
2010012000

Scan of the week – scan by country scan by continent

Yuri — Mon, 22 Mar 2010 21:15:09 +0000

Gooood morning everyone . Today I launch yet another weekly column “Scan of the week” and this will be all about scanning the Net. Tools will be many but they will not be the point, my wanting here is to show interesting/funny/unusual/useful things you can see on the Internet by going out there and exploring.
Dis+claimer – all this stuff I bring to your attention is for educational purposes only, and what may be fine and ok here and for me can easily get you somewhere else in trouble so use your discretion here .
Happy scanning.

“…Don’t know much about geography” as the song goes was ok in 1958 but can be embarrassing in our times of globalization. So let’s fill the gap using the NMAP . Say you
are investigating the issue of negative attitude towards foreigners in Russia , and as part of the research
you just have to see active members of the movement(s) in question voicing their opinions. Only that many
times access to such forums or messageboards is limited by their admins to Russian IPs only. So to get there you need a free open Russian proxy. So let’s see how to find one.

Round 1-Gimme the addresses. IP geolocation databases as it is known in the Net , or simply GeoIP databases are compilation of IP ranges per their assigned country. Take it with a bit of salt as accuracy is the issue here. The one of the most known and used free GeoIP source is Maxmind.com free database that is updated once per month (good enough for this).
The Maxmind database comes as binary proprietary format file you can work with using 3rd party tools or as CSV file I will be using here. Download it as Geolite country , unzip and you have GeoIpCountryCSV.csv . Format of the records in it goes like this -

"1.0.0.0","1.0.0.255","16777216","16777471","AP","Asia/Pacific Region"
"1.1.1.0","1.1.1.255","16843008","16843263","AU","Australia"
"1.2.3.0","1.2.3.255","16909056","16909311","AU","Australia"
"1.50.0.0","1.50.3.255","20054016","20055039","AP","Asia/Pacific Region"

The purpose here is to :

Find all IP ranges that belong to the country of interest
Reformat found IP ranges into the presentation suitable for the NMAP

awk -F, ‘/RU/ { gsub(/”/,”",$0); print $1 “-” $2} ‘ GeoIPCountryWhois.csv > IPs.data

head IPs.data
62.5.128.0-62.5.255.255
62.12.80.0-62.12.81.255
62.16.32.0-62.16.66.255

- After I found all Russian IPs reformat it to the NMAP eatable form

awk -F\. ‘{split($4,aaa,”-”); print $1″-”aaa[2]“.”$2″-”$5 “.” $3″-”$6″.”aaa[1]“-”$7}’ IPs.data > scan.me

 head scan.me
62-62.5-5.128-255.0-255
62-62.12-12.80-81.0-255
62-62.16-16.32-66.0-255
62-62.16-16.68-127.0-255
62-62.32-32.64-95.0-255

Round 2 – find me some proxy Here I will use LUA script from NSE repository of the nmap called http-open-proxy

nmap -n -PN -oN proxy-check.grep –script=http-open-proxy -iL scan.me -p 8080,3128

That completes this opening article of the Scan of the week united with Awk weekly . Hope you found it educational enough and see you next time.

awk weekly – rule hits statistics . Checkpoint again

Yuri — Sat, 13 Mar 2010 10:50:36 +0000

I know , I know, I have to get out of this Checkpoint-only land and am working on this ,but for today again comes the awk travelling diary on the Checkpoint firewall roads. As I mentioned before once you export firewall logs into human-readable format you can do lots of interesting things – for example script that gives statistics of how many times each Security rule was hit .
Be aware that this counts explicit Security rules only – i.e. the ones you see in Security tab of the Smartdashboard. No other rules you usually see in Smartview Tracker are counted – e.g. SmartDefense,Web Filtering etc. Also afterwards I sort it by number of hits to see what rules are used most:

awk -F\; ‘ {match($0,/rule: +([0-9]+)/,rules);rule_count[rules[1]]++} END {for (rule_number in rule_count) print ” Rule number: ” rule_number ” Hits: ” rule_count[rule_number]}’ ./fw.log.txt | sort -n -k5

Rule number:  Hits: 1197330  Ignore this line as it counts non-matched lines I dont want to filter with additional conditions and added time processing
 Rule number: 2 Hits: 9
 Rule number: 5 Hits: 366
 Rule number: 11 Hits: 12296
 Rule number: 9 Hits: 14457
 Rule number: 0 Hits: 17094
 Rule number: 1 Hits: 44066
 Rule number: 7 Hits: 233643
 Rule number: 10 Hits: 366275
 Rule number: 6 Hits: 424639

awk weekly – Checkpoint Anti Spam statistics or viva la Open Relays

Yuri — Mon, 08 Mar 2010 20:56:38 +0000

Goooood day everyone again,
today I have had another fight with the spam cartel that my client fell victim of. Once upon a time there was not so powerful UTM providing internet to not so crowded office in not so security-aware Central Europe.
All would be good and well if not this problem – they could not send emails outside as the IP of the firewall entered every imaginable blacklist on the Earth. Hmm, but the firewall has AntiSpam subscription service up and running.
LAN is blocked on port 25 outbound except the Exchange. Antivirus is everywhere so low chance of spam coming from LAN. In SmartView Tracker lots of SMTP rule logs in red – spam entering Exchange is blocked .

So what the …? tcpdump with -w option for 5 minutes was all I needed to see that Exchange was open relay and kindly offered to relay spam from everyone to everywhere.
To really measure the impact of the event I had to have some statistics and Checkpoint didn’t help me much with that , eventhough this UTM has also SmartView Monitor license it is not suited for the task. So I exported fw.log on the UTM into text human-awk-readable format , that took some 40 mins on 300 Mb log file and produced text file of 475 Mb, and then did whatever I wanted with the data using awk.
Now get some action:
Script 1 – Find all mails rejected in direction from LAN (interface Internal, remember it is UTM) to the Internet (interface External) , then gather statistics of how many mails came from what ip [less relevant here as all mails come from Exchnage, but in environment where hosts send mails directly outside it is] and show us :

# awk -F\; ‘/Internal to External/ && /reject/ {print $2}’ ./fw.log.txt | awk ‘ {match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/,IP); IPS[IP[0]]++ } END { for (spammer_ips in IPS) print spammer_ips ” ” IPS[spammer_ips]}’ | sort -n -k2,2

192.168.14.12 402804

Yahooooo! In the timeframe of 28 hours there were blocked 402804 mails as spam coming from Exchange!
Not bad at all – all this without any malware installed on the client side [my educated by Wireshark guess here as I dont have access to the Exchange],just amazing!

Now let’s have a look at overal number of mails that was accepted and sent outside to the Internet :
Script 2 – Find all mails accepted in direction from LAN (interface Internal) to the Internet (interface External) , then gather statistics of how many mails came from what ip and show us :

# awk -F\; ‘/Internal to External/ && /accept/ {print $2}’ ./fw.log.txt | awk ‘ {match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/,IP); IPS[IP[0]]++ } END { for (spammer_ips in IPS) print spammer_ips ” ” IPS[spammer_ips]}’ | sort -n -k2,2
192.168.14.12 257940

Wow! in addition to 402804 mails blocked by Checkpoint firewall as spam 257940 mails were sent out as clean, given that this is a very small office hardly sending 300 mails a day we get ratio of 39% spam passing through the Checkpoint Antispam , pity . Antispam blocking rate of 61% ? In 21st century ? Wake up !

Just for statistics I also calculated how many spam emails were blocked from outside inbound:
Script 3 – gather how many mails from outside coming in were rejected by Checkpoint as spam.

# awk -F\; ‘/External to Internal/ && /reject/ {print $2}’ ./fw.log.txt | wc

# 5593 11186 112648

So only 5593 incoming spam emails and almost half a million outgoing ones – that’s what I call effectiveness.
Script 4 – gather statistics on blocked emails and IPs it came from:

# awk -F\; ‘/External to Internal/ && /reject/ {print $2}’ ./fw.log.txt | awk ‘ {match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/,IP); IPS[IP[0]]++ } END { for (spam_ips in IPS) print spam_ips ” ” IPS[spam_ips]}’ | sort -n -k2,2

And here are some results

93.81.26.2 75
91.121.114.1 81
220.168.57.1 87
58.9.205.2 129
122.102.101.1 149
58.137.99.7 160
189.35.231.6 189
60.248.174.6 631

PS I know Checkpoint folks visit here , so if you think I get some data wrong – don’t be shy to correct me,all fixes are gladly welcome.

awk weekly – how to see Checkpoint logs on command line

Yuri — Fri, 26 Feb 2010 20:12:57 +0000

Hey Everyone, I decided to start a weekly column of awk scripting where I will bring interesting (I am being subjective I know) short scripts that made my life easier in dealing with actual problems in the wild or just look cool.

Until recently I had never had any need to work with Checkpoint log files without SmartView Tracker , namely on the command line. But there is always first time . Client complained on some dropped mail traffic and to even say if there is any problem or not I had to look at relevant logs, not a big deal except that I had only ssh access to the firewall . Checkpoint provided for such cases fw log command line log extracting utility that reads the binary log file ( fw.log by default) you feed in and outputs it in human-readable format. That’s good, but its filtering possibilities are quite bad . You can see all available options with fw log –h , but selection is limited to source, start/end time,action (drop/reject/etc) . Not that much to say the least . No port/direction filtering . And specifically it was a very busy firewall – some 80 mbytes of traffic passing through at any given moment and log is the default action on any rule. So using fw log filters would help me not.
Here is how I solved this with the help of awk – I exported to text format all logs using

# fw log -n> fw_log.txt &

Note –n option to fw log here – it prevents resolving IP/ports to names , shortens processing time by ~70%
Then I just used all-powerful awk to search the text file to show the client what was the reason (Exchange in LAN was sending heaps of spam that Anti-Spam stopped at its best but nevertheless some spam leaked and caused RBL blocking of the external firewall IP) :

[Expert@Orlean] # awk ‘/Anti Spam/ && /Internal to External/’ fw_log.txt | awk -F: ‘ {print $5 $6}’

192.168.143.12; dst 65.55.37.88; proto
192.168.143.12; dst 65.55.92.136; proto
192.168.143.12; dst 65.55.92.136; proto
192.168.143.12; dst 203.216.247.184; proto

Here:
External, Internal – UTM appliance interface names and direction of the Anti-Spam scanning
NOTE: exporting logs from binary to text takes a bit of time, depends on situation. Enabling name resolving sky-rocketed the processing time to 15 minutes , but on the other hand gave some additional insight :

Exchange; dst col0-mc2-f.col0.hotmail.com; proto
Exchange; dst mx1.hotmail.com; proto
Exchange; dst mx1.hotmail.com; proto
Exchange; dst mta19.mail.vip.tnz.yahoo.co.jp; proto
Exchange; dst bay0-mc2-f.bay0.hotmail.com; proto
Exchange; dst mx3.hotmail.com; proto