yurisk.info » Awk weekly

Check duplex and speed settings of all interfaces in one go

Yuri — Thu, 16 Feb 2012 18:36:39 +0000

One of the first things you do when checking connectivity issues on the Checkpoint (or any networking gear for that matter) is to see speed and duplex parameters of the interfaces. But have you tried to do it on a firewall with 15-20 interfaces ?
No fun entering one by one interfaces’ names. Here is the one-liner I use to get speed and duplex settings of all interfaces in one go.

# for ii in $(ifconfig | awk ‘/Ethernet/ {print $1}’) ;do ethtool $ii; done | egrep ‘eth|Speed|Duplex’
Settings for eth0:
Speed: 100Mb/s
Duplex: Full
Settings for eth1:
Speed: 1000Mb/s
Duplex: Full
Settings for eth1.150:
Speed: 1000Mb/s
Duplex: Full
Settings for eth1.160:
Speed: 1000Mb/s
Duplex: Full
Settings for eth1.161:
Speed: 1000Mb/s
Duplex: Full
Settings for eth1.270:
Speed: 1000Mb/s
Duplex: Full
Settings for eth1.271:
Speed: 1000Mb/s
Duplex: Full

Settings for eth1.281:
Speed: 1000Mb/s
Duplex: Full
Settings for eth1.35:
Speed: 1000Mb/s
Duplex: Full
Settings for eth2:
Speed: 100Mb/s
Duplex: Full
Settings for eth3:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.112:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.211:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.311:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.71:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.72:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.73:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.413:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.419:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.451:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.407:
Speed: 1000Mb/s
Duplex: Full
Settings for eth4.408:
Speed: 1000Mb/s
Duplex: Full
Settings for eth5:
Speed: 1000Mb/s
Duplex: Full
Settings for eth7:
Speed: 1000Mb/s
Duplex: Full

awk weekly – Security rule hits statistics . Checkpoint

Yuri — Tue, 31 Jan 2012 08:50:36 +0000

As I mentioned before once you export firewall logs into human-readable format you can do lots of interesting things – for example script that gives statistics of how many times each Security rule was hit .
Be aware that this counts explicit Security rules only – i.e. the ones you see in Security tab of the Smartdashboard. No other rules you usually see in Smartview Tracker are counted – e.g. SmartDefense,Web Filtering etc. Also afterwards I sort it by number of hits to see what rules are used most:

awk -F\; ' {match($0,/rule: +([0-9]+)/,rules);rule_count[rules[1]]++} END {for (rule_number in rule_count) print " Rule number: " rule_number " Hits: " rule_count[rule_number]}' ./fw.log.txt | sort -n -k5

Rule number:  Hits: 1197330  Ignore this line as it counts non-matched lines I dont want to filter with additional conditions and added time processing
 Rule number: 2 Hits: 9
 Rule number: 5 Hits: 366
 Rule number: 11 Hits: 12296
 Rule number: 9 Hits: 14457
 Rule number: 0 Hits: 17094
 Rule number: 1 Hits: 44066
 Rule number: 7 Hits: 233643
 Rule number: 10 Hits: 366275
 Rule number: 6 Hits: 424639

Update 2012 Below is the script to use Rule ID instead of Rule sequential numbers – this way changing rules order will not affect statistics. The script matches also non-security rules – e.g. email session id, that are a bit shorter then Rule ID, but I didn’t want to slow down the processing with additional formatting .

awk -F\; ' {match($0,/{([[:print:]]+)}/,rules);rule_count[rules[1]]++} END {for (rule_number in rule_count) print " Rule number: " rule_number " Hits: " rule_count[rule_number]}' ./fw.log.txt | sort -n -k5

Rule number: D199972C-ED3E-4EB4-8B83-813333156D18 Hits: 175
 Rule number: 85A905A7-951E-4100-A4BA-E13333151D29 Hits: 219
 Rule number: 81333316-E942-4313-BB7D-E1333315802F Hits: 1519
 Rule number: 71333215-2DB5-4A3A-95BC-5080AD0F5564 Hits: 2298
 Rule number: 11331315-AE52-44E0-A42A-711029B5768E Hits: 3755
 Rule number: 01333315-D290-4B05-AFE7-23BF24D889FF Hits: 4116
 Rule number: 121FA62F-3885-4328-8090-BF1333315eB1 Hits: 399793
 Rule number: FE40E076-BAEB-4979-8E41-5EF1333315e6 Hits: 440101
 Rule number: BB3F6772-4D38-4D5A-952A-301333315de8 Hits: 1354341
Running time for a file of 900 Mb with 4.7 million records
real    5m50.287s
user    4m22.890s
sys     0m3.190s

IP address pools of Facebook to block, if you need to

Yuri — Mon, 15 Nov 2010 12:14:15 +0000

Once upon a time I mentioned that blocking Facebook is easy as they have a uniform IP addresses pool . Since then they added more , here is the new and old pools:

NetRange: 69.63.176.0 – 69.63.191.255
CIDR: 69.63.176.0/20
OriginAS: AS32934
NetName: TFBNET2
NetHandle: NET-69-63-176-0-1
Parent: NET-69-0-0-0-0
NetType: Direct Assignment
NameServer: NS5.FACEBOOK.COM
NameServer: NS3.FACEBOOK.COM
NameServer: NS4.FACEBOOK.COM
RegDate: 2007-02-07
Updated: 2010-07-08

NetRange: 66.220.144.0 – 66.220.159.255
CIDR: 66.220.144.0/20
OriginAS: AS32934
NetName: TFBNET3
NetHandle: NET-66-220-144-0-1
Parent: NET-66-0-0-0-0
NetType: Direct Assignment
NameServer: NS5.FACEBOOK.COM
NameServer: NS3.FACEBOOK.COM
NameServer: NS4.FACEBOOK.COM
RegDate: 2009-02-13

See what your users are doing – awk one-line scripts to parse eSafe logs

Yuri — Tue, 19 Oct 2010 20:49:46 +0000

As most of the posts here this one is also inspired by a client. There was an unassuming shy and not making any troubles eSafe 8.5 appliance XG-200. Then one day Security Admin of the company complained to me about ‘high CPU utilization’ – getting somewhere up to 60% . eSafe looked absolutely fine and doing its work.
Also I noticed it was kinda working hard , nothing special but just general feeling that it handles a lot of load. And boy was I right – it was doing 200 Mb of logs per day . Given the number of stations in LAN and working hours that was huge. I looked with awk at the logs and happily updated the SecAdmin that eSafe is doing its work and blocks all the users trying frantically to visit various porno sites ignoring the “Site is blocked ..”message by the eSafe. “What? my users to pron sites, can’t be, can you show me the logs, who does it ?” . No problem, and so the awk one-liners you see below were written to parse esafe Aladdin logs to get some insight. Also at the end of the post see eSafe log format in case you want to develop your own scripts.
All logs are located at /opt/eSafe/eSafeCR/SessionLog/
One-liner number one – Gather IPs that sent spam , count number of spam messages per IP, sort the list in ascending order .

# awk -F"|" '$4~SMTP && ( $6~/Spam blocked/ || $6~/Mail rejected/ ) { print $11} ' *.log | sort -n | uniq -c | sort -n | tail -10

29 80.179.60.37
32 41.59.0.205
41 74.63.80.6
41 83.16.167.14
48 125.163.188.31
54 125.163.192.184
57 113.168.9.224
78 89.223.56.16
80 212.143.70.26
104 212.143.70.27

One-liner number two – Mail sender fileds of spam messages , just for fun , no real value for security purpose:

# awk -F"|" ' $4~SMTP && ( $6~/Spam blocked/ || $6~/Mail rejected/ ) { print $15} ' *.log | sort | uniq -c | sort -n

8 Stephan@117.40.136.73
10 bsb@bsbinfo.in
13 info@all-free.co.il
13 Janette@2.90.58.204
15 Ronnie@178.34.19.174
17 KellieClements@cramerspointmotel.com
22 notifs@m.snapinteractiveapps.com
60 ezrachmudag26@gmail.com
25 Simone@187.63.223.21
102 Angelo@31.subnet125-163-188.speedy.telkom.net.id

One-liner to see all the blocks/rejects reasons and respective statistics.

# awk -F"|"' {print $6}' *.log | sort -k1,1 | uniq -c

8 Application blocked
21967 File allowed
360 File blocked
114891 File clean
1731 File modified to remove malicious content
3650 Mail clean
111 Mail modified to remove malicious content
13 Mail rejected #912 – Anti-spoofing – Mail rejected. Attempt to impersonate a local user
164 SMTP error
803 Spam blocked

Now let’s move to HTTP browsing.
One-liner number four – blocked access to websites : number of blocked attempts per website, hostname of the website, internal LAN IP of PC that tried to access the resource.
I do not bring examples here as they are quite embarrassing, even to be brought anonymously, so just trust me – run it on your esafe and you will blush.

# awk -F"|" '$4~HTTP && /File blocked/ { print $7,$17} ' *.log | sort -k1,1 | uniq -c | sort -n -k1,1

Same as above but with full path to the prohibited file.

# awk -F"|" '$4~HTTP && /File blocked/ { print $8,$17,$11} ' *.log | sort -k1,1 | uniq -c | sort -n -k1,1

And finally as promised the format of eSafe logs. All the fields in logs are separated by vertical bar (as you probably guessed awk –F”|” accounts for that). All the fields are present, while irrelevant fields are empty. So it is really scripting-friendly. I broke down the fields into separate lines with field number of each field. Enjoy.

# awk -F"|" ' { for (i=1;i<=NF;i++) print i,$i}' header.txt

1 Date (yyyy-mm-dd HH:mm:ss)
2 eSafe name
3 Record ID
4 ProtocolType
5 Method
6 Event
7 URL host
8 File Name\Mail Subject
9 File Type
10 #File Size
11 Source IP
12 Destination IP
13 #VLAN
14 #Port
15 Mail Sender
16 Mail Recipients
17 URL category
18 User
19 LDAP domain
20 Host
21 Decision By
22 Profile
23 Policy
24 #Policy ID
25 Details
26 Extended result
27 SessionID
28 MessageID
29 #Rule
30 #File Binary Family
31 File container path
32 File name
33 #File parameter
34 #Engine code
35 #Activity code
36 Blocked URL category code
37 URL category mask
38 Result name
39 #Result code
40 #Server type ID
41 #Application code
42 #Action
43 #Risky
44 #Source IP
45 #Destination IP
46 #MachineIP
47 #Duration
48 #AID
49 Referrer
50 UUID
51 #Has CMF
52 Date
53 Time
54 #Mail status
55 DLP profile

Darknet can’t lie – most of the attacks, scans and other interesting things indeed come from behind the Great Firewall of China.

Yuri — Tue, 12 Oct 2010 10:33:01 +0000

Working for ISP entitles me to various perks, one of them is unlimited connection to the Internet with wealth of unallocated yet IP addresses. So to use it somehow I set up a little Darknet (details what it means can be found here Darknet Project ) and gather some statistics. First the volume of unsolicited and malicious traffic is staggering . Mostly it is traffic to Windows sharing – port 445 , then brute force – port 22, then strange ports used by new malware in the wild .Second, the interesting information pretty much stops here – as nothing listens on my side of the Darknet I don’t get more insight. As comes from this I am working on the next stage of the Darknet – HoneyNet. Once done, I’ll post here the findings.
To give you a glimpse of the Ips and ports involved in probes here is the non-sanitized sorted list of the alien IPs , destination ports, protocols and number of packets seen.This is the day’s worth statistics Bad guys and gals IPs
To get this list from Tcpdump capture I used one-liner: [root@darkstar]# tshark -n -r honey_bunny.cap42 | awk ' $3~/[0-9]+\./ {print $3,$6,$9}' | sort -n -k1,1 | uniq -c > Darknet_probing_IPs.txt

List of valid domain names for load testing DNS

Yuri — Sat, 14 Aug 2010 09:45:19 +0000

I am currently running a bunch of tests on DNS resolver software called Unbound to see what it is worth and for that needed a list of valid domain names in different but controllable TLDs. The only resource to download such list I could find was 3 million records file from Nominum Sample query data file for use with resperf . Only that it contains all kinds of record types : A, PTR, AAAA and I want list of domain names where I can modify query type but also that it will be of a specific TLD sample.
Say all domains in .ASIA only TLD . To compile such list I took a word list , added to each word specific extensions and then run against some DNS server. Then I filtered the answers to include only existing resolvable domains that return at least 1 answer to query ANY. So far I did it for extensions : .ASIA .COM .CA .BIZ .EDU .EU .FR .INFO .MIL .NET .ORG .RU and it brought 831903 valid domains.
You can download the final list of those domains here : Domain list 831903 domains

Top 10 usernames used in SSH brute force

Yuri — Fri, 04 Jun 2010 09:08:23 +0000

In continuation to yesterday’s post I thought it would be interesting to know statistics of the usernames used in those bruteforce probes. I thought and I did . Find below awk/sed script to get usernames for failed ssh login attempts and sort it for statistics and also list of the usernames I got from my server. The full list of usernames can be found at the end.
The script:

awk '/Failed password for/ ' /var/log/secure* | sed 's/.* $[[:print:]]\+$ from .*/ \1 /g ' | sort | uniq -c | sort -n -k1

And the winners are:

The table listing top 10 usernames used in real cracking attampts on SSH service
Username	Number of times seen
mysql	232
info	252
postgres	317
guest	435
nagios	452
user	459
oracle	598
admin	884
test	1017
root	22058

Full list of the usernames Usernames.log

SSH brute force is on the rise

Yuri — Thu, 03 Jun 2010 19:31:43 +0000

SSH brute forcing is still in high demand. I have , for my own testing and pleasure, virtual servers scattered around the world. All of them being of the Linux/BSD family I manage by SSH. The other quirk of mine is that I have on purpose no static IP at home for various reasons (saving me money being one of them). And to manage those servers by SSH I implement a very simple security rule – from Any to SSH port allow. Port is left to be standard one – 22. After all that time my server was broken into just once , when I gave access by SSH to the colleague of mine and later he changed the password to something crackable in 5 secs. Since then I – first don’t give ssh access to colleagues , and second – look from time to time at ssh failed attempts logs for amusement.

My observations so far are :
- ssh brute forcing is still/yet/again extremely popular and increasing . On average after unfirewalled access to port 22 is discovered it goes to ~ 5000-6000 attempts per day .
- crackers do have some means of communicating between them (market economy ?) – my servers have static IPs and first days after its set up brute force login attempts are as low as 2-10 a day. But once the server IP has been discovered by determined crackers it goes up in numbers very quickly.
- origins of the attacks correlate pretty well with the known sources of Spam/Malware : Brazil, China, US etc.
If you’d like to look at your SSH logs and do some stats on failed attempts here is the awk one-liner I use. Enjoy.

awk --re-interval '/authentication failure/ {}
/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]/ {match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]/,IP); IPS[IP[0]]++ } END { for (cracker_ips in IPS) print cracker_ips " " IPS[cracker_ips]}' /var/log/secure.1 | sort -n -k2

190.202.85.3 1
194.192.14.7 1
212.111.199.3 1
222.124.195.1 1
210.71.71.1 2
89.138.195.1 5
212.156.65.7 25
202.117.51.2 32
210.51.48.7 32
115.146.138.5 47
60.191.98.5 88
174.120.208.5 107
61.129.60.2 165
202.103.180.4 175
213.251.192.2 239
91.82.101.4 242
220.173.60.6 264
12.11.210.3 271
144.16.72.1 291
212.118.5.1 360
66.11.122.1 384
211.160.160.1 703
190.12.66.1 999
83.19.184.3 1176
67.213.8.2 4955
199.187.120.2 5312
95.0.180.2 6680
85.131.163.5 7685

NB Crackers IPs are not sanitized

Schedule fw monitor to run unattended

Yuri — Sat, 29 May 2010 08:43:48 +0000

Not a groundbreaking idea but worth remembering that you can also run scheduled fw monitor using the cron. In case you have some problem occurring at the late night hours or you want to run debug at night when system is loaded less or put your case here this is one of the ways to do it.
First, the script named timed_fw_monitor.sh that starts the fw monitor:

#!/bin/bash
# We have to source Checkpoint environment variables for fw monitor to work
. /etc/profile.d/CP.sh
/opt/CPsuite-R71/fw1/bin/fw monitor -o /home/lambada/capture.cap -e 'accept icmp or port(25);'

Then of course I will want to stop fw monitor , here is the script named stop_fw_monitor.sh that I also put in cron jobs that stops previously started fw monitor :

#!/bin/bash
ps ax | grep 'capture.cap' | grep -v grep | awk '{ print ("kill -s 3 " $1) | "/bin/bash" }'

Now my crontab looks like this:

# DO NOT EDIT THIS FILE – edit the master and reinstall.
# (/tmp/crontab.4760 installed on Sat May 29 11:00:22 2010)
# (Cron version — $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
03 23 * * * /home/lambada/timed_fw_monitor.sh > /dev/null
17 23 * * * /home/lambada/stop_fw_monitor.sh > /dev/null

Alert on change of SOA in domain

Yuri — Sat, 22 May 2010 12:27:57 +0000

This comes from unpleasant experience of mine. One of my clients’ domain records (MX for the case involved) was mistakenly changed. While it was a human error and trying to fix humans is rolling the rock of Sisyphus,damage would be much lesser had I known about the change immediately . To take care of this side of the story I wrote awk one-liner that when invoked by cron compares SOA of the domain with the one saved locally in the file. If there is a discrepancy the mail is sent.
I tried to find more elegant solution on Google but found zillions of tools too complex for such a simple task.

#!/bin/bash
awk ' BEGIN {"dig +short soa yurisk.info"| getline
SOA_NOW=$3
getline SOA < "serial-yurisk.info"
if (SOA_NOW != SOA) { print (" mail -c yurisk@yurisk.info -s \"SOA of domain yurisk.info has changed\" president@whitehouse.gov") | "/bin/bash" }}'

serial-yurisk.info:
2010012000