yurisk.info

Yuri Slobodyanyuk's blog on IT Security and Networking sharing experience and expertise

Page 10 of 24

See what your users are doing – awk one-line scripts to parse eSafe logs

As most of the posts here this one is also inspired by a client. There was an unassuming shy and not making any troubles eSafe 8.5 appliance XG-200. Then one day Security Admin of the company complained to me about ‘high CPU utilization’ – getting somewhere up to 60% . eSafe looked absolutely fine and doing its work.
Also I noticed it was kinda working hard , nothing special but just general feeling that it handles a lot of load. And boy was I right – it was doing 200 Mb of logs per day . Given the number of stations in LAN and working hours that was huge. I looked with awk at the logs and happily updated the SecAdmin that eSafe is doing its work and blocks all the users trying frantically to visit various porno sites ignoring the “Site is blocked ..”message by the eSafe. “What? my users to pron sites, can’t be, can you show me the logs, who does it ?” . No problem, and so the awk one-liners you see below were written to parse esafe Aladdin logs to get some insight. Also at the end of the post see eSafe log format in case you want to develop your own scripts.
All logs are located at /opt/eSafe/eSafeCR/SessionLog/
One-liner number one – Gather IPs that sent spam , count number of spam messages per IP, sort the list in ascending order .

# awk -F"|" '$4~SMTP && ( $6~/Spam blocked/ || $6~/Mail rejected/ ) { print $11} ' *.log | sort -n | uniq -c | sort -n | tail -10
29 80.179.60.37
32 41.59.0.205
41 74.63.80.6
41 83.16.167.14
48 125.163.188.31
54 125.163.192.184
57 113.168.9.224
78 89.223.56.16
80 212.143.70.26
104 212.143.70.27

One-liner number two – Mail sender fileds of spam messages , just for fun , no real value for security purpose:

# awk -F"|" ' $4~SMTP && ( $6~/Spam blocked/ || $6~/Mail rejected/ ) { print $15} ' *.log | sort | uniq -c | sort -n
8 Stephan@117.40.136.73
10 bsb@bsbinfo.in
13 info@all-free.co.il
13 Janette@2.90.58.204
15 Ronnie@178.34.19.174
17 KellieClements@cramerspointmotel.com
22 notifs@m.snapinteractiveapps.com
60 ezrachmudag26@gmail.com
25 Simone@187.63.223.21
102 Angelo@31.subnet125-163-188.speedy.telkom.net.id

One-liner to see all the blocks/rejects reasons and respective statistics.

# awk -F"|"' {print $6}' *.log | sort -k1,1 | uniq -c
8 Application blocked
21967 File allowed
360 File blocked
114891 File clean
1731 File modified to remove malicious content
3650 Mail clean
111 Mail modified to remove malicious content
13 Mail rejected #912 – Anti-spoofing – Mail rejected. Attempt to impersonate a local user
164 SMTP error
803 Spam blocked

Now let’s move to HTTP browsing.
One-liner number four – blocked access to websites : number of blocked attempts per website, hostname of the website, internal LAN IP of PC that tried to access the resource.
I do not bring examples here as they are quite embarrassing, even to be brought anonymously, so just trust me – run it on your esafe and you will blush.

# awk -F"|" '$4~HTTP && /File blocked/ { print $7,$17} ' *.log | sort -k1,1 | uniq -c | sort -n -k1,1

Same as above but with full path to the prohibited file.

# awk -F"|" '$4~HTTP && /File blocked/ { print $8,$17,$11} ' *.log | sort -k1,1 | uniq -c | sort -n -k1,1

And finally as promised the format of eSafe logs. All the fields in logs are separated by vertical bar (as you probably guessed awk –F”|” accounts for that). All the fields are present, while irrelevant fields are empty. So it is really scripting-friendly. I broke down the fields into separate lines with field number of each field. Enjoy.

# awk -F"|" ' { for (i=1;i<=NF;i++) print i,$i}' header.txt
1 Date (yyyy-mm-dd HH:mm:ss)
2 eSafe name
3 Record ID
4 ProtocolType
5 Method
6 Event
7 URL host
8 File Name\Mail Subject
9 File Type
10 #File Size
11 Source IP
12 Destination IP Continue reading

RBLs – more hassle than benefit

as I wrote earlier Google mail servers got blacklisted , but it was only the beginning. Now I got complains from the client that hotmail.com users also get mail bounced because of this list . So, as running after each and every IP blocked by this list endlessly wasn’t the best use of my time, I disabled this specific RBL in eSafe. So far no complains neither about bounced mail nor increased spam. The screenshot below shows where to find this RBL in the eSafe.
How to disable specific RBL server

Do not miss the long awaited addition to the Fortigate 4 MR2 – sFlow data export

Great news – now Fortigate supports exporting data flows statistics to an external server using sFlow protocol (twin of Netflow from the Cisco world). I configured it in about a minute and it just works. To collect the sFlow data I use nfdump/Nfsen , that I found to be the most stable and versatile, not to mention being the rare one supporting both Netflow and sFlow.
You first set external server IP and destination port , here it is 10.99.99.158 and UDP 7774, and then enable flow export per interface. Example follows, here I did it on Fortigate 100.

# show system sflow
config system sflow

set collector-ip 10.99.99.158

set collector-port 7774

end

# show system interface dmz1

config system interface

edit “dmz1”

set vdom “root”

set ip 10.99.99.254 255.255.255.0

set allowaccess ping https ssh snmp
set type physical
set wccp enable
set sflow-sampler enable
next
end

Fortigate article

Darknet can’t lie – most of the attacks, scans and other interesting things indeed come from behind the Great Firewall of China.

Working for ISP entitles me to various perks, one of them is unlimited connection to the Internet with wealth of unallocated yet IP addresses. So to use it somehow I set up a little Darknet (details what it means can be found here Darknet Project ) Most malware comes from Chinaand gather some statistics. First the volume of unsolicited and malicious traffic is staggering . Mostly it is traffic to Windows sharing – port 445 , then brute force – port 22, then strange ports used by new malware in the wild .Second, the interesting information pretty much stops here – as nothing listens on my side of the Darknet I don’t get more insight. As comes from this I am working on the next stage of the Darknet – HoneyNet. Once done, I’ll post here the findings.
To give you a glimpse of the Ips and ports involved in probes here is the non-sanitized sorted list of the alien IPs , destination ports, protocols and number of packets seen.This is the day’s worth statistics Bad guys and gals IPs
To get this list from Tcpdump capture I used one-liner: [root@darkstar]# tshark -n -r honey_bunny.cap42 | awk ' $3~/[0-9]+\./ {print $3,$6,$9}' | sort -n -k1,1 | uniq -c > Darknet_probing_IPs.txt

Funny things people do – how to turn Checkpoint UTM 450 into Windows Media player

Someone has finally found the best use of the Checkpoint UTM 450  –   turned it into the Windows Media player and recorded the instructions so others may follow.  In case you still wonder – yes , it certainly voids the warranty. Enjoy youtube.com And to those very few that will try to do it – word of caution: UTM 450 makes such noise you will not be able to put it in your living room.

You need no MX record to get mails

That one is funny. One client of ours that is actually themselves provide ISP services
in a far-far-away land asked to add PTR record for their mail server . But that was dull,
the interesting part was that their domain had absolutely NO MX record ! Only A record for the mail server host . I had always thought if there is no MX record for the destination domain sending mail server should bail out and I was wrong. A SMTP RFC 5321 actually states that if there no MX record exists for the domain the sender should try delivering the mail to A record of the domain RFC 5321 section 5 . Be aware though that MX record should be completely absent, so say if MX record does exist but points to a not responding server is a different case – in such case sender should fail the delivery.
The funny thing about that is that they have been working without MX record for about 2 years and have had no problems with receiving the mails, just amazing how  RFC-compliant mail servers in the wild are.

Skynet got blacklisted – Google mail servers entered RBL of Sorbs.net

When yesterday my client sent me the headers of blocked by eSafe (Aladdin) mails I was quite surprised – the message said ” Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 574 574 MAIL REFUSED – IP (74.125.82.172) is in RBL black list recent.spam.dnsbl.sorbs.net (state 18). ” What? Google servers got blacklisted ? No way .
I also expected Sorbs.net to be wiped out from the Earth rather quickly for such act of aggression against Skynet , also known as Google.com but nothing actually happened. So just for the fun of it I checked another IP of theirs – 74.125.82.48, also blocked. In short the class-C 74.125.82.0 got listed (screenshot follows). From
practical point of view – make sure if your device is using www.Sorbs.net to put this pool in exclusion list, as I did in the eSafe of the client.

« Older posts Newer posts »

© 2016 yurisk.info

Theme by Anders NorenUp ↑