Jan
26
2010

Enabling antispam or antivirus on the Checkpoint gateway blocks smtp or http traffic

Recently I was unplesantly presented with “it is not a bug ,it is a feature” case with the Checkpoint .
There was some UTM with TS (Total Security) valid license that includes antivirus and antispam services that client paid for and even asked to enable. So far so good. Part of the routine I checked on Gateway properties Antivirus and Antispam features , in Content inspection picked this UTM as enforcing Antispam/Antivirus policy , did install and .. got a call from the client that they can’t send/receive mails . In SmartView Tracker I saw the error of invalid license (it was the most clever disguise Checkpoint could come up with) , on command line fw monitor proved connections to port 25 arrive perfectly and pass pre/post insert points inbound but then nothing happens. Trying to telnet port 25 to the external ip of the mail server got me opened session , then connection was reset.
Only with the help of Checkpoint support (that actually were surprised that after all these years with their
product I haven’t seen this “feature” yet) did I find that issue is known one and caused by that to represent the mail server in LAN I created a MANUAL NAT rule . And ANY security server inside Checkpoint has to
know from security rules or from object properties its ip before and after NAT. Of course this info is
not to be located in any guides.
So to fix the situation you have to either :

  • replace manual NAT rules with automatic ones;
  • in security rules relevant to the server in question use BOTH internal and external IPs (that was
    what I did and it works ever since – see screenshot below).

I did the rules similar to this:
rulebase for SMTP server inside

NB there exist Secureknowledge base articles for it :
sk34862
sk32198

PS I talk here about SMTP but enabling Antivirus for the webserver in LAN with static NAT will have the same
devastating result.

Jan
23
2010

IP Options are evil – drop them , drop them on Cisco Asa/IOS Microsoft ISA Juniper or Checkpoint

As you probably noticed IP header has variable length placeholder for the IP Options field. It has been there since the beginning , once a good idea for debug now turned into trouble. RFC 791 states that hosts/routers supporting IP protocol must implement Ip Options filed . It is up to the vendor to decide what to do with this optional field, but it must understand it. Still, wouldn’t be a problem if not modern architecture of the routing equipment that was designed to do most efficiently Routing , i.e. pass from interface to interface gigabytes of traffic. Therefore routing functions are highly optimized and most of the time are implemented in hardware . All other types of traffic unfortunately are not, and in most of the cases processing , lets call it Control traffic, is being left to poor router CPU and done in software. That brought the troubles into the IP world – relatively small amounts of control traffic (including Ip Options packets) may bring down otherwise
powerful router in just minutes.
To prevent this attack vendors implemented protection measures to drop entirely or selectively IP packets that has Ip Options filed set. Below is quick cheat sheet how to do it in some gear :

Checkpoint NG/NGX – packets with Ip Options are dropped by default except for the “Router Alert” option (0×94) for the IGMPv2 and PIM protocols [or so CP claim, will have to verify later] and not even logged. To start logging dropped packets go to Policy -> Global Properties -> Log and Alerts -> check Ip dropped packets : Log

There is a value related to it that is on by default : Global Properties -> SmartDashboard customization -> Advanced Configuration -> Configure -> Firewall 1 -> Stateful inspection -> enable_ip_options (check/uncheck) but unchecking it removes from firewall VM chain module that inspects these Options at all and all Ip Options packets are dropped . So all packets bearing Ip Options are happily dropped even before security rules , here:

[Expert@splat60]# fw ctl chain
in chain (9):
0: -7f800000 (9095dd60) (ffffffff) IP Options Strip (ipopt_strip)
1: – 1fffff6 (9095ee80) (00000001) Stateless verifications (asm)

Also Checkpoint say you can decide which Ip Options will be allowed later BUT only when installing the firewall: “The set of permitted options must be configured during installation … the enable_ip_options setting in SmartDashboard is then used to enable or disable this functionality. Contact Check Point support for instructions on configuring the set of allowed IP options.”

Microsoft ISA 2000 server:
– If Enable Packet Filtering is not checked then do it in IP Packet Filters -> Properties – > General tab. On the Packet Filters tab check Enable Filtering IP Options .
Microsoft ISA 2004 Server:
- IP options filtering is enabled by default
- Go to Configuration node of the server in question in Management console -> General -> Additional Security Policy
Define IP Preferences . Here you will have 3 options to deal with Ip Options packets:
a) Deny packets with any IP options;
b) Deny packets with selected IP options;
c) Deny packets with all except selected IP options
The same options are available in ISA 2006 , click on Configure IP Protection link – > IP Preference settings
IOS Cisco router :
see my other blog – to be filled later
Cisco ASA :
see my other blog – to be filled later
Juniper router:
You just add ip-options term to the filter and apply it to the interface of interest. In the example below I block only Route Record type of Ip Options, if you use any then it will block any type:

[edit firewall family inet filter NOICMP term 3]
firewall {
    family inet {
        filter NOICMP {
            term 1 {
                from {
                    address {
                        192.168.2.100/32;
                    }
                }
                then {
                    reject;
                }
            }
            term 2 {
                from {
                    ip-options route-record;
                }
                then {
                    reject;
                }
            }
            term 3 {
                from {
                    address {
                        192.168.2.0/24;
                    }
                }
                then accept;
            }
        }
    }
}

Apply to the interface:

interfaces {
    em0 {
        unit 0 {
            enable;
            family inet {
                filter {
                    input NOICMP;
                }
                address 192.168.2.133/24;
            }
        }
    }

Other possible arguments to ip-options clause:

set term 3 from ip-options ?

Possible completions:

              Range of values
  [                    Open a set of values
  any                  Any IP option
  loose-source-route   Loose source route
  route-record         Route record
  router-alert         Router alert
  security             Security
  stream-id            Stream ID
  strict-source-route  Strict source route
  timestamp            Timestamp 

Windows 2008.
By default it doesnt allow/forward packets with Source Routing set, and that's good. For completeness
here is how to enable (or check whether it is enabled) source-routed forwarding:
BillG> netsh interface ipv4 set global sourceroutingbehavior=drop| forward| dontforward
- or-
Registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameter
Key: DisableIPSourceRouting
DWORD value: 0

Verify:
In Security any measure/protection/method is as good as the proof you can present that it actually works.

Windows:
- Ping with Record Route field set:
BillG> ping –r 9 192.2.2.1
- Ping with Strict Routing field set:
BillG> ping –k <1st_hop_router_IP> <2nd_hop_router_IP…> <target>
- Ping with Loose Routing field set:
BillG> ping -j <1st_hop_router_IP> <2nd_hop_router_IP…> <target>
- Ping with Timestamp option set:
BillG> ping –s 3 8.8.8.8

Linux:
- Ping with Record Route field set:
root@darktstar:~/nmap# ping -R 8.8.8.8
- Ping with Timestamp option set:
root@darkstar:~/nmap# ping -T tsonly 8.8.8.8

Linux,BSD,Unix :
This handy utility sends bunch of packets to the target to test what Ip Options the target supports:
freebsd# fragtest ip-opt 192.168.2.133
ip-opt: sec lsrr ts esec cipso satid ssrr

I run fragroute above against Juniper (8.3) that was configured in the example earlier to block only Record Route option, as you can see it is indeed missing in the output list that enumerates what Ip Options the target supports [ see Reference for fragroute details]

References for further details:
Juniper: JUNOS Enterprise Routing, 1st Edition, By Doug Marschke; Harry Reynolds, 2008
Microsoft ISA : Microsoft® ISA Server 2006 Unleashed ,By Michael Noel, 2007
Fragroute http://monkey.org/~dugsong/fragroute/
Windows 2008: Windows® Server 2008 TCP/IP Protocols and Services,By Joseph Davies, 2008

Jan
21
2010

Cisco log: Missing cef table for tableid 65535 during CEF samecable event

Today I’ve noticed some strange error on my Cisco 1841 router :

%FIB-4-FIBCBLK: Missing cef table for tableid 65535 during CEF samecable event

After searching the net, i’ve found some Cisco bug that describes this.

“FIB-4-FIBCBLK errors with dns view
Symptoms

Message “%FIB-4-FIBCBLK: Missing cef table for tableid 65535 during CEF samecable event” displayed on the console logs.

Conditions

The message seems to be generated anytime a dns request is made to the router where the router then has to use the dns forwarder

Workaround
No workaround”

(Source)

This happens when you have DNS server on the device and it needs to grab the answer from the device configured DNS server (A.K.A DNS Forwarder), and each request will cause this error log.

According to Cisco, the affected device list does not include my MD release – 12.4(25b), however i do see it . (List)

List of IOS with the Fix :
12.2(33)XNE
12.4(24.6)T1
12.4(15)T9
12.2(32.8.11)SR183
12.2(32.8.1)REC186
12.4(20)T3
12.2(33.1.3)MCP5
15.0(1)M
12.4(24)T1
12.2(32.8.1)REE186
12.4(22)T2
12.4(22)MDA1
12.4(24)YG
12.4(24)GC1
12.4(22)XR
12.4(24)MD
12.4(22)YE2

Jan
19
2010

Scheduled/Daily Reboot of FortiGate

Recently I had to do late night restart of a Fortigate and was looking for “Reload in…”
I found it, but in Fortigate it is a little different.
It’s called Daily Restart, and if you want to use it once you need to remember to remove this command.

config system global
set daily-restart enable
set restart time 04:00
end

Now the FortiGate is configured to reboot at 4 AM (System Time).
Don’t forget to update the system clock (Use NTP, Always keeps it synced)

Jan
18
2010

Cisco ASA privilege separation for a local user or read only user on ASA

Today I had the need to create a user in ASA that would have read-only permissions and also could issue
only 2 commands: show run and show conn. Here is how to do it.
We talk here about user with local authentication (with TACACS it is much easier).
Just as in Cisco routers you assign specific command to some privilege level different from its default level , then create user with this privilege level :

1) Assign command to specific privilege level ( I pick here level 3 , but it may be any but 15):

(config)# privilege show level 3 mode exec command running-config
(config)# privilege show level 3 mode exec command conn

2) create username with privilege of the command you want him to give
(config)# username Joedoe password asdlgfuwe privilege 3

Now you have 2 options – create general enable password for this given level (3 here) ,so
any user after successful login can enter > enable 3 and enter it to get to level 3 enable
mode. Or , as I did here, not creating enable level 3 password at all and the user will have to enter its
privilege level using login command.
3) now user can connect by ssh (if allowed by Ip of course) :
#ssh Joedoe@10.10.10.7
Joedoe@10.10.10.7password:<enter user’s pass here>
ASA> login
Username: Joedoe
Password: **********
# sh curpriv
Username : Joedoe
Current privilege level : 3
Current Mode/s : P_PRIV

Reference:
Cisco ASA Configuration Guide 8.0

Dec
31
2009

Print rulebase in Checkpoint

The best place to hide something is to place it before your eyes. Thanks to theacademypro.com I discovered a cool feature of the SmartDashboard – ability to print rules directly from the Dashboard , you just go to File -> Print -> Rule Base.. and that’s it. Just amazing , I have been using Dashboards  throughout these years hundreds of times and never noticed it. Seems like you have to learn all your life to just return to the place you started from :) .
Happy New Year All!

Dec
30
2009

Checkpoint – back up centrally for recovery.

Backing up firewall configs for disaster recovery  is tedious and mundane task. And if you have enough firewalls doing it manually becomes impractical . To address this case I set up a highly secured server that periodically runs script backing up the clients’ firewalls.

I use here poll model – this central server connects by SSH to the remote firewalls ,issues upgrade_export command then downloads backup using SCP and finally deletes the backup from the firewall itself.
Advantage of such a schema as opposed to the push model where firewalls push backups to the central server I see in that:
 - I can secure this server much more as no remotely accessible services are running (so no remote exploit is possible)
 - I can have rule in firewall before this server Inbound – > Deny Any Any
 - I centrally manage the backup script , if something changes I fix just one script .
Disadvantage – password to enter the firewalls is stored clear text in the script.
Now to the script – I did it in Expect to make life easier , in short it just emulates interactive login by SSH, then runs upgrade_export command, downloads by SCP the backup file, also creating file with md5sum of the backup and downloading it as well. The final action is to login by SSH back and remove the backup file from the firewall.
Naming it does by adding current date to the IP of the firewall. No error checking is done.
 
Files used in script:
hosts  – file containing IPs of the firewalls to backup in the form <IP of firewall> one per line .

The script goes next (at the end you can download script as file to fix lines wrapping):

#!/usr/local/bin/expect
#set timeout to suffice for the largest backup file to download
set timeout 3000
 
#set password to enter the firewall
set password “password”
set username  “admin”
#set format for naming files
set timeand_date [clock format [clock seconds] -format %B-%Y-%m-%d]
#open hosts file that contains IPs of the firewalls and read it in a loop
set ff [open "hosts" r]
while {[gets $ff hostName] >= 0} {
 
 puts “Entering $hostName”
 spawn ssh -l $username $hostName
 expect {
        {[Pp]assword:} { send “$password\r” }
 ”(yes*no)” { send “yes\r”
              expect {[Pp]assword:} {
 send “$password\r”
 }
}}
 
#increase timeout of SSH session
 expect {*#}  {
 send “TMOUT=900\r” }
 expect {*#}  {
 send “export TMOUT\r”}
#Create backup directory
 expect {*#}  {
 send “mkdir /var/Upgrade_export_backups\r”  }
 expect {*#}  {
 send “cd /var/Upgrade_export_backups\r”  }
#Issue the upgrade_export command
 expect {*#}  {
 send “\$FWDIR/bin/upgrade_tools/upgrade_export $timeand_date$hostName\r”  }
 expect {
{ready} {
 send “\r”      }
 {(y/n) [n]} {
 send “yes\r” }
}
#Calculate md5sum of the newly created backup file and save it to file
expect {*#} {
send “md5sum $timeand_date$hostName.tgz > $timeand_date$hostName.md5sum\r”}
 
expect {*#} {
  send “exit\r”}
  spawn  scp  $username@$hostName:/var/Upgrade_export_backups/\{$timeand_date$hostName.md5sum,$timeand_date$hostName.tgz\}    .
expect {
        {[Pp]assword:} { send “$password\r” }
}
 expect {#}   {
 #send “exit\r”
}
 
 spawn ssh -l $username $hostName
 expect {
        {[Pp]assword:} { send “$password\r” }
 ”(yes*no)” { send “yes\r”
              expect {[Pp]assword:} {
 send “$password\r”
 }
}}
 
#remove created backup file
 expect {*#}  {
 send “cd /var/Upgrade_export_backups\r”  }
 expect {*#}  {
 send “rm -f $timeand_date$hostName.tgz\r”  }
 

 expect {*#}  {
 send “exit\r”  }
 

}
close $ff
 interact

Script as a file

Dec
19
2009

Checkpoint winscp troubles

Checkpoint firewalls have 3 means of  transferring files in/out – ftp (client ) , SCP and SFTP (haven’t tried it yet) .

At some stage of the debug/upgrade process you will have to move files in either direction. The most secure is SCP protocol. On windows platforms picking the GUI SCP client is not hard – you only have WinSCP as your choice. And being otherwise  reliable and easy to use software it just doesn’t work with Checkpoint many times.  To fix this is easier than you can think of.

But first few prerequisites:

  To allow SCP connection to the firewall you have to :

       – create file named /etc/scpusers

       – add to it user per line – with which user you will be connecting for SCP session

       – make sure that for this user(s) shell is set to /bin/bash in /etc/passwd file

       – and of course allow SSH protocol connection from your host to the firewall.

After all the above done you connect using WinSCP, all goes well, try to download some file and …

Winscp fails when trying to download/upload some file from/to firewall

Error happens…
The easiest way (and the only one I found so far ) is to .. NOT use WinSCP but instead use wonderful
software PSCP from Putty authour that doesn’t have GUI but works flawlessly with Checkpoint.
Download it here www.chiark.greenend.org.uk/~sgtatham , read instructions and have no regrets ever after.

Dec
15
2009

ARP table overflow in Checkpoint and Linux in general

Not specific to the Checkpoint but rather any Linux-based system issue, still people often
forget  about that and look for the Checkpoint-specific solutions to that , so to help with  this search I wrote the note 
how  to fix it  below:
Problem  usually shows itself in randomly distributed inability of stations to pass the firewall, slowness and other network problems follow.
In /var/log/message you see the following record:

kernel: Neighbour table overflow.
That means ARP table has reached its maximum allowed limit and no new ARP entries are being learnt.

You can either find reason for sudden ARP requests influx or adjust ARP table limts accordingly.
You adjust ARP table limits either editing  this file (then change survives reboot):

/etc/sysctl.conf
If not present add these lines at the end, and try not to delete by mistake anything:
net.ipv4.neigh.default.gc_thresh1 = 1024
net.ipv4.neigh.default.gc_thresh2 = 4096
net.ipv4.neigh.default.gc_thresh3 = 16384

 - Then issue command:
  # sysctl -p
- Or if you want to increase it temporarily until reboot:
#echo 1024 > net.ipv4.neigh.default.gc_thresh1
#echo 4096 > net.ipv4.neigh.default.gc_thresh2
#echo 16384 > net.ipv4.neigh.default.gc_thresh3

And the short explanation follows.
gc in the above means Garbage Collector (GC).
net.ipv4.neigh.default.gc_thresh1  – sets minimum number of ARP entries in the cache.
Until this value is reached GC doesnt run at all.
net.ipv4.neigh.default.gc_thresh2  – sets soft maximum number of ARP entries in the cache.
GC allows ARP cache to pass this limit for 5 seconds and then starts cleaning.
net.ipv4.neigh.default.gc_thresh3  -  sets hard limit of ARP entries in the cache.
After it is reached no more ARP entries are being added.

Dec
14
2009

Increase log size in eSafe

Session logs in eSafe are  essential for debugging  . By default ,nevertheless each Session log file is limited to 100 megabytes in size , after reaching this limit eSafe stops writing the Session logs until the next log rotation – that is midnight.

To fix this , edit  the file /opt/eSafe/esafecfg.ini:

[ALERT GENERAL]  
Size limit=2
Last overflow=0
Minimum free disk space=2000
Block if internal error=1
File name=^M
Report days=10
Session log days=7    = >  Session log days= 365
Report max length=100
Session log max length=100  =>  Session log max length=500
Log sessions=1
Detailed log sessions=0
Log System Info Interval=10
MMS block if internal error=1
SessionLog To EventLog=0