Fortigate to Fortimanager management tunnel connection debug how-to

in Fortigate
#Fortigate #Fortimanager

When the policy install fails on Fortimanager, it may mean many things as the process is quite complex with database/policy verification. But frequently, it happens because the communication tunnel between Fortimanager and Fortigate is down. The tunnel works on port 541, is encrypted (so we cannot see the contents) and can fail for various reasons. The first step I do is to check whether the tunnel is up or down. Here is example of such situation where client suspected tunnel was down, but I showed her it was not. I run all the commands on Fortigate only.

First, let's make sure configuration is correct on the Fortigate:

show sys central-management

config system central-management
    set type fortimanager
    fmg "10.72.38.11"   <--IP address of FMG
    set fmg-source-ip 10.13.91.99 <-- setting source IP is not a must of course, depends on the admin decision
end

All seems in place, so the next step is to try and ping the Fortimanager from the Fortigate. This does NOT pass inside the management tunnel, but at least verifies reachability:

exe ping-options source 10.13.91.99

exe ping 10.72.38.11

PING 10.72.38.11 (10.72.38.11): 56 data bytes
64 bytes from 10.72.38.11: icmp_seq=0 ttl=59 time=22.8 ms
64 bytes from 10.72.38.11: icmp_seq=1 ttl=59 time=27.3 ms
64 bytes from 10.72.38.11: icmp_seq=2 ttl=59 time=32.7 ms

Ping works, looks good so far, next is the sniffer to actually see if port 541 packets are being exchanged:

diag sniffer packet any 'host 10.72.38.11' 4

NOTE: Management tunnel works on port 541, but logs, if you enabled Fortianalyzer feature on the FMG are still being sent on the usual for Fortigate port 514. So make sure not to confuse them, as logs sending is a different daemon's work.

And the last step to make sure that management tunnel works as expected is to run debug:

diag deb app fgfm 255

diag deb enable

upd_cfg_extract_ids_db_version[367]-version=05006033NIDS02402-00006.00741-1512010230
FGFMs: client:send:
keepalive
checksum=57 ba 31 52 f0 a2 0c 7a c3 e7 b8 ff 40 d5 1a 22
ipsversion=6.00741(2015-12-01 02:30)


FGFMs: client:
reply 200
request=keepalive
cur_tun_serial= 

FGFMs: [fgfm_chan_msg_handler:607] peer close channel: local=5502, remote=10151, body_len=0
FGFMs: Destroy tcp channnel local_id=5502, remote_id=10151, sock_rd=6076, sock_wr=203, sock_size=0.
FGFMs: Destroy stream_svr_obj
FGFMs: Destroy chan local=5502, remote=10151, in=203, ack=203, out=6076,acked=6076,inbuff=-1.
FGFMs: client:
get connect_tcp
localid=10152
chan_window_sz=32768
deflate=gzip
tcp_port=80


FGFMs: [fgfm_chan_msg_handler:607] peer close channel: local=5503, remote=10152, body_len=0
FGFMs: Destroy tcp channnel local_id=5503, remote_id=10152, sock_rd=581, sock_wr=204, sock_size=0.
FGFMs: Destroy stream_svr_obj
FGFMs: Destroy chan local=5503, remote=10152, in=204, ack=204, out=581,acked=581,inbuff=-1.

upd_cfg_extract_ids_db_version[367]-version=05006033NIDS02402-00006.00741-1512010230
FGFMs: client:send:
keepalive
checksum=57 ba 31 52 f0 a2 0c 7a c3 e7 b8 ff 40 d5 1a 22
ipsversion=6.00741(2015-12-01 02:30)


FGFMs: client:
reply 200
request=keepalive
cur_tun_serial=

The important part here is keep-alive exchange. And as we can see it works just fine - Fortigate received reply 200 which means "Success". The tunnel works correctly, case closed.

Follow me on https://www.linkedin.com/in/yurislobodyanyuk/ not to miss what I publish on Linkedin, Github, blog, and more.