Fortigate to Fortimanager management tunnel connection debug how-to


When the policy install fails on Fortimanager, it may mean many things as the process is quite complex with database/policy verification. But frequently, it happens because the communication tunnel between Fortimanager and Fortigate is down. The tunnel works on port 514, is encrypted (so we cannot see the contents) and can fail for various reasons. The first step I do is to check whether the tunnel is up or down. Here is example of such situation where client suspected tunnel was down, but I showed her it was not. I run all the commands on Fortigate only.

First, let's make sure configuration is correct on the Fortigate:

show sys central-management

config system central-management
    set type fortimanager
    fmg "10.72.38.11"   <--IP address of FMG
    set fmg-source-ip 10.13.91.99 <-- setting source IP is not a must of course, depends on the admin decision
end

All seems in place, so the next step is to try and ping the Fortimanager from the Fortigate. This does NOT pass inside the management tunnel, but at least verifies reachability:

exe ping-options source 10.13.91.99

exe ping 10.72.38.11

PING 10.72.38.11 (10.72.38.11): 56 data bytes
64 bytes from 10.72.38.11: icmp_seq=0 ttl=59 time=22.8 ms
64 bytes from 10.72.38.11: icmp_seq=1 ttl=59 time=27.3 ms
64 bytes from 10.72.38.11: icmp_seq=2 ttl=59 time=32.7 ms

Ping works, looks good so far, next is the sniffer to actually see if port 514 packets are being exchanged:

diag sniffer packet any 'host 10.72.38.11' 4

interfaces=[any]
filters=[host 10.72.38.11]
4.065959 port1 out 10.13.91.99.2871 -> 10.72.38.11.514: udp 600
5.095764 port1 out 10.13.91.99.2871 -> 10.72.38.11.514: udp 392
8.175664 port1 out 10.13.91.99.20987 -> 10.72.38.11.514: psh 4115781506 ack 888833763 
8.196360 port1 in 10.72.38.11.514 -> 10.13.91.99.20987: ack 4115781547 
8.196386 port1 in 10.72.38.11.514 -> 10.13.91.99.20987: psh 888833763 ack 4115781547 
8.196449 port1 out 10.13.91.99.20987 -> 10.72.38.11.514: psh 4115781547 ack 888833850 
8.259182 port1 in 10.72.38.11.514 -> 10.13.91.99.20987: ack 4115781600 
8.425671 port1 out 10.13.91.99.17622 -> 10.72.38.11.514: psh 1225476456 ack 239423256 
8.447417 port1 in 10.72.38.11.514 -> 10.13.91.99.17622: ack 1225476497 
8.447438 port1 in 10.72.38.11.514 -> 10.13.91.99.17622: psh 239423256 ack 1225476497 
8.447500 port1 out 10.13.91.99.17622 -> 10.72.38.11.514: psh 1225476497 ack 239423343 
8.504843 port1 in 10.72.38.11.514 -> 10.13.91.99.17622: ack 1225476550 
9.085853 port1 out 10.13.91.99.2871 -> 10.72.38.11.514: udp 793
10.115769 port1 out 10.13.91.99.2871 -> 10.72.38.11.514: udp 392
14.105856 port1 out 10.13.91.99.2871 -> 10.72.38.11.514: udp 603
15.135762 port1 out 10.13.91.99.2871 -> 10.72.38.11.514: udp 392

Indeed, Fortimanager talks with the Fortigate on the expected port 514.

And the last step to make sure that management tunnel works as expected is to run debug:

diag deb app fgfm 255

diag deb enable

upd_cfg_extract_ids_db_version[367]-version=05006033NIDS02402-00006.00741-1512010230
FGFMs: client:send:
keepalive
checksum=57 ba 31 52 f0 a2 0c 7a c3 e7 b8 ff 40 d5 1a 22
ipsversion=6.00741(2015-12-01 02:30)


FGFMs: client:
reply 200
request=keepalive
cur_tun_serial= 

FGFMs: [fgfm_chan_msg_handler:607] peer close channel: local=5502, remote=10151, body_len=0
FGFMs: Destroy tcp channnel local_id=5502, remote_id=10151, sock_rd=6076, sock_wr=203, sock_size=0.
FGFMs: Destroy stream_svr_obj
FGFMs: Destroy chan local=5502, remote=10151, in=203, ack=203, out=6076,acked=6076,inbuff=-1.
FGFMs: client:
get connect_tcp
localid=10152
chan_window_sz=32768
deflate=gzip
tcp_port=80


FGFMs: [fgfm_chan_msg_handler:607] peer close channel: local=5503, remote=10152, body_len=0
FGFMs: Destroy tcp channnel local_id=5503, remote_id=10152, sock_rd=581, sock_wr=204, sock_size=0.
FGFMs: Destroy stream_svr_obj
FGFMs: Destroy chan local=5503, remote=10152, in=204, ack=204, out=581,acked=581,inbuff=-1.

upd_cfg_extract_ids_db_version[367]-version=05006033NIDS02402-00006.00741-1512010230
FGFMs: client:send:
keepalive
checksum=57 ba 31 52 f0 a2 0c 7a c3 e7 b8 ff 40 d5 1a 22
ipsversion=6.00741(2015-12-01 02:30)


FGFMs: client:
reply 200
request=keepalive
cur_tun_serial= 

The important part here is keep-alive exchange. And as we can see it works just fine - Fortigate received reply 200 which means "Success". The tunnel works correctly, case closed.