Fortigate BGP - configure and debug
Everyone today speaks BGP: Cisco ,Juniper and ScreenOS firewalls, Fortigate does it, even SonicWall have it as planned feature. The opportunity to see how it works on Fortinet Fortigate firewall recently presented itself and here is the sum up of how I configured and debugged Fortigate BGP set up.
Task at hand: configure on Fortigate the BGP peering with Bogon Route project by Team Cymru https://team-cymru.com/community-services/bogon-reference . In few words they advertise to you routes that are never to be seen in your network for legitimate reasons. Those are networks not only from RFC 1918 but those reserved by RIPE for special purposes, and those unallocated to anyone as of now. What we need to know for this set up is this:
- They advertise all the networks with
no-exportcommunity - also they attach
65333:888community - they use md5 password authentication
- they don't expect you to advertise to them anything
- in advertised networks next hop is their advertising router
- their AS number is 65333
Based on all the above my Fortigate BGP peer had to :
- enable multihop eBGP peering
- use MD5 password authentication
- have route-map to attach
no-exportcommunity so that we don't inadvertently advertise learned routes to other peers ( just safety net , in case their BGP peer stops attaching no-export community to their routes) - set next hop for the learned routes to Null 0 interface (Cisco naming, Fortigate has 'blackhole' instead).
Let's start configuring something. Important surprise here – in Fortigate GUI regarding BGP you can only set 3 parameters: As number , Peer Ip and networks to be advertised, the rest is to be done on the command line (new versions of FortiOS add more, but still CLI is the way to go). So here it goes:
1.Configure route-map to set no-export community on learned networks and force next hop to be some reserved Ip (192.0.2.1 ) that in turn is statically routed to Null interface ,
config router route-map
edit "NO-EXPORT"
config rule
edit 3
set set-community "no-advertise"
set set-ip-nexthop 192.0.2.1
next
end
next
End
2.Configure BGP peer
(root) # show router bgp
config router bgp
set as 65002
config neighbor
edit 84.22.96.5
set ebgp-enforce-multihop enable
set remote-as 65333
set route-map-in "NO-EXPORT"
set password "yuiyui"
next
end
config redistribute "connected"
set status enable
end
3.Configure static blackhole route for the reserved IP used as the next hop for this.
(root) # sh router static
config router static
edit 3
set blackhole enable
set dst 192.0.2.1 255.255.255.255
next
End
Verification.
All configs are as good as the proof that they work.
- List briefly all the peers
(root) # get router info bgp summary
BGP router identifier 10.250.250.2, local AS number 65002
BGP table version is 159
2 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
84.22.96.5 4 65333 4 6 159 0 0 00:00:48 0
Total number of neighbors 1
- List all BGP neighbors and their peering state:
My-FG (root) # get router info bgp neighbors
BGP neighbor is 84.22.96.5, remote AS 65333, local AS 65002, external link
BGP version 4, remote router ID 84.22.96.5
BGP state = Established, up for 00:00:58
Last read 00:00:58, hold time is 180, keepalive interval is 60 seconds
Configured hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
Route refresh: advertised and received (old and new)
Address family IPv4 Unicast: advertised and received
Received 4 messages, 0 notifications, 0 in queue
Sent 6 messages, 0 notifications, 0 in queue
Route refresh request: received 0, sent 0
Minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
BGP table version 160, neighbor version 159
Index 3, Offset 0, Mask 0x8
Community attribute sent to this neighbor (both)
Inbound path policy configured
Route map for incoming advertisements is *NO-EXPORT
0 accepted prefixes
19 announced prefixes
Connections established 1; dropped 0
External BGP neighbor may be up to 255 hops away.
Local host: 10.250.250.2, Local port: 9188
Foreign host: 84.22.96.5, Foreign port: 179
Nexthop: 10.250.250.1
- See the routes learned through the BGP protocol:
(root) # get router info bgp network
BGP table version is 161, local router ID is 10.250.250.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 5.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 14.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 23.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 31.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 36.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 37.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 39.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 42.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 49.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 100.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 101.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 102.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 103.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 104.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 105.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 106.0.0.0 192.0.2.1 0 0 65333 65333 i
*> 169.254.0.0 192.0.2.1 0 0 65333 65333 i
*> 172.16.0.0/12 192.0.2.1 0 0 65333 65333 i
*> 176.0.0.0/8 192.0.2.1 0 0 65333 65333 i
*> 177.0.0.0/8 192.0.2.1 0 0 65333 65333 i
*> 179.0.0.0/8 192.0.2.1 0 0 65333 65333 i
*> 181.0.0.0/8 192.0.2.1 0 0 65333 65333 i
*> 185.0.0.0/8 192.0.2.1 0 0 65333 65333 i
- List routes that are currently installed in the routing table that were learned by BGP:
(root) # get router info routing-table bgp
B 5.0.0.0/8 [20/0] via 192.0.2.1 (recursive is directly connected, unknown), 00:00:19
B 14.0.0.0/8 [20/0] via 192.0.2.1 (recursive is directly connected, unknown), 00:00:19
B 23.0.0.0/8 [20/0] via 192.0.2.1 (recursive is directly connected, unknown), 00:00:19
B 31.0.0.0/8 [20/0] via 192.0.2.1 (recursive is directly connected, unknown), 00:00:19
B 36.0.0.0/8 [20/0] via 192.0.2.1 (recursive is directly connected, unknown), 00:00:19
B 37.0.0.0/8 [20/0] via 192.0.2.1 (recursive is directly connected, unknown), 00:00:19
B 39.0.0.0/8 [20/0] via 192.0.2.1 (recursive is directly connected, unknown), 00:00:19
B 42.0.0.0/8 [20/0] via 192.0.2.1 (recursive is directly connected, unknown), 00:00:19
After all is configured and saved (and probably doesn't work) comes the bgp debug round.
- Set to INFO level the debug (by default the debug level is ERROR, which is usually not enough):
(root)# diagnose ip router bgp level info
- Enable bgp debug on the Fortigate:
(root)# diag ip router bgp all enable
- To verify that debug is on:
(root)# diag ip router bgp show
BGP debugging status:
BGP events debugging is on
BGP debug level: INFO
- If nothing happens you may try clearing all BGP sessions (WARNING: tears down all BGP sessions established on the Fortigate):
(root)# exec router clear bgp all
- To stop the debug:
(root)# diagnose ip router bgp all disable
-or-
(root)# diagnose debug reset
The good way to judge something new is to compare it with something you already know. To continue with that logic I cross-reference BGP debug output seen on Cisco with the one seen on the Fortigate BGP peer. That way you can decide what is more informative and who wins the race (Cisco of course, what you thought?).
Case 1
One of the peers is configured with wrong AS number.
In Fortigate you see this:
BGP: 84.22.96.5-Outgoing [FSM] State: Idle Event: 3
BGP: 84.22.96.5-Outgoing [NETWORK] FD=15, Sock Status: 0-Success
BGP: 84.22.96.5-Outgoing [FSM] State: Connect Event: 17
BGP: 84.22.96.5-Outgoing [ENCODE] Msg-Hdr: Type 1
BGP: 84.22.96.5-Outgoing [ENCODE] Open: Ver 4 MyAS 65002 Holdtime 180
BGP: 84.22.96.5-Outgoing [ENCODE] Open: Msg-Size 45
BGP: 84.22.96.5-Outgoing [DECODE] Msg-Hdr: type 3, length 23
BGP: %BGP-3-NOTIFICATION: received from 84.22.96.5 2/2 (OPEN Message Error/Bad Peer AS.) 2 data-bytes
Now let's compare to the debug from Cisco:
#debug ip bgp events
Mar 24 13:14:55.572: %BGP-3-NOTIFICATION: sent to neighbor 10.250.250.2
2/2 (peer in wrong AS) 2 bytes FDEA FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 FAEA
01B4 0AFA EA02 1302 0201 1400 0100 0132 0222 0012 0222 00
Case 2
MD5 authentication is set on Cisco but not on the Fortigate. Again for comparison BGP
debug from Fortigate and from Cisco.
Cisco:
Jan 5 10:42:14.299: %TCP-6-BADAUTH: No MD5 digest from 10.250.250.2 (1037) to 84.22.96.5(179)
Fortigate:
84.22.96.5-Outgoing [FSM] State: Connect Event: 9
BGP: [RIB] Scanning BGP Network Routes...
84.22.96.5-Outgoing [FSM] State: Connect Event: 9
BGP: [RIB] Scanning BGP Network Routes...
Case 3 (that actually happened when I configured this Fortigate) is mismatched MD5 password on either side
Fortigate:
Doing summary listing showed peering as down :
84.22.96.5 4 65333 934 1036 0 0 0 never Connect
Cisco:
*Mar 24 13:40:28.800: BGP: Regular scanner event timer
*Mar 24 13:40:28.800: BGP: Import timer expired. Walking from 1 to 1
*Mar 24 13:40:42.764: %TCP-6-BADAUTH: Invalid MD5 digest from 10.250.250.2(11064) to 84.22.96.5(179)
Case 4 On Cisco ttl-security is enabled while on Forigate ebgp multi-hop is not . There is no such thing as TTL security on the Fortigate by the way, all you can do to handle this state is enable ebgp-multihop and then it starts sending BGP packets with ttl = 255 .
Cisco:
Jan 7 13:01:36.992: %BGP-4-INCORRECT_TTL: Discarded message with TTL 2 from 10.250.250.2
Forigate:
BGP: 84.22.96.5-Outgoing [FSM] State: OpenConfirm Event: 11
BGP: 84.22.96.5-Outgoing [ENCODE] Msg-Hdr: Type 4
BGP: 84.22.96.5-Outgoing [ENCODE] Keepalive: 13548 KAlive msg(s) sent
84.22.96.5-Outgoing [FSM] State: OpenConfirm Event: 10
BGP: 84.22.96.5-Outgoing [ENCODE] Msg-Hdr: Type 3
BGP: %BGP-3-NOTIFICATION: sending to 84.22.96.5 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes
BGP: 84.22.96.5-Outgoing [FSM] State: Idle Event: 3
BGP: 84.22.96.5-Outgoing [NETWORK] FD=14, Sock Status: 111-Connection refused
BGP: 84.22.96.5-Outgoing [FSM] State: Connect Event: 18
Bonus Case Bug-not-a-feature thing on the Fortigate – when configuring MD5 password for BGP authentication you get Cross-Site vulnerability protection for free :) Don't ask me how XSS is connected to cli configuration of BGP …
set password <2AEARep>
Error:The string contains XSS vulnerability characters
value parse error before ''
Command fail. Return code -173
Update 2020: You can find more example configurations in my new article Fortigate BGP Cookbook of example configurations
And all Fortigate BGP debug commands in my Fortigate Complete Cheat Sheet: Fortigate debug and diagnose commands complete cheat sheet | PDF
Follow me on https://www.linkedin.com/in/yurislobodyanyuk/ not to miss what I publish on Linkedin, Github, blog, and more.