Failed to connect to Fortiguard servers verification and debug
Fortiguard is a subscription based service from Fortinet, where your Fortigate queries their servers in real-time for various services:
- Periodic checking of Fortigate subscription/license validity for Web Filtering/AppControl/AntiVirus/AntiSpam/DNS Filtering.
- Real-time querying for visited by users web sites rating.
- Periodic signatures updates for IPS/AppControl/AntiVirus.
Most critical of them is Web Filter rating query - if your Fortigate cannot get answer what category the web site belongs to, access to this web site will be blocked by default. It means that if for any reason Fortigate cannot reach Fortiguard servers and it has security rules with Web Filtering by Category configured - those rules will BLOCK users access to ANY website, not just malicious ones.
First, as emergency but not advisable measure, you can click in Security Profiles -> Web Filter -> <Profile Used in Security Rules on Allow websites when a rating error occurs. This will ALLOW access to any website if a Fortigate cannot get rating from the FortiGuard. Also you can remove Web Filter Profile from Security Rules, but it is even worse security-wise.
So how do you debug such a situation?
First, check status of license/subscription and FortiGuard connection status in System -> FortiGuard - the Web Filtering status should be in green. This checks subscription license status, but not always detects connection to the FortiGuard status. If you see it red, it is most probably a license/subscription issue to be checked with Fortinet TAC, as subscription checks are done once in a while and are cached. To check actual connectivity to the FortiGuard servers - on the same page, under Filtering subsection, there is Test Connectivity button to push. It should return status as Up/green. Also pay attention to the widget on the same page in the right bottom corner FortiGuard Filter Rating Servers, it shows real time stats and IP addresses of the servers the Fortigate is trying to reach. If timings are unusually high and in red, there could be network connectivity problem, we will look at next.
First step in checking connectivity to FortiGuard servers is successful DNS resolving by Fortigate of the following hostnames:
- service.fortiguard.net
- update.fortiguard.net
- guard.fortinet.net
Even better check is to run ping exe ping to all the hostnames above to see if the Fortigate can resolve AND can reach them. The most important of them being service.fortiguard.net.
It is also worth checking if there is a known outage on the Fortinet side - https://status.fortimonitor.forticloud.com/fortiguardsdns
If the resolving is OK, next step is this:
diagnose debug rating
This will show a list of FortiGuard servers this Fortigate is trying to reach for Web Filtering rating and their status.
The output will look like:
Locale : english
Service : Web-filter
Status : Enable
License : Contract
Service : Antispam
Status : Disable
Service : Virus Outbreak Prevention
Status : Disable
Num. of servers : 6
Protocol : https
Port : 443
Anycast : Disable
Default servers : Included
-=- Server List (Sun Feb 21 09:20:14 2021) -=-
IP Weight RTT Flags TZ Packets Curr Lost Total Lost Updated Time
149.5.232.18 10 84 1 1060088 0 815 Sun Feb 21 09:19:19 2021
12.34.97.18 70 566 -5 34828 0 6 Sun Feb 21 09:19:19 2021
65.210.95.234 70 572 -5 34165 0 418 Sun Feb 21 09:19:19 2021
210.7.96.18 70 1439 9 30847 0 1 Sun Feb 21 09:19:20 2021
96.45.33.68 100 801 -8 33731 0 8 Sun Feb 21 09:19:19 2021
173.243.138.210 100 821 DI -8 33874 0 22 Sun Feb 21 09:19:19 2021
Here:
Status - shows if Web Filtering as a service is enabled.
Protocol - via what protocol this Fortigate is trying to reach FortiGuard servers (more on this below).
Anycast - whether this Fortigate is trying to reach Anycast servers of FortiGuard (more on this below).
Server List - actual list of FortiGuard servers that this Fortigate was/is trying to reach. Here most important is status legend:
- F: failed, bad - Fortigate tried few times to reach this server to no avail. Note that it is bad only if ALL servers in the list have this status. It is OK if only few of the servers are unreachable.
- D: this server was successfully resolved from FQDN to its IP address, but it does not indicate its reachability yet.
- I: server to which Fortigate tries to initiate connection, most frequently goes with D,it does not indicate if a server is working or not yet.
- T: server was found, it answered, and is now being "timed", i.e. its answer time/RTT is being measured.
- TZ: Time Zone, while not a status indicator, Fortigate tries and prefers servers with the least time zone difference in hope of geographic proximity. Therefore, it is quite important to set correctly the time zone for your Fortigate.
Fortigate communicates for its functions with just one server at a time - the one on top of the list. The rest of the servers are being constantly monitored and their RTT, and packet loss measured. If the top-list server fails, it will be replaced with the next best one and so on. We do not have capability to influence this server list manually.
So if all servers in the list have F(ailed), what do we do next?. This may mean either all Fortiguard servers at the Fortinet side are down (less likely), or that this Fortigate has the problem of reaching them at the network level.
Fortigate can use several ports to talk to Fortiguard servers (or Fortiguard Distribution Network as they call it) - 53, 8888, 443, the default being 8888. The port 53 is a well known DNS protocol/port, only that Fortigate uses proprietary UDP/53 obfuscated/encrypted protocol to query the servers, and for this reason some IPS/anti-DDoS/etc protections on the way from Fortigate to FortiGuard may mark such traffic as malicious and drop it. You can check if it is the case by going to System -> FortiGuard -> Filtering and change (if set so) from port 53 to port 8888. On newer FortiOS versions (6.4 and up) they moved this to CLI only: config sys fortiguard then set port 53|8888|443. So, as first debug measure it is recommended to try all possible ports and see if status of connection to the FortiGuard servers changes. Note about protocol I mentioned before - in 6.4 and newer they added option to force the communication to FortiGuard servers to be a valid HTTPS traffic, which is most likely to pass the Internet successfully. For this you have to enable it (in addition to setting port to 443) via CLI: config sys fortiguard, then set protocol https end.
Important note if you have VDOMs enabled - all communication to the Fortiguard network is initiated from management/root VDOM only! The frequent human error I've seen - someone by mistake changes management domain to the VDOM that has no/limited access to the Internet and as a consequence, it cannot reach FortiGuard network. Very common, indeed. To verify who is the management VDOM:
config sys global
# show full | grep management-vdom
set management-vdom "root" <-- THIS IS THE VDOM THAT WILL COMMUNICATE WITH FORTIGUARD
Anycast servers - starting with FortiOS 6.4 the default setting to reach FortiGuard is anycast. The intention was good - to improve reachability of FortiGuard servers, but unfortunately the implementation did not live up to the expectations. More often than not it actually creates a problem in reaching the Fortinet servers. It may improve in the future, but for now my advice is to disable anycast and switch back to unicast servers. You do so in CLI:
config system fortiguard
set fortiguard-anycast disable
set protocol udp
set port 8888
set sdns-server-ip 208.91.112.220 <-- IMPORTANT TO ADD THIS OR ANY OTHER FDN SERVER TO PREVENT DOWNTIME!
end
This configuration above will cause Fortigate to disable anycast, then reach the specified server (here 208.91.112.220), download from it the full list of available unicast servers and use them.
Sum up of steps to fix FortiGuard failed connection situation:
- Check that FortiGuard license on the Fortigate is in green.
- Check if there is an outage on Fortinet side: https://status.fortimonitor.forticloud.com/fortiguardsdns
- Make sure Fortigate can DNS resolve update.fortiguard.net, service.fortiguard.net
- Make sure Fortigate can ping service.fortiguard.net
- Try changing communication with FortiGuard port between 53, 8888, 443
- Make sure (if VDOMs are enabled) that management VDOM has access to the Internet
- Disable anycast and enable unicast for FortiGuard services.
Follow me on https://www.linkedin.com/in/yurislobodyanyuk/ not to miss what I publish on Linkedin, Github, blog, and more.