Fortianalyzer diagnose and debug cheat sheet

Mon 03 May 2021 in Fortinet

#Fortianalyzer

This cheat sheet as PDF: Fortianalyzer diagnose and debug cheat sheet

General Health

Command Description

Command	Description
get sys status	Get general information: firmware version, serial number, ADOMs enabled or not, time and time zone, general license status (Valid or not).
get sys performance	Detailed performance statistics: CPU load, memory usage, hard disk/flash disk used space and input/output (`iostat`) statistics.
exe top	Display real time list of running processes with their CPU load.
diag log device	Shows how much space is used by each device logging to the Fortianalyzer, including quotas.
exe iotop -b -n 1	Display and update every 1 second READ/WRITE statistics for all the processes.
diagnose system print cpuinfo	Display hardware CPU information - vendor, number of CPUs etc.
diagnose hardware info	Even more hardware-related info.
diagnose system print df	Show disk partitions and space used. Analog of the Linux `df`.
exe lvm info	Shows disks status and size
diagnose system print loadavg	Show average system load, analog to the Linux `uptime` command.
diagnose system print netstat	Show established connections to the Fortianalyzer, as well as listening ports. Every logging device can (and usually does) have multiple connections established.
diagnose system print route	Show routing table of the Fortianalyzer.

get sys status

Get general information: firmware version, serial number, ADOMs enabled or not, time and time zone, general license status (Valid or not).

get sys performance

Detailed performance statistics: CPU load, memory usage, hard disk/flash disk used space and input/output (iostat) statistics.

exe top

Display real time list of running processes with their CPU load.

diag log device

Shows how much space is used by each device logging to the Fortianalyzer, including quotas.

exe iotop -b -n 1

Display and update every 1 second READ/WRITE statistics for all the processes.

diagnose system print cpuinfo

Display hardware CPU information - vendor, number of CPUs etc.

diagnose hardware info

Even more hardware-related info.

diagnose system print df

Show disk partitions and space used. Analog of the Linux df.

exe lvm info

Shows disks status and size

diagnose system print loadavg

Show average system load, analog to the Linux uptime command.

diagnose system print netstat

Show established connections to the Fortianalyzer, as well as listening ports. Every logging device can (and usually does) have multiple connections established.

diagnose system print route

Show routing table of the Fortianalyzer.

Communication debug

Command	Description
diagnose test application oftpd 3	List all devices sending logs to the Fortianalyzer with their IP addresses, serial numbers, uptime meaning connection establishment uptime, not remote device uptime, and packets received (should be growing).
*diagnose debug application oftpd 8 <Device name>* diagnose debug enable	Real time debug of communicating with the Device name device.
*diagnose sniffer packet any "host IP of remote device"*	Sniff packets from/to remote device, to make sure they are sending each other packets. The communication is encrypted.
diagnose sniffer packet any "port 514"	Sniff all packets to/from port 514 used by Fortianalyzer to receive logs from remote devices.

Command

Description

diagnose test application oftpd 3

List all devices sending logs to the Fortianalyzer with their IP addresses, serial numbers, uptime meaning connection establishment uptime, not remote device uptime, and packets received (should be growing).

diagnose debug application oftpd 8 <Device name>

diagnose debug enable

Real time debug of communicating with the Device name device.

diagnose sniffer packet any "host IP of remote device"

Sniff packets from/to remote device, to make sure they are sending each other packets. The communication is encrypted.

diagnose sniffer packet any "port 514"

Sniff all packets to/from port 514 used by Fortianalyzer to receive logs from remote devices.

Logs from devices

Command	Description
diagnose test application oftpd 50	Show log types received and stored for each device.
diag log device	Shows how much space is used by each device logging to the Fortianalyzer, including quotas.
diagnose fortilogd lograte	Show in one line last 5/30/60 seconds rate of receiving logs.
diagnose fortilogd lograte-adom all	Show as table log receiving rates for all ADOMs aggregated per device type (i.e. rate for all Fortigates will be as one data per ADOM).
diagnose fortilogd lograte-device	Show average logs receive rate per device for the last hour, day, and week.
diagnose fortilogd lograte-total	Show summary log receive rate for all devices on this Fortianalyzer.

Command

Description

diagnose test application oftpd 50

Show log types received and stored for each device.

diag log device

Shows how much space is used by each device logging to the Fortianalyzer, including quotas.

diagnose fortilogd lograte

Show in one line last 5/30/60 seconds rate of receiving logs.

diagnose fortilogd lograte-adom all

Show as table log receiving rates for all ADOMs aggregated per device type (i.e. rate for all Fortigates will be as one data per ADOM).

diagnose fortilogd lograte-device

Show average logs receive rate per device for the last hour, day, and week.

diagnose fortilogd lograte-total

Show summary log receive rate for all devices on this Fortianalyzer.

Licensing

Command	Description
diagnose dvm device list	Look for the line There are currently N devices/vdoms count for license.
diagnose debug vminfo	Show report on Virtual Machine license: whether valid or not, type, licensed storage volume, licensed log receive rate, licensed maximum device count.

Command

Description

diagnose dvm device list

Look for the line There are currently N devices/vdoms count for license.

diagnose debug vminfo

Show report on Virtual Machine license: whether valid or not, type, licensed storage volume, licensed log receive rate, licensed maximum device count.

Example debug session on Fortianalyzer

Show connected to the FAZ devices

FAZ-AWS # diagnose test application oftpd 3
now = 1713716940(2024/04/21 19:29:00)
 #  DEVICE            CONN      HOSTNAME       IP             UPTIME    IDLETIME  #PKTS              BUFSZ (curr,avg,advice)
-------------------------------------------------------------------------------------------------------------------------------
 1  FGTAWSN1JDGCU42E  65535: 0  FGT-Perimeter  10.100.104.13  3h18m2s   1s        3048               512,0,0  (1)
 2                                             10.100.104.15  6h56m37s  1m42s     2027 Plain-Syslog  512,0,32768 (2)

Fortigate named "FGT-Perimeter" (IP 10.100.104.13) sending logs via OFTP native protocol
Linux server (IP 10.100.104.15), sending its logs via Syslog.

General state of FAZ (version, serial, HA status, license status)

FAZ-AWS # dia deb dis

FAZ-AWS #   get sys status
Platform Type                   : FAZVM64-AWSOnDemand
Platform Full Name              : FortiAnalyzer-VM64-AWSOnDemand
Version                         : v7.4.2-build2397 231220 (GA)
Serial Number                   : FAZAWSTA23002441
BIOS version                    : 04000002
Hostname                        : FAZ-AWS
Max Number of Admin Domains     : 5
Admin Domain Configuration      : Enabled
FIPS Mode                       : Disabled
HA Mode                         : Stand Alone
Branch Point                    : 2397
Release Version Information     : GA
Current Time                    : Sun Apr 21 19:39:34 IDT 2024
Daylight Time Saving            : Yes
Time Zone                       : (GMT+2:00) Jerusalem.
x86-64 Applications             : Yes
Disk Usage                      : Free 70.79GB, Total 78.19GB
File System                     : Ext4
License Status                  : Valid

Performance stats (appliance FAZ will have more data)

FAZ-AWS # get sys performance
CPU:
        Used:                   5.50%
        Used(Excluded NICE):    5.50%
                  %used   %user   %nice  %sys    %idle %iowait  %irq %softirq
        CPU0       5.05    3.24    0.00    1.80   94.95    0.00    0.00     0.00
        CPU1       5.96    4.69    0.00    1.08   94.04    0.00    0.00     0.18
Memory:
        Total:  10,041,896 KB
        Used:   5,416,028 KB    53.9%
        Total (Excluding Swap): 7,944,748 KB
        Used (Excluding Swap):  5,079,124 KB    63.9%
Hard Disk:
        Total:  81,983,896 KB
        Used:   7,742,552 KB    9.4%
        Inode-Total:    5,242,880
        Inode-Used:     26,347  0.5%
        IOStat:   tps     r_tps    w_tps    r_kB/s    w_kB/s    queue   wait_ms
                   8.9      3.2      5.7     236.9     312.8      0.2    20.6
Flash Disk:
        Total:  1,006,252 KB
        Used:   444,916 KB      44.2%
        Inode-Total:    65,536
        Inode-Used:     43      0.1%
        IOStat:   tps     r_tps    w_tps    r_kB/s    w_kB/s    queue   wait_ms
                   0.1      0.1      0.0      17.5       0.0      0.0     0.8

Running processes and CPU load

top - 19:42:52 up  7:13,  0 user,  load average: 0.24, 0.23, 0.19
Tasks: 234 total,   1 running, 232 sleeping,   0 stopped,   1 zombie
%Cpu(s):  2.3 us,  1.3 sy,  0.0 ni, 96.2 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7758.5 total,    135.6 free,   7118.2 used,   2722.2 buff/cache
MiB Swap:   2048.0 total,   1707.5 free,    340.5 used.    640.3 avail Mem

  PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND
 9859 root      20   0  177.6m 127.0m   2.0   1.6   7:41.94 S /bin/python /usr/local/lib/python3.11
 1727 root      20   0  259.9m  46.9m   0.7   0.6   1:08.81 S /bin/logfwd
 7833 postgres  20   0 1436.5m  27.8m   0.7   0.4   0:02.26 S postgres: postgres airflow 127.0.0.1(
 9886 root      20   0  176.5m 124.8m   0.7   1.6   1:52.72 S airflow scheduler -- DagFileProcessor
  245 root      20   0   95.4m  28.1m   0.3   0.4   0:35.78 S /bin/cmdbsvr
  750 redis     20   0   57.6m   9.5m   0.3   0.1   0:51.04 S /bin/redis-server 127.0.0.1:6379
 1573 root      20   0 1217.3m 874.5m   0.3  11.3   0:03.68 S scheduled
 1579 redis     20   0  131.6m  11.2m   0.3   0.1   0:33.65 S /bin/redis-server 127.0.0.1:6380
 1580 redis     20   0  131.6m  10.4m   0.3   0.1   0:26.31 S /bin/redis-server 127.0.0.1:6383
 1757 root      20   0  226.5m  39.7m   0.3   0.5   0:09.40 S /bin/clusterd
 1785 root      20   0  210.6m  57.8m   0.3   0.7   0:11.08 S /bin/sqlrptcached
 1789 root      20   0  283.2m  70.3m   0.3   0.9   0:25.49 S /bin/sqlplugind

Logging devices with quotas for each ADOM

FAZ-AWS # diag log device
Device Name          Device ID            Used Space(logs / quarantine / content / IPS) Allocated Space  Used%
FGT-Perimeter        FGTAWSN1JDGCU42E        4.4MB(   4.4MB/   0.0KB/   0.0KB/   0.0KB) unlimited        n/a
SYSLOG-Linux         SYSLOG-0A64680F        76.0KB(  76.0KB/   0.0KB/   0.0KB/   0.0KB) unlimited        n/a
Total: 2 log devices, used=4.5MB quota=unlimited


AdomName           AdomOID  Type                                   Logs
                                 [Retention   Quota      Used(  logs/quaranti/ content/     IPS) Used%] [Retention
FGT-only           193      FGT    365days unlimited    4.4MB( 4.4MB/   0.0KB/   0.0KB/   0.0KB)   n/a     92days
FortiAnalyzer      133      FAZ    365days   300.0MB    8.0KB( 8.0KB/   0.0KB/   0.0KB/   0.0KB)  0.0%     60days
FortiAuthenticator 149      FAC    365days   300.0MB    8.0KB( 8.0KB/   0.0KB/   0.0KB/   0.0KB)  0.0%     60days
FortiCache         137      FCH    365days   300.0MB    8.0KB( 8.0KB/   0.0KB/   0.0KB/   0.0KB)  0.0%     60days
FortiCarrier       129      FGT    365days   300.0MB    8.0KB( 8.0KB/   0.0KB/   0.0KB/   0.0KB)  0.0%     60days
FortiClient        139      FCT    365days   300.0MB    8.0KB( 8.0KB/   0.0KB/   0.0KB/   0.0KB)  0.0%     60days

Total usage: 17 ADOMs, logs=4.6MB(4.6MB/0.0KB/0.0KB/0.0KB) database=181.3MB(ADOMs usage:24.3MB(96.6KB, 0.0KB)

Total Quota Summary:
*** Warning: Total Allocated Quota is bigger than Total Quota! Please check the quota configuration of ADOMs!
    Total Quota      Allocated        Available        Allocate%
    63.2GB           64.6GB           0.0KB            102.3%

System Storage Summary:
    Total            Used             Available        Use%
    78.2GB           7.4GB            70.8GB           9.5 %

Reserved space: 15.0GB (19.2% of total space).

Run Linux df -h command

FAZ-AWS #  diagnose system print df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    3.8G      1.8G      1.9G  49% /
none                      3.8G         0      3.8G   0% /dev
none                      6.9G      1.1M      6.9G   0% /dev/shm
none                     64.0M     72.0K     63.9M   0% /tmp
/dev/nvme0n1p1          982.7M    434.5M    548.2M  44% /data
/dev/mdvg/mdlv           78.2G      7.4G     70.8G   9% /var
/dev/mdvg/mdlv           78.2G      7.4G     70.8G   9% /drive0
/dev/mdvg/mdlv           78.2G      7.4G     70.8G   9% /Storage
/dev/loop0                8.6M     19.0K      8.1M   0% /var/dm/tcl-root
none                    512.0M         0    512.0M   0% /drive0/tmp/sql_bat
none                    128.0M         0    128.0M   0% /drive0/private/dbcommit

Show disks and partitions

FAZ-AWS # exe lvm info
LVM Status: OK
LVM Size: 80GB
File System: ext4 78GB

Disk1 :         Used       80GB
Disk2 :  Unavailable        0GB
Disk3 :  Unavailable        0GB

Print average load, only meaningful for comparing

FAZ-AWS # diagnose system print loadavg
0.08 0.19 0.18 1/695 9241

netstat - Open connections and Listening ports

FAZ-AWS # diagnose system print netstat
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:7080            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN
tcp        0      0 :::443                  :::*                    LISTEN
tcp        0      0 :::26443                :::*                    LISTEN
tcp        0      0 ::1:8123                :::*                    LISTEN
tcp        0      0 ::ffff:10.100.104.17:514 ::ffff:10.100.104.13:4128 ESTABLISHED
tcp        0      0 ::ffff:10.100.104.17:514 ::ffff:10.100.104.15:60170 ESTABLISHED
udp        0      0 127.0.0.1:6001          0.0.0.0:*
udp        0      0 127.0.0.1:6003          0.0.0.0:*
udp        0      0 0.0.0.0:31167           10.100.0.2:53           ESTABLISHED
udp        0      0 10.100.104.17:52222     10.100.0.2:53           ESTABLISHED

Print routing table

FAZ-AWS #  diagnose system print route
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.100.104.1    0.0.0.0         UG    0      0        0 port1
0.0.0.0         10.100.104.13   0.0.0.0         UG    1      0        0 port1
10.100.104.0    0.0.0.0         255.255.255.0   U     0      0        0 port1
169.254.169.254 169.254.169.254 255.255.255.255 UGH   0      0        0 port1

Real time debug of communication between FAZ and FGT (not much of help)

FAZ-AWS # diagnose debug application oftpd 8 FGTAWSN1JDGCU42E
oftpd debug filter:     filter(string)==FGTAWSN1JDGCU42E

FAZ-AWS #
FAZ-AWS #
[T3993:oftps.c:1933 FGTAWSN1JDGCU42E:10.100.104.13] SSL socket[20] pid[1754] ssl[0x5600c4048980]
received [12] bytes:

[T3993:main.c:4174 FGTAWSN1JDGCU42E:10.100.104.13] handle KEEPALIVE

[T3996:oftps.c:1999 FGTAWSN1JDGCU42E:10.100.104.13] SSL socket[20] pid[1754] ssl[0x5600c4048980]
sent [21] bytes:

[T3996:oftp_restapi_sched.c:1785] FGTAWSN1JDGCU42E

[T3996:oftp_restapi.c:2333 FGTAWSN1JDGCU42E:10.100.104.13] ret = 0.

Sniffer of packets in real time `dia sni pa`

FAZ-AWS # diagnose sniffer packet any "port 514"
interfaces=[any]
filters=[port 514]
1.383021 10.100.104.13.9334 -> 10.100.104.17.514: udp 646
3.640615 10.100.104.13.4128 -> 10.100.104.17.514: psh 2709416742 ack 1556741276
3.640752 10.100.104.17.514 -> 10.100.104.13.4128: psh 1556741276 ack 2709416776
3.640870 10.100.104.13.4128 -> 10.100.104.17.514: ack 1556741319
6.383617 10.100.104.13.9334 -> 10.100.104.17.514: udp 592
8.646227 10.100.104.13.4128 -> 10.100.104.17.514: psh 2709416776 ack 1556741319
8.646360 10.100.104.17.514 -> 10.100.104.13.4128: psh 1556741319 ack 2709416810
8.646492 10.100.104.13.4128 -> 10.100.104.17.514: ack 1556741362

What type of logs are being received from each device

FAZ-AWS # diagnose test application oftpd 50

Showing logtypes of all cached devices ......

 SN                VDOM  RETENTION-HOUR  LOGTYPES/ERROR

 FGTAWSN1JDGCU42E  root  2208            app-ctrl|ips|anomaly|dlp|emailfilter|event.system|event.vpn|
 event.user|event.wireless|event.endpoint|event.ha|event.security-rating|event.connector|
 traffic.forward|traffic.local|traffic.multicast|traffic.sniffer|virus|voip|webfilter|dns|
 ssh|ssl|security
 SYSLOG-0A64680F   root  1440            generic

Log received stats in last 5, 30, and 60 seconds

FAZ-AWS # diagnose fortilogd lograte
last 5 seconds: 0.0, last 30 seconds: 0.1, last 60 seconds: 0.1

Logs per second per device

FAZ-AWS # diagnose fortilogd lograte-device

Logs per second
Totals                    Last Hour       Day      Week
-------------------------------------------------------
        FGTAWSN1JDGCU42E:      0.16      0.11      0.02
         SYSLOG-0A64680F:      0.03      0.02      0.00
        FAZAWSTA23002441:      0.00      0.00      0.00

Show license for VMs

FAZ-AWS # diagnose debug vminfo
VM license is valid.
fds_code: 200

Type: Full
Licensed GB/Day: 1
Max devices: 2
Serial Number: FAZAWSTA23772441
VM UUID: ec211ef8-3328-358f-f78f-9450cf09a51d

Follow me on https://www.linkedin.com/in/yurislobodyanyuk/ not to miss what I publish on Linkedin, Github, blog, and more.

General Health

Communication debug

Logs from devices

Licensing

Example debug session on Fortianalyzer

Show connected to the FAZ devices

General state of FAZ (version, serial, HA status, license status)

Performance stats (appliance FAZ will have more data)

Running processes and CPU load

Logging devices with quotas for each ADOM

Run Linux df -h command

Show disks and partitions

Print average load, only meaningful for comparing

netstat - Open connections and Listening ports

Print routing table

Real time debug of communication between FAZ and FGT (not much of help)

Sniffer of packets in real time dia sni pa

What type of logs are being received from each device

Log received stats in last 5, 30, and 60 seconds

Logs per second per device

Show license for VMs

Sniffer of packets in real time `dia sni pa`