Nfdump netflow/sflow cookbook of examples


Introduction

Few facts to know before diving into examples:

  • nfdump packet filter syntax is tcpdump-compatible, and it should come as the last argument on the line.

  • nfcapd daemon receives Netflow streams and saves them into local files, switching to a new file every 5 minutes (configurable). The naming starts with nfcapd, then dot, and finally date and time stamp.

  • The newest version for 2022 is 1.7, which is multi-threaded.

  • There is a GUI web based front end nfsen, which is a separate install.

  • nfdump reads files from filesystem and outputs to either STDOUT or to binary files (if used with -w option). If it runs out of host memory or free disk space for temporary files, it will crash.

Start nfcapd netflow collector in a daemon mode listening on port 5001 with all extensions enabled and saving received netflow data into the named folder NFS-cisco-rtr. Accept netflow records only coming from the sender with the IP of 13.13.13.137

nfcapd -D -T all -n NFS-cisco-rtr,13.13.13.137,/var/flows/NFS-cisco-rtr -p 5001

Read and print all records form a single file

Here the records file is nfcapd.202004221040 nfdump prints record in random order, not sorted by any means.

nfdump -r nfcapd.202004221040

Display cumulative statistics about all the flows in a records file

nfdump -I -r nfcapd.202004221040

Output:

Ident: NFS-cisco-rtr
Flows: 378330
Flows_tcp: 318586
Flows_udp: 54743
Flows_icmp: 3864
Flows_other: 1137
Packets: 11162669
Packets_tcp: 8681920
Packets_udp: 2163252
Packets_icmp: 34346
Packets_other: 283151
Bytes: 6315310484
Bytes_tcp: 5677222352
Bytes_udp: 467682299
Bytes_icmp: 3717079
Bytes_other: 166688754
First: 1587551972
Last: 1587552299
msec_first: 950
msec_last: 914
Sequence failures: 0

Read and print all records from a range of files, starting at nfcapd.202004221040 and up to but not including the current file still being written to nfcapd.current.1609

nfdump -R nfcapd.202004221040

Read all records from a range of files, starting at nfcapd.202209242120 and finishing at nfcapd.202209242150

This works if files are in the same directory. If they are not, also specify -M for directories list.

nfdump -R nfcapd.202209242120:nfcapd.202209242150

Print sessions where the source or destination IP is 8.8.8.8

nfdump -r nfcapd.202004221040 'host 8.8.8.8'

Print sessions where the destination port is 53, the destination IP is 8.8.8.8, and the protocol is TCP

nfdump  -r nfcapd.202004221040  'host 8.8.8.8 and dst port 53 and proto tcp '

Show top 10 flows sorted by the bits per second statistics

Note: -o extended sets output to include also bps column. -n 10 limits output to top 10 rows (which is default as well). Finally, -O bps tells nfdump to sort the output by bits per second value in descending (default) order.

nfdump -r nfcapd.202004221050 -n 10  -O bps -o extended

Output:

Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port     Flags Tos  Packets    Bytes      pps      bps    Bpp Flows
2020-04-22 12:19:58.824     0.004 TCP      44.30.248.239:443   ->     44.244.6.114:54044 ...AP...  16      141   204984    35250  410.0 M   1453     1
2020-04-22 12:22:37.845     0.004 TCP       175.68.86.47:80    ->     44.244.6.114:53717 ...AP... 128      133   184649    33250  369.3 M   1388     1
2020-04-22 12:20:37.844     0.004 TCP       175.68.86.47:80    ->     44.244.6.114:53717 ...AP... 128      133   184649    33250  369.3 M   1388     1
2020-04-22 12:24:37.845     0.004 TCP       175.68.86.47:80    ->     44.244.6.114:53717 ...AP... 128      132   184609    33000  369.2 M   1398     1
2020-04-22 12:22:59.517     0.008 TCP      44.244.195.12:443   ->     44.244.6.114:58302 ...AP...  16      212   302672    26500  302.7 M   1427     1
2020-04-22 12:23:15.541     0.036 TCP      44.244.195.12:443   ->     44.244.6.114:58302 ...AP...  16      915    1.3 M    25416  298.9 M   1469     1
2020-04-22 12:20:03.728     0.004 TCP        50.62.32.42:80    ->    216.88.40.116:52054 ...AP...  40       99   135345    24750  270.7 M   1367     1
2020-04-22 12:23:08.773     0.012 TCP      44.244.195.12:443   ->     44.244.6.114:58302 ...AP...  16      255   371935    21250  248.0 M   1458     1
2020-04-22 12:22:58.377     0.004 TCP        50.62.32.25:80    ->    216.88.40.116:52157 ...AP...   0       77   109616    19250  219.2 M   1423     1
2020-04-22 12:21:25.568     0.028 TCP     158.255.172.17:443   ->     44.244.6.114:55324 ...AP...  16      483   669748    17250  191.4 M   1386     1
IP addresses anonymised
Summary: total flows: 492540, total bytes: 7.4 G, total packets: 14.6 M, avg bps: 184.8 M, avg pps: 45237, avg bpp: 510
Time window: 2020-04-22 12:19:37 - 2020-04-22 12:24:59
Total flows processed: 492540, Blocks skipped: 0, Bytes read: 39403680
Sys: 0.216s flows/second: 2277989.2  Wall: 0.404s flows/second: 1219146.3

Show all flows sorted by the bits per second statistics

nfdump -r nfcapd.202004221050 -n 0  -O bps -o extended

Aggregate all flows to/from host 8.8.8.8 based on source IP

nfdump -r nfcapd.202004221005 -A srcip ' host 8.8.8.8'
Date first seen          Duration       Src IP Addr   Packets    Bytes      bps    Bpp Flows
2020-04-22 10:05:01.183   241.032    113.166.180.142       122     6938      230     56   122
2020-04-22 10:05:00.915   295.020   18.113.121.204      1493    96860     2626     64    62
2020-04-22 10:05:03.819   289.848    18.113.43.130       750    63000     1738     84    54
2020-04-22 10:05:02.887   289.828    113.166.180.139       750    63000     1738     84    49
2020-04-22 10:05:01.455   295.148    113.166.180.138       812    50458     1367     62   810
2020-04-22 10:05:03.507   289.852    113.166.180.137       750    63000     1738     84    56
2020-04-22 10:04:55.799   300.484    89.12.212.116       417    33261      885     79   417
2020-04-22 10:05:00.667   289.868    113.166.180.141       750    63000     1738     84    55
2020-04-22 10:04:56.047   303.116           8.8.8.8      6730   768784    20290    114  1825
2020-04-22 10:05:01.127   291.740      113.166.88.58       886    70796     1941     79   172
Summary: total flows: 3622, total bytes: 1.3 M, total packets: 13460, avg bps: 33731, avg pps: 44, avg bpp: 95
Time window: 2020-04-22 10:04:37 - 2020-04-22 10:09:59
Total flows processed: 426270, Blocks skipped: 0, Bytes read: 34102112
Sys: 0.036s flows/second: 11784203.7 Wall: 0.036s flows/second: 11560177.9

Calculate statistics for port 443 traffic and sort by bps to see bandwidth abusing hosts

We can include as many -s as needed, each statistics table will be printed independently. Statistics will be calculated for the flows located in this specific nfcapd. file, to count statistics over longer periods of time see -R & -M

nfdump -r nfcapd.202004220705 -s srcip/bps -s dstip/bps  ' port 443'

Output:

Top 10 Src IP Addr ordered by bps:
Date first seen          Duration Proto       Src IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2020-04-22 12:19:50.336   309.513 any      219.149.22.196     8957( 2.6)   995109(10.7)    1.3 G(21.6)     3215   34.2 M  1327
2020-04-22 12:19:50.336   309.361 any      219.149.22.201     8167( 2.4)   837173( 9.0)    1.1 G(18.1)     2706   28.6 M  1321
2020-04-22 12:19:37.828   322.081 any        44.244.6.114    58555(16.9)    2.1 M(22.3)  978.3 M(16.0)     6471   24.3 M   469
2020-04-22 12:21:31.120     0.496 any       128.73.82.164        6( 0.0)      958( 0.0)    1.3 M( 0.0)     1931   21.7 M  1406
2020-04-22 12:19:49.064   310.609 any       244.34.184.28     6849( 2.0)   411384( 4.4)  369.3 M( 6.0)     1324    9.5 M   897
2020-04-22 12:23:01.213     2.244 any      148.161.85.162        3( 0.0)     1322( 0.0)    1.9 M( 0.0)      589    6.9 M  1469
2020-04-22 12:19:49.860   309.909 any       244.34.184.29     4425( 1.3)   270828( 2.9)  250.0 M( 4.1)      873    6.5 M   922
2020-04-22 12:19:52.984   306.313 any      219.149.22.228    12205( 3.5)   245171( 2.6)  244.1 M( 4.0)      800    6.4 M   995
2020-04-22 12:21:08.360     6.460 any        92.34.211.23        4( 0.0)     3080( 0.0)    4.5 M( 0.1)      476    5.6 M  1472
2020-04-22 12:19:37.828   321.421 any      219.149.22.229    16208( 4.7)   252091( 2.7)  220.4 M( 3.6)      784    5.5 M   874

Top 10 Dst IP Addr ordered by bps:
Date first seen          Duration Proto       Dst IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2020-04-22 12:19:37.828   322.081 any        44.244.6.114    57453(16.6)    3.5 M(37.6)    3.1 G(51.2)    10892   77.8 M   893
2020-04-22 12:19:43.316   316.633 any       216.88.40.117    56198(16.2)    1.1 M(11.8)  939.0 M(15.4)     3470   23.7 M   854
2020-04-22 12:19:48.900   311.033 any       216.88.40.116    56349(16.3)    1.0 M(11.0)  835.6 M(13.7)     3295   21.5 M   815
2020-04-22 12:19:54.760   300.929 any      93.161.105.117      833( 0.2)    83367( 0.9)  108.3 M( 1.8)      277    2.9 M  1299
2020-04-22 12:19:47.736   310.013 any      210.249.165.16       89( 0.0)    79489( 0.9)   92.7 M( 1.5)      256    2.4 M  1165
2020-04-22 12:19:59.256   298.577 any        70.35.238.51      180( 0.1)   100020( 1.1)   85.8 M( 1.4)      334    2.3 M   858
2020-04-22 12:20:07.024     0.004 any      209.213.75.111        2( 0.0)       28( 0.0)     1120( 0.0)     7000    2.2 M    40
2020-04-22 12:21:16.636     0.004 any       216.88.58.165        2( 0.0)       22( 0.0)      968( 0.0)     5500    1.9 M    44
2020-04-22 12:19:59.472   299.353 any      207.176.46.233       94( 0.0)    42353( 0.5)   45.7 M( 0.7)      141    1.2 M  1079
2020-04-22 12:19:50.356   309.537 any      219.149.22.196     9055( 2.6)   274394( 2.9)   47.0 M( 0.8)      886    1.2 M   171

IP addresses anonymised
Summary: total flows: 346143, total bytes: 6.1 G, total packets: 9.3 M, avg bps: 151.8 M, avg pps: 28957, avg bpp: 655
Time window: 2020-04-22 12:19:37 - 2020-04-22 12:24:59
Total flows processed: 492540, Blocks skipped: 0, Bytes read: 39403680
Sys: 0.139s flows/second: 3539124.8  Wall: 0.136s flows/second: 3598781.3

Sort presented flows by duration, longest at the bottom

nfdump itself has no provision to sort flows by their duration, but we can easily pipe the output to any Linux sorting tool. Let’s display top 10 flows by duration:

echo 'Date first seen          Duration Proto      Src IP Addr:Port Dst IP Addr:Port      Packets    Bytes    Flows ' ; \
nfdump -r nfcapd.202209281905 | sort -n -k3,3 | tail -10

Output:

Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port      Packets    Bytes    Flows
2022-09-28 18:10:08.820  3360.080 TCP      172.17.12.130:57095 ->   20.199.120.182:443          4      306     1
2022-09-28 18:10:08.820  3360.080 TCP     20.199.120.182:443   ->    172.17.12.130:57095        2      428     1
2022-09-28 18:11:48.620  3360.130 TCP      172.17.12.164:49836 ->   20.199.120.151:443          4      304     1
2022-09-28 18:11:48.620  3360.130 TCP     20.199.120.151:443   ->    172.17.12.164:49836        2      426     1
2022-09-28 17:06:18.140  7202.630 ICMP     172.17.80.245:0     ->       172.20.0.2:0.8        120     7200     1
2022-09-28 09:09:36.580 35232.610 PIM    100.100.100.100:0     ->    172.17.46.254:0            0        0     1
2022-09-27 20:34:35.550 81030.200 ICMP     172.17.80.245:0     ->    87.128.226.58:0.8       1362    81720     1
2022-09-27 20:34:35.550 81030.200 ICMP     87.128.226.58:0     ->    172.17.80.245:8.8       1362    81720     1
2022-09-27 15:45:26.850 98610.750 ICMP       172.17.7.12:0     ->    172.17.24.127:8.8       9860   433840     1
2022-09-27 15:45:26.850 98610.750 ICMP     172.17.24.127:0     ->      172.17.7.12:0.8       9859   433796     1

Anonymize IP addresses in all the flows in the file, overwrite in-place

Use bundled with nfdump tool named nfanon. To use it we have to specify a random ASCII of 32 characters or hexadecimal string of 64 characters. The -K option accepts the random key.

To generate random 32 chars:

dd if=/dev/urandom  bs=16 count=1  |  hexdump -v -e '/1 "%02X "' | tr -d ' ' ; echo
1+0 records in
1+0 records out
16 bytes copied, 0.000491685 s, 32.5 kB/s
E9C11DC6F92488E7A13A1F42EF6A9E87
nfanon -K E9C11DC6F92488E7A13A1F42EF6A9E87  -r nfcapd.202004220710

Find records in the time range of 12:19:00 - 12:20:00 matching the filter of protocol = UDP and port = 53

Using -t option we can limit the time range of the records to look into. nfdump puts 0 for any missing time part, e.g. 12:19 means 12:19:00.

 nfdump -r nfcapd.202004220920 -t 2020/04/22.12:19:00-2020/04/22.12:20:00 'port 53 and proto udp'

Output:

Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2020-04-22 12:19:55.564     0.000 UDP      56.30.111.241:53    ->    216.88.40.117:64842        1      636     1
2020-04-22 12:19:55.564     0.000 UDP      216.88.40.117:64012 ->    158.174.33.78:53           1       81     1
2020-04-22 12:19:54.852     0.000 UDP      216.88.40.117:63044 ->      70.158.34.8:53           1       80     1
2020-04-22 12:19:55.712     0.000 UDP     219.154.149.77:53    ->    216.88.40.116:49880        1       89     1
2020-04-22 12:19:55.716     0.000 UDP      216.88.40.117:65172 ->   246.220.77.233:53           1       82     1
2020-04-22 12:19:55.152     0.000 UDP      216.88.40.117:63463 ->  177.234.225.103:53           1       79     1
2020-04-22 12:19:55.364     0.000 UDP      216.88.40.117:63493 ->       51.11.3.16:53           1       73     1
IP addresses anonymised
Summary: total flows: 7, total bytes: 1120, total packets: 7, avg bps: 10370, avg pps: 8, avg bpp: 160
Time window: 2020-04-22 12:19:37 - 2020-04-22 12:24:59
Total flows processed: 492540, Blocks skipped: 0, Bytes read: 39403680
Sys: 0.050s flows/second: 9713068.7  Wall: 0.048s flows/second: 10233959.4

Follow me on https://www.linkedin.com/in/yurislobodyanyuk/ not to miss what I publish on Linkedin, Github, blog, and more.