FortiGate FortiOS CLI Troubleshooting: Cheat Sheet

FortiGate's power lies in the diagnose command family - the CLI toolkit that reveals exactly what the firewall is doing with every packet, session and daemon. This cheat sheet, condensed from the official FortiOS CLI troubleshooting documentation, organizes the most useful diagnose commands by subsystem: IPS engine, WAD proxy, routing and BFD, SD-WAN link monitoring, authentication, IPsec VPN, logging and SIP ALG. Keep it handy for the next time a FortiGate misbehaves and you need hard evidence fast.

Everything after the next section is indexed by symptom rather than by subsystem, because in an outage you start from "what is broken", not from "which daemon". Jump to the flow that matches the complaint, run it top to bottom, and stop when the evidence points at a cause.

Symptom Index: Start Here

Packet-Path Debugging: The Two Commands That Answer Most Tickets

Before touching any daemon-specific command, establish whether the FortiGate saw the packet and what it decided to do with it. Two tools do that: the flow trace, which prints the policy lookup and drop reason for a session, and the sniffer, which shows raw packets in and out of an interface.

diagnose debug flow filter addr 10.10.10.50        # narrow to one host
diagnose debug flow filter port 443
diagnose debug flow show function-name enable       # show which function decided
diagnose debug flow show iprope enable              # show the policy (iprope) lookup
diagnose debug flow trace start 20                  # trace the next 20 packets
diagnose debug enable                               # turn output on
diagnose debug disable                              # turn output off
diagnose debug reset                                # clear all debug settings

diagnose sniffer packet any 'host 10.10.10.50 and port 443' 4 20 a
                                    # iface | filter | verbose | count | timestamp

Read the flow trace from the top: find a route tells you whether the routing table matched, find a policy tells you which firewall policy matched, and a line such as iprope_in_check() check failed, drop means no policy allowed the session. A trace that shows packets entering but never leaving usually points at an asymmetric route or an SD-WAN rule sending the return path elsewhere - which is where the SD-WAN commands further down earn their keep.

Symptom: Traffic Is Dropped or an Application Fails

diagnose debug flow filter addr <source-ip>
diagnose debug flow trace start 50
diagnose debug enable
diagnose firewall iprope list 100004            # installed policy list
diagnose sys session filter src <source-ip>     # narrow the session table
diagnose sys session list                       # session detail for that host
diagnose sys session stat                       # session table totals
diagnose sys session full-stat                  # per-protocol/state counters
diagnose ip arp list                            # verify the next-hop MAC is learned

Session exhaustion is the most common "random" complaint on a busy FortiGate: the table is full, so new sessions are dropped while existing ones keep working. Check diagnose sys session stat against the platform's session limit, then look for a host or a subnet filling the table with diagnose sys session full-stat before you consider raising limits.

System Diagnostics: IPS Engine and WAD

diagnose test application ipsmonitor 1      # show IPS engine information
diagnose test application ipsmonitor 2      # enable/disable IPS engines
diagnose test application ipsmonitor 97     # start all IPS engines
diagnose test application ipsmonitor 98     # stop all IPS engines
diagnose ips session list                   # show IPS sessions per engine
diagnose test application wad 1000          # show all WAD processes
diagnose test application wad 2             # show total memory usage
diagnose test application wad 99            # restart all WAD processes

On a FortiGate running proxy-based inspection, "browsing is slow" is usually a WAD problem rather than a policy problem: WAD handles explicit proxy, web filtering and SSL inspection, and a busy WAD process shows up as a memory spike in diagnose test application wad 2 long before users report anything. If one WAD process is consuming memory while others are idle, note the process ID and follow it with the WAD debug commands below rather than restarting blindly.

Symptom: Users Cannot Browse or Browsing Is Slow (WAD)

diagnose test application wad 1000                      # process list and memory
diagnose test application wad 1001                      # per-process detail
diagnose wad debug enable category http
diagnose wad debug enable level verbose
diagnose debug enable
diagnose wad debug disable                              # stop when finished
diagnose debug reset

WAD debugging is verbose - filter it to a single source or destination before you enable it, or you will lose the relevant lines in the flood. When the problem clears as soon as the affected user retries, suspect a WAD worker that ran out of resources rather than a filter that is blocking traffic.

Network and Routing Diagnostics

diagnose ip arp list                        # show ARP entries
diagnose netlink brctl list                 # show all switch interfaces
get system interface                        # interface configuration and status
get router info6 bfd neighbor               # show BFD neighbors
diagnose test application bfd 1             # show BFD statistics

get router info routing-table details <prefix>   # which route will be used
diagnose ip address list                          # all IPs, including VIPs
diagnose ip route list                            # kernel routes
execute ping-options source <interface-or-ip>     # pin the source of a ping
execute ping <destination>
execute traceroute <destination>

Always set ping-options source before testing reachability from a FortiGate: an unqualified ping uses the outgoing interface address, which may not be the address your remote side expects, producing a "the firewall cannot ping the peer" conclusion that is simply a source-selection artefact.

SD-WAN Link Monitoring

diagnose sys link-monitor interface <interface>    # link monitoring statistics
diagnose debug application link-monitor -1
diagnose debug enable                               # real-time link monitor debugging
diagnose test application lnkmtd 1                  # link monitor statistics

Symptom: SD-WAN SLA Failures or Tunnel Flapping

diagnose sys sdwan health-check             # health-check status and latency/loss
diagnose sys sdwan member                   # member link state and scores
diagnose sys sdwan service                  # service (rule) selection
diagnose sys sdwan route                    # resulting SD-WAN routes
diagnose sys link-monitor interface wan1    # underlying probe statistics
diagnose netlink interface list             # interface up/down and IP state

Work the chain in order - health-check, member, service, route. An SLA that fails while the physical link is up is nearly always a probe problem (wrong source interface, probe blocked upstream, or a jitter budget below the real network), not a routing problem. If diagnose sys sdwan health-check shows a member with no probes at all, check that traffic from the health-check source address is actually permitted out of that link.

Authentication and FSSO

diagnose firewall auth list                  # list authenticated IPv4 users
diagnose firewall auth filter <filter>       # set filter for the list
diagnose wad user list                       # users authenticated by proxy
diagnose debug application fnbamd -1
diagnose debug application authd -1
diagnose debug enable                        # real-time auth debugging
diagnose test authserver <auth_protocol> <server> <user> <password>   # test auth from CLI
diagnose debug authd fsso list               # show FSSO logged-on users
execute fsso refresh                         # refresh FSSO user list (may cause outage)

Symptom: Users Re-Prompted for Authentication, or FSSO Is Empty

diagnose debug authd fsso list              # is the user actually logged on?
diagnose wad user list                      # proxy-side user table
diagnose firewall auth list                 # firewall-side user table
diagnose debug application fnbamd -1
diagnose debug application authd -1
diagnose debug enable
diagnose test authserver ldap <server> <user> <password>   # test the auth server
execute fsso refresh                        # re-pull the FSSO user list

Two tables matter, and they are not the same: diagnose firewall auth list shows users authenticated by the firewall itself, while diagnose wad user list shows users authenticated through the proxy. A user present in one and missing from the other explains "the policy should have matched but it didn't". If FSSO is empty, start with the collector-side conversation using the fnbamd debug, not with the firewall policies. Note that execute fsso refresh can interrupt active sessions - schedule it accordingly.

IPsec VPN Troubleshooting

diagnose vpn ike gateway clear               # clear IKE gateways (Phase 1)
diagnose vpn ipsec status                    # encryption counters
diagnose vpn ike log filter <filter>         # filter IKE daemon debugs
diagnose debug application ike -1
diagnose debug enable                        # real-time IKE debugging

Symptom: Tunnel Up but No Traffic Passes

diagnose vpn ipsec status                   # are there ANY encrypted packets?
diagnose vpn ike gateway list               # phase 1 / phase 2 detail per peer
get vpn ipsec tunnel summary
get router info routing-table details <remote-subnet>
diagnose debug flow filter addr <remote-subnet>
diagnose debug flow trace start 20
diagnose debug enable

If the Phase 1 SA is up but the encryption counters are flat, the interesting question is which side never selected the tunnel. Run the flow trace for traffic toward the remote subnet and look at the outgoing interface in the trace output: if the packet leaves via the WAN default route instead of the tunnel interface, the static route or the policy route for the remote subnet is missing or has a longer prefix match somewhere else. Asymmetric routing across two tunnels to the same peer is the usual culprit in hub-and-spoke designs.

Logging and SIP ALG

diagnose log test                    # generate test logs
execute log filter <filter>          # set log filters
execute log filter dump              # show log filters
exec log display                     # show filtered logs
execute log delete                   # delete filtered logs
diagnose sys sip-proxy calls list    # active SIP proxy calls
diagnose sys sip-proxy stats         # SIP proxy statistics
diagnose debug application sip -1
diagnose debug enable                # real-time SIP debugging

Remember the golden rule of FortiGate debugging: enable the daemon debug with diagnose debug application <app> <level>, then activate output with diagnose debug enable, and disable it with diagnose debug disable when you are done to avoid console flooding.

Symptom: SIP Calls Drop or One-Way Audio

diagnose sys sip-proxy calls list   # is the call tracked at all?
diagnose sys sip-proxy stats        # registration and call counters
diagnose debug application sip -1
diagnose debug enable
# then inspect the SDP for the advertised media address:
diagnose sniffer packet any 'host <pbx-ip> and portrange 5060-5061' 4 30 a

SIP problems are usually address problems: the PBX advertises a private address in SDP that the far end cannot reach, or the SIP ALG rewrites the contact differently from what the carrier expects. The call list tells you whether the firewall is tracking the call, and the sniffer shows whether the media address in SDP is routable from the other end. If the ALG rewrites are the problem, disabling the SIP helper on that policy and letting the PBX handle NAT traversal is often the fastest fix.

Symptom: High CPU, High Memory or Session Table Full

get system performance status          # CPU, memory, uptime, session counts
diagnose sys top 2 30                  # interactive top: press q to quit
diagnose sys top-summary               # one-line per-process CPU summary
diagnose sys mpstat 1 10               # per-CPU statistics
diagnose hardware sysinfo memory       # memory detail
diagnose sys session stat              # sessions vs configured limit
diagnose sys session full-stat         # breakdown by protocol and state
diagnose sys npu-session list          # offloaded hardware sessions (NP models)

Interpretation matters more than collection here. High CPU in diagnose sys top with a matching daemon name points at inspection (ipsengine, wad), while high CPU with no dominant process usually means interrupt or softirq pressure from packet rate - a completely different problem. Compare the session count against the limit before assuming a resource problem: a full session table looks like a CPU problem from the user's seat but shows up as dropped new sessions in the counters.

Symptom: HA Cluster Members Out of Sync or Failover Problems

get system ha status                    # role, priority, uptime, sync state
diagnose sys ha status                  # detailed HA state per VDOM
diagnose sys ha checksum show           # per-VDOM config checksum
diagnose sys ha checksum show root      # checksum for one VDOM
diagnose sys ha dump-by all-vcluster    # cluster-by-cluster detail
diagnose sys ha history read            # HA event history
execute ha manage 0 admin               # jump to the other cluster member

An out-of-sync member is a silent failure mode: everything works until a failover, then half your configuration is missing. Compare checksums between members with diagnose sys ha checksum show, and remember that some objects (certificates, addresses embedded in policies) routinely cause checksum mismatches without being functionally different - always verify against the actual config with show, not just the checksum numbers.

Debug Hygiene Checklist

1. Filter before you enable. A debug without a filter on a busy firewall produces thousands of lines per second and can itself degrade service.
2. Know the three-step pattern. Configure the filter, enable the debug application, then diagnose debug enable.
3. Always turn it off. diagnose debug disable followed by diagnose debug reset. An abandoned debug left running is the most common cause of "the firewall got slower after the last engineer looked at it".
4. Capture to a file. Use execute tac report or redirect console output to a host; debugging that exists only in a scrollback buffer is debugging you will run twice.
5. Record the baseline. Session count, CPU and memory before the change - otherwise you cannot prove the change fixed anything.

Related Reading

• FortiGate active-passive HA cluster: CLI setup guide
• IPsec IKEv2 debugs: SA_INIT and IKE_AUTH troubleshooting
• Linux 网络故障排查：从网卡到路由的定位思路
• BGP Network Troubleshooting: Best Practices and Commands

原文链接：https://docs.fortinet.com/document/fortigate/8.0.0/cli-troubleshooting-cheat-sheet/420966/cli-troubleshooting-cheat-sheet