Cisco Aironet Won’t Connect to Wireless LAN Controller

2023年3月16日 LOG admin

原文：Cisco Aironet Won’t Connect to Wireless LAN Controller — theDXT (Daniel Keer)

I ran into an issue where some older Cisco Aironet APs (Access Points) stopped connecting to a Cisco WLC (Wireless LAN Controller). No config changes had been made and some of the Cisco Aironet APs would connect and some wouldn’t. All of them were the same model, the Cisco Aironet APs were able to ping the Cisco WLC and vice versa.

What happened is the Cisco MIC (Manufacture Installed Certificate) expired and the default setup of a Cisco WLC is to reject any Cisco Aironet AP with an expired MIC.

It looks like this could impacts every Cisco WLC when used with older Cisco Aironet APs that have an expired Cisco MIC. Cisco has a Field Notice about this issue, you can read it here FN63942.

Any Cisco Aironet AP that was manufactured from July 18, 2005 until 2017 will have a Cisco MIC that expires 10 years after the manufacture date. There seems to be no way to replace or renew that Cisco MIC, this will keep being an issue that could randomly show up until 2027 when all of them should be broken.

The reason some of my Cisco Aironet APs worked and some didn’t is because they were manufactured at different times even though they have the same model number.

The fix is super quick we just need to tell the Cisco WLC to ignore expired Cisco MICs.

Before we just ignore the expired Cisco MICs we should confirm that it is the actual issue. Here are a few ways to do that.

How AP Certificate Validation Works on a WLC

To understand why an expired certificate takes an access point off the air, it helps to know what identities a Cisco AP actually carries. Every Aironet AP leaves the factory with two certificates in flash:

MIC — Manufacture Installed Certificate. Burned in during manufacturing and signed by the Cisco Manufacturing CA, which chains up to the Cisco Root CA. It is unique per device and cannot be exported, copied or re-issued in the field.
SSC — Self-Signed Certificate. Generated locally by the AP on its first boot. It is self-signed, so it is only trusted once the WLC has learned it.

When an AP tries to join a WLC it runs CAPWAP (formerly LWAPP) discovery, then sends a join request. During that exchange the WLC validates the identity presented by the AP. If the AP presents a MIC, the WLC checks the signature chain up to a trusted Cisco CA, checks the certificate validity period (the not-before / not-after fields), and then uses that certificate to build the DTLS tunnel that protects the CAPWAP control channel on UDP 5246/5247.

The important part is that the validity period is checked by default. Once the MIC passes its not-after date, the join is rejected outright — no amount of VLAN, routing or DHCP work will fix it, because the control plane handshake never gets far enough to care about those things. On the AP console you will typically see the AP cycle between discovery and join, and the WLC will log something like:

(Cisco Controller) > show msglog
%SSHPM-3-GENERIC_CERT_ERROR: [SA]sshpmPkiApi.c:2237 Certificate validation failed!
Reason Cisco user certificate not verified by cisco root., Certificate type : MIC,
Certificate issuer :Cisco Certificate

Because the symptom is "the AP will not join", it looks identical to the classic CAPWAP problems — wrong management VLAN, missing DHCP option 43, DNS not resolving CISCO-CAPWAP-CONTROLLER, a firewall blocking UDP 5246/5247, or a country-code mismatch. That similarity is exactly why this fault eats so much troubleshooting time: engineers check the network first, and the network is fine.

(Cisco Controller) > show ap summary
AP Name   Slots  AP Model         Ethernet MAC        State
--------  -----  ---------------  ------------------  ----------
AP01      2      AIR-CAP3602I-A   xx:xx:xx:xx:xx:xx   Joining

Which Aironet APs Are Affected

The rule from the field notice is simple: any Cisco Aironet AP manufactured between 18 July 2005 and 2017 has a MIC that expires ten years after the manufacture date. That gives a rolling wave of failures rather than one single outage day — an AP built in March 2013 drops off the air in March 2023, one built in November 2015 drops in November 2025, and the last of the affected fleet breaks sometime in 2027.

The affected platforms include the 1040, 1130/1140/1260, 1600, 2600, 3500, 3600 and similar generation indoor and outdoor Aironet APs — basically anything pre-Wave-2 that relies on a MIC for identity. Newer 802.11ac Wave 2 and Wi-Fi 6/6E/7 access points use SUDI (Secure Unique Device Identifier) with certificates that can be renewed through software, so they are not part of this class of problem.

Two practical consequences follow from this. First, the model number tells you nothing about when the MIC expires. A site with ten AIR-CAP3602I APs can have five working and five failing, because they came from different manufacturing batches. Second, the fault appears "randomly" from the operator's point of view: nothing changed in the configuration, and the AP was fine yesterday. There is no configuration event to correlate with, which is why the WLC message log is the fastest place to start.

It is also worth checking the WLC software release. Older AireOS releases treated this condition differently, and if you have not upgraded in a long time your behaviour may differ slightly from the examples here — but the default policy is always to reject an expired certificate, and the fix below is the same regardless.

Confirming with Logs

You can run the command show msglog on the Cisco WLC to see if you have entries like this.

%SSHPM-3-GENERIC_CERT_ERROR: [SA]sshpmPkiApi.c:2237 Certificate validation failed! Reason Cisco user certificate not verified by cisco root., Certificate type : MIC, Certificate issuer :Cisco Certificate

Confirming with Serial Numbers

You can also use the serial number of the Cisco Aironet AP to confirm your manufacture date to see when the Cisco MIC might expire.

To get the serial number you can do so from the Cisco WLC or from the Cisco Aironet AP.

On the Cisco WLC run the command show ap inventory all

The output should look something like this.

Inventory for AP01NAME: "AP3600" , DESCR: "Cisco Aironet 3600 Series (IEEE 802.11n) Access Point"PID: AIR-CAP3602I-A-K9, VID: V01, SN: FGL1712xxxx

On a Cisco Aironet AP run the command show version

The Processor board ID is the serial number or you can use the Top Assembly Serial Number

The output should look something like this

cisco AIR-CAP3602I-A-K9 (PowerPC) processor (revision A0) with 188398K/60928K bytes of memory.Processor board ID FGL1712xxxxPowerPC CPU at 800Mhz, revision number 0x2151Last reset from power-on1 Gigabit Ethernet interface2 802.11 Radios32K bytes of flash-simulated non-volatile configuration memory.Base ethernet MAC Address:Part Number : 73-14521-02PCB Serial Number : Top Assembly Part Number : 800-35852-02Top Assembly Serial Number : FGL1712xxxxTop Revision Number : C0Product/Model Number : AIR-CAP3602I-A-K9

With the serial number we can decode it to figure out when it was manufactured to get an idea of when the Cisco MIC might expire.

Use the Cisco Field Notice FN63942to gather the decoding info.

In my example my serial number is FGL1712xxxx the first 3 characters mean nothing to us so we can ignore them. We are left with 1712xxxx.

The last 4 characters mean nothing to us so we can ignore them. We are now left with 1712 if we decode it then we know 17 means 2013 and 12 means March.

Now we know our Cisco Aironet AP was manufactured in March 2013 and because it was before 2017 that means the Cisco MIC will expire in 10 years leaving us with an expiry of March 2023.

Working through a whole fleet this way is tedious but worthwhile — build a spreadsheet with AP name, model, serial, decoded manufacture date and calculated MIC expiry date, then sort by expiry. Your replacement plan writes itself: the sites whose APs were built in 2012–2014 fail first, the 2015–2016 units give you a couple more years of runway, and 2027 is the hard stop for everything in this generation.

Confirming with certificates

Another way to confirm the issue is to look at the config on the Cisco Aironet AP itself and decode the certificates on it.

On the Cisco Aironet AP run the command show running-config

The part we care about will look something like this

crypto pki certificate chain chain Cisco_IOS_MIC_cert certificate ca 02 6F20526F 6F742043 41204D32 301E170D 31323131 31323133 35303538 5A170D33 37313131 32313330 3031375A 3036310E 300C0603 55040A13 05436973 636F3124 30220603 55040313 1B436973 636F204D 616E7566 61637475 72696E67 20434120

Cisco stores the certificates in hex. To decode the certificate we need to convert from hex to base64 then add the normal -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- parts to it and then we can read it. I used CyberChefto do all of this. Here’s the recipe I usedTo_Base64('A-Za-z0-9%2B/%3D')Pad_lines('Start',28,'-----BEGIN%20CERTIFICATE-----')Pad_lines('End',26,'-----END%20CERTIFICATE-----')Parse_X.509_certificate('PEM')).

It should look something like this

Looking at the output we can see that the Cisco MIC for that Cisco Aironet AP expired on 7 March 2023.

If you prefer to do the decode on a workstation rather than in a browser, the same pipeline works in openssl: strip the hex into a binary DER file, then inspect the validity window directly.

# convert the hex blob from "show running-config" into DER
xxd -r -p mic_hex.txt mic.der

# read the certificate and check the validity dates
openssl x509 -inform DER -in mic.der -noout -subject -issuer -dates
notBefore=Nov 21 13:50:58 2013 GMT
notAfter=Nov 21 13:00:17 2023 GMT

The notAfter value is the MIC expiry date the WLC is comparing against. If that date is in the past and the AP refuses to join, you have your root cause.

The Fix

SSH into the Cisco WLC
run the command config ap cert-expiry-ignore mic enable

We can also tell it to ignore expired Self Signed Certificates too.

run the command config ap cert-expiry-ignore ssc enable

Save your changes by running the command save config

After that everything should just start working again.

Restart the APs that are stuck rather than waiting for them to retry. An AP that has been cycling through discovery tends to back off between attempts, so a power cycle (or a reset system from the AP console) makes the next join attempt immediate and lets you confirm the fix in seconds instead of minutes.

Verifying That the APs Rejoin

Do not trust the "everything should work now" step. Confirm it on both the WLC and the AP, and confirm it for the whole fleet rather than the one AP you happened to be watching.

(Cisco Controller) > show ap cert-expiry-ignore
MIC Certificate Expiry Ignore.............. Enabled
SSC Certificate Expiry Ignore.............. Enabled

(Cisco Controller) > show ap summary
AP Name   Slots  AP Model         Ethernet MAC        State
--------  -----  ---------------  ------------------  ----------
AP01      2      AIR-CAP3602I-A   xx:xx:xx:xx:xx:xx   Registered
AP02      2      AIR-CAP3602I-A   xx:xx:xx:xx:xx:xx   Registered

(Cisco Controller) > show ap join stats summary all

The State column should read Registered for every AP that was previously stuck. If an AP still shows Joining or Downloading after ten minutes of retries, the certificate was not the only problem — go back to the CAPWAP fundamentals on that specific AP (management VLAN reachability, DHCP option 43, DNS, UDP 5246/5247, MTU).

For a per-AP view of what is actually happening during the join, the debug that matters is the CAPWAP/LWAPP state machine on the WLC, plus the AP console.

(Cisco Controller) > debug lwapp events enable
(Cisco Controller) > debug capwap events enable
(Cisco Controller) > show msglog

Remember to turn the debug off again (debug disable-all) — CAPWAP debug on a busy controller is noisy and can itself drive CPU utilisation up, which is a separate problem worth avoiding.

Security Trade-Offs Before You Disable MIC Validation

The fix is a one-line command, but it is a security control you are switching off, so it is worth being explicit about what changes. config ap cert-expiry-ignore mic enable disables only the expiry-date check. The certificate still has to chain to a trusted Cisco CA and the signature still has to validate — you are not opening the controller up to arbitrary self-signed certificates. It is a narrow change, not a wide one.

What you do lose is the guarantee that an AP joining your controller was built recently enough to be within its certified lifetime. The practical risk is an attacker with physical access to a long-decommissioned AP who wants to make it look like a legitimate one. That is a real but limited scenario, and it can be compensated for with controls you should already have on an AP management segment:

Put every AP on a dedicated management VLAN with no user traffic and no routing to the user VLANs beyond what is required.
Enable DHCP snooping, dynamic ARP inspection and IP Source Guard on the AP access ports so a rogue device cannot simply take an AP's address.
Enable 802.1X or MAB on the switch ports that serve APs — see our Cisco 802.1X and MAB configuration walkthrough for the port-based authentication setup.
Monitor the controller's AP inventory for new MAC addresses appearing where they should not, and alert on it.
Treat the flag as a temporary bridge, not a permanent configuration. Record it, date it, and pair it with a hardware replacement plan.

If the underlying problem you are chasing is a certificate chain rather than an expiry date — wrong issuer, missing intermediate, clock skew — the diagnostic sequence is different and it is covered in TLS certificate chain problems: diagnosing with OpenSSL.

Long-Term Options and Fleet Planning

Disabling the expiry check buys time. It does not buy hardware. The MIC cannot be renewed in the field, there is no firmware trick that resets the clock at the controller for the certificate's benefit, and every affected AP will still be affected in 2027. Realistic options, roughly in order of cost:

Replace the oldest APs first. You already have the decoded expiry dates from the serial-number step; work the list oldest-first. Replacing the MIC-era APs with Wi-Fi 6/6E access points removes the problem permanently and gives you a capacity and security upgrade at the same time.
Consolidate onto newer AP models only. If a site has a mix of affected and unaffected APs, standardise on the unaffected ones. You will not hit 2027 if the MIC-era units are already out of service.
Plan a controller migration alongside the AP refresh. If you are still on a legacy AireOS controller, a refresh is a natural moment to move to a Catalyst 9800 or an embedded wireless controller on a switch, and to re-examine your overall wireless design. Our Wi-Fi 7 MLO enterprise deployment guide covers the channel, cabling and design realities you will face on the new hardware, and 802.11k/v/r fast roaming configuration is worth reading before you re-plan mobility domains.
Document the exception. Leave a note in the runbook saying the expiry-ignore flag is enabled, why, and when it is expected to be removed. Otherwise in three years nobody will know whether it is safe to turn it back off.

None of this is complicated, but it is the difference between a firefight every few months as another batch of APs silently drops off the air, and a controlled migration you can schedule on your own terms.