原文：ESXi Config Restore Bug — theDXT (Daniel Keer)

ESXi can back up its host configuration to a small bundle file and restore it later, which is the fastest way to rebuild a host after a boot device failure or a bad upgrade. The bundle is a tarball containing a manifest and the individual configuration archives. What most people never inspect is the validation code that runs during the restore: it turns out the check is weaker than the documentation implies, and a config from a different build can be restored without complaint. This article walks through the restore logic, shows what the manifest actually contains, and explains why you should still refuse to restore a mismatched bundle even though the tool lets you.

What an ESXi Config Backup Actually Contains

A host config backup created with the standard tooling - vicfg-cfgbackup from the vSphere CLI, the vCenter host-profile mechanism, or the DCUI/keyboard-driven backup - produces a single gzipped tar archive. Unpack it and you find a Manifest.txt plus the state archives that hold the host settings: networking, storage, and the hostd/vpxa configuration.

The manifest is the gatekeeper. Everything the restore script validates comes from this one small text file, so it is worth knowing exactly which keys it carries.

Usage: RestoreConfiguration(<bundleFile>, <force>)

The bundled restore helper, historically firmwareConfig.sh, takes two arguments: the bundle path and a force flag. Its flow is disarmingly simple - unpack to a temporary directory, read the manifest, and compare two values against the running host before doing any real work.

RestoreConfiguration()
{
   local bundle=${1} force=${2}
   local tmpdir="/tmp/firmware-restore.$$"
   local host_uuid=$(esxcfg-info -u) bundle_uuid=
   local host_release=$(vmware -l) bundle_release=
   local manifest_file="${tmpdir}/Manifest.txt"

   mkdir -p "${tmpdir}"

   tar xzf "${bundle}" -C "${tmpdir}"
   if [ ! -e "${manifest_file}" ] ; then
      echo "Invalid Bundle: Missing Manifest File"
      rm -rf "${tmpdir}"
      exit 1
   fi

   bundle_uuid=$(GetValue "${manifest_file}" UUID)
   bundle_release=$(GetValue "${manifest_file}" RELEASELEVEL)

   # validate version and UUID
   if [ "${bundle_release}" != "${host_release}" ] ; then
      echo "Mismatched Bundle: Host release level: ${host_release} Bundle release level: ${bundle_release}"
      rm -rf "${tmpdir}"
      exit 1
   fi

   if [ "${host_uuid}" != "${bundle_uuid}" ] && [ ${force} -ne 1 ] ; then
      echo "Mismatched Bundle: Host UUID: ${host_uuid} Bundle UUID: ${bundle_uuid}"
      rm -rf "${tmpdir}"
      exit 1
   fi
}

Two guards exist: a release-level match and a UUID match, the latter bypassable with the force argument. There is no comparison of the build number anywhere in this script, and that omission is the whole story.

What Changes in ESXi 6.7 and 7.x

Due to this I did not do any further testing in ESXi 6.7. I believe it is safe to say that ESXi 6.7 has no validation on the build numbers only the release numbers.

It looks with the release of ESXi 7 VMware added build numbers to the ESXi config backups as the ESXi config backup files now list the build number in the Manifest.txt file.

Manifest.txt from ESXi 7.0 Update 3d Build Number 19482537:

RELEASELEVEL=VMware ESXi 7.0 Update 3
BUILDNUMBER=19482537
UUID=00000000-0000-0000-0000-AC1F6B950B36
KERNELOPTS=autoPartition=FALSE
USEROPTS=

Another change with ESXi 7 is that the firmwareConfig.sh script is now a python script called firmwareConfig.py and there are mentions of build numbers in it. Here is the restore part of the firmwareConfig.py script:

Validate RELEASELEVEL and UUID

  # Error message format will be parsed by FirmwareSystemImpl:
  # 'Mismatched Bundle: Host KEY VALUE Bundle KEY VALUE'
  hreleaseLevel = _hostReleaseLevel()
  if mDict['RELEASELEVEL'] != hreleaseLevel:
     logger.error('Mismatched Bundle: Host release {} Bundle release {}'
                  .format(hreleaseLevel, mDict['RELEASELEVEL']))

     # FirmwareSystemImpl want's to throw a MismatchedBundle exception
     # which reports the build number if they don't match.
     build = getBuildNum()
     if mDict['BUILDNUMBER'] != build:
        logger.error('Mismatched Bundle: Host build {} Bundle build {}'
                     .format(build, mDict['BUILDNUMBER']))
     return False

  huuid = getHardwareUuid()
  if not mDict['UUID'] == huuid and not force:
     logger.error('Mismatched Bundle: Host UUID {} Bundle UUID {}'
                  .format(huuid, mDict['UUID']))
     return False

Notice the indentation. The build-number comparison is nested inside the if block that already fired when the release level did not match. In other words, the build check only executes on the path where the release level check has already failed - and on that path the function returns False a few lines later regardless.

My Thoughts

I think what might be happening is the build number check only happens if the release level check fails as that seems to be the only time it throws an error that includes the build number.

Even though you can restore an ESXi config when the build numbers don't match I would not recommend doing so, as it could cause unexpected behavior and based on VMware's documentation would likely not be supported if something goes wrong later on.

I've place a support request with VMware to report this and to find out for sure if this is actually a bug or maybe the documentation is wrong.

Why This Matters in Practice

The practical consequence is straightforward. Two hosts running the same nominal release - say ESXi 7.0 Update 3 - can carry different build numbers because one has been patched and the other has not. Their manifests will report the same RELEASELEVEL, so the restore validation passes, the configuration is applied, and the host boots. Nothing stops you.

That is exactly the danger. A configuration captured on build 19482537 and restored onto an older build of the same release can reintroduce assumptions the older host code does not share: new configuration keys, changed defaults, storage and network stack parameters that landed in a later patch. The host may come up cleanly and then misbehave days later, at which point the two events are no longer associated in anyone's mind - and support may decline to help because you restored a config from a build you were never running.

A Safe Restore Procedure

The way to avoid the whole class of problem is to make the build number, not just the release level, part of the workflow:

• Record the host build with vmware -v (or vmware -l for the release line) at the moment you take the backup, and store it beside the bundle, not only inside it.
• Before a restore, compare that recorded build against the current host build. If they differ, patch the host to the matching build first, or rebuild from scratch.
• Verify the manifest yourself - tar tzf bundle.tgz followed by extracting and reading Manifest.txt - rather than trusting the tool's own validation.
• Never use the force flag to paper over a UUID mismatch unless you are deliberately cloning a reference configuration onto new hardware and fully understand the consequences.

If you want to automate that recording step, the same idea drives a scheduled host config backup job that timestamps the bundle and logs the build alongside it; see the vCenter ESXi config backup script for a working example you can adapt.

Confirming the Behaviour Yourself

You can test the claim in a lab without risking production. Take a config backup on a host, note the build with vmware -v, then install a different patch level of the same release on a second host and attempt the restore with that bundle. Inspect the manifest directly rather than trusting the tool:

vmware -v
vmware -l
tar tzf /tmp/configBundle-esx01.tgz
tar xzf /tmp/configBundle-esx01.tgz -C /tmp/inspect
cat /tmp/inspect/Manifest.txt

If the release strings match, the restore proceeds even though the build numbers in the manifest differ. Repeat with a bundle from a genuinely different release and the restore stops at the release check, printing the message that also reports the build - which is exactly the behaviour described above, and the reason the build mismatch only ever appears inside that error path.

Documentation vs Implementation

VMware's guidance treats a host config backup as tied to the host and the build that produced it. The shipped implementation enforces that tie at the release level only, and leaves the build comparison on a code path that has already decided to fail. That gap is what a bug report is for. It is also worth checking the behaviour on the build you actually run rather than assuming a fix has landed, because the validation logic has changed shape across releases - from a POSIX shell script to a Python module - and a future patch could tighten either or both checks without changing the surrounding documentation.

Alternatives When the Build Does Not Match

If the recorded build cannot be matched, rebuilding from documented intent is safer than restoring a mismatched bundle. The host profile mechanism in vCenter can apply a curated set of settings without dragging along build-specific state, and a documented network and storage baseline lets you rebuild networking by hand in minutes. Keep the config bundle as a reference to diff against and a last resort, not as the primary recovery path. Where the configuration matters enough to restore, it matters enough to keep documented in a form that does not carry a build number.

Related ESXi Administration Reading

Once a restored host is back online, re-check the parts of the configuration that fail silently: the vSwitch and port-group layout (ESXi vSwitch, port groups, VLAN IDs and teaming) and the host certificates (regenerating the ESXi self-signed certificate). Both are stored in the configuration bundle, and both are common causes of "the restore worked but vCenter will not manage the host" afterwards.

Key Takeaways

The restore helper validates the release level strictly and the UUID softly, but it does not independently enforce a matching build number - the build comparison is only reached on the failure path where the release check already returned an error. VMware added the build number to the manifest in ESXi 7, which is presumably the first step toward enforcing it properly, but the shipped logic does not yet reject a same-release, different-build bundle. Until that changes, the safe assumption is that any mismatch the tool does not catch is yours to catch. Track the exact build at backup time, patch before restoring when they diverge, and treat the configuration bundle as version-locked to the build that produced it.