What's Inside an ESXi vm-support Bundle?

When you open a Support Request (SR) at VMware, the Global Support Services usually requests you to collect diagnostic information. You can create this log bundle with a special command line tool (vm-support), with the vSphere (Web-)Client or with the API. No matter how you create the bundle the result is always the same: a .tgz file containing your ESXi Host name and the creation date with a size of about 30 - 300MB.

This post explains what's inside this log bundle, but instead of simply extracting the file and looking at the result, I am going to take a look on how the file is created and what's part of the process.

Why is it important to know? Because it does not only contain data that allows VMware GSS to identify your issue, but also to help yourself to enhance your troubleshooting skills. The script gathers the output of many useful commands and there is nearly no black magic involved. You do not need special tools or internal VMware knowledge to make use of this log bundle.

Manifests Definition

The basic function of the vm-support command is to run a bunch of commands and copy files. The definition and configuration of commands and files to copy is done with manifests. Manifests (*.mfx files) can be found in /etc/vmware/vm-support/.

Each manifest has 3 main configuration parameters:

• Manifest name - User-friendly visible name for this manifest.
• Manifest group - Organizes manifests into logical groups.
• Manifest default - Specifies whether or not to gather this data by default. (Enabled/Disabled)

Currently, there are 20 manifest groups containing 61 manifests (ESXi 6.0 Update 1). Not all manifests are enabled by default because they might run slow or the output is uncommon. They can be activated on demand. This is a grouped list of all manifests. For better readability I've omitted the manifest name in manifest groups that contain a single manifest only. Manifests disabled by default are marked with a *.

• ActiveDirectory
• CIM
• Configuration
• Core
• CrashDumps
• FaultTolerance
• FileSystem
  •  base
  •  VMFSDiskDump
  •  vmfsheader*
• Hardware
  •  base
  •  usb
• hostProfiles
  •  hostProfiles
  •  SystemImageCacheHostProfile
• HungVM*
  •  Coredump_VM*
  •  Send_NMI_To_Guest*
  •  Suspend_VM*
• Installer
• IntegrityChecks
• Logs
• Network
  •  base
  •  dvs
  •  lacp
  •  tcpip
  •  nscd
• PerformanceSnapshot*
  •  performance-start*
  •  performance-end*
• Storage
  •  Multipathing
  •  iodm
  •  FCoE
  •  iscsi
  •  NAS
  •  base
  •  VSANHealth
  •  VSAN
  •  VSANMinimal*
  •  VSANTraces*
  •  VVOL
• System
  •  BaseMinmal*
  •  Base
  •  EsxImage
  •  Modules
  •  ntp
  •  ResourceGroups
  •  VFlash
  •  VmiofPlugins
  •  VMKernel_Cores
  •  VMKernel_Old_Cores*
  •  VMKernel_Stdout_Cores*
  •  VMKernel
• Testing*
• Userworld
  •  FDM
  •  HostAgent
  •  ProcessInformation
• VirtualMachines
  •  corefiles
  •  diskinfo
  •  GuestState*
  •  logs
  •  sidecarfiles*
  •  SnapshotsAndMemoryImages
  •  VirtualMachineStats
  •  base

When the vSphere Client is utilized to export system logs you see a similar output:

 Manifest Commands

Manifest commands are used to define which information are gathered in each category. This is an example of a manifest file that gathers information about the Network Time Protocol:

~ #  cat /etc/vmware/vm-support/ntp.mfx
% Manifest name: ntp
% Manifest group: System
% Manifest default: Enabled
% Manifest tag: base
# action Options file/command

copy IGNORE_MISSING /etc/ntp.drift
copy - /etc/ntp.conf
run TIMEOUT=10,IGNORE_ERROR /bin/ntpq -p
run TIMEOUT=10,IGNORE_ERROR /bin/ntpdc -c kerninfo

The following commands are supported:

• copy - Adds the specified files into the bundle. Wildcard (*) allowed.
• safecopy - Adds the specified files into the bundle but internally uses "cat <filename>" to prevent errors on files like proc nodes.
• run - Runs the specified command and puts the output into the bundle. Both stdout and stderr are redirected to the same file.
• vmfile - Adds the specified file from the VM directory of all VMs into the bundle.
• vmcmd - Runs the specified command against all registered vms.
• vmfscmd - Runs the specified command against all volumes under /vmfs/volumes.
• prune - Ensures the specified file doesn't end up in the final log bundle.

vm-support.tgz Structure and what's not inside the log bundle

Any files copied with the copy or safecopy command are in the same folder structure as on the original ESXi host. Any command where the output has been gathered are in the /commands/ directory. For better automatic processing, command outputs are also gathered in JSON format. This output is placed in the /json/ directory. Additionally, all commands that have been used to create the log bundle are logged to a file in the root directory called actions.log. Any errors are written to error.log and errors-ignored.log.

The following files that contain sensitive information are pruned from all log bundle:

• /etc/vmware/vmkiscsid/vmkiscsid.db
• /etc/vmware/ssl/rui.crt
• /etc/vmware/ssl/rui.key
• /etc/vmware/ssl/rui.bak
• /etc/vmware/vmkiscsid/.#vmkiscsid.db
• /etc/vmware/ssl/.#rui.crt
• /etc/vmware/ssl/.#rui.key
• /etc/vmware/ssl/.#rui.bak
• /etc/ssh/ssh_host_dsa_key
• /etc/ssh/ssh_host_rsa_key
• /etc/shadow
• /etc/passwd
• /etc/vmware/vmware.lic

What's inside the log bundle

As you can see from the manifest definition, the number of commands are quite enormous. Altogether there are about 300 commands executed and files copied to create the log bundle. All information are in the sequence used in the upper section (alphabetically sorted by manifest groups). The heading is "Manifest Goup > Manifest Name (Manifest File)". I've omitted commands where only lots of files are copied.

The following list is quite long. If you are intrested in the mostly common troubleshooting commands, search this page for the following terms:

• FileSystem > base (filesystem.mfx)
• Network > base (network.mfx)
• Network > tcpip (networking-tcpip.mfx)
• Storage > base (storage.mfx)
• System > Base (base.mfx)
• System > EsxImage (esxsoftware.mfx)

ActiveDirectory > base (ad.mfx)
Useful when your ESXi host is joined to an Active Directory domain.

/usr/lib/vmware/likewise/bin/lw-lsa ad-cache --enum-users
/usr/lib/vmware/likewise/bin/lw-lsa ad-cache --enum-groups
/usr/lib/vmware/likewise/bin/lw-lsa enum-users
/usr/lib/vmware/likewise/bin/lw-lsa enum-groups
/usr/lib/vmware/likewise/bin/lw-lsa get-metrics
/usr/lib/vmware/likewise/bin/lw-lsa get-status
/usr/lib/vmware/likewise/bin/lw-lsa trace-info

CIM > base (cim.mfx)
The Common Information Model (CIM) is used on ESXi hosts instead of installing hardware agents. CIM providers are available for the server itself and different hardware such as network cards, HBAs or RAID controllers.
These commands and files are useful to identify built-in hardware, firmware versions and hardware status.

/bin/cim-diagnostic.sh
/usr/sbin/localcli hardware ipmi sel get -p -i -n all
/usr/sbin/localcli hardware ipmi sel list -p -i -n all
/usr/sbin/localcli hardware ipmi sdr get -p -i -n all
/usr/sbin/localcli hardware ipmi sdr list -p -i -n all
/usr/sbin/localcli hardware ipmi fru get -p -i -n all
/usr/sbin/localcli hardware ipmi fru list -p -i -n all
/usr/sbin/localcli system slp stats get
/usr/sbin/localcli system slp search
/usr/lib/vmware/vm-support/bin/swfw.sh
/bin/host-ind --show

Configuration > System (config-system.mfx)
General ESXi system configuration. Mainly copies ESXi configuration files and kernel parameters.

/usr/lib/vmware/vm-support/bin/localtgz.sh
/usr/lib/vmware/vm-support/bin/altlocaltgz.sh
/sbin/localcli system settings kernel list -d
/sbin/localcli system settings advanced list -d

Core > RDMA (rdma.mfx)
Gathers RDMA (Remote Direct Memory Access) device information. The RDMA protocol allows you to connects two communicating host channel adapters (InfiniBand for example) to reduce latency.

/usr/lib/vmware/vm-support/bin/rdmainfo.sh

CrashDumps > userworld (userworld-cores.mfx)
Copies VMkernel Coredumps (/var/core/) to the support bundle

FaultTolerance > Base (fault-tolerance.mfx)
Gathers VMware Fault Tolerance (FT) statistics. FT provides continuous availability by creating a live shadow instance of a virtual machine.

/sbin/ft-stats -lv

FileSystem > base (filesystem.mfx)
Gathers information about both, the filesystem where the ESXi is running at and VMFS datestores. This information can help you to identify disk usage problems for example. Some commands are running against an datastore directely, so you have to replace {VMFS} with your datastore name when you rung it manually. Issues with datastores or the ESXi itself are often cause by filesystem issue, so this is one of the most important manifests.

df
ls -isla /
ls -islR /dev
ls -isl /vmfs/volumes
vmkvsitools lsof
/usr/sbin/localcli system visorfs get
/usr/sbin/localcli system visorfs ramdisk list
/usr/sbin/localcli system visorfs tardisk list
/bin/vdu -a /
/usr/sbin/localcli storage filesystem list -i
/usr/sbin/localcli storage vmfs snapshot list
/usr/sbin/localcli storage vmfs extent list
/usr/sbin/localcli storage vmfs lockmode list -i
vmkfstools -P -v 10 "/vmfs/volumes/{VMFS}"
/usr/lib/vmware/vm-support/bin/dump-vmdk-rdm-info.sh "{VMCFG}"
vmkfstools -P -v 10 /bootbank
vmkfstools -P -v 10 /altbootbank
vmkfstools -P -v 10 /scratch
vmkfstools -P -v 10 /productLocker

FileSystem > VMFSDiskDump (vmfs-diskdump.mfx)
Copies VMFS dumps (/var/core/*vmfsdump.gz) to the support bundle.

FileSystem > vmfsheader (vmfs-header.mfx)
Copies VMFS volume headers (/vmfs/volumes/*/.vh.sf) to the support bundle.

Hardware > base (hardware.mfx)
Gathers information about the hardware where the ESXi host is running at. This includes BIOS configuration, PCI devices and firmware parameters.

/sbin/smbiosDump
/sbin/lspci
/sbin/lspci -v
/bin/irqinfo
/usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -a
/usr/sbin/localcli hardware trustedboot get

Hardware > usb (usb.mfx)
Lists connected USB devices with their specification (eg. USB Version).

lsusb
lsusb -t
lsusb -v
cat /vmfs/devices/char/vmkdriver/usbdevices

hostProfiles > hostProfiles (hostProfiles.mfx)
hostProfiles > SystemImageCacheHostProfile (system-image-cache.mfx)
Host Profiles are used to ensure consistency for ESXi configuration. This manifests copies files to the support bundle that are used during Host Profile adaptation.

HungVM > Coredump_VM (vm-coredump.mfx)
HungVM > Send_NMI_To_Guest (vm-nmi.mfx)
HungVM > Suspend_VM (vm-suspend.mfx)
Warning: HungVM manifests will by design crash virtual machines. This commands can be used to create a virtual machine core dump for troubleshooting purposes. {VMCFG} has to be replaced with the location of the virtual machine config file. Use vm-support -V to receive a list of all registered VMs.

vm-support -V
/usr/lib/vmware/vm-support/bin/debug-hung-vm "{VMCFG}" nmi
/usr/lib/vmware/vm-support/bin/debug-hung-vm "{VMCFG}" suspend
/usr/lib/vmware/vm-support/bin/debug-hung-vm "{VMCFG}" coredump

Installer > base (installer.mfx)
Copies onetime.tgz to the support bundle. This file is used during automated Kickstart installations. After the installation this file is kept for post-install references.

IntegrityChecks > md5sums (md5sums.mfx)
Creates an MD5 checksums of binaries and libraries to make sure that they havn't been modified.

Logs > System (system-logs.mfx)
Copies Log Files (/var/log/*) to the support bundle.

Network > base (network.mfx)
Gathers information about the network configuration including physical interfaces, virtual switches and network statistics. The commands also verify external components with LLDP (Link Layer Discovery Protocol) and pings to the default gateway.

/usr/sbin/esxcfg-vswitch -l
/usr/sbin/esxcfg-vmknic -l
/usr/sbin/vmkping -D -v
/usr/sbin/net-stats -l
/usr/lib/vmware/vm-support/bin/nicinfo.sh
/sbin/lldpnetmap
/sbin/summarize-dvfilter

Network > dvs (networking-dvs.mfx)
Gathers information about the vNetwork Distributed Switch (dvSwitch) including configuration files placed in datastores.

/usr/lib/vmware/bin/net-dvs -l

Network > lacp (networking-lacp.mfx)
Gathers information about Link Aggregation Control Protocol (LACP) configuration. LACP is a method to control the aggregation of physical network links to form a logical channel for increased bandwidth and redundancy purposes.

/usr/sbin/localcli network vswitch dvs vmware lacp config get
/usr/sbin/localcli network vswitch dvs vmware lacp status get
/usr/sbin/localcli network vswitch dvs vmware lacp stats get

Network > tcpip (networking-tcpip.mfx)
Gathers information about the TCP/IP network stack including ARP tables and routing tables.

/usr/sbin/localcli network ip connection list
/usr/sbin/localcli network ip neighbor list
/usr/sbin/esxcfg-route -l
/usr/sbin/esxcfg-route -f V6 -l
/usr/sbin/tracenet

Network > nscd (nscd.mfx)
Copies the NSCD (Name service caching daemon) configuration file (/etc/nscd.conf) to the support bundle. NSCD is a daemon that provides a cache for DNS requests.

PerformanceSnapshot > performance-start (performance-start.mfx)
PerformanceSnapshot > performance-end (performance-end.mfx)
Gathers performance information that might help to investigate unresponsive ESXi hosts or virtual machines. This manifest is disabled by default and can be started with vm-support -p.

Storage > Multipathing (multipath.mfx)
Gathers information about current storage paths and claim rules.

/usr/sbin/localcli storage core path list
/usr/sbin/esxcfg-mpath -b
/usr/sbin/localcli storage core claimrule list --claimrule-class=all
/usr/sbin/localcli storage nmp device list
/usr/sbin/localcli storage nmp path list
/usr/sbin/localcli storage nmp satp rule list

Storage > iodm (storage-driver.mfx)
Gatheres iodm (IO Device Management) information for diagnosis of various storage protocol issues including Fibre Channel, FCoE, iSCSI and SAS. This enables an administrator to determine whether a storage issue is occurring at the ESXi, HBA, fabric or storage-port level.

/usr/sbin/localcli storage san fc events get
/usr/sbin/localcli storage san fc list
/usr/sbin/localcli storage san fc stats get
/usr/sbin/localcli storage san fcoe list
/usr/sbin/localcli storage san fcoe stats get
/usr/sbin/localcli storage san sas list
/usr/sbin/localcli storage san sas stats get
/usr/sbin/localcli storage san iscsi list
/usr/sbin/localcli storage san iscsi stats get

Storage > FCoE (storage-fcoe.mfx)
Gathers information about FCoE (Fibre Channel over Ethernet) capable CNA devices and FCoE HBAs.

/usr/sbin/localcli fcoe nic list
/usr/sbin/localcli fcoe adapter list

Storage > iscsi (storage-iscsi.mfx)
Gathers information about iSCSI (internet Small Computer System Interface) adapters and iSCSI Initiator and Target Configuration

/usr/sbin/localcli iscsi software get
/usr/sbin/vmkiscsid --dump-db
/usr/sbin/localcli iscsi physicalnetworkportal list
/usr/sbin/localcli iscsi logicalnetworkportal list
/usr/sbin/localcli iscsi networkportal list

Storage > NAS (storage-nas.mfx)
Displays NFS (Network File System) volumes currently known to the ESX host

/usr/sbin/localcli storage nfs list
/usr/sbin/localcli storage nfs41 list

Storage > base (storage.mfx)
Gathers information about the storage used by the ESXi hosts including partition tables, LUNs, VAAI status, HBAs and S.M.A.R.T. values from local drives.

fdisk -lu
/usr/lib/vmware/vm-support/bin/partedUtil.sh
/usr/sbin/esxcfg-scsidevs -m
/usr/sbin/localcli storage core device list
/usr/sbin/localcli storage core device partition list
/usr/sbin/localcli storage core device partition showguid
/usr/sbin/localcli storage core device vaai status get
/usr/sbin/localcli storage core adapter list
/usr/sbin/localcli storage core adapter stats get
/usr/sbin/localcli storage core device stats get
/usr/sbin/localcli storage core device detached list
/usr/sbin/localcli storage core plugin registration list
/usr/lib/vmware/vm-support/bin/smartinfo.sh

Storage > VSANHealth (vsan-health.mfx)
Commands that provide a brief overview of the Virtual SAN health.

python /lib/python2.7/site-packages/pyMo/vim/vsan/VsanLsomHealth.pyc
python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc

Storage > VSAN (vsan.mfx)
Storage > VSANMinimal (vsan-minimal.mfx)
This manifest provides comprehensive Virtual SAN configuration an status information. Virtual SAN trace files are also copied to the support bundle. VSANMinimal is a short version where some rarely used commands and traces are missing.

/sbin/localcli vsan cluster get
/sbin/localcli vsan network list
/sbin/localcli vsan storage list
/sbin/vdq -q -H
/sbin/cmmds-tool find -f python
/sbin/cmmds-tool -f json readdump -p -d /scratch/log/cmmds.dump
/sbin/cmmds-tool -f json readdump -p -d /scratch/log/cmmds.dump.backup
/sbin/vsish -e get /vmkModules/tracing/stats
/usr/lib/vmware/vsan/bin/clom-tool fitness-stats-get
/usr/lib/vmware/vsan/bin/clom-tool fitness-config-get
/usr/lib/vmware/vsan/bin/clom-tool stats

Storage > VSANTraces (vsan-traces.mfx)
Copies Virtual SAN Traces to the Support Bundle.

Storage > VVOL (vvol.mfx)
Gathers VVol (Virtual Volumes) information. Virtual Volumes is a new storage architecture introduced in vSphere 6.0.

/sbin/localcli storage vvol vasaprovider list
/sbin/localcli storage vvol storagecontainer list
/sbin/localcli storage vvol protocolendpoint list
/sbin/localcli storage core device list --pe-only
/sbin/localcli storage nfs list --pe-only

System > BaseMinmal* (base-minimal.mfx)
System > Base (base.mfx)
Gathers general system information from the ESXi host including version, date, time, coredump configuration. This also includes a comprehensive configuration and status dump. Some ESXi VMkernel state information are not included in the BaseMinimal manifest.

vmware -vl
/bin/date
/usr/sbin/localcli system stats uptime get
uname -a
/usr/sbin/esxcfg-info -a
/usr/sbin/esxcfg-info -a -F 'xml'
/usr/sbin/localcli system coredump partition get
/usr/sbin/localcli system coredump partition list
/usr/sbin/localcli system coredump file get
/usr/sbin/localcli system coredump file list
/usr/sbin/localcli --plugin-dir /usr/lib/vmware/esxcli/int systemInternal coredump configuration check
/usr/sbin/vmkerrcode -l
/usr/sbin/secpolicytools -d
/sbin/bootOption -oC

System > EsxImage (esxsoftware.mfx)
Gathers information about vib packages that have been installed on the ESXi host.

localcli software acceptance get
localcli software profile get
localcli software vib get
localcli software vib list
localcli software profile get --rebooting-image
localcli software vib get --rebooting-image

System > Modules (modules.mfx)
Gathers information about VMKernel modules and module configuration

/usr/sbin/vmkload_mod -v10 -l
/usr/sbin/esxcfg-module -q

System > ntp (ntp.mfx)
Gathers NTP (Network Time Protocol) information including configured NTP Servers and drift. Time drift information are useful to identify timing issues.

/bin/ntpq -p
/bin/ntpdc -c kerninfo

System > ResourceGroups (resource.mfx)
Gathers information about resource groups. This scheduling information are specific to all processes including Virtual Machines that are running on the ESXi host.

/usr/sbin/esxcfg-resgrp -l

System > VFlash (vflash.mfx)
Gathers information about vSphere Flash Read Cache vFRC also known as vFlash. vFRC uses local SSDs and turns them in to a caching layer for virtual machines.

/usr/lib/vmware/vm-support/bin/vFlash.sh

System > VMKernel_Cores (vmkernel-cores.mfx)
System > VMKernel_Old_Cores (vmkernel-old-cores.mfx)
System > VMKernel_Stdout_Cores (vmkernel-stdout-cores.mfx)
Copies VMKernel Core Dumps (/var/log/*) to the support bundle. Manifest are prepared to copy all, or the latest core dump only. The default setting is to include the latest core dump only.

System > VMKernel (vmkernel.mfx)
Copies the VMKernel logfile (/var/log/vmkernel.log) to the support bundle.

Userworld > FDM (fdm.mfx)
The Fault Domain Manager (FDM) provides VMware High Availability (HA), the feature that restars Virtual Machine in case of ESXi failures. This manifest copies FDM based logfiles and gathers information about the configuration. The term "userworld" refers to a process running in the VMkernel.

/opt/vmware/fdm/fdm/fdm-dump.sh
/opt/vmware/fdm/fdm/prettyPrint.sh hostlist
/opt/vmware/fdm/fdm/prettyPrint.sh vmmetadata
/opt/vmware/fdm/fdm/prettyPrint.sh clusterconfig

Userworld > HostAgent (hostd.mfx)
The ESXi Host Agent (hostd) is the service that communicates with the vCenter Server. This manifest copies a full virtual machine inventory maintained by the host agent and host agent statistics.

Userworld > ProcessInformation (userworld.mfx)
Gathers information about running processes and userworld memory usage.

ps -cPTgjstz
/sbin/memstats -r uw-stats

VirtualMachines > corefiles (vm-cores.mfx)
Copies Virtual Machine Core files from all Virtual Machines to the support bundle.

VirtualMachines > diskinfo (vm-disks.mfx)
Copies descriptor files from all Virtual Machine Disks (.vmdk) to the support bundle.

VirtualMachines > logs (vm-logs.mfx)
Copies Virtual Machine log files (vmware.log from the VM directory) to the support bundle.

VirtualMachines > sidecarfiles (vm-sidecars.mfx)
Copies Virtual Machine virtual disk sidecar files (.vmfd) to the support bundle. Sidecar files contain metadata.

VirtualMachines > SnapshotsAndMemoryImages (vm-snapshots.mfx)
Copies Virtual Machine files (.vmsn, debug.vmss and debug.vmem) to the support bundle.

VirtualMachines > VirtualMachineStats (vm-stats.mfx)
Copies Virtual Machine stats, kstats and callstack profiling information to the support bundle.

VirtualMachines > base (vm.mfx)
Copies Virtual Machine configuration files and a list of running Virtual Machines to the support bundle.

/usr/sbin/localcli vm process list