Solved: VM synched time with host during vmotion

This came up while upgrading a client’s VMWare environment from 5.5 to 6.5. The upgrade was finished and working fine. VM/Host Rules had been defined and DRS turned on.

A day later alarms for the on-prem email began alerting around 2am. (Of course!)

The external email checking system was receiving email responses back but the time stamp was 3 hours off and being interpreted as outside of the alarm thresholds.

A quick check of the Exchange servers showed the time was indeed off. The main concern, though, was the domain controller time being off which was the authoritative time server for the domain computers. I went ahead and fixed the time and checked email. An added complication was the barracudas’ outbound queue was high and test emails were only intermittently being delivered although incoming email was unaffected. This had to be due to the time stamps also. With the time fixed the queue started to come down. Next I started finding the root cause.

Since the latest change was the VMWare update and DRS configuration I went there to look at the logs. I could see where VCenter log times jumped back 3 hours. I narrowed in on what else had happened around that time.

I saw DRS had VMotioned the domain controller about 20 minutes before the log times changed. I verified the VM settings had ‘Synchronize guest time with host’ unchecked. I was perplexed, I did not see how a vmotion would cause the time to go off.

I looked through Windows logs and saw the time changes but nothing else substantial. Eventually I checked the time settings of the VMWare hosts and noticed 1 of them had ntp turned off and was showing a time 3 hours off.

I double checked the vmotion log and saw this host was the one DRS had vmotioned the domain controller onto. Given the matching time difference I knew I had found a correlation but still wasn’t sure why.

As a precaution I vmotioned the domain controller off the host and disabled DRS. I checked the vmotion did not affect the server time (it didn’t).

I started doing internet research and finally landed on this VMWare KB:

https://kb.vmware.com/s/article/1189

Ah ha!!! This was new information to me

” The time synchronization checkbox controls only whether time is periodically resynchronized while the virtual machine is running. Even if this box is unselected, by default VMware Tools synchronizes the virtual machine’s time after a few specific events that are likely to leave the time incorrect.’

One of those specific events was vmotion.

I checked all the hosts and made sure ntp was running and the host time was correct.

I had opened a case with VMWare to get the support ball rolling but by now I had a pretty good idea of what had happened and why. I ran it by VMWare who verified it.

Who knows how long this host was misconfigured but with DRS off it was pure luck no one had moved the DC to this host before.

So there you go, keep those host time settings correct!